OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code
Authors: Maxence Faldor
Jenny Zhang
Antoine Cully
Jeff Clune
Venue: ICLR 2025 Poster (2025)
Keywords: Open-endedness
Environment Generation
Reinforcement Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A framework that uses large language models to automatically generate diverse and interesting learning environments through code.

Abstract: Open-ended and AI-generating algorithms aim to continuously generate and solve increasingly complex tasks indefinitely, offering a promising path toward more general intelligence. To accomplish this grand vision, learning must occur within a vast array of potential tasks. Existing approaches to automatically generating environments are constrained within manually predefined, often narrow distributions of environments, limiting their ability to create any learning environment. To address this limitation, we introduce a novel framework, OMNI-EPIC, that augments previous work in Open-endedness via Models of human Notions of Interestingness (OMNI) with Environments Programmed in Code (EPIC). OMNI-EPIC leverages foundation models to autonomously generate code specifying the next learnable (i.e., not too easy or difficult for the agent’s current skill set) and interesting (e.g., worthwhile and novel) tasks. OMNI-EPIC generates both environments (e.g., an obstacle course) and reward functions (e.g., progress through the obstacle course quickly without touching red objects), enabling it, in principle, to create any simulatable learning task. We showcase the explosive creativity of OMNI-EPIC, which continuously innovates to suggest new, interesting learning challenges. We also highlight how OMNI-EPIC can adapt to reinforcement learning agents’ learning progress, generating tasks that are of suitable difficulty. Overall, OMNI-EPIC has the potential to endlessly create learnable and interesting environments, further propelling the development of self-improving AI systems and AI-Generating Algorithms.

Published as a conference paper at ICLR 2025
OMNI-EPIC: OPEN-ENDEDNESS VIA MODELS OF HUMAN NOTIONS OF INTERESTINGNESS WITH ENVIRONMENTS PROGRAMMED IN CODE
Maxence Faldor'
Jenny Zhang"
Department of Computing
Department of Computer Science University of British Columbia
Imperial College London
London, United Kingdom
Vector Institute
m. ww.faldor222. 22@ iemimiiiaaaa ac.
jennyzzt@cs wbc. ca
Antoine Cully!
Jeff Clune?
Department of Computing
Department of Computer Science University of British Columbia
Imperial College London London, United Kingdom
Vector Institute
a. cuclll 1 ly
e ieeimperiaa al ac.uk
Canada CIFAR Al Chair jeff. clune @ellune@ubb.... ca
ABSTRACT
Open-ended and Al-generating algorithms aim to continuously generate and solve increasingly complex tasks indefinitely, offering promising path toward more general intelligence. To accomplish this grand vision, learning must occur within a vast array potential tasks. Existing approaches automatically generating envi- ronments constrained within manually predefined, often narrow distributions of environments, limiting their ability to create any learning environment. To address this limitation, introduce novel framework, OMNI-EPIC, that augments pre- vious work in ppeneenndd via Models of human Notions of Interestingness (OMNT) with Environments Programmed in Code (EPIC). OMNI-EPIC leverages foundation models to autonomously generate code specifying the next learnable (i.e.. not too easy difficult for the agent's current skill set) and interesting (e.g., worthwhile and novel) tasks. OMNI-EPIC generates both environments (c.g., an obstacle course) and reward functions (e.g., progress through the obstacle course quickly without touching red objects), enabling it, in principle, to create any simu- latable learning task. We showcase the explosive creativity of OMNI-EPIC, which continuously innovates to suggest new, interesting learning challenges. We also highlight how OMNI-EPIC can adapt to reinforcement learning agents' learning progress, generating tasks that are of suitable difficulty. Overall, OMNI-EPIC has the potential to endlessly create Icarnable and interesting environments, fur- ther propelling the development of self-improving AI systems and Al-Generating Algorithms. Project website with videos: https::/ sh co///picicccccccc
INTRODUCTION
In recent years, the field of artificial intelligence (AI) has witnessed groundbreaking achievements, particularly with advancements in reinforcement learning (RL) (Akkaya et al. 2019: Silver et al.)
Colas al., 2019) and foundation models (FMs) (Brown 2020 Radford et al., 2019). Yet, the most ambitious goal of AI building generalist agents capable autonomously perceiving, reasoning, deciding, and acting within complex environments remains formidable challenge.
2olas
The creation general-purpose agents promises immense societal benefits, provided we can address the critical safety and existential risks involved (Bengio al. 2023), including those unique
"co-authors
co--ennior authors

---

Published as a conference paper at ICLR 2025
open-ended and Al-generating algorithms (AI-GAs) Clluee 2019 Ecoffet et al., 2020 While
researchers have made substantial progress in developing powerful learning architectures (Hafner at 2023 Jaegle et al., 202T Vaswani 2017), these algorithms have been applied to relatively narrow range of tasks due to the limited availability of diverse datasets. Compounding this issue, current models require extensive amounts of data and fine-tuning to be trained. Consequently, over the past decade, the bottleneck in advancing Al has shifted from improving learning algorithms to acquiring the necessary data to train them (Jiang et al. 2023). In other words, the main challenge has become: how to effectively acquire large amounts of data for diverse tasks.
fundamentally different approach involves developing open-ended algorithms that continuously generate and solve new challenges endlessly (Stanley et al. 2017). The objective of open eneedended algorithms is to ignite an explosion of creativity and complexity in a computer, akin to the processes observed in biological evolution and human culture, including science and technology. Replicating open-ended evolution in silico extends beyond understanding biological life or developing engaging simulations. It delves into fundamental questions about the nature of creativity, the potential for machines to exhibit life-like characteristics, and the emergence of general intelligence. Given that biological evolution is the only known process that has successfully created general intelligence, its principles hold invaluable insights for advancing the field of AI, and creating AL-GAs (Clune) 2019).
For open-ended algorithms and AI-GAs to succeed, they must operate within a vast task space capable of generating an infinite array of potential challenges. Previous open-ended algorithms (Sudhakaran et al. 2024 Wang et al., 2023a 2019: 2020, /Zhang et al. 2023) have been applied to limited domains with specific types of worlds (e.g., obstacle courses) and confined to predefined parameterizations (e.g., obstacle size and type), restricting their potential to exhibit true open-endedness. Achieving Darwin Completeness- - the potential to generate any possible learning environment - is essential for realizing the full potential of AI-GAs (Clune 2019). However, Darwin Completeness presents a significant challenge: the ability to generate infinitely many environments makes the search potentially intractable, or even impossible. Exploring this immense environment search space may involve endlessly generating trivial, redundant, or overly complex tasks that do not effectively contribute to the agent's 's lcarning progress. The main challenges are to ensure that generated environments are novel, interesting, solvable, and appropriately matched to the current capabilities of the learning agents. Biological evolution required an unfathomable amount of computation over billions of years to produce intelligence. Therefore, one key scientific challenge is to determine how can optimize the process of generating and solving interesting tasks within a Darwin Complete environment search space, as create an AI-GA given the computational capabilities we expect to have in the future.
Previous work in Open-endedness via Models of human Notions of Interestingness (OMNT) (Zhang et al. 2023) leverages FMs to improve open-ended learning by focusing on tasks that both learnable and interesting. However, OMNI, like all prior open-ended works (Sudhakaran et al.) 2024 Wang et al. 2019; 2020 Zhang al. 2023 Wang et al.) 2023a). was confined generating tasks within a narrow environment search space, inhibiting the generation of any possible learning environment. This paper introduces a novel framework, OMNI-EPIC, that augments OMNI with Environments Programmed in Code (EPIC). OMNI-EPIC utilizes FMs to choose the next interesting and learnable task and subsequently generate environment code to enable the agent to learn how to solve that task. Our approach generates not only the simulated world but also the reward and termination functions, allowing it, in principle, to create any simulatable task. We take advantage of pre-existing simulators and OMNI-EPIC writes code to create tasks within it. For example, if the task is to kick a ball to hit a moving target, OMNI-EPIC would generate the environment code to simulate the physics, the agent, the ball, and the moving target, rewarding the agent when the ball hits the target. Conversely, for a task involving maneuvering the ball around a moving target, the simulated world remains the same, but the reward function could differ, penalizing the agent for any contact with the target. A model of interestingness (Mol) is employed both when generating the next task and checking if any newly proposed task is interestingly new compared to similar ones in the archive. Finally, we introduce a success detector that can automatically determine whether the agent has successfully completed any proposed task.
Our vision is for this algorithm to generate any code, including installing and modifying any existing simulator, or even writing the code for new simulator. Given that the programming language used here (Python) is Turing complete, OMNI-EPIC could potentially create any computable environment (e.g., logic and math problems to quests in virtual worlds, such as building a computer Minecraft). As first step toward this ambitious goal, in this work, we constrain our method to write code for
2

---

Published as a conference paper at ICLR 2025
one simulator, namely PyBullet (Coumans & Bai 2016). By continuously generating learnable and interesting environments, OMNI-EPIC advances the development of self-improving Al systems, bringing closer to achieving Darwin Completeness and realizing AI-GAs.
2 RELATED WORK
Unsupervised Environment Design. Unsupervised environment design has garnered increasing interest in RL. Several works have explored the development of auto-curricula to perpetually generate new training environments for agents (Dennis et al. 2020 Jiang et al., 2021; Parker-Holde et al.
2022 Samvelyan et al. 2023; Wang et 2019 2020). However, significant limitation of these methods is their reliance on predefined or manually curated distributions of tasks or environment
parameters (Dennis et al. 2020; Heess et al. [2077 Jiang et al. [02T Barker-Holdes et al. 2022
Samvelyan et 2023] Wang et al. 2019 2020), inhibiting the generation of any possible learning environment. Regret-based approaches (Jing al. 202T Parker-Holder
al., (202)] Samvelyan
et al. 2023) prioritize tasks with high regret, measured by the difference between the highest known return and the mean return across simulations. Alternative methods calculate learning progress by measuring the difference in the agent's 's task success rates across training steps (Kanitscheider et al.
2021]. By focusing on tasks with high learning progress, these approaches aim to guide the agent's 's learning towards the most promising areas of the task space (Oudeyer et al. [207)] Oudeyer & Kaplan 2007 Baranes& & Oudeyer 2013). However, a critical challenge remains distinguishing which environments are genuinely interesting (Jiang et al. [023] Zhang et al. 2023) Colas et al. (0022) In the vast space of any task describable in natural language, there may be countless learnable but not meaningful environments (c.g., kicking a ball into goal at slightly different positions). Inspired by Zhang et al. (023). OMNI-EPIC uses human notions of interestingness distilled into FMs to generate environments that are not only learnable, but also interesting.
Foundation Models for Environment Design. Recent advancements in FMs have showcased their remarkable ability to capture extensive knowledge across diverse subjects by training on vast text corpora (Bommasani et al. 2021). Consequently, this has sparked interest in applying FMs to environment design. Ma et al.
(023) utilize FMs to generate code for reward functions while Waag iet al. (2023b) employ FMs to generate simulation environments and expert demonstrations. However, these methods do not build upon the agent's previous performance on different tasks, and hence lack an auto cutto-curiculum that can provide an endless stream of environments and tasks. In procedural content generation (Shaker et al. 6 Juliani et al., 2019 Justesen al.. 2018), Sudhakaran et al. (2024) and Todd et al. (2023) fine-tune FMs to create domain-specific levels. Bruce et al. (2024) propose training a world model to generate game environments. These approaches focus on level creation but do not train agents or adapt the difficulty based on the agent's performance learning progress. Zala et al. (2024) generate environments as a curriculum to learn a fixed set of tasks but is limited in its ability to generate truly open-ended environments and adapt to the agent's evolving capabilities. Waan et al. (2023c) rely on a predefined set of objects and is limited in reflecting real-world dynamics in its simulation. Despite significant progress in applying FMs to environment design, opportunities remain for further exploration, such as integrating the generative capabilities of FMs with adaptive auto-curricula. OMNI-EPIC addresses this challenge, aiming unlock the potential for truly open-ended and effective learning environments.
Foundation Models in Open-Endednees. The field of open- endednesss seeks to create algorithmic systems that produce never-ending innovation (Stanley et al.) 2017). There has been increasing interest in leveraging FMs to generate intelligent variations for code or text in evolutionary algorithms. However, these approaches are often confined to fixed archive with hand-crafted characteris- ties (Bradley al. 2023) Ding et al., 2023 [Lehman et al.) [003) Lim et al.) (022)). The core objective of open-endedness algorithms is to generate and solve an endless stream of tasks. One way to do so is to keep an ever-expanding archive of tasks or solutions. Zhang et al. (2023) and Wang et al. (223a) have adopted FMs as mechanism for auto-curricula, proposing both learnable and interesting tasks for agent training. By restricting the agent's 's interactions to a predefined range of environmental conditions, these methods may hinder the development of truly adaptable and versatile agents capable of handling the complexities of real-world scenarios. OMNI-EPIC addresses this limitation by leveraging FMS to generate not only tasks but also the simulated worlds and reward functions, potentially exposing agents to a wider range of challenges and learning opportunities.
3

---

Published as a conference paper at ICLR 2025
3 METHOD
Task Archive
simktr carned falled ttaks from the archtve
Task Generator (LLM]
Generaae next Icamable and imtresiing 2054
Chosen takk
Trik deszaptton ratural raturalinggo imguuge
complatin
Add
Hernre for the max
tisk desccigaaon Reseiring for bellire
Environment Generator [LLM)
keraso the
Success Detector
Train agent with RL
Post- costt-ennration Model of Interestingness
Frsfrrmmntt candle Semileeed sentt Reward luncsion
Sucocss, aichiv
Agont behavior
Taakk creeztal
Ves. interesting
Comparc aga nos muse saniie teaks
Figure 1: OMNI-EPIC overview. OMNI-EPIC continuously generates and solves new, interesting tasks in simulation. Our approach maintains a task archive of learned and failed tasks.
OMNI-EPIC leverages FMs, including large language models (LLMs) and vision- lision-language models (VLMs), to autonomously create learnable and interesting tasks for open-ended learning (Figure [] OMNI-EPIC maintains a growing task archive (Section 3.1) that catalogs successfully learned and completed tasks, well as unsuccessfully attempted ones. The task generator (Section 3.2) information from the archive about what has been learned and what has not, proposing the next interestingly new task, described natural language, for the agent to attempt. Because the model has distilled sense of what is interesting from training on internet data, it has Mol that emulates the human capacity for making nuanced judgments of interestingness in open-ended learning (Zhang et al., 2023). The task generator utilizes this Mol when generating tasks. These tasks are then translated into environment code by an environment generator (Section (.3)) specifying the simulated world and functions required for RL. The newly generated task and its environment code are assessed by a second, post-generation Mol (Section (.4). to ensure the task is indeed interesting given what has come before (it compares the new task to the most similar tasks already in the archive with retrieval-ause auggented generation (Lewis et al. 2020). Tasks deemed interesting are then used to train an RL agent (Section 3.5) If deemed uninteresting, the task is discarded, and new task is generated. After training, a success detector (Section assesses whether the agent has successfully completed the task. Successfully completed tasks are added to the archive. Failed tasks are iterated upon maximum number of times and added to the archive as failed tasks if the RL agents are not able to solve them. Then, the cycle of generating the next task restarts. Each component of OMNI-EPIC is explained in more detail below, and hyperparameters are shown in Appendix K OMNI-EPIC's iterative process ensures continuous generation and learning of new interesting tasks, forming a potentially never-ending growing collection of environments and learned agents.
3.1 TASK ARCHIVE
OMNI-EPIC maintains a continuously expanding archive of tasks, including successfully learned ones and those attempted but failed. Successful tasks serve as stepping stones for creating more complex yet learnable ones, while failed tasks provide insights into generating new tasks within the agent's current capabilities. OMNI-EPIC uses past experiences to generate novel and diverse challenges, continuously pushing the boundaries of what's already learned. The task archive is initialized with a few task description seeds in natural language (Appendix N This archive is unbounded and can grow indefinitely as new tasks are generated and learned. Given the generality, interpretability, and universality of natural language and programs (Hopcroft) 2001), each task is represented by its natural language description and the corresponding environment, represented as executable code.
3.2 TASK GENERATOR
Open-ender algorithms require focusing on tasks that are both learnable (i.e., not too difficult or too easy for the agent to learn) and interesting (i.e., worthwhile and sufficiently novel). Previous attempts have resulted in pathologies when optimizing against definitions and quantifications of interestingness (Zhang al. 2023). Inspired by Zhang et al. (2023), we harness FMs to model
4

---

Published as a conference paper at ICLR 2025
ineffable human notions of interestingness, gleaned from large text corpora of Here, the task generator is an LLM, which proposes novel task descriptions in natural language that distinct from those already discovered while remaining learnable (full prompt in Appendix L.I].
data.
To ensure the task generator remains open-ended and continuously suggests new and diverse tasks, uses the content of the task archive as context. Given the limited context length of current LLMs, we retrieve a predefined number of tasks that are most similar to a randomly selected task from the archive (Appendix D]. These retrieved tasks include both those that were successfully completed and those that were attempted but failed. These similar tasks serve as examples and are input into the LLM, which then generates the next learnable and interesting task. We opt for the most similar tasks rather than the most different ones to use previous tasks as stepping stones. As LLMs and FMs improve, we expect that larger portion of the task archive, or potentially the entire archive, could be used as context. However, partial knowledge of stepping stones might be advantageous for creativity and diversity, much as human scientists and artists benefit from not being aware of everything that has come before. The task generator outputs a natural language description of the next task, crafted to be both achievable and interesting for the agent. This description serves as the basis for the subsequent environment generation step, where the natural language task descriptions are translated into executable code to create learning environments.
3.3 ENVIRONMENT GENERATOR
The environment generator, powered by an LLM, translates a given natural language task description into executable (here, Python) code defining the learning environment. Appendix L.2 shows the full prompt. This code includes specifications for creating the simulated world and functions needed for RL (Sutton 2018) based on the standard API Gymnasium (Towers al., 2024): reset, step, reward, rd. and t terminated. The reset function resets the environment to an initial state, including setting up the initial positions and orientations of the agent and objects. The step function updates the environment according to the simulated physics, the agent's action, and any other dynamic behaviors (e.g., moving platforms or activated doors). The reward function returns a scalar number, whose cumulative maximization defines the task. The termi na nated function indicates whether the agent has reached a terminal state.
For example, if the task is to "cross a bridge with moving segments", bridge should be created in the simulated world. The reset function should initialize the agent the start of the bridge and each segment's position. The step function should update the environment based on the agent's is actions and each segment's movement. The reward function should reward the agent's progress across the bridge, and the termination function should indicate when the agent falls off the bridge.
compilation errors occur when generating the environment code, the errors (with the traceback) are fed back into the environment generator, which then modifies and improves the environment code. Appendix L.3 shows the full prompt. This loop is limited to a maximum of five iterations per task. If the code still fails to compile after these attempts, the uncompiled code is discarded, and the task generator proposes new task, potentially one that is less complex or differently structured.
3.4 POST-GENERATION MODEL OF INTERESTINGNESS
While ideally all tasks proposed by the task generator should be interesting (owing to its Mol), the limited context length of current FMS prevents the task generator from considering the entire task archive at once. Furthermore, it is often easier for an FM to evaluate whether a solution is good rather than generate good solution (Bradley et al. 2023). OMNI-EPIC draws inspiration from the dynamics of human culture. For example, researchers often study a specific subset of previous works to inspire new ideas. Once a new idea is produced, is crucial to check the literature to determine whether the contribution is truly novel. If the idea is deemed interestingly new, it is published, adding to the ever-gre gre-iggowing archive of human knowledge. This process creates a growing set of stepping stones to leap off of, which is a key ingredient of open-endedness. (Stanley & Lehman 2015). OMNI-EPIC captures these important dynamics open-ended algorithms, considering a small batch of related stepping stones when creating new task (Section B.2) and then verifying its novelty against the task archive. Given a newly generated task and its corresponding environment code, we compare it against predefined number of the most similar tasks from the archive (Appendix [) The post-generation Mol then evaluates if the new task is interesting (e.g., novel, surprising, diverse, worthwhile) (full prompt Appendix L.4). If the task is deemed interesting, we proceed train an RL agent on it. If not, the task is discarded, and a new one is generated.
5

---

Published as a conference paper at ICLR 2025
3.5 TRAINING AGENTS WITH REINFORCEMENT LEARNING
A key objective of open-ended learning is to enable agents to master an ever-expanding set of tasks. To achieve this, OMNI-EPIC generates diverse array of tasks along with their corresponding environments programmed in code. Agents are then trained in these environments using RL to solve the generated tasks. In this work, we utilize the PyBullet physics simulator (Coumans & Bai 2016). While any RL algorithm could be used, we employ DreamerV3 (Hatner et al.) 2023). Appendix K details the hyperparameters and compute resources used. Agents receive proprioceptive (joints positions and velocities) and visual information (images of size 64 x 64 x 3). While OMNI-EPIC can be applied to any robot (Appendix PP). due to computational limitations, we demonstrate results using an R2D2 robot with discrete action space of six actions: do nothing, go forward, go backward, rotate clockwise, rotate counterclockwise, and jump. R2D2's simple action space allows the agent to learn tasks more efficiently, requiring fewer time steps than agents with complex action spaces.
Following Wang et al. (2020), we train one specialist RL agent for each task. For tasks used to initialize the archive, the RL agents are trained from scratch. For newly generated tasks, the agent continue training from an existing policy previously trained on tasks in the archive. The trained policy is selected from successfully completed task with the closest embedding to the new task (Appendix D This approach allows the agents to build upon their existing knowledge and adapt more efficiently to challenges presented by the new tasks. Additionally, color variation in the environments is a form of domain randomization, as an RL agent trained on an environment with specific colors may not perform well in an identical environment with different colors (Tobin et al.).
3.6 SUCCESS DETECTOR
In this infinite task space of any task describable in natural language, an essential ingredient training agents to learn any generated task is a universal reward function, which can evaluate if any proposed task has been completed or not. We employ a success detector, instantiated as an L.MM or VLM, that assesses whether the agent has successfully completed the given task. Since our preliminary testing found that current VLMs are not yet accurate enough to be used as success detectors (Appendix (). we use code generated by LLMs for this purpose instead.
When using an L.M as the success detector, we ask the environment generator to generate an additional success-checking function get_success, alongside the environment code, that checks the agent has successfully completed the task. The success-checking function serves a different purpose from the reward function. The reward function is designed to enable the RL agent to learn efficiently, which often results in a more complex function than the success-checking function. The complexity of the reward function arises from the need to shape rewards to facilitate optimal learning (Krakovna et al. 2020). Meanwhile, the success-checking function aims to evaluate whether the agent has accomplished the task. This function is less susceptible to reward hacking, as it does not affect how the agent learn. For example, consider the task of "run forward". The puicess-checking function would evaluate whether the agent's 's x-axis position has exceeded a certain threshold. In contrast, the reward function should encourage efficient learning by rewarding the agent's 's x-axis velocity and promoting cyclic joint movements that resemble a natural running motion.
If a task is deemed to have been successfully learned by the trained agent, it is added to the archive. If not, the task is returned the environment generator for modifications to aid in the agent's 's learning (e.g., changing the reward function or making the physical world less complex). This feedback loop repeated for a max number of attempts. If the task remains unlearned after these attempts, is added to the archive failed task and a new task is generated.
LONG RUN WITH SIMULATED LEARNING
To illustrate the creative explosion of generated tasks, we run OMNI-EPIC without training RL agents, assuming all generated tasks can be successfully completed. This allows us to showcase a larger number of generated tasks, as training RL agents on each task is more time-consuming. Excluding tasks that did not generate executable code, [ 200 iterations of OMNI-EPIC with the R2D2 robot. Each node represents a generated task, which includes the natural language task description and environment code. The color of the nodes corresponds to the generation number. For better visualization, each task (natural language description and environment code) is encoded using a pre-trained encoder language model (OpenAl's text-embedding-3-se (OpenAl 2024)

---

Published as a conference paper at ICLR 2025
Ted-04 Stay Tate Sencion
260 (ud Chullongs
7M622 Vall-foon CAN DRSTORS
N0 120 30
Successhilly lasand NCt intteosss
Commntcion Deanag
prdeess bogge with groert
Thek 1E8 fon Delvery
That Shu4 Scoting
Job 125 Printing
PAARS u aoko partter.
Trac Becout the platurmss
XIE Corrcinll rch antiher rending
Joch Dudggtul Aruma.
210 en Teo ZIE Awciding SAAA steakchs
Kicc kiccbailtenng
gites 2010s ailedd movin
Imaii
Tusk SIYP Bed
Recele pase moiinng ptendiec.ss gudli Bdc
IOEK Bulll BID
JACOO
Figure 2: Long Run with Simulated Learning. OMNI-EPIC generates a diverse array of tasks, ranging from wildly different objectives to interesting variations of similar overarching tasks. The node color reflects the generation number of the task. A check mark in the node means that the task was successfully learned. 77Z symbol means that the task was deemed uninteresting and discarded. The node connections illustrate which tasks were conditioned on when asking an FM to generate similar yet new and interesting task. Grey nodes show task description seeds that initialized the run.
and then projected into a 2-dimensional space using t-SNE (Van der Maaten & Hinton 2008). We manually selected a few tasks that are well-distributed across the embedding space show a sample of the diversity and creativity of OMNI-EPIC. Connections between nodes indicate which tasks were used as stepping stones generate the next task, meaning they were provided in context the task generator. For better readability. Figure only displays the parent task that is closest to the child task the embedding space and the header text of task descriptions. Appendix shows the full natural language descriptions of the magnified tasks and provides a more comprehensive visualization of all parent-child task connections. Appendix shows the three task descriptions that seed the run.
OMNI-EPIC creates tasks that evolve and diverge across the embedding space, forming clusters of related and increasingly complex challenges (Figure Db. For example, the bottom-right region features tasks involving kicking a ball into goal. Moving towards the bottom-left, the tasks gradually include dynamic elements such as moving obstacles or targets, while still focusing on kicking a ball. As we traverse the top-left region, the focus shifts to tasks that require pushing delivering objects into designated receptacles or target places. This marks a significant departure from the ball-kicking tasks and showcases the diversity of the generated environments. The top-right portion of the space is characterized by generations that emphasize navigation challenges, such as traversing (often moving) platforms or navigating across varied terrains.
Furthermore, there are niches where some generations appear to be variations of each other (Figure 2 For example, in the top-right corner, the generations surrounding task 90 involve retrieving an object and returning it to a designated location. While the overarching objective surrounding that niche remains consistent, the tasks and their learning environments exhibit notable differences. These variations manifest in several aspects, such as the simulated world in which the task takes place (e.g., task 90 outdoors, task 24 is in connected rooms, task 27 in two levels connected by ramp). Other variations include the number of objects that need to be retrieved (e.g., task 102 requires multiple objects to be retrieved, while task 90 only requires onc), and the allotted time for completing the task (e.g., task 27 has time limit of 3 minutes, while task 24 has time limit of minutes). These task variations can potentially allow for the development of robust and adaptable agents capable of handling wide range of scenarios (Adaptive Agent Team al. 2023
Overall, OMNI-EPIC generates tasks that significantly diverge from the seed tasks used to initialize the archive (the grey nodes in Figure 2). For example, despite the absence of dynamic objects in the
7

---

Published as a conference paper at ICLR 2025
seed tasks, OMNI-EPIC generates substantial number of tasks that incorporate dynamic objects and interactions, such as platforms that move borizontally and vertically, buttons and levers to activate, and moving obstacles. OMNI-EPIC not only explores different task niches (e.g.. navigating across different terrains vs. retrieving objects) but also generates interesting variations within each niche (e.g., retrieving objects in different simulated world settings).
SHORT RUN WITH LEARNING
bridge platform.
plafoon.
platform.
Köek
obstacles.
with segments gape moving x moving x to reach a platform.
bridge ball into goal.
X
Nevigate obstacle reach
target.
Thik Push lccaioon on dynamic platform.
X
Game
reach largel zone platform.
obsilaclea reach
X
obstacles reach target zons.
zone.
Task Jump from statle platform to stadic platform to cross lava.
Tank Jump from static v plarform static platform while Jank 18: Jump rolling loga bridge
Taak 12: Touch button activate drawbridge.
Issk Push cube target, zone static platform.
avoiding moving obstacllss.
Thie 1B: Push a cube in large Task 20: Push domino to start open area
Task 2t: Destroy tower of blocka
Task 22: Navigate dynamic platform to touch target.
Thes 23: Push a box next
chain reaction.
wall climb on tep of
Figure 3: Short Run with Learning. OMNI-EPIC adapts to current capabilities trained agents, generating tasks that both interesting and learnable. Tasks deemed interesting that successfully learned are marked by a check and failures by cross. Uninteresting tasks are not trained and hence not included here. Arrows between tasks indicate instances where OMNI-EPIC modified task that the RL. agent failed to learn, adjusting the task difficulty to facilitate learning.
To demonstrate OMNI-EPIC's ability generate tasks of suitable difficulty for training RI. agents, we conducted 5 short runs with RL agent training. Due to limited computational resources in our academic lab, the runs are shorter, but still show the creative potential of OMNI-EPIC and its ability to tailor tasks to the agents' abilities. The success detector (Section 3.6) evaluates if the agent has successfully completed each task. All short runs are initialized with task description seeds (Appendix N We find that OMNI-EPIC can effectively generate tasks and environment code that are not only creative and interesting but also learnable and appropriately challenging for RL agents (Figure 3 Appendix F In the run shown in Figure 3 the RI. agents successfully completed 16 tasks, failed at tasks, and task was deemed uninteresting by the post-generation Mol. We conducted user study with 50 participants and found 72.7% alignment rate between human evaluations and the success detector's assessments (Appendix H). Figure shows images the trained agents and their corresponding tasks, displaying only the header text of the task descriptions for better readability. Appendix shows the environment code with full task descriptions, images, and a task graph.
OMNI-EPIC leverages previously learned tasks stepping stones generate and master more challenging tasks. This iterative process allows RL agents to build upon existing skills to tackle increasingly complex environments. For example, since RI. agents in the archive successfully learned
8

---

Published as a conference paper at ICLR 2025
to cross pride-colored bridge with gaps (seed 1) and cross bridge with moving segments (task 1), OMNI-EPIC branched off these tasks to generate a more challenging task (task 4) combining these elements. This new task required the agent cross a rainbow bridge with gaps and moving segments. By continuing training from the policy learned on task 1, the RI. agent successfully completed task 4. Furthermore, OMNI-EPIC considers tasks that the RI. agents agents failed to learn. For example, since the RL agent failed to learn how to push a box to target location on dynamic platform (task 9), future tasks involving crossing platforms did not include pushing objects across them (e.g., tasks 11, 14, 15). This ensures that the generated tasks remain learnable and do not repeatedly incorporate overly difficult challenges. Similarly, when the RL agent failed to navigate through a terrain with obstacles (task 12), OMNI-EPIC generates an easier obstacle course (task 13) that maintains the same objective but reduces the number of obstacles. By using the agent's 's past experiences (both successes and failures) as building blocks, OMNI-EPIC generates curriculum of tasks that progressively and interestingly increases in difficulty while remaining learnable (additional ablations in Appendix G). This adaptive task generation process showcases OMNI-EPIC" capacity to create a tailored curriculum that maintains an appropriate level of challenge, ensuring that the generated tasks are neither too simple nor too complex.
QUANTITATIVE RESULTS
Stemty
0:35
GNNI-ERIC leariing only
ONNIEEPIC
0.332
-DINI-PPCC who Archee
AMNECS-DMM
0.322
0.325
0.212
OMN-EPCC
ONNIEEPIC
CANI-EPIC w/o auchive
leaming progress only
Suozessh.lp
Figure 4: OMNI-EPIC generates significantly more diverse tasks and continues to innovate throughout the run. (Left) Cell coverage of archive diversity plots in long runs with simulated learning by OMNI-EPIC and the controls. (Right) ANNECS- IMMNI measure of progress for OMNI- EPIC and the controls. Dotted lines are median values, shaded regions are 95% confidence intervals.
To evaluate the impact of having a task archive and the contribution of the notion of interestingness in OMNI-EPIC, we compare against two controls: (1) OMNI-EPIC without the task archive (OMNI- EPIC w/o archive), and (2) OMNI-EPIC without the models of interestingness (by removing the request for interesting tasks from the task generator prompts and skipping the post-generation Mol step) (OMNI-EPIC Learning Progress Only). The quantitative measures for comparisons are described next, and OMNI-EPIC significantly outperforms the controls on both metrics.
Task Diversity. To measure the diversity of generated task archives, we plot the cell coverage of task archive a 2D discretized plot. First, we use a pretrained text embedding model (OpenAl's text- embedding-3-small (OpenAl 2024)) encode the generated tasks (natural language description and environment code), then reduce the dimensionality to two via PCA (Maćkiewicz & Ratajezak 1993) (across all tasks from all methods) for easier visualization. We create 2D discretized plots by selecting a discretization level (e.g.. 50 in Figure 4 and generating uniform bins across the minimum and maximum values from the PCA embeddings. Each task in the archive is placed in the appropriate bin, and we count the number of unique bins the algorithm fills, a standard measure of diversity (Mouret Clune 2015) Pugh al. 2016). Each method is run with simulated learning, repeated 3 times. OMNI-EPIC achieves significantly higher cell coverage than the controls (p < 0.05, Mann- HannnWWitnee Whitney test). This shows that both the task archive and the Mol significantly and quantifiably contribute to OMNI-EPIC's ability to generate more diverse tasks (Figure 4). Appendix [ shows the archive diversity plots for each method and that the same results hold for different discretization levels.
Measure of Progress. While open-ended systems aim to endlessly create and learn new tasks, the question of how to measure progress in such systems remains. Wang et al. (2020) proposed tracking the Accumulated Number of Novel Environments Created and Solved (ANNECS) throughout the run of an open-ended system. Specifically, ANNECS requires that an environment created at a particular iteration (1) is not too hard or easy for the agent to learn and (2) must eventually be solved by the

---

Published as a conference paper at ICLR 2025
system. However, ANNECS lacks measure of how interesting or novel the newly generated task is. To address this limitation, we introduce a new metric to measure progress in open-ended systems, ANNECS-OMNI. Inspired by ideas in Zhang et al. (023), ANNECS- MANNCS-OMNI adds a third criterion to ANNECS: the new task must be considered interesting compared to previous tasks (approximated here by asking FM if the task is interesting given the archive of already tasks).
Each method is run with RL. training, repeated 5 times. OMNI-EPIC achieves significantly higher ANNECS-OMNI scores than the controls (p 0.05, Mann-Whitner U test). As the run proceeds, the SINECS-OMNT metric consistently increases for OMNI-EPIC, indicating that, for as long as we could afford to run it, the algorithm continuously creates meaningfully new and interesting tasks without stagnation (Figure 4. That is a new high watermark in our field's ' longstanding quest to create open-ended algorithms. The ANNECS metric for the different methods can be found Appendix
DISCUSSION, FUTURE WORK, CONCLUSION
OMNI-EPIC provides a general recipe for generating a potentially endless stream of learnable and interesting environments. By leveraging FMs generate tasks and their corresponding environment (and reward) code, OMNI-EPIC eliminates the need for handcrafted parameters or predefined task distributions, unshackling algorithms to have the potential to be truly open-ended. OMNI-EPIC can produce a wide variety of tasks, from simple to complex, and allows for the creation of environments that continuously adapt in response to an agent's 's developing capabilities. The results demonstrate its effectiveness in generating diverse and creative tasks, highlighting the potential of this approach for training more capable and intelligent agents. By leveraging the generative capabilities of FMs to create vast array of unique and challenging environments, OMNI-EPIC brings us one step closer to achieving Darwin Completeness, the ability to create any possible learning environment.
While we cannot rule out that OMNI-EPIC is creating environments similar to those in its training data, (1) even if it were, this would still be useful for open- endedness; (2) we believe it is generating novel environments, as we were unable to find the same environments when searching online; and (3) with longer runs, we hypothesize it could generate environments endlessly. Its task generator and Mol could generalize well outside the distribution of human data when conditioned on a growing archive of discoveries, though studying this remains a fascinating area for future research.
However, it is important to acknowledge that the current implementation of OMNI-EPIC is not yet Darwin Complete, primarily due to the limitations imposed by the choice of simulator. While OMNI- EPIC can generate a wide variety of tasks and environments. it is constrained by the capabilities and assumptions of the underlying simulation platform. To achieve true Darwin Completeness, OMNI-EPIC could simply be allowed to generate any code, including the ability to download, install, use, or modify any existing simulator, or even write the code for an entirely new simulator from scratch. Since the programming language used here (Python) is Turing Complete, generating code can, in principle, create any computable environment. Enabling the generation of arbitrary code would unlock the full potential of OMNI-EPIC and bring us closer to realizing the vision of Darwin Completeness. Of course, ever smarter FMs are required to take advantage of this opportunity.
The current implementation of OMNI-EPIC trains a population of specialist agents. We do not provide evidence of their generalization across tasks as it is not the focus of the paper. Future research could explore alternative training modalities. One promising direction is to train a single policy across all environments (Adaptive Agent Team et al., 2023), which could encourage the development of more versatile and adaptable agents (Appendix O). Another approach is to prioritize environments based on learning progress (Baranes & Oudeyer 2013 Kanitscheider et al. 202T) focusing on tasks that provide the most opportunities for improvement Each of these strategies introduces unique dynamics into the open-ended environment generation process, and understanding their effects on agent performance and generalization represents an exciting avenue for future research.
In conclusion, OMNI-EPIC represents a leap towards open-ended learning by generating a potentially endless stream of learnable and interesting tasks. Intriguingly. it also provides a new way of creating human entertainment and educational resources by offering a limitless supply of engaging challenges (Appendix B OMNI-EPIC could potentially be applied in myriad ways, covering anything from math problems and poetry challenges to games and virtual worlds. By leveraging FMs to create tasks and environment code, OMNI-EPIC opens up vast space of possibilities for Al and human agents to explore and master. By combining that expressive power with human notions of interestingness, OMNI-EPIC presents promising path towards the development of truly open-ended and creative Al.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was supported by the Vector Institute, the Canada CIFAR AI Chairs program, a grant from Schmidt Futures, a grant from G-Research, an NSERC Discovery Grant, the Center for AI Safety Compute Cluster, DARPA and generous donation from Rafael Cosman. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors. We also thank Aaron Dharna, Arthur Braida, Ben Norman, Cong Lu, Gabriel Béna, Luca Grillotti, Rach Pradhan, and Shengran Hu, for insightful discussions and feedback.
REFERENCES
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie -Schmie,, Michael Chang, Natalie Clay, Adrian Collister, et al. Human timesaeee adaptation in open-ended task space. arXiv preprint 1xXi:2301100600 2023.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv: 2019.
Adrien Baranes and Pierre- Yves Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49-73, 2013.
Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shaaar-Shwartz, Gillian Hadfield, et al. Managing AI risks in an era of rapid progress. arXiv preprint arXiv:2310. 17688, pp. 18, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108..
2021.
Herbie Bradley, Andrew Dai, Hannah Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth Stanley, Grégory Schott, and Joel Lehman. Quality-diversity through AI feedback. arXiv preprint arXiv:2310.1 2023.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 2020.
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference Machine Learning, 2024.
Jeff Clune. AI-GAs: Al-generating algorithms, an alternate paradigm for producing general artificial intelligence, arXiv preprint 1rx.vv1905..09900 2019.
Cédric Colas, Pierre Fournier, Mohamed Chetouani, Olivier Sigaud, and Pierre-Yves Oudeyer. Curious: intrinsically motivated modular multi-goal reinforcement learning. In International conference on machine learning, pp. 1331-1340. PMLR, 2019.
Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre Yves Oudeyer. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: short survey. Journal of Artificial Intelligence Research. 74:1159-1199 2022.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 13:13049-13661 2020.
Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, and Joel Lehman. Quality diversity through human feedback. arXiv preprint arXiv:23 10. 12103, 2023.

---

Published as a conference paper at ICLR 2025
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-langee languale models as success detectors. arXiv preprint arXiv:2303.07280, 2023.
Adrien Ecoffet, Jeff Clune, and Joel Lehman. Open questions in creating safe open-ended AI: tensions between control and creativity. In Artificial Life Conference Proceedings 32, pp. 27-35. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info .... 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04 10044,, 2023.
Nicolas Heess, Dhruva Tb, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint 1 2017.
JE Hopcroft. Introduction to Automata Theory, Languages, and Computation, 2001.
Andrew Jaegle, Sebastian Borgeaud. Jean Jean-Baptiste Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint 11X1v:2107.17795 2021.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp. 4940-4950 PMLR, 2021.
Minqi Jiang, Tim Rocktäschel, and Edward Grefenstette. General intelligence requires rethinking exploration. Royal Society Open Science, 10(6):23033,, 2023.
Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: generalization challenge in vision, control, and planning. arXiv preprint 1rXiv:1902.01138,, 2019.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint arXiv: 1006. 10729, 2018
Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton, Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, et al. Multi-task cur- riculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint arXiv:2/06. 14876, 2021.
Victoria Krakovna, Jonathan Uesato, Vladimirand Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of AI ingenuity. https: /ewppeiddd woogle////oooooooooooo wealle///icooo disovvr/bboog specification gaming t-e-ttee-iiee side - of a maai-iinnnnnityy/ 2020.
Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Ych, and Kenneth Stanley. Evolution through large models. In Handbook of Evolutionary Machine Learning, pp. 331-366. Springer, 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval-augmented genera- tion for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Bryan Lim, Manon Flageat, and Antoine Cully. Large Language Models as In-context Al Generators for
arXiv preprint arXiv:2404.15794 2024.
Yecheng Jason Ma. William Liang, Guanzhi Wang. De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.1293. 2023.
Andrzej Mačkiewicz and Waldemar Ratajezak. Principal components analysis (PCA). Computers Geosciences, 1933):303-342, 1993.
12

---

Published as a conference paper at ICLR 2025
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXivv1504.04909, 2015.
OpenAl. Text embedding small. https::/plaaformm openai. com http///cc///cc//c/eeeeses//s//
humbbeddiigg///mmeedd.og
2024. Accessed: 17 May 2024.
Pierre- Oudeyer and Frederic Kaplan. What is intrinsic motivation? A typology of computational approaches. Frontiers in neurorobotics, 1:108, 2007.
Yves Oudeyer, Frdric Kaplan, and Verena Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions evolutionary computation, 11(2):265-286. 2007.
Jack Parker-Holder, Minqi Jiang. Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In International Conference on Machine Learning, pp. 11473-17498, PMLR, 2022.
Justin K Pugh, Lisa Soros, and Kenneth O Stanley, Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:202845, 2016.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8766 PMLR, 2021.
Mikayel Samvelyan, Akbir Khan, Michael Dennis, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Roberta Raileanu, and Tim Rocktäschel. MAESTRO: Open-ended environment design for multi- agent reinforcement learning. arXiv preprint arXiv:2303.03376 2023.
Noor Shaker, Julian Togelius, and Mark J Nelson. Procedural content generation in games. 2016.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Pannecrshelvam, Mare Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Kenneth O Stanley and Joel Lehman. Why greatness cannot be planned: The myth of the objective. Springer, 2015.
Kenneth O Stanley, Joel Lehman, and Lisa Soros. Open-endedness: The last grand challenge you'te 've never heard of. While open-endedness could be a force for discovering intelligence, it could also be a component of Al itself, 2017.
Shyam Sudhakaran, Miguel González- González-Duque, Matthias Freiberger, Claire Glanois, Elias Najarro, and Sebastian Risi. Mariogpt: Open-ended text2level generation through large language models. Advances in Neural Information Processing Systems, 36, 2024.
Richard Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference on intelligent robots and systems (IROS).
Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius. Level generation through large language models. In Proceedings of the 18th International Conference the Foundations of Digital Games, pp. 1-8, 2023.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John Balis, Gianluca De Cola, Tristan Delcu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17033 2024.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
13

---

Published as a conference paper at ICLR 2025
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.1629) 2023a.
Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang. Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. arXiv preprint arXiv:2310.01361, 2023b.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poct): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv: 2019.
Rui Wang. Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International conference on machine learning, pp. 9940-9951. PMLR, 2020.
Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arX1v:2311.01455, 2023c.
Abhay Zala, Jaemin Cho, Han Lin, Jaehong Yoon, and Mohit Bansal. EnvGen: Generating and Adapt- ing Environments via LLMs for Training Embodied Agents. arXiv preprint 0rXiv:2403.12014 2024.
Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open-endedness, via models of human notions of interestingness. arXiv preprint 1rX11:230.001111,200
14

---

Published as a conference paper at ICLR 2025
A REPRODUCIBILITY
For full transparency and to ensure reproducibility, we have open-sourced all code associated with this work. The code can be found at hhttts://ggiieee http:///hhhaa..commmaeeeeeeeaaaa
B GAME INTERFACE
Instructione bost
Burn right
jurp
Luvvl
the Largel Use
Thoust Levat
Figure 5: OMNI-EPIC in game interface.
Apart from training agents, OMNI-EPIC also be good form of human entertainment! We created a game interface whereby players can control the R2D2 robot with keyboard inputs, and play the generated tasks (Figure 5. OMNI-EPIC dynamically generates the next interesting level for the player, adjusting based on the player's 's skill level by suggesting tasks that are not too easy or difficult. OMNI-EPIC opens
era for games, where procedural content is automatically generated and tailored to the player's abilities, ensuring a consistently engaging experience.
15

---

Published as a conference paper at ICLR 2025
C VLM AS SUCCESS DETECTOR
The success-checking function can easily implemented simulated environments where infor- mation can be readily accessed through code (Section 3.6). However, it may face challenges in real-world scenarios, off-the-shelf closed-source video games, or even within simulated environments for tasks that involve visual assessment (e.g.. building a castle, arranging boxes to resemble an elephant), where the required information is not directly available or difficult to evaluate using only code. A natural solution is use VLMs as success detectors (Radford et al. 202T). VLMs can potentially detect success on wider range of tasks (e.g.. tidying a room, doing backflip) (Du et al. 2023) than code generation. We input snapshots of the agent's behavior every second, the natural language task description, and environment code (sce below). Since preliminary testing found that current VLMs are not yet accurate enough to be used as success detectors, we use code generated by LLMs for this purpose instead. However, expect VLM capabilities to rapidly improve time, eventually achieving higher accuracy and making them viable for future use.
System Prompt:
You are an expert in Python programming and reinforcement learning. Your goal is to evaluate if a robot has solved a task. You will be provided with the task description, the corresponding environment code and an image containing snapshots of the robot attempting to complete the task. Your objective is to describe the image, reason about whether the task has been completed and determine if the robot has solved the task.
Instructions:
In the description of the image, describe the environment and the behavior of the robot.
In the reasoning, analyze if the environment corresponds to the task description and if the behavior of the robot meets the requirements for task success.
The task is considered failed if the environment is constructed in way that makes solving the task impossible.
unsure, make educated guess and always provide answer. If you are unsure, say that it has failed.
If you
Robot description: [ROBOT_DESC)
Desired format: Description of the image: <image description>
Reasoning for task success/failure: <reasoning>
Did the robot solve the task?: <Yes/No>
User Prompt:
Task description and envirnnment code: {ENV CODE}
Image Examples:
16

---

Published as a conference paper at ICLR 2025
Task Description: Push box a target area.
Task Description: Cross pride colored bridge with gaps while avoiding moving obstacles to reach platform.
Task Description: Navigate multi-evel environment by ascending and descending multiple staircases.
D SUPPLEMENTARY MATERIALS FOR LONG RUN WITH SIMULATED LEARNING
Takid Hoskaarr
12s1 Sunza Rzon
Succnschily
Tedl TO
ZZZ
Seed Chrsu prilore and wAch plattemm.
Commeer Bet Ios Delkery
Upsuri Taget
Seed2 ts reeeh plite.r.
The robot FiEE galk lnpinng mver pape.
Choncinlee with 20.99 turge
Tanic sepalate pail Will
Vikcea Reccbslihioag noxing syeee lagata
Resove Tacc 24
searrroiigg obsscies and
1-S-- robet
Seed goal.
Figure 6: Long run with simulated learning task graph with all parent-child task connections. This figure presents the same task graph as Figure 2 but with all parent-child task connections displayed. The node color reflects the generation number of the task. A check mark in the node means that the task was successfully learned. A 777. symbol means that the task was deemed uninteresting and discarded. The node connections illustrate which tasks were conditioned on when asking an FM to generate similar yet new and interesting task. Due to the high density of tasks and connections, visualizing all relationships clearly in a static image is challenging. To better understand and navigate the intricate web of task relationships, interactive version of the task graph is available at hthttps://dub sh/ omn ompiepic
17

---

Published as a conference paper at ICLR 2025
The natural language descriptions and environment code for all tasks shown in Figure are available at https: /ubb sh/ omniepic For conciseness, we present the full natural language descriptions of all magnified tasks, and the environment code of one magnified task (Task 76):
import numpy as np
from oped. envs. r2d2. base import R2D2Env
class Env (R2DZENV)
Construction Site Cleanup
Description:
The environment is a construction site with dimensions 10 x 10 m. There are 3 piles of different types of debris scattered around the construction site:
bricks (each 0.2 m x 0.2 m x 0.2 m)
5 metal scraps (each 0.5 m x 0.25 m x 0.25 m)
5 wooden planks (each m 0.2 m x 0.1 m)
There are also 2 robotic construction vehicles moving around the site in pre programmed patterns:
A bulldozer that pushes dirt piles around in rectangle, 4 x
m
An excavator that swings its arm and bucket back and forth in a 180 degree arc, 2 in radius
one side the construction site are 3 square receptacle bins, each m x 1 m x 0.5 m tall, labeled for each type of debris ( bricks, metal, wood).
The robot starts in one corner of the site.
Task:
The robot needs to pick up each piece of debris and place it in the correct receptacle, while avoiding collisions with the moving construction vehicles. The robot can only carry one piece of debris at a time.
Success Conditions:
The task is complete when all pieces of debris have been placed in their correct bins, and the robot has returned to its starting position.
Time Limit:
The robot has 10 minutes to complete the cleanup task.
Rewards:
Provide a small reward for each piece of debris successfully picked
up.
Provide moderate reward for each piece of debris placed in the correct bin.
Provide a large reward for completing the task and returning to the start position.
Provide small penalty for each collision with the construction vehicles.
Termination:
The episode ends if the robot flips over, or if the time limit is exceeded.
def is
iuper()...nit___
self. 110.0, 10.0, 0.1]
self self.siteeposition 10.0, 0.0, 0.0]
self.site-id self.
0, half_extents=[see (oreasse ee
site size[0] 2, self.site size / 2, self. size /
2], positionsself.sia position, color= 0.5, 0.5, 1.0))
18

---

init Published as a conference paper at ICLR 2025
self httpp//wwwgggyyooooooooooooooooe lateralFriction=0.33 ,8, restitution=0.5)
linkIndex= 1,
self.brick.size [0.2, 6.2, 0. 0.2]
self.metal_size = [0.5, 0.25, 0. 25]
self.wood = [1.0, 0.2, 0.1]
self.debris. sizes [self. size,
size, self.
woodd_iize
delf.nun.debris.types
Len(self.deerrs..sil
self htt..nundeerisseecc =
self seff.debris_ids--
for i in rangeeself.... debris types):
for range(self.num debris each)
debris positioosiionnn = [np. randoanddoommm wnwform(-sslfffs.eemoo wwt.seeeff.eee.eeeeeeeeeeeeeeeet s
2 + 2.0, self.site_sizeffe / 2.0), np. random.
uniform( self.site.sizelle
2.0, self.site. size
[1] 2 2.0), debris_id melf.create..box(mss===_. sebris_sizestill0) self.debris.sizesiiee
2]
half_extents= / 2,
self.dGbris_.iiessi]]]]] 21, position posiiiionn
sebrisspositiin, color=[0.8, 0.8, 0.8, 1.0))
self.debris..ds..ppeml. (debriis id, i))
self.bulldozer size [1.0, 0.5, 0.5]
=
self wtt.bulldozzrr.osittiioo init seleff.sit size[0] / 4, 0.0, self bbu.ldozerrssizei2]
self.bullldzer::dd
half_ ealffexxtets==self. 2, self.
bulldozer_size(0] / 2, Luff.bulldozer-sizzee
bulldozer.size(2] / 2],
color=[1.0, 1.0, 0.0, 1.0))
self.excavator.raddus 2.0
seffeeecaavttr.ppsttion.nit
size[0] / 4, 0.0, 0.0] wtw. (ma radius=0.5,
posittooseet.ecccaattrrp..ooooo. init, color=(0.00 1.0, 1.0, 1.01)
self. www.erettcleesiiee [1.0, 1.0, 0.5]
self reeeeppaaaaeee.ssiiionss [[self. size[ / self.
receptacle size(0
www...ellste.. site size[1] 3, self.
size[2] / 2], [self.site size[0] / self.
receptacle size[0] ,, 0.0, self. size[2] 2],
self.site.sizelf/ 2
self self. [0] / 2, self. 2]]
site site_ seep.rreeetacee
self. receptacee.dc
for i in receptacle_id b (oa.s= half exteett=sss=sff
receptacle.sizel0]] / self. receptacceee size[]] / 2, self
Treceptacle.sizee222 / 2], position=self.
[i], color=(0.2, 0.2, 0.2, 1.0))
self. refffrrcertteeeee.d..pppeeerrree r rottootttooosittioniiitt (-slf
/ 2.0, self. self. robot. links [
Site_sizee1) 2.0, setf.site.sieel111
0.1]
w p-getQuatermionnrromeee romEuler self. relo. ortetttoieennioooom init = self.
(10.0, 0.0, np.pi 4])
self.time limit 600.0
self.debris.ppik.rewarr reward = 1.0
self self.debris.place_reee reward = 10.0
Baf.ttask.completeerreeees
100.0
self.collission.eenatte -1.0
def create_box mass, half_extents, totttexttettssspiiiiinn color):
collision id Srte GEOM BOX, halfExtents=halff extents)
shape id self pet.fe.p.creeeeesssssuaaaaaaaaap GEOM_BOX, halffxttents-halF.exteettss rgbacolor=color) oopcooooooo
19

---

Published as a conference paper at ICLR 2025
pette shapseeiioosss____ppeeea_e shape_id,
return hrt..ppccreeeettttbbooylllssssss Siseoolisssssshppeeddooooooooooo hBBoVUssuuShhggennneeevv/ssssssp htg.a//wwweeeaaaaooooooooooooaor
basefositionpposttion)
def create ssetteesspeeeesel.., mass, poeiluiin
collision shape id sste.paccrattllllsssooooohhehhee
-p. SPHERE, radiussradiuss
visual_shape_ize = aat..ppperratlllaaaasaiacheeeeee
GEOM_ racbusssrrddusrrgggoolloooooooo
return BBt.-
basedolLissooshhappnnnxxooooLLll shap
baselüissuaShhhpphnnneevvvssssss htt.s//sss....pppiioooooooaoooor
bssePositionnposittinn)
def get_ooject_positiinnsellll (ojjct_id): ;
return np. wtt.arrwww.........eeeesstttoooa [0]]
def reset(self):
observation = super() reset()
tilf.time 0.0
bicked picked [False]
ids)
self.debris placed = [False] con(self.debris ids)
self hwt.etttsssosiiiooodddinnnaattto
robot id, self
.robot_ppostionniiiis self.rooot.orientttiinnnnnni
return observation
def step(self, action):
observation, reward, terminated, truncated, info super() (oper().stepp) ( action)
self. self.dt bulldozer-position
objeccttpssiiionsssle. set.iionnsseff.ubbuleeeeeeee bulldozer idd wuw www.uullboerrrpssttioniiiii
bulldozer
2.0
np.sin np.pi self.time / 20.0)
butldozer_position[]] =slffbultooerr_psiitioiittttt
1.5
(p.sinn22x self.time 30.0)
p.
id,
mulldozerr_position. (0.0, 0.0, 0.0, I.0])
excavator.posiition self. wttp.e/gt.iiieeeppssttooooollooe www.tionsself..o.mmmoooom ott.::/geggiitttppiiiooalleeeaa. wxt.iionnsell.eeeeattttttro id)
excavator.posittion00 ott.::/:oo.............oooooooo.
self.
excavator_raadius np. co. stm.
self. otg.://ooe..a.c.cttttttoooooooo. 10.00 10.0)
excavator-possitoonll
+ self. eeff.prceessssssttooooooiotitttr self. tee. 10.0)
excavator_radiiu * wtt.:aiiuuunniinninnnnn.ooooo
excavator.posiition, [0.0, 0.0, 0.0, 1.0])
return (observation, reward, terminated, truncated, info)
def get tast-tas_2oo fasskrrewardsselle action):
reward_pick 0.0
reward_place 0.0
reward_complett penalty collision 0.0
0.0
for i, (debris id, debris type) in enumerate(self debris ids): if not self.debris celf.debbisspickeelil and len Len getoontactPoinss (bodyAsselffrooot.robll) self.debris_piccee([]]==1 True
bodyB-debris. 0:
self.debris += s stt.
if self wtt.deisssppcceeeii]1. and (not self.debris.emmm placed and
Llentself....getcontttttooonnnns (bby:innndbodyydderiiss..... bodyB=self (recpptccee.ddssccerrrss.ppp)p))
0)
self. celf.debriss.paace(]]===== reward -waacerdpllaeeeeee www..debris..p.....eeeee... Silebrispplace..eo reward
)lacced if all(self deiff.deebis..m placed) and np. np. robot. Links ['
base 1 np. mp..aarreyseeef.rrrooooootttoo..
[:2])] < 1.0:
reward_ccmplete = self task cast
20

---

Published as a conference paper at ICLR 2025
if tenfsset..pp.eetotttttoootonss robot. robot_ bodyB=
self. [tff.bulldozer_idd] 0
Lenlseelf--D--ettonteetooo (bodyA menalty.collision self collision
=self. robot. robot_id
0:
return { reward -ickk'rddpickk seward-pick, pewce'-place': reward reward-place,
'reward complete'..... reward_complete penalty-collision penalty.collision)
def get_ (er.tmeeeeettteeee. action):
if self. time >= time-mimiiitttt
return True if np. dot dot (np. (np.asarray(11), 0, 11), np.asarray(self... petMatriifrromquateniion
robot. links ('base' orientation)
) res 3);; 2]) < 0.5:
return True
return False
def get_success(selff)
return all(sel dewbrispplaceeee and np. Linall..gggnnn Ganaag..ormmself robot.
links base position :2] mp.asarray(self.
robot position init[:2])) 1.0
Task 23
Search and Rescue in a Multi Room Environment
Description:
The environment consists of large, two two-story building with multiple rooms connected by doors, hallways, and staircases. The layout is fixed but contains challenging features:
8 rooms total (4 on each floor), connected by 2 hallways (one per floor)
Each room is
m. Hallways are 20 long and 2 wide. Staircase is 5 m long.
Rooms contain various static and movable obstacles like furniture. Doors are 1 m wide and initially closed. Robot must push them open. A target object (a 0.5
m box) representing person is located in one of the rooms on the second floor. Its location is fixed for now.
0.5
The robot starts at the entrence of the building on the first floor. The robot must systematically explore the building, locate the target object, attach virtual tether it, and guide it back to the entrance while avoiding obstacles.
Success Conditions:
The task is considered complete when the robot brings the target object rest within of the entrance location.
Time Limit:
The robot has time limit of 10 minutes to complete the task.
Rewards:
Provide a small reward for exploring new areas of the environment.
Provide moderate reward for locating the target object. Provide large reward for successfully bringing the target object to the goal location.
Provide small penalty for collisions with obstacles to encourage careful navigation.
Termination:
The episode terminates if the robot flips over or if the time limit is exceeded.
Task 60
21

---

Published as a conference paper at ICLR 2025
Simplified 1-vs-1 robot soccer.
Description:
The environment consists of large flat ground representing a soccer field.
Two goals are placed on opposite ends of the field, each defined by two posts 2 meters high and 3 meters apart.
The robot is placed on one side of the field, facing the opponent's goal.
A basic opponent robot is placed on the other side, initially stationary,
A soccer ball is placed the center of the field. The objective is for the robot to navigate to the ball, take possession of it, and kick it into the opponent's goal while defending its own goal.
In the first stage, the opponent robot remains stationary. In the second stage, the opponent robot also moves to chase after the ball and attempt to kick it into the robot's goal.
Success:
The task is completed successfully if the robot is able to kick the ball into the opponent's goal while preventing the opponent from scoring.
Rewards:
The robot is rewarded for possessing the ball, defined as being within 1 meter of the ball while the opponent is not.
The robot is rewarded for bringing the ball closer to the opponent's goal.
The robot is rewarded for kicking the ball with a velocity toward the opponent's 's goal.
The robot is penalized if the opponent takes possession of the ball or if the ball goes out of bounds.
The robot is greatly rewarded for scoring goal and penalized if the opponent scores.
Termination:
The task is terminated if goal is scored by either side, if the ball goes out of bounds, or if a time limit is reached.
Task 74
Receive pass and kick ball past moving obstacles and goalie into goal.
The robot is placed on large flat ground designed to look like soccer /football pitch with markings.
teammate robot is positioned 5 meters to the side of the main robot. After a fixed delay at the start of the episode, the teammate robot passes a ball towards a point 3 meters in front of the main robot at speed of 3 m/s. The pass target point is fixed, but the main robot' s initial position has some randomness, so it will need to adjust to align itself with the passed ball.
cylindrical obstacles of 0.5 meter radius and 1 meter height placed at random positions 3 8 meters front of the robot's initial position. The obstacles are colored to resemble soccer players and move randomly back and forth over a 2 meter range at speeds between 0.5 and 1.5 m/s. The obstacles do not actively chase the ball but will block it if it comes near them.
A goal area 5 meters wide and meters deep is placed 10 meters in front of the robot's initial position, centered along the front line.
22

---

Published as a conference paper at ICLR 2025
A goalie obstacle meter wide, 0.5 meters deep, and 1.5 meters tall moves side to side across the goal line at speed of 1.5 m/s, tracking the ball when it is within meters. The goalie is colored differently than the field obstacles.
The task is for the main robot to receive the teammate's pass and then kick the moving ball past the field obstacles and goalie to score goal, without touching any of the obstacles.
Task 84
Busy Restaurant Table Service
Description:
The environment is busy restaurant dining area, consisting of 12 x 12 room with tables, an entrance door, and a kitchen door. Restaurant patrons (simulated as cylinders) continuously enter from the entrance, move to a table, stay for a period of time, and then exit, making the environment dynamic.
The robot must perform two types of table service tasks:
1) Table setting: The robot must collect clean dishes and utensils from the kitchen and arrange them properly on the tables in preparation for new patrons. Dishes should be stacked neatly.
2) Table busing: After patrons leave, the robot must clear dirty dishes and utensils from the tables, load them into a bin, and return the bin to the kitchen. The bin has a maximum capacity that the robot must respect.
The robot must avoid colliding with patrons and restaurant furniture it navigates.
Dishes will break if dropped. The robot must handle them carefully.
Success Conditions:
For table setting, success means all necessary dishes and utensils are properly placed on empty tables prior to new patrons arriving. Dishes should be undamaged and neatly stacked.
For table busing, success involves clearing all dirty dishes from unoccupied tables, loading them into the bin without exceeding capacity, and delivering the bin to the kitchen. No dishes should be broken.
The robot should avoid any collisions with patrons or furniture throughout the episode.
Time Limit:
The episode lasts for 10 minutes of simulated time. The robot must continuously perform both table setting and busing tasks during this period as needed.
Rewards:
Provide a moderate reward for each successful table set, with all dishes placed neatly and undamaged.
Provide moderate reward for each successful table busing, with
broken dishes and the bin delivered to the kitchen.
Give small reward for carefully handling dishes without breaking them
Assign a substantial penalty for any collisions between the robot and patrons furniture.
Assign a small penalty for dropping and breaking a dish. Provide a small reward at each timestep for maintaining a tidy dining room, with no dirty dishes left on unoccupied tables.
Termination:
The episode ends if the robot crashes into a patron or furniture, or if the time limit is reached. The episode is considered successful if the robot consistently performs both table setting and busing throughout the full time period with no collisions or broken dishes.
23

---

Published as a conference paper at ICLR 2025
Task 88
Bounce Shot Scoring Challenge
Description:
The environment consists of walled field measuring 20 10 The field contains 3 ramps/mounds (triangular prisms) placed randomly, each measuring 2 m long 1 wide x 0.5 m high.
A single goal 3 m wide and 1 m high is placed centered on one end of the field.
A ball (0.5 diameter) is released from a random point along the opposite side of the field from the goal. The ball is given an initial velocity of 5 m/s at a random angle towards the goal end of the field.
Due to the ramps the field, the ball will bounce and roll unpredictably as it crosses the field.
The robot begins in the center of the field on the goal side.
Task:
The robot must intercept the bouncing ball and kick it into the goal to score a point. The robot should dynamically adjust its approach to the ball based on the ball's changing trajectory after each bounce off a ramp. The robot needs to time its kick to hit the ball at the optimal point in its bounce to redirect it into the goal.
Success Conditions:
The task is completed successfully if the robot causes the ball to fully enter the goal area.
Rewards:
Small reward for decreasing distance to the ball, scaled higher as the ball gets closer to the robot's 's side of the field
Moderate penalty if the ball's velocity drops to zero it leaves the field without scoring
Large reward for kicking the ball, scaled by the ball's post-kick velocity in the direction of the goal
Large reward for scoring goal
Termination:
The episode ends if the ball comes to rest, leaves the field, or enters the goal area.
Task 90
Outdoor Retrieval Challenge
Description:
The environment is a large outdoor field 50 x 50 m, with the robot starting in the center.
The terrain includes various features:
steep hill with 30 degree incline
A shallow stream, 0.5 m deep and m wide
Patches of loose sand and gravel
Scattered trees and boulders up to 2 tall 0.5 m diameter soccer ball is placed randomly between 10 and m from the robot's starting position.
The robot must locate the ball, pick it up or get it balanced on its back, and return it to the starting location.
The ball is heavy enough that it can't simply be pushed the whole way.
The robot needs to lift/carry it.
If the robot drops the ball or it rolls more than 5 m away, it is replaced at new random location.
24

---

Published as a conference paper at ICLR 2025
Success Conditions:
The task is completed when the robot returns to within 2 of its starting position with the ball balanced on its back or in its possession.
Time Limit:
The robot has minutes complete the retrieval.
Rewards:
Provide small reward for exploring the environment and locating the ball
Provide moderate reward for successfully getting the ball balanced on the robot's back or lifted off the ground
Provide large reward for returning to the start area with the ball Provide small penalty if the ball is dropped or lost, to encourage careful handling
Termination:
The episode ends if the robot flips over and can't right itself, or if the time limit is exceeded.
Task 103
Conveyor Belt Item Delivery
Description:
The environment consists of two m x 5 m platforms connected by a 5 m long conveyor belt. The conveyor belt moves from left to right at speed of 0.5 m/s.
the right platform, there are three target areas marked by m 1 colored squares (red, green, blue).
Also on the right platform, there is a dispenser that releases items onto the conveyor belt at random intervals between 2 5 seconds. The items are 0.5 m cubes colored either red, green, or blue. The robot begins on the left platform.
Task:
The robot must jump onto the conveyor belt, pick up the colored items, ride the conveyor to the right platform, and deliver each item to the target area matching its color. The robot should try to deliver as many items as possible to the correct targets within the time limit.
After delivering an item, the robot must jump back onto the conveyor belt and return to the left platform before the next item is released. The robot should wait on the left platform for the next item.
Rewards:
Provide small reward for picking up an item from the conveyor belt. Provide a large reward for delivering an item to the correct target
area based on color.
Provide small penalty for delivering item to the wrong target area
Provide a moderate penalty if the robot falls off the conveyor belt or platforms.
Success Conditions:
The task is considered complete if the robot successfully delivers at least 5 items to their correct target areas within the time limit.
Time Limit:
The robot has minutes to deliver as many items possible.
Termination:
The episode ends if the robot falls off the platforms or conveyor belt, if the time limit is reached.
25

---

Published as a conference paper at ICLR 2025
Task 122
Kick ball through moving gates to zones marked by moving targets.
The environment consists of a large flat ground. A ball is placed 5 meters directly in front of the robot. After a delay of 1-2 seconds ( randomized), the ball begins rolling straight ahead at constant velocity of 1 m/s (randomized).
Five vertical rectangular gates, each meters tall and 1 meter wide, are placed randomized positions between to 25 meters in front of the ball's initial position, at randomized lateral offsets up to meters on either side. The gates translate laterally with randomized constant velocities between 0.2 1 m/s, reversing direction each time they move 5 meters from their initial position. The gates' movement triggered at the start of the episode.
Five target zones are marked on the ground at randomized positions between 10 to 30 meters in front of the ball's initial position, at randomized lateral offsets up to 10 meters target zone a circle with radius meters. either side. Each the center of each target zone is a flat cylindrical marker, 2 meters in radius and 0.1 meters tall. The markers rotate in place with randomized constant angular velocities between 0.1 0.5 rad/s, switching between clockwise and counterclockwise rotation each time the ball passes through gate. The markers' rotation is triggered at the start of the episode.
The robot is initialized 3 meters behind the ball, facing the direction of the ball's initial velocity.
The task is for the robot to kick the ball through each gate in order, aiming to get the ball to stop within the corresponding target zone after it passes through the gate. The robot should decide when and how to kick the ball based the observed motion the gates and markers.
Passing through a gate awards points if the ball subsequently stops in the correct target zone, and 1 point otherwise. The episode ends when the ball passes through all gates, or maximum time limit is reached. The robot should aim to maximize its total score.
Task 155
Task: Dynamic Target Pushing Challenge
Description:
The environment is a 15 x 15 m open area with a flat surface. There are 3 dynamic targets (spheres with a 0.5 m diameter) that move along predefined linear paths. Each target moves back and forth along speed of 0.5 m/s.
5 path
The robot starts at the center of the area.
Three goal zones (1 m x m squares) are located at the edges of the
area: one at the north edge, one at the east edge, and one at the west edge.
Task:
The robot must push each moving target into its corresponding goal zone. The targets be pushed in any order, but each target must come to rest fully within its goal zone to be considered successfully delivered.
Initial Positions:
Robot: Center of the area at (0, 0, 0.5)
26

---

Published as a conference paper at ICLR 2025
Target 1:
Moving along the path from (-2.5, 5, 0.25) to (2.5, 5, 0.25)
Target 2: Moving along the path from (5, -2.5, 0.25) to (5, 2.5, 0.25) Target Moving along the path from (-5, -2.5, 0.25) to (-5, 2.5,
0.25)
Goal Zone 1: Centered at (0, 7, 0.05)
Goal Zone 2: Centered at (7, 0. 0.05)
Goal Zone 3: Centered at 7, 0, 0.05)
Success Conditions:
The task is considered complete when all three targets come to rest fully within their respective goal zones.
Time Limit:
The robot has time limit of minutes to complete the task.
Rewards:
Provide small reward for moving closer to a target.
Provide a moderate reward for reaching a target.
Provide large reward for pushing target towards its goal zone.
Provide significant reward for delivering target to its goal zone. Apply a small penalty for collisions with the targets the boundaries of the area.
Termination:
The episode terminates if the robot flips over, if a target is pushed out of bounds, or if the time limit is exceeded. The episode should terminate with success if all targets are delivered to their goal zones.
Task 176
Task: Push two balls into separate goals while avoiding static obstacles.
Description:
The environment consists of large flat ground.
Two balls, one red and one blue, are placed at fixed initial positions on the ground. The
(2.0, -2.0, 0.5] and the blue ball is positioned at [2.0, 2.0, 0.5].
ball is positioned
Two goal areas are defined pairs of posts. The red goal is defined by posts placed at [10.0, 2.0, 1.0] and [10.0, 1.0, 1.0]. The blue goal is defined by posts placed at [10.0, 1.0, 1.0] and [10.0, 2.0, 1.0].
Three cylindrical obstacles of 1 meter radius and 2 meters height are placed at fixed positions: 15.0, 0.0, 1.0], 17.0 -1.0, 1.01, and [7.0, 1.0, 1.0].
The robot is initialized at [0.0, 0.0, 0.5], facing the positive direction.
The task is for the robot to push the red ball into the red goal and the blue ball into the blue goal while avoiding the obstacles.
Success:
The task is completed successfully if the robot pushes both balls into their respective goals without colliding with any obstacles.
Rewards:
The robot is rewarded for decreasing its distance to each ball.
The robot is rewarded for pushing each ball towards its respective goal
The robot is penalized for colliding with obstacles. The robot is greatly rewarded if a ball passes between the goal posts of its respective goal.
Termination:
27

---

Published as a conference paper at ICLR 2025
The task is terminated if the robot collides with an obstacle, if a ball collides with an obstacle, if time limit of 60 seconds is reached.
Task 178
Task: Coordinate with another moving ball to hit target
Description:
The environment consists of large flat ground.
Two balls are placed 5 meters in front of the robot, spaced 2 meters apart laterally. Both balls begin rolling in parallel at 1 m/s after short delay of second.
A target area defined by a 2x2 meter flat vertical target is placed 10 meters from the balls' initial position. The target moves laterally back and forth at 0.5 m/s, covering 3 meters to each side before reversing direction.
The robot is initialized meters behind the balls, facing the direction of the target.
The task is for the robot to coordinate its movements to kick both balls such that they hit the moving target simultaneously.
Success:
The task is completed successfully if both balls hit the front face of the moving target simultaneously.
Rewards:
The robot is rewarded for decreasing the distance to each ball as they move.
The robot is rewarded for matching the velocity of each ball it approaches.
The robot is rewarded for kicking each ball with a velocity that is likely to reach the target.
The robot is greatly rewarded if both balls hit the front face of the moving target simultaneously.
Termination:
The task is terminated if either ball stops moving, if the balls do not hit the target simultaneously,, if a time limit of 30 seconds is reached.
Task 196
Task: Dodgeball Arena
Description:
The environment is 15 x walled arena with flat ground
surface.
There are 5 automatic ball launchers positioned around the perimeter of the arena. Each launcher can shoot foam ball (0.2 m diameter) at varying speeds (up to 5 m/s) and angles.
The launchers are programmed to shoot balls at random intervals ( between 1 to 3 seconds) and in random directions.
The robot starts in the center of the arena.
Task:
The robot must avoid being hit by the foam balls for as long as possible. The robot can move around the arena and jump to dodge the balls.
Success Conditions:
The task is considered successful if the robot can avoid being hit by any balls for a duration of 2 minutes.
Rewards:
28

---

Published as a conference paper at ICLR 2025
Provide small reward for each second the robot avoids being hit. Provide large reward for successfully avoiding all balls for the entire duration.
Apply a small penalty for each ball that hits the robot.
Termination:
The episode ends if the robot is hit by a ball, or if the time limit of minutes is exceeded.
Task 197
Task: Jumping Over Gaps
Description:
The environment consists of 10 10 platform with series of gaps that the robot must jump over. The platform is divided into three sections:
1. The first section (3 x 10 m) has three gaps, each 0.5 m wide, spaced m apart.
2. The second section (3
10 m) has two gaps, each wide, spaced
m apart.
3. The third section (4 m x 10 m) has one gap, 1.5 wide, located 2 m from the end of the platform.
The robot begins at the start of the platform, facing the positive axis.
Task:
The robot must navigate through the three sections of the platform, jumping over the gaps to reach the end of the platform.
Success Conditions:
The task is considered complete when the robot reaches the end of the platform without falling into any gaps.
Rewards:
Provide small reward for each gap successfully jumped over.
Provide large reward for reaching the end of the platform. Apply a small penalty for each failed j jmp (falling into a gap).
Termination:
The episode ends if the robot falls into gap, flips over, reaches the end the platform.
29

---

Published as a conference paper at ICLR 2025
E SUPPLEMENTARY MATERIALS FOR SHORT RUN WITH LEARNING
20
Generation Number
21
Tusk
Took
Teal
Tamac Good
ToEk
Task
Seed
Task 2
T
Tai 6
Than
Bood
Figure 7: Short run with Learning task graph. The node color reflects the generation number of the task. node with a check mark indicates successful task learning, while a cross mark denotes task that was attempted but failed be learned. A ZZZ symbol means that the task was deemed uninteresting and discarded. The node connections illustrate which tasks were conditioned on when asking an FM to generate similar yet and interesting task.
Task
import numpy as np
from oped envs. r2d2. base import R2D2Env
class Env (R222Env):
Cross a bridge with moving segments to reach platform.
Description:
A start platform and an end platform (each m in size and 0.5 in thickness) are placed 20 m apart.
The two platforms are connected by bridge (2 wide) divided into 5 segments of equal length (3 each).
Each segment moves up and down independently with sinusoidal motion. The amplitude is 1 and the period is seconds, with each segment offset in phase by 0.4 seconds from the previous one
The segments are colored red, orange, yellow, green, and blue. the start platform.
The robot is initialized The task of the robot is to cross the dynamic bridge to reach the end platform as fast as possible by timing its jumps between the moving segments.
30

---

Published as a conference paper at ICLR 2025
Figure 8: Task the OMNI-EPIC run presented in Section
Success:
The task is successfully completed when the robot reaches the end platform.
Rewards:
To help the robot complete the task:
The robot receives reward for each time step it remains on the bridge or platforms, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the end platform, incentivizing swift movement towards the goal.
Termination:
The task terminates immediately if the robot falls off the start platform, any segment of the bridge, or the end platform.
def
isllf
self All..plaatoomssizee.. [3.0, 3.0, 0.5]
self. wtt.pttoommstrrrpssitiiiooo [0.0, 0.0, 0.01
self polatform_end_positeee
slatfor otrriotttitto platform start positionsi])
20.0,
position[1], self.
self wel.fplatform.cmmmm start id self.create box 0, half extents=[
self. self.plaatormmsizee00
/ 2, self.ptlatfom.siieiii] 2, self.
platform_size(2) / 2], posttponsef..pattroomsstttpppppp color=[0.8, 0.8, 0.8, 1.0))
self wwlffpplafffreeemm end id self.
s (ortttee_bx(mas==00..00 half exlf_eextett=[ seelf.platform.sizettl 2, self. htt.://nsee...lattoooom.mppppppe posiiiiionn,,, color=[0.8, 0.8, 0.8, 1.0))
Slatform-size(2) / 2),
belf.bridgeeleegtth = self.platform.ent end [ self.
(aa..plaafoommsizelo))
witf. 2.0
self.num_segments 5
Self.segmett_lengte = self welf.bridge... lengthh / self num_segments
amplitude 1.0
self segment period 2.0
31

---

Published as a conference paper at ICLR 2025
self segment_phase offset 0.4
Segment_colorce I[1.0, 0.0, 0. 1. 01, 11.0, 0.5, 0.0, 1.0),
[1.0, 1.0, 0.0, 1.0], 10.0, 0, 0.0, 1.01, [0. 0, 0.0, 1.0,
1.01]
self Delf.segment ids = []
for i in range(self nuf.mmummmu seumensttttts::
segment id self. box (mass=0.0, half extents=(self.
segment.length 2, www.. biiff.bridgewwidth 2, self.
platform_sizei2] 2], positionsssef... = platform_star__psstttinn0.
self. self.plattoommssizeee
/ 2 +
self. segment..ntt.... segmentttLegghh
self self.segment length, self.
platform start startinnoitton]]] self.platform start position
[2]], color=seementtcoolrrss sel self st htt.:cccwhgegyaaaicccoooooooeeee
linkIndex= 1,
sel self. see
mass, def create_.oxx(self, mass,,, www.saaassaaaacaaaaaaaaaa half_extentss position, color):
-p.GEOM hel.ffstenntssshlllcceeeettt
GEOM B0X, extf orgpa//sssatttttitooooooooooorrr rgbacolor=color) return self. setf.ppppceeetttttttrooooooossss
def
object id)
return np. htt.arrrwy.......eeeeeeoooooooor [0])
def get ret_distance to objjojte object-id):
doject_posittin = self w.t.gt..bjec...ositiiiooo
(objectt_oo
robot-position wol.frrooo..o bbass ppsiition
return np. linalg. toonaall..mmmmoojctttttttttooo
robot_position :2])
def reset(self):
observation = pttpa/omo...ttrrrsssssooooolrrer
self.time 0.0
r htt.:urreeeeseeeiooooooooooneeer robot_.d,
[1], selffplatfom.saarrppsittii/// + self welf.plattormmsieee..
/ 2 + "taaaaeee..iiienatttooo return observation
robot.
def step,self, action):
self distance to www.snnceetoopptom..mmmmmomm
wwwtform..eeeo end self welf.get..aee get-ddistancee to obeect
platformLend_id
observation, reward, terminated, truncated, info super() step action)
self. time += self.dt
for i, segment id in enumerate(self.seeee set.geeeente..sssseeettetee ids):
segment position self. self.gee..ojeecttteeeeeeee....ee position (segment id)
new_ www.ssegmentppsittiee = [segment_.osiition(]]] segment_position
[1]
(trrmmssstart_ppptto[221]1 + self.
segment.ammllitude * nin * np.pi (self.time
self. hee...semeett_haaseeoffeeee
self. wew..
new.seegment_positiion [0.0, 0.0, 0.0, 1.0))
return (observation, reward, terminated, truncated, info)
def get task sowards(selff action):
new new_distance_r wtwpanceettolllafomeeee.. self. wtt.eeitttanceeeoooooeeeeeeeee (self.
platform_end
32

---

Published as a conference paper at ICLR 2025
on peridge_or-plattiin [2] wtw.:paaoooomttrtppooooo
1.0 1f robot. Links| 'base posoitioo self. platformmssizel2
2 else -1.0
@each_platformae end = shelf.distance to ho_pplatforment
www.tanceeto..llatttote.. new new dww.ancseecee..m to www.foratoomoooooo end) / self.dt
return bridgee or platforms': br.drid platforms,
reach reach.platform.endd' reach platform end]
def get_ Set_tterminatdd(eef,, action):
return
links| (bases''base') base') " position[[] self. platform start position[2]
def get (seces
os wwtoonnlaffomm_ommmm end = self, pet_disstance to_ cboojbeccsseff platform id) sel self.platform.sizeee / return platforrnno
Task
Figure 9: Task 3 of the OMNI-EPIC run presented in Section
import numpy np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2Env):
Kick ball into goal with obstacles.
Description:
The environment consists of large flat ground measuring 1000 1000 10 meters.
ball with radius of 0.5 meters placed randomly on the ground 5 meters in front of the robot's s starting position.
The goal is defined by two goal posts, each 2 meters high and placed 3 meters apart, forming goal area. The goal is positioned 10 meters in front of the robot's starting position.
Three box shaped obstacles, each measuring x x meters, are placed between the robot's starting position and the ball.
The first obstacle is placed 2 meters in front and 1 meter to the left of the robot.
33

---

Published as a conference paper at ICLR 2025
The second obstacle is placed 3 meters in front of the robot.
The third obstacle is placed meters in front and 1 meter to the right of the robot.
The robot is initialized at fixed position on the ground, facing the goal.
The task of the robot is to navigate around the obstacles, reach the ball, and kick it into the goal.
Success:
The task is successfully completed if the robot kicks the ball so that it passes between the two goal posts without colliding with any the obstacles.
Rewards:
To help the robot complete the task:
The robot is rewarded for survival each time step.
The robot is rewarded for decreasing its distance the ball while avoiding obstacles.
The robot is penalized for colliding with obstacles.
The robot is rewarded for kicking the ball towards the goal, with additional rewards for successfully kicking the ball into the goal.
Termination:
The task is terminated if the robot collides with any obstacle. Othervise, it does not have specific termination condition. ==.
def initf(selff))
super() in)er(())niii(
self.ground size [1000.0 1000.0, 10.0]
ltt.ggruud_pssittiioo = [0.0, 0.0, 0.01
self.ground_id ottp:/ddddd.......rrreeooooaaoaa
ground_sizee00 sel half_extentss<(sll
/ 2, self, self.ground ground_ size [2] 2], positiorseel..grrudoopptttii color= 0.5, 0.5, 1.0]) self. p. b.changgeyyaaniiccoooooyuuuullld ground id, linkIndex= 1,
lateralFriitionno..., r
self. bal.bbaa_rrdduus 0.5
self.ballidd = createreete.eeem spaeeeesssereeeeesspee radius=self. ball_radius, position= 0.0, 0.0], color= 0.0, 0.0, 1.01)
self self.goal_post.heihht 2.0
lelf.goal_post_raduus 0.1
self.goal.width 3.0
self self.goal [10.0, 0.0, self.ground p + self.
self self.goal_post_leftt.. self.goal hoal_postthhhghtt height 2]
ground /
cesf.crrate.cyLlidee(eeeeee self.ggal_.ost_radiius,
radius=
=[self..gal_pooition heigghtsell..ooll.ottt...eeeeee
position
self.gool self.
goal width 2, self.goal position[2]], (1, 1.0, 1.0, 1.01)
self post right id self. self.crraate.cyLinderrmmm (yLsnde((mass=::. radius= self gelff.goal_postrratt radius, heigght=eelF.ggollppstt...eppp
position self welff.goal_ppsstionni self. goal_width / 2, self -og.glpssiiioooi]]1111111 color=11.0, 1.0, 1.0, 1.01)
diself.goal_posite
self allfobsttccee.stt size [1.0, 1.0, 1.0] self Delf.obstaclee.1i0 = wtw.rrrrtteebbbmmsss0o...... hawf_aetefeteeeeeeeeeeeee helf_extents==self sisaccleesze((0]// 2,
sizee / 2, self.
obstacle sinecleesieee2e 2], position=l 1.000,11..0, self.
ground position[]] self.ground size|2] + self.
[istaacle_siz[[2] / 2], color= 0.0, 1.0, 1.0))
self.obstacle.2.id self. selff.creaee.boxmmss=.....
helf_extents==self .bbstacle.sizefo 2, self.obstacle_sizern 2, self.
34

---

Published as a conference paper at ICLR 2025
obstacle_size(2] / 21, position=[ 2.0, 0.0, self.
ground groundd.position[2] self.ground_ssizeiee
self.
obstacle_size(2] / 2], color=[0.0, 0.0, 1.0, 1.01)
self. .bstaale_...dd=
Poostacle.sizel/e / belf.createeboxx(mss=000
half exlf_eextets==sell self.obbstacle_size[]] / self.
Dostacle
2], (.
self.
ground position[]]
self. sizer] self..
obstacle_sizze[2] / 21, color=[0.0, 0.0, 1.0, 1.0))
self trobot_possiion_iiitt [ [0.0, 0.0, self..round, self..rrunn_.pssttinn]]]] +
self self.grouund_sizel2) position intton]nnn((]]]]]
2
ralffrobot.line "base").
self. robot orientation init = self. rogeroustermionnrrrreillrrr ([0.0, 0.0, 0.0])
def create_box mass, half_extents, position, color)
self http::/eccecc.....tteeeeooaaaaan
-P.GEOM BOX, hallferrentssaalf..ceeeett)))) id htt.:ppp...........ccrccaaaaaaap
GEOM_BOX, htw.aeeeeeatttttttoottttetteeeen
return self saf._pppccccttttttttooooaaaaaaas
BaseoolLsssooshppeedddeeoooLLss. shrpoon-hhheiiiii aaselissuulShhppennndeeessssssss id,
basePositionnpooition) org.opae..ooo.aaoooooooeeoooooor
def createe_shhere(self, mass, radius, positiinnnooiooooooii...oooo.ooe color):
collision_shape_.ie
........GMMMSPPPEES radius-radiss
visual_shape = =att_-ppcrrreesssssaabbeeppppepp
GEOM_SPHEERS aoduss==rdduuss
return hrt.ap/..rerttttllboooolaeeessss
baseoolLsssssseeenneeexxsossssss basePosittion=oosttion)) org.ap/eeeeeeaapoooo.ooo.oooooor hBassissuaahhppnnnneeevvssssssss
def create_cylinddr((self,, mass, hetisraddiiieeieeeeeee height, position, color):
collision_shape_idd ott.......rttttoooooooooeeeeeeee
-P_GEOM_CYLLTNDER radius-radius, heigheingttttiiiggggieninnniiini visual shapel aet..pp.rrateesssccccchhccceeeee
GEOM CYLINDER, radius=radius, Length-heighttt,, rgbaColor=color) return self p. wwt.pprrrreetttttttooooolllaaase baseol1sssoohhpppeeeeeoooLooooos
def get_ get_objectt wtt_objject_psssee jejjec_posiiioonseef.. object_
return np. oppasrrwwss....eeessssettooooorr [0])
def -et_ dit_ddistncee. to object object id):
object_poosiae position self wwt.:ggeeeoeettppooooeoeeee position (obj @ctbjeec....
robot_position self. L [lbaae]].poo ]
return np. linalg. (otog
robot_position
def reset(self):
observation super().
hat.:p.ccettssstttttooomrrrrrrar robot. robot id, self .robot_.osiition.iinit,, selff.robb..orienattioo...itt
self np.nnddm. 2.00, 2.
eet.pprrrestaaeesstttoooonrrrrrr (al.eeattttttossee..a.all..,, [5.0,
ba ball_yiinit, self (tt.grounn_pssition2))) self felf.ground.co sizul2_ /
+ self.bail.radiuss) 10.0, 0.0, 0.0, 1.0))
return observation
def step(self, action):
self. dist.distance....m ball self get distance object(sels fallf.bblll id) ballid
self.get
nelf.ballLposition self self. observation, reward, terminated, truncated, info super(). step action)
35

---

Published as a conference paper at ICLR 2025
return (observation, reward, terminated, truncated, info)
def task_ rewards (self, action):
new diw.anie to bolball = self self.get distance toacce__ttooo abjjct(seel Barooff.ballli ballid)
ww-bball_positiee
self.get get_object postttooooooosssttfee..ooo positionn/elf.ball...
survival 1.0
reach ball (self www.aaiceeeee...mm to ball
new www.distanceetto to ballll))
self.dt
collision_obstalle_1
robot..rooot..22
oollision.obstatle
obstacle 2 (rgisaaceeeellllll..o.ooooooorrd
robot. id, bocys=sell..bstaacee
oollision.obstaller
robot. boceeselffobbsttceee3...)..)
collision_pemalty 10.0 if collisiion.obttalle.l or
collision-obstacle.. or www.lssoonoosttaalllooooooooiiie else 0.0
kick_ball (new.ball .dt
self self.ball
self
ball_inggoal = 0.0
if self position self.goal goal width / self.goll_pssition[11 self.
new bal. positions
goal width / and new ball position[0] > self.goal. position [0]:
ball_in-goal 10.0
return {'survival': survival, 'all: reach_ball,
collision_penaltyy collision_penalty 'kick_ball's kick_ball 'ball ball_in_goall
def get terminated action):
collision.obstacle-.i http::/wwooo.tt.......o..aooooar
robot. robot. Lb) collision robot_ddb htg.a/ooooooeett..ooool.eeeeell" co robot. body8-self.
return collision obstacle or collision obstacle 2 or
collision obstacle
def get_success(self)):
ball_position self return self. goall_ossiiiio|]] self. Loiddgh /
bal ball_position < self 1ggp___osiiioo11]1111111 + self self.goal_width /
and balLposition self 1
Task 4
import numpy np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2Env):
Cross rainbow bridge with moving segments and gaps to reach a platform.
Description:
A start platform and an end platform (each in size and 0.5 in thickness) are placed m apart.
The two platforms are connected by bridge (2 wide) divided into segments of equal length (3 each).
Each segment moves up and down independently with sinusoidal motion. The amplitude is and the period is seconds, with each segment offset in phase by 1/3 seconds from the previous one
The segments are colored red, orange, yellow, green, blue, and purple, like rainbow.
36

---

Published as a conference paper at ICLR 2025
Figure 10: Task the OMNI-EPIC run presented in Section
The segments have 1 gaps between them. The robot initialized on the start platform. The task of the robot is to cross the dynamic bridge to reach the end platform as fast as possible by timing its jumps between the moving segments and over the gaps.
Success:
The task is successfully completed when the robot reaches the end platform.
Rewards:
To help the robot complete the task:
The robot receives reward for each time step it remains the bridge or platforms, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the end platform, incentivizing swift movement towards the goal.
Termination:
The task terminates immediately if the robot falls off the start platform, any segment of the bridge, or the end platform. =**
def i (self):
super()..init) self. plwttoommssizeeeeeeeeemee [3.0, 3.0, 0.5]
self www.lptfmm.strttppssiiccee
= [0.0, 0.0, 0.0]
self Sollaform_end_ppee
25.0, positiio = paaeff.lattoonssttrt.oosttooolo +
self.
p [latfommsstartpppiittnn]]]]]]]]
sel 1t...plattoom.strr.idd obr.r//d@@_s.......eeee.bomaaaaa
half_extents= self. self.pplatoommsizeel0] / 2, www....laaffm.iiieeeeeeeeeoe / 2, self.
platform.sizei22 2], posittonnself,,laatfmm.ees sttpaaattfoomsssstttooooooooooo positionn,
color=[0.8, 0.8, 0.8, 1.0))
self, self aal..pllatfomeedd.dd www..ccreate.boomeeeeeeeeee
half_extents= / 2, self. platform size[2] 2), desition=self.platfoo end position, color=[0.8, 0.8, 0.8, 1.0))
self. seeff.platfommsizettt
/ 2,
37

---

ids): Published as a conference paper at ICLR 2025
self wwt..brrieeeeee Lengghh self platform 1 self. self.platform.size
self. wwf.bbidde.wiittt 2.0
self.num.segmeeee segments 6
self lenf.segment-lenghh = 3.0
self.gap
1.0
self self.segent..apllttude
1.0
self.segment-periood 2.0
self.seement.phhsse.offffss
1.0 3.0
gggmmenttcollos== I11.0, 0.0, 0.0, 1.0), 11.0, 0.5, 0.0, 1.0),
[1.0, 1.0, 0.0, 1.0), 10.0, 1.0, 0.0, 1. 1.0], 10.0, 0.0, 1.0,
1.01, [0.5, 0.0, 0.5, 1.0]]
self segment ids for i in segment-id celef.creeteebboxmassseseee
half (xtff platform_size/// 2] (osition==self.. F http://tiiooooilssfafaaaaroaaasa
segment.length 2, self. www.slll.biiiiee weeff.briige.widdt. 2, self.
/ 2 +
self 1egg (ta (self.segment__enggt) + self
t startt self.
.gap plaaffommsttrrtppsittoo111,, colorssemeet
[ colors(i])
p. Sedteralfiicttonnn....eeettttto self SetT.seeeenttii..appp(((eeeeeere
id, LinkIndex= 1,
def create_box mass, half_extents, posiiittttttssssoiioooo color):
id collision Shh.ppaccccttt....crretoooossssp -P.GEMM h visual_shape id hatppppp...aaaaaaccccaaaaaaaaaaa BOX, h robccoooooo
return self. setfeppp.rrattttttriooooooaaaass Ihaeoollsssossaaaannoocroooousss
def get_object_posiiiio((sfff object_id] ;
return np. www.rr/w/ee.....ettateeettttttts [0])
object_id): ;b)
object_position = self.gee.ojeec_.pstttiooooofeef
robot-position
return np. hainalg..omm(checttpssttooooloo
robot_position
def reset
observation = reseet
self.time time 0.0
self. p. wtw.rttseestttttooooiorrarnrrrra wwwp:nnssese.e...oooooooooooo.ee robot id,
st aat self.platfomm.ssart..ooo
[1], self self.plarfommsstarpppiiiiiiii
+ self
/ 2 self. Links
links [iniii((((beesrppiiiiiiiiiiiii/
robot.
links baass''bae''eeeo "oasee).orientation init)
return observation
def step(self, action):
wtw..eeddistaaeetoooooooooeeeeee observation, reward, terminated, truncated, info super step action)
self diisdanceeeettooooo t plat
self. time += self.dt
for i, segment id in enumerate(self. cessstiionssementt segment id)
segment position self object position
new swwment position
segment_posiiion [segment position[0]]
segment positttossiioooo
[1], selff
+ self.
segment_amplittue * sin * np.pi * (self. time i
self.seemeet__haseeooffssst self self.segment_perriod))
38

---

Published as a conference paper at ICLR 2025
self (rr.reeteeeesttttooooooorraraaan
new posttiion,osiiinn,, [0.0, 0.0, 0.0, 1.01)
return (observation, reward, terminated, truncated, info)
def tasktttr_wwwardd (self, action):
new distance to to. wwwplllafoommm end self get distance (ob platform sentform_eeddiiin id)
mm.brioge.orpplatfome
1.0 if self. robot. links 'baee posittion
[2]
hll..plaaformmsizze
/ 2 else -1.0
reach platform end (self distance to platform end new distance to platform end) / self.dt
return {'on bridge or platforms': on bridge or or._pfooros, reach-platform.ent
platforrato
def get_.temiiateed(slff action):
return self. robot. links bbase' I. position[2] self.
def g succes
is on platform end self get distance to object(self platform end id) < self.platfor size[2] / return is on platform end
Task
Figure 11: Task of the OMNI-EPIC run presented in Section
import numpy as np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2EnV))
==*
Kick a ball into a moving goal.
Description:
The environment consists of large flat ground measuring 1000 x 1000 x 10 meters.
A ball with radius of 0.5 meters is placed on the ground, 5 meters directly in front of the robot's starting position.
39

---

Published as a conference paper at ICLR 2025
The goal is defined by two goal posts, each 2 meters high and placed 3 meters apart, forming goal area.
The goal moves side-to-side (along the y-axis) in sinusoidal motion with an amplitude of meters and a period of 10 seconds.
The center of its motion is 10 meters in front of the robot's
starting position.
The robot is initialized at fixed position on the ground, facing the goal.
The task of the robot is to move towards the ball, time its kick correctly, and kick the ball into the moving goal.
Success:
The task is successfully completed if the robot kicks the ball so that it passes between the two goal posts.
Rewards:
To help the robot complete the task:
The robot is rewarded for survival at each time step.
The robot is rewarded for decreasing its distance to the ball. The robot is rewarded for kicking the ball towards the goal, with additional rewards for successfully kicking the ball into the goal.
Termination:
The task does not have specific termination condition. The episode ends after a fixed amount of time.
def Seiiii (self):
super()..inii)
self.ground size [1000.0 1000.0, 10.0]
ltt.groudd_psittiiooo = [0.0, 0.0, 0.01
self.ground_id oote
half_extents=<
pround_sizee[00 2, www.:seel..fonnoe..ee self / 2, self, self.ground ground_
[2] 2], posittoossel..rrruddopstttto color= 0.5, 0.5, 1.0]) self. p. bhannggelyyaaniissoooo//ouuueeee ground id, linkIndex= 1,
lateralFrritionno0.., r
bal.bbla_rrdduss 0.5
self.ballidd = createreete.eeem smpetteeessheeeeeesseee radius=self. ball_radius, position= 0.0, 0.0], color= 0.0, 0.0, 1.01)
self.goal self.goal-post.heightt 2.0
self.goa 0.1
welf. 3.0
self.goal_psiiion.iiit = [10.0, 0.0, gelf.ground.
wtt.:grronnppooittioooooo position[2] + gelff.ground.sizell 2 haa.f.golllpoostthoee
2] radius=
self.gool_post_leff..dd selff
self.ggol_.osttradiuss, ooooooooo,,,,ii..aoooooooooooooo position
iiself.goal_ppsitionnnntttee selff.oal--piiitionniittt color=
self.goal width / 2, self goal
1.0, 1.0, 1.0))
self right = self. create welf.ccratee.cylinneereeeee cylinder((ass==0000 radius=
self self.ggol-_ost_radiuus heighht=eelf.oool.oottt.eeeeee position
self..goll_posiiionnnitttt self self.goalwidthh 2, self self.goal goal_pposiiiooniint[]]]] color= 0.,
1.0, 1.0, 1.0))
self self.goal_amplitude = 2.0
self. 10.0
robot relf.robot.ppoition init = weelf.ground.oo
(0], self.
ground position[]], self. grouud positiinnss] + self. ground size[2] [2]]
self.robot Links baas positionn init
def cseate.box(self, mass, half_extents, position, color)
40

---

Published as a conference paper at ICLR 2025
collision_shapee
-p.GEOM helffxxtentsshhll..xxeeett))
A.sual_shape__. id htt.pp/.....attc..cccecccaeaaaaa
GEOM GEOM_BOX, wtw.a//eetesstttttoootttttttonoa
return self set.. osscoollssssshheeeenexxcooolossd
bassPosittio==posttiin)) baseliisauhhhhpppnneeeevvvssalbp otg.apaa....apppoooooooooooooorr
def create_sphereesel1f mass, o
collision_shape._. htt.teec.crtteeeellooaaaheeheeee
P.O. SPHERE, radiussradiis
disual_shape id = oott-pecceeeetssssssagghhhoheeer
GEOM SCCMESSPHERE.. roddussrraduus
return sel.__...cccrrreetttooooo/osssss
bassPositioonpooiiiion htgps//see..ppppooooooooooooooor
def create..cylinder(self,, mass, radius, height, position, color):
collision_shape.
....GEOM.CYLLINDEE radius=radius, heighthhhhtthhhhhhhhhhhhhhhh))h
sisual_shape_iiz setf.pp.crareessscccchhpaaafpppe Lat rgbacolo==cooon
GEOM CYLINDER, radius=radius Lenghhhheiht,,hh return self. p setf. baseollissossscccnccccoouuuusssd
def wwwp:ocejessiiiioo....
object_ id)
return np. patprp/eg......eeaeeeooooootoonr [0])
def get det_distance to objjojt object-id):
doject-position = self.get_on gget_object.positionn (objjett_ooooo
robot-position wol.frrooo..o Links "bankss("
return np. linalg. www.nnal..ommmoojettpssittoooooo
robot_position! :2])
def reset(self):
observation = super() reset()
self 0.0
htt.:ppceeettttttttt.ooooooooote robot. robot_id, self .robot_.ositiion.inii,, self.robot. (base
orientat.on__init)__..
[self.
self self.ground_positionn/ 5.0, self. self.robot_posi tion (
robot_positiion_iiito01111
2 self.
radius,, [0.0, 0.0, 0.0, 1.0))
return observation
def step(self, action):
self self. cilstance_to_balle self.ball_positinn = self observation, reward, terminated, truncated, info super step action)
self.time self.dt
goal_y sel.==oellaoaaooo www..goal.ammmltttu np.sin(2 * np.pi self. time
self.gool_period)
self hat..pprceeessssssttooorrrrrartr
self.goal i self selFFggal self .goalwwithh
goal_y, self seff.goal.posiiion.nnttt.... [0.0,
0.0, 0.0, 1.0])
self. htt.::/pp..tttteetoooooooooorrrr
wwtpootttrrrttiiiiee right_id id, +
[self. position
self width 2 goal pelf.ggal init([]),
[0.0, 0.0, 0.0, 1.01)
return (observation, reward, terminated, truncated, info)
41

---

Published as a conference paper at ICLR 2025
def get- rewsrds (self, action):
diwaanes to
tt_ ball self self.get. wet.ggetddaa distance to oseect weasseff.alll... id)
new.ball_position self wtt.eee..heecc..ssttiooeeeeee btt.iioonnsee..aaala.ii. survival = 1.0
reach-ball. = Sdelf.distance.fff to ball new new.distance..e to new.distance.tt.ball)
self.dt
kick-ball (new ball positions)) self ball wtt.:balllllllol posiiiioii))))) self .dt
goal_y self. Self.gool_mmplitude np.sin(2 * np.pi. pi self.time
self self.goal_perodd)
ball in -n. 0.0
if cellfgoal_pposte
init gelf.goal posturadius
radius < init self.
new new.ball_posittonn[] <
goal-post_raaius and celf.goal.pssttonnintttt goal_width goal_position_intt[]] self.goaLwidthh
self. goalyy [ew.ball[pssition]]]] self.
goal_y:
ball_in_goal 10.0
return ('survival': survival, 'reach_ball': reach_ball, kick_ball'; kick_ball, 'ball_in-goal'' ball_in-goal)
def get terminated action):
return False
def get_successssell))
ball_position self oeeeffeet.ojettt.psstttoooo ball_bb
goal_y self.goal_amllituuce np.sin(2 np.pi self time
self.goal_period)
return [ttppptiiiooinii[]__ self self.goal_post_ radius <
ball_position(0] wwt.:/gaaloosiionniileo
goal_post_radius and pow.oaopossiiiooniiiiteeee.oo self.
self.
goal_width goalyy bbll_position[1] self.
ioaL-position.nnit/] + self.goal.witth / 2 goal_y
Task
Figure 12: Task of the OMNI-EPIC run presented in Section
import numpy as np from oped. envs. r2d2. base import R2D2Env
42

---

Published as a conference paper at ICLR 2025
class Env (R2D2EEN)))
Kick a ball into a moving goal.
Description:
The envi ronment consists of large flat ground measuring 1000 1000 10 meters.
ball with radius of 0.5 meters is placed on the ground, 5 meters directly in front of the robot's starting position. The goal is defined by two goal posts, each 2 meters high and placed 3 meters apart, forming goal area.
The goal moves side tods (along the y-axis) in sinusoidal motion with an amplitude of meters and a period of 10 seconds.
The center of its motion is 10 meters in front of the robot's starting position.
The robot is initialized at fixed position on the ground, facing the goal.
The task of the robot is to move towards the ball, time its kick correctly, and kick the ball into the moving goal.
Success:
The task is successfully completed if the robot kicks the ball so that it passes between the two goal posts.
Rewards:
To help the robot complete the task:
The robot is rewarded for survival at each time step.
The robot is rewarded for decreasing its distance to the ball. The robot is rewarded for kicking the ball towards the goal, with additional rewards for successfully kicking the ball into the goal.
Termination:
The task terminates if the ball goes out of bounds if the agent fails to kick the ball within certain time frame.
=**
def
init :self):
super() (uperr)...iii__1 ()
self.ground_siie = [1000.0] 1000.0, 10.0]
self.ground_positionn = [0.0, 0.0, 0.0]
self.ground_id== self.
ground_size(0 hea.ffccreteebbx(massseeeee box 0, half_ [ilf-eetents==seff
2, self.ground_siieer size[l] / 2, self. golf.ground...m size
[2] / 2], wow.tptooossllff.rrrooooooooooo. 0.5, 1.0]) self. w. htw.pphhgggggyllloooooooo/oottue ground id, linkIndex= 1,
color=(0.5, 0.5,
1ateralFFricionn00.8,, restitution=0.5)
self.balL_adiiu- 0.5
self.ball.id self. www.reeeeeeeeeeeeeeeeeeeeeeeeeee
radius=self.
ball_radius, position= 0.0, 0.0], color= 0.0, 0.0, 1.01)
self post_heiggt 2.
hatf.goal_posttradie.e 0.1
self.goal.width 3.0
self.goal_positonniiitt = [10.0, 0.0, self. wtw.gggoooooooiiiiigeooeoeeeooee +
self. self..gal__post_heiintt// 2] self self.gool_.psttleft.i. welf.create.ccyllce crt..rree_cciinndrmmmasss.... radius=
selffggroun__siee22
self.ggal heg.o position
=Iself.goal_positioniinttee set.
Siif.gooa / gelf.goal petffpoll color=[1.0, 1.0, 1.0, 1.0])
self foal. rost id self www.ccetre.ciiiinieee
cyiiiddeemedmdeas...oo radius=
self self. saalf.goal-psttrrdduus heigghtssel...ooLLoootLnLLLiiii position =Iself.gaal_ssitioniiiteee
self.ggal_positiionnnntttt +
43

---

Published as a conference paper at ICLR 2025
self. wet.ggollwwddt 2, self. eet..poll.psittoonnntt111]. color=[1.0, 1.0, 1.0, 1.0])
self self.goal_ame -oplta = 2.0
self Self.goal_period = 10.0
Cll..robotppoittion init = wtw.:grrooooditttdee
self.
grounddppssiteetd position[1], self self.ground prroonddpositioonie + self.
ground size[2] [2]]
self. Links ('baee' ] init
def create_box mass, half_extents, position, color):
collision
id self s (aaassassSSSpppsssppppeeeeeesr -p.GEMM BOX, halfExtents=hallf extentss extents)
visual visual_ shape id httpaa/aee......araaaaaaaaaaaaaa hat.aaaeeeessstattatettttetttrrr
return self sef..ppocraaeeeittooooo///ssssss
def create_.shere(sel]) mass, p
collision shape id wtt..e...crtttlllllaaaaaaaahheee
GEOM SPHERE, radiussradiss)
visual_shape id = setff.ppccraatesssssuahhhhhhhhep
GEOM SPHERE, roddus--rrdduus, return get._pp..ccarteeelilioyyyllsssss
def create (ylnneesclinddrseel.,,, mass, radius, height, position, position, color):
id
-P.GEOM raduus-radiuu,, hegggettgnaauuuuu..ggggeteeenini sisual__saae__id GEOM CYLMNDER,,,,RR, raduueerssssssssssssssseeoonoioo ing otbp rgbacolor=color)
return saf......ccarrtttttttoooo//bbasa baseolLSSSSShheeneeccooLLLLLsso.
def get dejeectpositionsseeeeeee object id) return np.aaa... wtwpr//sss.............peppeeots wwt.:passsstttttoooorrrttttttota o id [0])
def wttpsttannnetttooeeeeeeeeee
object_id):
jiject_position = wew.f.get_object_peee. posiitionobbectt
robot_position self. Links jabasse]ppssition
return np. nom nomm(ohject__pssstinn
robot_posittion_ :2])
def reset(self):
observation super() reset()
self 0.0 .
wtt.peettessssttttooooooatrtrtta robot. robot_id, self .robot__postiion.iiii, self.robot. base
orientation init)
self. p. p.w.eerlsssssstttoooooiaaattttte bal...ll........................ id, [self.
robot pocitttpsiiion.oco init[0] 5.0, self.robor rolt position init[l], self.ggruund_psiition]] self..round grouud_size((] 2 self.
rall_r, (0.0, 0.0, 0.0, 1.0))
return observation
def step(self, action):
self.distane www.ddicetttoooooooooo self bolf.ball_pp position self get object (see ball folllid id)
welffgee_disstee qet_distance_eeee toreeettooojjjjeeeee toaeieeettoooecc(sseseee bateef.bball... ball_dd)
observation, reward, terminated, truncated, info super () step action)
+= self.dt
goal_y self www.fgooolammplitttoio np.sin(2 * np. pi self. time /
self. self.goallperiood
44

---

Published as a conference paper at ICLR 2025
self ._.. htt.://eeeeaappp...oooooaattttte
self.goal_posiie ptwtllpoooooniiiiitttttteeeee init[o]] self.ggoi_posiitioniiiite self
.goal_ 2 goal_y, self self.gal_positionniintzz]. [0.0,
0.0, 0.0, 1.0])
self.
id, weelf.goat position -itittiitonnn self.goal-positionninntte +
self.goal width / goaly,, self.goal_poiitionnnnittt]] 10.0, 0.0, 0.0, 1.0])
return (observation, reward, terminated, truncated, info)
def get task rewards (self, action):
new distance to ball self self.get get distance to object ball id) new foallpositionn
self.ger object position(self ball. id)
survival = 1.0
reach_ball do.f..distanceeto.bbll
to
self.dt
kick_ball bnw.bball - self wal.baaspposiiioo00 self .dt
goal_y self goal_ampliiuue np.sin(2 np.pi pi self.time Sellfggoal-periine
ba goal 0.0
if gelf.gonl-posiiton
position init[0] golf post radius <
new ball [ < self. position init[0] self.
goal_post_raddus and self.ggal_pssitionniitt..
self.
goal_width / goaly <ew.ballLpositionnll self.
goal_postiioninii[]] + self.goal.with / goaly:
b 10.0
return ('survival': survival, 'reach_balls ball': teach_ball. kick_ball': kick_ball, 'ball_in ball_in
def get- (sef action):
ball_position self.get objjectposiiiionssee..beee
out_of_bounds ball_positionl0] 0 or ball_position(0] > self.
ground_ ball_position[1] 0 or (balLposiition[1)
self.ground_ (r)
return out of bounds
def get success
ball-position self self.
wtt..e.eeettttppsitiiiooooooooo. (self.ball_dd) bball_id)
goal_y self.goal..mplltuuee nin * np.pi self.time self.goal_period)
return selff.gall_psiiionninne ball_position(e] self.ggal..siiiionniiit
sttlllllosttl....pppptttrt.o. radius <
self.
goal_post_radius and self.goal.pssiiinnniiie
self.
width / goaly ball posttions self.
poalttoniiion + self widhh / 2 goal.y
Task
import numpy as np
from oped. envs, r2d2. base import R2D2Env
class Env (R2D2Env):
=
Cross bridge with moving segments and kick ball into goal.
Description:
A start platform and an end platform (each m in size and 0.5 m in thickness) are placed 25 m apart.
The two platforms are connected by bridge (2 wide) divided into each).
segments of equal length
Each segment moves up and down independently with sinusoidal motion. The amplitude is 1 m and the period is 2 seconds, with each segment offset in phase by 1/3 seconds from the previous one
45

---

Published as a conference paper at ICLR 2025
Figure 13: Task the OMNI-EPIC run presented in Section
The segments have 1 gaps between them.
spherical ball (0.5 m diameter) is placed on the start platform, 1 m in front of the robot's initial position.
A goal area (3 m wide and indicated by two posts) is placed on the end platform.
The robot
initialized on the start platform, facing the ball and bridge.
The task is for the robot to kick or push the ball across the dynamic bridge segments, navigating the gaps, to get the ball into the goal area on the other side. The robot must also successfully cross the bridge and reach the end platform.
Success:
The task is successfully completed when the ball reaches the goal area on the end platform and the robot also reaches the end platform intact.
Rewards:
Reward for decreasing distance between robot and ball, encouraging the robot to approach and interact with the ball.
Reward for increasing the forward progress of the ball across the bridge.
Large reward for getting the ball into the goal area.
Reward for robot staying on the bridge segments or platforms and avoiding falling.
Reward for robot reaching the end platform.
Termination:
The episode terminates if the robot or ball falls off the bridge or platforms. It also terminates if the ball goes off the sides of the bridge or platforms.
_sllff def
self f.l..platform.mmmm size [3.0, 3.0, 0.5]
self platfor start position [0.0, 0.0, 0.0]
46

---

Published as a conference paper at ICLR 2025
self leaaform_end_positeee = [self. pwa...ppaaformssttttpottttttet + 25.0, potplatfrrmssarttppssiissss ilatfom__ttrt_.osittinnl
self.
self wwt.tptoomstsatttttao..
half_extents" 2, self.
self Sel.f.platfommmsizeee / 2, wel..pplatfommmsieeeeeeeeeee
Seatformssize[2 2], positionsself.plttfom start posittooitiiooo.onn
color=[0.8, 0.8, 0.8, 1.0))
2elf.platformmeno_idd ott.://ooo...........rroooooooa.
half_extents=( selt.pllatfommsizee(0]] [ 2, self.
platform / 2], ott.a//eeaaooooooooooaoooaaaaaar wwwpppoooosa....atttooo.eooooooa
color=[0.8, 0.8, 0.8, 1.0))
welf.bridge..engeeeeeee self.platform end position(0] self.
pplatformsstarfe start:ttttttttiiinn position self helf.platform ]
self.bridge.wiil width = 2.0
self.num.segmenne
6
self seits 3.0
self.gap 1.0
selfeseegmett_ompLituuue 1.0
self self.segment__erriod = 2.0
self. self.segmentt.hasee.offff pfa.tet--paeeeefffeeeeeee offset 1.0 3.0
self. segment
for i in range(self.num segments):
segment id = seef.craatee box (oxas==0.0, half extents=[self. segment-length 2, self belf.bridge_width 2, self.
platform-sizel2) posttiosssss platformsstart_ppsttiio00// sttp::0:ssslllppfffffossssll /
(ssl==ssssegeeeeeeteeeee + self platffrmssttrr_oosiitton2),,,,,,
self.
.gap
self.
colo= 0.5, 0.5, 1.01)
sel htt..-/ccagggooaatccoooooooeeeee linkIndex= latteralFFictiions0.... setf.segmentt.0..ppppdddeeeeeeee
self self.ballLradduus
self.balt self.ball-position..itt self
1.0,
self.plaafommsstarrppsttttt
self.
platform start position[2] self Seff.plaffom size[2] / 2 + self
hall_radius,
self self.ball self.creat sphere radius=self. ball_ra position=self 0.0, 0.0, 1.0]) self welf.goalwidth 3.0
color=[1.0,
self.goal__ooition [self palatformeen_pooittiee self. htt.ffommeen_pssitiiooooooe
self.platform.ent wnw.:m_meeddpopiooi__111m
self.plattommsiiee22] 2 belf.ball_radius]
=0.1, height=1.0, self. (mLinder 0, radius
self.
goal_position[1]
2, self ltw.lgoolllppsiooooo
[2]]. color=[0.0, 1.0, 0.0, 1.0])
self relf.goa__rigit..ottidd (osttooosssoto pogeff.gol-posiiioon00 self.
create.ceeeeccommm crr.atte.ccyiindermmass........
radius
=0.1, height=1.0,
positiionn]] + +elil]=eeellooo width 2, self goal position
[2]]. color=(0.0, 1.0, 0.0, 1.0))
def create..ox(self,, mass, half_extents, pow.tttennssssppiiooio.ooo position, color): htt.pphh......aoaa.......eeeoeer
BOX ) orsssiae.ec.sseeeccccccceeeataan
GEOM httpop/pnm...tttttttttsaaaattttt return baseeolllsssossppeennddsssooosss i. DaseVissuaShhhppelneeevvvvsssaa id, basepositioonpooiition) hrg.apaee....appii.oaoooooooooor
def create_spherresell mass, radius, polpslspppiiiiioio color):
collision_shape-i self.-p. seltf_pp.rrateeolllsssissshee sesapppyessssppeeeeesseseee
-P.GEOM_SPHERE, radius=radius)
47

---

Published as a conference paper at ICLR 2025
visual_.hape = oet..pp...eeesssaabbppechhaeeeee
GEOM_SPHERE, radius=radius, r return p. hrt.
Shsecollssaashhoonoooooooosssssd basePosition=poostioon baseWWissuabhhpeenneeeevvvsaal.p org.aaas...apppoooooooooooooooor
def create create_cylinder(seeff mass, radius, height, position, color) :
vollision_shape__ii htt.::/ic...........toooooooonna
GEOM GEOM CYLINDER, radius=rradis,, hegg rgbaooloocooooooo
prr basecolllssoshhhenndddooooooooos shae http://hhhhhooooooooooooo.oo...a
return set.
def get.obbect :
object
return np hrp.arrwwyeat.....eesssttttttoor [0])
def get distance to object object id):
object- poscitinsi sel. object oot.eeeptttsiitiinoootit id)
robot position = self. robot. ('nks('basel [base'].positton
return np. linalg. norm(ojjetttt positionl robot postttosssiiooot :2])
def reset(self):
observation super() reset()
self.time = 0.0
self..p. ott.perreetsssttttooooooorrtrrnr ball..iddd self.
ball_posiiion intpaattooonnnitttooe [0.0 0.0, 0.0, 1.0])
robot-
self.platfomm.starttposssttt
celf.platform.start..poo
[1], self. set.:plaffommtttrrppsttttooooo. + self. wwwff.lattorr..... siietfrrmmsize21
2 + links htspeassssoossppppiiiiiotittttt
positiio robot.
return observation
def step action):
self.distance to ball self. get distance to object ballfbbbbel. id) self ball_positionn self self teeettobeectppsitiiiee ball_id) self delsstanceeto_platffie
platform_end observation, reward, terminated, truncated, info super() step action)
self, time += self.dt for i, segment id in enumerate www.:eeeeelseeeetttiooo ids):
Degment_positile
= welf.get.objeectppeeee object wwwitionissssiiiiii. (ww.iiietinsssennttt... id)
new www.seegentt..mmm position segmennt_psiitoon htt..erllatfoomssrrrrppootoooooo
D'segment_positiionn
segment segment_posittinn position
self.
segment_amplittude * np.sin(2 * np.pi * Aself.time i *
self seefsssegment_hhhse.ofssss// self selffeegnetee_eerrillllll id,
self segment position, [0.0, 0.0, 0.0, 1.0))
return (observation, reward, terminated, truncated, info)
def setttasskreewadds (selff, action):
new.distance. to_ ball self.get_dee distance caaitranneettooooooo
new.ball_poiitonn self setf..gee_ojjcct oet..oojectppssiioosssstlll positionsself..f..ooooooo ball_id) idll new_distance plapfoormmeee platform_endd.oo id) approach ball self.dt
deelf.distance to ball new distance to to ball))
alll.fowwrrd velocity (new ball position[0]] - self. ball_position(0]] / self.dt
ball_in-goal= = 10.0 if self www.goollppooo goal_poiition[1] self goal_width /
2 < bww.ball_osiitonn
self self.goal-posittion/ self.
48

---

Published as a conference paper at ICLR 2025
goal_width / and new.ball_posittion > self getf.goal--osiition
[0] else 0.0
on
1.0 self robot. links| ' possition 'base
[2]
+ self Se..ppaatfrmmsiizel//
/ 2 else 1.0
reach platform -
(self. .ie.ffddittnceeeccmm to platform
new new distance plapform end) / self.dt
return <'approach.balll: approach_ball, batt_forward "ball_in_goal'' ball_in-goal,
on.briddgeor_plaaffmme reach platform end': reach platform end}
po.bbrigge.r__plaffrrrs.
def get terminateddself, action):
is_ rs_rooot_on_briddee 1ss.roott.o..brr..eee000001ffoos = robo.. L'akss'base'' "base'].
position[2] > = wtt...aaafff...ttttttssttooo....
is 1r self.get..0jeet_ppstttioee..
ort.loff.iieeee.oo.....oorrrrr.p _tart_pssiiion1]2]
self.platform.siietll / is [1]) > self self.bridge platforms or not
ball_id)
return not is bnb.ige..bbbrigeeeee....o platffor is ball on.ballo.bbbrdgg.orr or or is of sides
def get success(self):
ball_position set..ggt..ooeeet.ootttoooooff...
is_bbal_inngoal self self_gooll_ooiitton11111- self widt /
ball_position [ + self.goal_widf /
2 and ball ball_ positiooi robot_ co_.oooooo_llaattteeee__e ow..lotoormmenndeee self.get distancedsttteeee_ee to seetesatcettttoocceesseee....
self..goal_posiiiio]]]
=eelf.gei.ddssanceettt
platform_c endrrm_end_dd) self.plaaform.sizee222 return is_ball_in_ge and is.rrbot.on
Task
Figure 14: Task of the OMNI-EPIC run presented in Section
import numpy np from oped envs. r2d2. base import R2D2EnV
class Env (R2D2Env):
49

---

Published as a conference paper at ICLR 2025
Navigate a dynamic obstacle course to reach target.
Description:
The environment consists of a large flat ground measuring 1000 x 1000 x 10 meters.
The robot starts at fixed position on the ground, facing the positive axis.
The target is a cylindrical object (0.5 meters in diameter and meter in height) placed 20 meters directly in front of the robot' starting position.
The obstacle course consists of dynamic obstacles, each moving in predictable pattern:
Obstacle 1: A box (1 x x 1 meters) moving side-to-sie to-side (along the y axis) with an amplitude of meters and a period of 4 seconds. It is placed meters in front of the robot's starting position.
Obstacle A box (1 x1x1x 1 meters) moving up and down (along the z z-axis) with an amplitude of meter and a period of 3 seconds. It is placed meters in front of the robot's starting position.
Obstacle
box (1 x 1 meters) rotating around its center with a radius of 1 meter and a period of seconds. It is placed 11 meters in front of the robot's starting position.
Obstacle 4: A box (1 x x1x1x
meters) moving diagonally (along both and y axes) with an amplitude of meter and period of 6 seconds. It is placed starting position.
meters in front the robot's
Obstacle 5: A box x1x1 (along the x y plane) with a radius of 1 meter and period of 7 seconds. It is placed starting position.
meters) moving in a circular pattern
meters in front the robot's
The task of the robot is to navigate through the dynamic obstacle course and reach the target as quickly as possible.
Success:
The task is successfully completed when the robot reaches the target.
Rewards:
To help the robot complete the task:
The robot receives a reward for each time step it remains in the environment, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the target, incentivizing swift movement towards the goal.
The robot is penalized for colliding with any obstacles.
Termination:
The task terminates immediately if the robot collides with any obstacle or reaches the target. =*.
def
(niit=seelf)- super().... init ()
self self.ground size [1000.0, 1000.0, 10.0]
self @eff.ground_position = [0.6 0.0, 0.0]
self.ground_id self. wt..eate_bbx(mmsss0__._____ dalf_extents=(sell
size [2] 2), posittonssel..grrnndopstttto. color=[0.5, 0.5, 0.5, 1.01) self .....ppp.. hhp.aaggyyyymiiiisoooooooouuuetd id linkIndex= 1, IateralFriction=o.0 restitution=0.5)
set..uuu..s... / 2,
self radiguutrrddiu 0.25
self target height 1.0
self.targge_posiitin [20.0, 0.0, self.
self
.ground_size(2] 2 self target_ / 21
50

---

Published as a conference paper at ICLR 2025
self. tal.ttrget_id self. cwwateeecylindrrnnnnnet.
radius=self. harget_radius, beightsself..trgettheigghh position=self.
target_position, color=[0.0, 1.0, 0.0, 1.0])
siff.obstacle.sizee [1.0, 1.0, 1.0]
position sel woostacle.sizeellleeee 2]
[5.00 0.0,
[2] self.ggound.ssize[2111
self
self eaf..osttacee22ppstttnnnnnit [8.0, 0.0, self.ground position
[2]
oot.iiittteeee..........tt...... position
position
.obstacle.sizee00
/ self
[ostacle_size(2]] 2], oosttonnser..ootttale..pppo.o...
color=[1.0, 0.0, 0.0, 1.01)
hal 2, self.
[isstaclessize(0))
abstacle 2], positionsself.obstacl. position init
color= 0.0, 0.0, 1.0])
self
oot. box half extents=(self 2, self. obstacle_sizee22 2], post:osssaf.oostttte..p...p.....
self.create .obstacle.sizeee]
self.obstacle
color=[1.0, 0.0, 0.0, 1.0])
= self.
half extents=[self 2, self.
obstacle_size(011
obstacle_size(2] 2], position=self.oostaclleo possitiinnnooiiiiooooooo init 00 0.0, 0.0, 1.0])
self wtw.oottaalee..dddeeaeee sel.:s..ddess....eteeeeeee wew.::eeeeeec.ceebbbbooooooooee
htw.eeettttstteeeeetettttsss wowstacle.sizeeggeee S self.obstacle size[[] 2, self.
obstacle_ 21, pos.tionsell...stttee...o.oooois
color=[1.0, 0.0, 0.0, 1.01)
rolffrrbot_posiiiionnntt [0.0, 0.0, self.ground_position222 +
/ self. robot. links| (base')
position init[2]]
self. orientation init self. (10.0, 0.0, 0.01)
ourr p.getQuatermionnFrrmmileeer
def create_box mass, half_extents, postiioon
self collision_ halffettents--hal_eeeeeetts - BOX, kisual_shapp__. htt.hhno..........aaaaaaaaaaaaaa
return seT.pppssrrrttttttooooo//aaaasss basecollsssachheerrrrroooosossss hhel/sssashhhppnndeeervvsaallsbs htw.:/ppppp..ioaaaeeoosstttoooot
def create__cLLinder(sell mass, radius, h wrwiightttpiiiiionnoo.oo position, color): colot))
collision id self ott.::a..c.tttttooooaiaaaaaaaaaa
GEOM CYLINDER, radius=radius, htg.s//etttttiussrruusssiiiiiiin heighghhghhtttghhhhhhhhhhhh))
sisual__happe_1 = wet..ppccreeteessaaaaaaaaaaaaaae
GEON_CYLINDER, Ladus tn rgbacolor=color)
return baser/ssaaShhhpenneeeevvvseesssp
def ge object_id):
;
return np. het.rp/eeaarrrrt......eeeeeeeoor ) [0])
def get_ :tt
object_id):
object_position = selffeeet.ojjeccppoiiiooooooddod
robot_position L
51

---

Published as a conference paper at ICLR 2025
return np. Linagg..oommropeettppppstttt.o.. robot_positionl :2])
def reset (self):
observation = super reset() reset))
self. time 0.0
self ret.efaseessttooooonrnaaaattttto roboot.oooooooot...oooooooooooo. id, self
robot position init, self. robot orientation initt return observation
def step(self, action):
self distaance to target self get distance to object(self. target id)
observation, reward, terminated, truncated, info super() step action)
self. self.dt
obstacle--nnew.y self.obstacle wtt.foostalleee.pooitttti [111 2.0 np.
sin(2 np self.time 4.0)
1dddd, [self self.
| obstaclee_Leew
osstactee_ppsstioonnnt((]]]]]
[[00]]),][[00,,,,00 [0.0, 0.0, 0.0, 1.0))
obstacle new self.osstacleee position init[2] 1.0 np.
* 3. 3.0)
2 id, [self
[1], 0bstaaciee22nee.22,,10000 0.0,
0.0, 1.01)
@bstacle_3_new_xx
1.0 * np.
cos * np.pi * self.time 5.0)
3 = self.bbfoccee33psstttonnnttttt
1.0 * np.
sin * np.pi *
id,
obstacle 3. new obstacle_3.new.y self.
obstacle 3 3 position_init([]], 10.0, 0.0, 0.0, 1.01)
obstacle_4_new.__ = self.obstaciee__posttoon...ttt
1.0 * np.
cos np.pi self /
@ostacle_4_new___
seli.osstaclee_.poiition..ttt
1.0 np.
sin(2 * np.pi * 6.00 self. p. opp.sstssessttthhhhhooorrrrrrrtt obstacle 4 id,
6.0)
obstacle 4 new www.ooobtttlee.e new new self
obstacle (bsteceeAAppstttoonnmt1111.11111 10.0, 0.0, 0.0, 1.01)
Dostacle_5_new.x1 new___________ htt.::/ss..e..sttttceeeessstoio.
1.0 * np.
cos * np.pi * self.time
obstacle_5_new.y> = haaf.obstaccee__ppoiiiiinnnnnti
1.0 * np.
sin np.pi
7.0)
self self._p. http://ttaeettttttooooooooaaaaaa
obstacle 5 Senew 5. self.
o posiiiion [0.0, 0.0, 1.0))
return (observation, reward, terminated, truncated, info)
def new-distance_ to_ new.pisttance.to.target = self.get (it.t
action):
target id)
survival 1.0
reach target = (self distance target new distance to target)
/ self.dt
return { f'survival': survival, 'reaeh reach_target)
def get_ (erl,n action): collision_.bstactler oot.apaeeeaiiiiieeeeeeooaaooaana robot. oot.o 0
collision [ll.apootttttta2222.meeeeererrrr
collision robot. robot_id, 3o orr.:poottcccccelalitteeeeeeeeer botye-self..ostacleeeeeee
52

---

Published as a conference paper at ICLR 2025
collision_obstaclee otre:
robot. robot id, tarlis robot_id, -aalaatee....ettttttttttteeroaoa robot. robot.
id))
return collision collision.oostacllee or collision obstacle.2 or collision.obstacle coltision.obstacle or collision target
collision obstacle or collision obstacle or
def get_succeessseelf):
contact-points--aageet het.-pogetCorracccoonnntttoooaae robot robot id, bodyB=self. target id) pointss
return len(contact points target)
Task
Figure 15: Task of the OMNI-EPIC run presented in Section
import numpy np from oped envs. r2d2. base import R2D2EnV
class En2(R222nnv):
Push a box target location on a dynamic platform.
Description:
The environment consists of start platform and an end platform (
each in size and 0.5 m in thickness) placed 20 apart.
The two platforms are connected by a dynamic bridge (2 m wide)
(3 m each). Each segment moves up and down independently with sinusoidal motion. The amplitude is 1 and the period is seconds, with each segment offset in phase by 0.4 seconds from the previous one
divided into 5 segments of equal length
A box (0.5 m in size) is placed on the start platform, 1 m in front of the robot's initial position.
The robot is initialized on the start platform, facing the box and bridge.
The task of the robot is to push the box across the dynamic bridge segments to reach a target location on the end platform.
53

---

Published as a conference paper at ICLR 2025
Success:
The task is successfully completed when the box reaches the target location on the end platform.
Rewards:
Reward for decreasing the distance between the robot and the box, encouraging the robot to approach and interact with the box. Reward for increasing the forward progress of the box across the bridge.
Large reward for getting the box to the target location on the end platform.
Reward for the robot staying on the bridge segments or platforms and avoiding falling.
Reward for the robot reaching the end platform.
Termination:
The episode terminates if the robot or box falls off the bridge or platforms.
def init (neit((self)))
super(). init ()
self Self.plaffrmm sizeoormmiiiiee--e = [3.0, 3.0, 0.5]
self hel.fpplafformmstc www.pooo..tat...oiittiee..
[0.0, 0.0, 0.0]
self self. wtw.patfomeen..osstiiooo.
20.0, self.platfommstarrrppsttttoo selff.platoommsttrrr.ossttttto sttp:mmmsttatttssstiioo
self
sel wwwtfoommstart.iii....
half_extents== 2, self.
self self.plattom..szzef00 /
platform.size(2) 2], positionssef..lattomm staerfoommstaattp.osssss
color=[0.8, 0.8, 0.8, 1.0])
self plattpooloooomdddid self.
self self.pllatfrm.sizeet) - selff
halt_extents=(( 2, self.
/ 2, self.platform
platform 21, pot.:tttosss....lttttomooooaaoao
color=(0.8, 0.8, 0.8, 0.8, 1.0))
self.bridge length = self self.platform end [osition self.
platform start position[0] self helf.platform size[0]
self.bridge width 2.0
5
self.segment-Length = Seff.bridge_len
self num_segmentss
self.seegmett.mmlltteee 1.0 1.0
self.segment__eriio = 2.0
self halfsseementt_haaeeofffsee
0.4
segment_colors [[1.0, 0.0, 0.0, 1.01, [1.0, 0.5, 0.0 1.0),
[1.0, 1.0, 0.0, 1.0], [0.0, 1.0, 0.0, 1.0], [0.0, 0.0, 1.0, 1.0]]
self segment ids [] for i wawggesell..uunnsgggeeeee (mass==000 half_extents=<sell
segment_id
segment_length, 2, welf.bbrdge / 2, self.
platform [ / 2], position=[self. I
platform ttaeformmssaar...m start_position(0)
self self.platform size[0]
self www.seeeeeeeeggtteeeeeee sttp:liiai..sllllaaaaaaarrre self.pplatfom.sttrrppoottti
self.segment.lengee self.
platfomm.start.psstttinff/
[2]], () self bet..pcccaeeyyaaalllooovvieeeeee
linkIndex=
latteralFrictiion00..,, restinutiooonn
self setf=sggeent_loooappppeeeeeeeee.
self. bot.bboo.sizeeeee [0.5, e.5 0.5]
self positiinnn..... init (selff start ] + 1.0,
self.box.posittion
self.platform.starre
Celf.platform.ttaee start position
self.box.size(2] / self.box oooo:idd=_ss....crrre..ooooosss. half_extents=
[2] www..platform..iiee.....
box_sizel0] / 2, self self.boo..sizel1 2, self belff.box.size(22 / 21, position=self. htw.:e/.....ooo.pooiiioo.o..ooo color= 0.0, 0.0, 1.01)
54

---

Published as a conference paper at ICLR 2025
self twtgtttppssttiiieeee [self. wwwttoormeeedd..oiittttto
self.
platform_ position [1], ctneffommeeenppstiiiiooooo
end_ position[2]
plwppooomoiiiee22222 / + self.box.size(2] / 2]
def create (self,,,, mass, postittttittssssiinnno position, color):
collision id htt.aiaeeee...eeeeeeeeaaaaaaaaaa
-p..EM BOX, halfExteents-hal__eexteeett visual_shape_id= = oegelpa...............eaaaaaaaae
return sel seff.
baseoolissooshhpeendbxxooollllll shap
baseliissaaShpppphodeeevvvsssssa org.ap/eeeeeaaapapppcceoeoooooor
basePosition-poiittin))
def set_objeet_.positionsselll
return np. ibb [0]]
object
def get_distance_.to (on.staanett.oojjec(((slle... of posittono = self object_id):
cobot_position self Liake.. I
return np. linalg. nor
robot cooot_position :2]]
def reset(self):
observation
self. 0.0
r wwt.rrrtsssssstttoooooooonnnnnnn robor. robot_id, robot.
starttposittion self. sell:pplafoomssstrppstttttiee statt_posiition((] self.platfor pollatformsstart.... position
[1], self haaf.pltfoom.sstatpppiittooooo
+ self welf.platforr platform-sizel22
2 2 Lelff init(2]), rebot.
"inks("base').orrr (oaisttt..rreettttiioo...tt)
self reseffasePsstinnnnnoieeettttttto
bbx id, self.
box wwwpppsssiiiiiiii init, 10.0, 0.0, 0.0, 1.0])
return observation
def step(self, action):
self dilffdistnnce to box = self. distance to object box id) self box position self.get object position(self (swlllsbo.oooeeeeeeeeeeeeeooolle id) observation, reward, terminated, truncated, info super step action)
self. time += self.dt
for i, segment-id in sruuereatessell..semeett__t. (eatttiiinnnsemtnnntie. id) segment_positii
self wellf.get.ooll peee_tbjject_ppsitiiioo
new gew.ssemmet_posin
Degment_positton [segment + self.
[1], wtt.rpplatoommsssttpssitttiioioo
segment-amplitut * np.sin(2 >npnnppppppp * (self.time
welf..segment_phaseeoo pffsptasssseeet self.segment.pp period)]
setf.pp.resttsseesttttooorrrrrin (segmett... new_segmentt..osiiion [0.0, 0.0, 0.0, 1.0))
return (observation, reward, terminated, truncated, info)
def get task rewards (self, action):
new distance to box self get distance to object box id) new box. poxitionn self self.geetobbectttt..cc.e position (self. box id) approach_box = (self dwwtance to_ new.distance.tt.box)) self.dt
forward_progress.u
self.box
[0]) self.dt on
1.0 if self robot. links Linksl'base' "
[2]
+ self.platform
/ 2 else 1.00
reach end platform 1.0 if wtlff.robotllink... links 'base' " position[0]
self self end position[0] else 0.0
reach_target_ 10.0 if np. winalg.norm
[:2]
< bolf.bo / else
0.0
55

---

Published as a conference paper at ICLR 2025
return ('approach_box box': approach_box, rrorward_progress_boxx
(rwward_prggres box, www.bddgg.oopppafffo..e
'reach_eed_
Bichh_end_llaffrr treachhtarget torgtttttettttcatioonn
def get terminated(self, action):
robot_fell cetrform_starttppsstionn
links[ baase' " possition [2] < self.
box_ celffgeetcojeettppsittoooc box [2] self.
platform start position[2]
return robot fell or box fell
def get_su)cesssselff:
box_position self return np. linalg, nwrm www.noox-ppoiiiioooooooooooooo.. pocp:/ooosiiiiioon iissiiion :2] self. target-position
[:2]] self. box. size /
Task
Figure 16: Task of the OMNI-EPIC run presented in Section
import numpy as np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2EEnv)
Dodgeball Game
Description:
The robot spawns in square arena measuring 10 meters by meters
Bails with a radius of 0.25 meters spawn randomly within the arena at height of 1 meter and are launched toward the robot with an initial velocity of meters per second.
The balls spawn at a rate of one ball every 2 seconds.
The robot's task is to dodge the balls for as long as possible.
Success:
56

---

Published as a conference paper at ICLR 2025
The task is successfully completed if the robot survives for predefined duration of 60 seconds without being hit by any balls.
Rewards:
The robot receives a reward for each time step it remains standing
and avoids being hit by ball, encouraging it to dodge effectively.
small survival reward is given for each second the robot remains in the game, incentivizing prolonged survival.
A penalty is applied if the robot is hit by ball, encouraging it to avoid collisions.
Termination:
The task terminates immediately if the robot is hit by a ball. The task also terminates if the robot survives for 60 seconds without being hit by any balls.
def
(self)
super(). init ()
self.arena size [10.0, 10.0]
self.arena position = 00.0, 0.0, 0.0]
self.arena id self.create cratt (oa half extents=[self.
arena_size(0]
/ 2, 0.1], position=
selffarena_posiitinn, color=[0.5, 0.5, 0.5, 1.01)
self..p..changeDynaag lateralFriction-00., restitution=0.5)
linkIndex= 1,
self.ball.radius self
self.ball_.elocityy 5.0
self.ball_spawn_inteevaa 2.0
self.ball.ids
self self.robot_positionnnng = [0.0, 0.0, self.arena position[2] 0.1 rint. 'base'] possition_init[21] self anffrrooot.orreenttiio..mmm
2 +
self relf=--p.getuuatemiioooroooooo
([0.0, 0.0, 0.0]) self.survival duration 16.0
def create box(s, mass, half extents, position, color) callision_shap_lii self otgprp/eeeaae.......aaooooooaaaa (sapppyes BD..EEMMBOX, hallfEstenttshhlll.eextttttt]
GEOM_BOX, helfsfstennssssllffoottttss rog. return SatTipppceeetteiitttaoo//oaassss paceollssssshhphhherxcoolllss... http:ppwwhhaaaaoooo.aoo..ooi.aaa
def create ff mass, r wewocity...o.ooooo color)
collision_shape._ ott.ret....rrttttoooooaaaaaaaaaa
-P.GEOM_SPHEREE rradigs-rraiusss
visual_shape_ii id = setff./cccceeeeessauaaaahehhheee
GEOM SPHERE, ratiussrraduuss
ball self p. www.preeeeehhttttlooooyaaaaaaaas oa
self oet..p.rreetessseeetttoovoooeeee Linerrellccityyy
velocity) return ball_id
def reset(self):
observation =
self.time 0.0
po..ettaeeestttoooooraaaaaaaaree robot. robot id, self
.robot_position.inii,, htt.:/ttt...s....oooooooootttaia
for ball_id in self.ball..ds:
self. reef-ppp.remveBBooybbaLLL..
57

---

Published as a conference paper at ICLR 2025
self. [] return observation
def (self,,,f, action):
self belf.hit.by.bale False
observation, reward, terminated, truncated, info super () step( action)
self. time += self.dt
if self. time &
< self.dt:
ball_position
+ self. radius, high=self.arenee size[0] self.
ball radius), np. random www.rrndoo.uiiormmoooooooo selo=wsllf..oo arena size[1]
+ wew..ball radius, bigh=self..reene s /
self
ballura 1.0]
bal vallvelocity = www.rranndom.iiffoo((C.....
1.0, high=1.0), np.
fandomm.uniform((oos) 1.0, high=1.0), 0.0]
ball_velocity selftttaeeeelbbbaa.eo..ooc..
mp.arrayyball
/ /mmp.unalgg.ommbbllleelcccccii.
bal [l__====ssf.crrttte..perremssss
radius=self.
ball postaiiou position, velocity=
ball wow.oiittycoooor._111111 0.0, 0.0, 1.0])
self.ball.ids..peeeddeeee for ball id in self.ball contact_points s - hrt.pp/wwe..aaaa.....o.ooo..ooor robot_id, bodyB=ballidd if Len(Contact_pointts > 0:
sy by = True break return (observation, reward, terminated, truncated, info)
def tatktass_eeeardss
action):
survival = 1.0 hit-penalty = -10.0 if self. self.hit-by-bal. else 0.0
return { ['survival': survival, 'hit_penalty': hit-penalty)
def get terminated action):
if self.hit by ball:
return True if self. time self. eul..suuvvvaa. return True return False
def get_successsself))
return self. time self survivaL and (not self. hit_by.ball)
Task 11
import numpy as np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2Envv:
J Jump from moving platform to moving platform to cross water and reach target zone on a platform.
Description:
The envi ronment consists of a start platform (3 m x 3 m 0.5 m)
and an end platform (3 m x 3 m 0.5 m) placed 20 meters apart. The two platforms are connected by series of moving platforms 0.5 m) placed meters apart.
(2 m 2
Each moving platform follows a sinusoidal motion along the y-axis with an amplitude of 1 meter and a period of 3 seconds. The motion of each platform is offset by 0.6 seconds from the previous one.
58

---

Published as a conference paper at ICLR 2025
Figure 17: Task 11 of the OMNI-EPIC run presented in Section 5
The platforms are placed above water surface, and the robot must jump from one moving platform to the next to reach the end platform.
Success:
The task is successfully completed when the robot reaches the end platform.
Rewards:
The robot receives reward for each time step it remains on the platforms, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the end platform, incentivizing swift movement towards the goal.
The robot is penalized for falling into the water.
Termination:
The task terminates immediately if the robot falls into the water or reaches the end platform
def
super(). init ()
self. wif..water_size [1000.0, 1000.0, 10.0]
self self.water.positionn = [0.0, 0.0, 0.0] = sel....ceeteeebbxmmmass.....
solf_extents=(sel..
water_size[0] 2, self.water
/ 2, self wwwer_wizeeeeeeeeeeeseee
2], poitiionselffwwaeerppsstttoo color=(0.0, 0.0, 1.0,
0.5])
self htfffplltffom.sttrtssieeeee
3.0, 0.5]
self wtt.f/.tafoomstartppsstttt
= (0.0, 0.0, self..atterpooittion [2] self self.waatrr.siee222
start-size(21 2]
self -
box box half_extents= self.platforn.sttrttsseee
self sel.fplatfommssttrtssieeel// / www.. www.f.ppaatoom.sttrttsiiseeeeeee platform start position, color=[0.8, 0.8, 0.8, 1.0))
2], position=self. f...
self. eelf.platform end size [3.0, 3.0, 0.5]
59

---

Published as a conference paper at ICLR 2025
self Seaatorm_enddpositteee = [self.platfore Seetf.paafoommsstrttpstttttooo +
20.0, setaattormmstattpssiiiiiii
self.
platform_start_ppe [tatrommsttrttpssttin]]]]]]]]
self plldtform_eenddddd self. wtt.rraataebbbomassreeeee (mass=00. ealfeext
self wwwpppomm.e.ddiiiiee.p.
2, self.platform.eem end size[[ / 2,
2], position=self.
platform end position, color=(0.8, 0.8, 0.8, 1.0))
self net..run.movinngplaaffrne
self..vving.llatffmm.szze [2.0, 2.0, 0.5]
self eet..noving.platffmmmmmmmmcee
1.0
self. platfoorroppatffommmmmmmm period 3.0
self.moving platform
self wwt..mvvnggplaatfrrro
self.moving platform phase offset 0.6
ids
for i httpppppwooooooteteoeeeeeeeeeett
+ 3.0, [trtommmsstttttttstion]11]]
h.t.==ee....lfffomm.mtttttttttoo self.
moving Saanngplaatii platform.id wtt.ii/deee......eebe..ooaaaesss
half_extents=((
self mww.ing.plaaffommsszeoo)/ / self.
moving platform.size(1]
self.moving platform siittorrmsiiee1 [2]
2] position=moving paatinggppaffrrmmooo platform-posiiioon, position, color=(0.8, 0.8, 8, 0.8, 1.0)) self. moving platform ids append platform id)
def create_ mass, half_extents, callision_shape_le
posttion,, color):
self
....EMM BOX, halffestenttshhalfffetttttt visual_ id httpaaaa......a.a.......eaaaaaaa GEOM BEE0M_BOX,haaileee haw.eeeeeeeeeeeaa....t.ttttttttn return sef.e haaslssssusshnppphheeeeevvssssss
def sf object
obb return np. htw.:aawwwyy..t...aaaeeeeetotooa [0])
def get distance to object object id): object_position self oe..e..oeeet..psiiiiooooooe
robot_position self. Linkot. J .position
return np. .om.iiig..oombboeeett... nommooheect..osstiinll
robot_position :2])
def reset(self):
observation super() reset()
self.time 0.0
self. B rrt.rrreeesssiiiiiooooooeieeiitr http:///eeeetttttoooeseoeeoroorr robot. robot id,
self httprfoom.starrtppssstiiooo//ooo
start
self.platform.st start position + self.
[1], sel.ffplaffommsttrr.psssttooool
platform.start_sizet22
robot. [
position_init(21] self. robot. links base .rroeenttiion.iiit)
return observation
def step action):
self. distance toatice__t_ppaa. tortafootttollaooomeeee self. oto.feget.disttnnettooooooppppp @eatform_ed_.1e
observation, reward, terminated, truncated, info super step action)
self.time time += self.dt for i, moving_platformm.n in enumerate(self..movv www.moiinnggpttffmiiisss.
moving wwt.pplllafooossttiiiiooo potittpossiiiooonoo self. .ee...et..... object position(
moving platform id)
new moving wwwppplatao..ooo
wwwiig.platoorm.mmmccccm position [moving platform position[0], self.
cereforrmppsitioon
gelf.platfomm.startpppiiiioo
moving_platform.am....
* np.sin(2 * np.pt. * (self. time
60

---

Published as a conference paper at ICLR 2025
i seL.. -fffass self.
moving moving-plattorne pecitfoommperiiidd. moving 1 self reeeffaeeooittonnnnoeeeeaattttre parrraaaaaanonoooooonaaaaaoooaaa new new_moving-piaee platform position, [0.0, 0.0, 0. 0, 1.01)
plaatfommppoiition
return (observation, reward, terminated, truncated, info)
def get task rewaddssss action) www.iiisttnne-topplafffrro platform_end_id)
self get.ddistaree.to..ojeettllll
on-platform = 1.0 if rebot Links| base' positiion[]] > self. platform start position[2] else -1.0
reach platform end (self. distance to platform end new distance to platform end) / self.dt return {'on_ p'otfoo'aooom'. on ml_ppaffrmm 'reach.platfommendd' reach.platform.end)
def get_ (sef, action):
is_ is_in_water = self.robot. Links["based " p.position[[] < self. water_position + self water-sizei2] / is on platform end self.get distance to to. object (self. platform end id) self end size[0] / 2
return is in water or is on platform end
def get_success is=on_platform_enn self to platform_end_ia id) self plattoormmeeed / return os_on_ pla.tooo end
Task 12
Figure 18: Task of the OMNI-EPIC presented in Section 51
import numpy as np from oped. envs. r2d2. base import R2D2EnV
class (R2D2EEvv)
===
Navigate a terrain with obstacles to reach a target zone.
Description:
61

---

Published as a conference paper at ICLR 2025
The envi ronment consists of start platform (3 3 x 0.5 m) and end platform (3
0.5 m) placed 30 meters apart. The terrain between the platforms is filled with static obstacles such as rocks (cylinders) and trees (cylinders with a height of 3 meters).
The obstacles are randomly placed with minimum distance of 1 meter between them, ensuring the robot cannot go around them easily.
The robot is initialized on the start platform, facing the end platform.
Success:
The task is successfully completed when the robot reaches the end platform.
Rewards:
The robot receives a reward for each time step it remains on the platforms or the terrain, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the end platform, incentivizing swift movement towards the goal. The robot is penalized for collisions with obstacles.
Termination:
The task terminates immediately if the robot falls off the start platform, the terrain, or the end platform, or if the robot collides with an obstacle. .*.
def
Seiiiit_ii. (self:
(nperr))-init__))
self. wla..ppatfoomssieeeeeee wwwptooomsttttttpppooooo
[3.0, 3.0, 0.5]
= [0.0, 0.0, 0.0]
self colf.plattoommen.psitttte = ptaafpptatoommsstrttpoossss
+
30.0, self.plaffommmsttrrrpsitttoooooo plattom_starr__pssitoo[]]]]
self.
self.platform.stt start idd self.create (mass=000 half extents=[ self self.platform size[0] / /, self.platform size[1] / 2, self.
platform size[2] 2], positionself.plattorr start position, color=[0.8, 0.8, 0.8, 1.01)
half_extents=- self.plattommeend..dd s2 2, self.
self self.plaatorm.sizeel]]
(latform-size(2] p ptmp posiiiiionnnenddppiitiiooo
color=[0.8, 0.8, 0.8, 1.0))
sel ert.:cchhhyyyyicloolylllllataate
self self aet.acttooeeonnaiicooooooooeeeee ott.::tcco...corr....ttttttttoos self.
set..erraainnssieeeeeeeeetrrprre self
oo 0.1]
+ self.
terrain 2,
htt.:ppplafformsttttpppsssssss start self.
/ self
-terrainnszze/// 2]
self.terrain.id w (rraaee_bxxmmss=00..,, exw.textenttt
terrain_size(0) 2, www.terrain..iizeiiii 2, self.
terrain.size(2] 2], positionsell...errain.ooooii
color
=[0.5, 0.5, 0.5, 1.01)
sel self. htt.:p/chgggoononiiooooooooeeeee
linkIndex
=-1, Lattera(Frictiono... 8, restitutioo=0055)
self.wall.size [ 0i 1.0,
self self.wall lefftaloole
position tself.terrain po self.
terrain (rraain-position11 self.terrain size[1] 2 self.
wall_size[]] 2, self. taaf.teerrainpssttion22
self.
terrain.size(2] 2 + self.wall_sizei21 / 2]
62

---

Published as a conference paper at ICLR 2025
self wel tssetftterrannnssittoono self. terrain_-osiiionnl]] wall_size[1] 2, self.terrain.pssttion2) self.
2 self.
terrain.sizze[2] 2 self.wall-size(2] 2]
self LLff.walLleeftidd
"olf_eextents=(eeef wa wall sall_size(0]] 2, self.wall.size[[]] / 2, self welf.wall... size[[2 2], Sesattoosself.wallllef....o
color=(0.5, 0.5, 0.5,
1.0]
sel self. wel.f.allltc
half wwal_size(0]] 2, self.wall-size(1] 2, welf.wall_szell/ee 2]. possotoonsseef wall right position, color=(0.5, 0.5, 0.5, 1.0]) self.num obstacles = 10
wall_right_id self. self.createe.box(msss=00.
self obstacle_ = 0.5
elffobbstalle_height
1.0
tef.ttree_hiihtt 3.0
self.obstacle ids []
for in range(self. mum_obstalles):
while True: obstacle position [ rww.om.nooo.mmmmm wnw.ormmseef.ff....mmmmm
platform start [otattoooiiiin(]] + =elff.laaform size[0] /
2 self.obstacle radius, self.platform end position size[0] / sise [o] self.platform
self.
srrraa/ppostttolllleeeettttrarar o oww.rrauuuuu.....rrnoomomuummmmo Cerrain_size(1] /
self
.obstacle_radius, self.terrain_position]]] self.
self self.obstacle
(esstucle_raiuus) self.
terrain..oosition[2]
/ 2 self
.bsstacle_heiigt 2]
if aat[app__.angg.ooommtttorrrrrooo
np. np.arrraylotherrobbstacll position[:2])) 2 * self.
cistacle_radiure 1.0 for other_obstace oosttooitteepposiiii
position in
seeff.eet.ojeett.ooitttnooottsss obstacle_id self self.obstacle break
for
if np. random. rand 0.5:
obstacle id = self. create cylinder radius=self. obstacle radius, wwgghtssllf.oosteceeee height, orientation= 0.0, 0.0, 1.0], color= 0.0, 0.0, 1.0])
ieaitiin=oosttaie.pssiitt
else:
obstacle id = self createrreeeecoo crnnteecclinderrmasstt obstacle_ height=self tree heigh,, position= obstacle posittonn orientation= 0.0, 0.0, 1.0], color= 0.5, 0.0, 1.0)) self.obstaclle ids ( (bbttaclel
radius=self.
def create_box mass, half_extents, postiittentts,ppsitiioo
position, color):
collision_shape.iio ott.pppp......a......ooooooooooo htgp:p/eeerrrrrooooooooooooeeeee (shapeType=self.
GE0M |
p.
GEOM BOX, halffxxtents-malfffottrrst orgaaaaaaetttttttt.s..rrroooorrn rgbacolor=color)
return self set..pprrraardtttttbaayllllaaass
basePositiooopooiitiin
def creatte_cylinder((s) mass, radius, height, position, orientation color):
collision_shape.. oet..p..ccttttolllooooocccchhhee
COGEOMMCYLLNDEEL radius=radius, height=heightt)
visual_shape-1 id het.fppp.crattteeuuuaaaaaaaaaaap
GEOM CYLINDER, radius=radius, length=height, rgbacolor=color)
return sels bacecollsSssshheennercccoooooo..
63

---

Published as a conference paper at ICLR 2025
baseWiisuusShhppelnneeevvvssseb_ oosesaai_hhhaaaooooooossooooooor baselrientaaiioooorriinttttt
basePositioonpooiition
def
object id)
return np. hpt.sarww.........eeeeeeeeooooer [0]]
def get.disttane..o..ojeetifs. objectct.dd
object_position = peeff_eee_obbeectppsiiioooooooo
robot_position self. ww..ooot..inkssbaasss 1.psiiiion
return np. wwiianggooomoooeee..........ee.e 2tspjjicctossiiio[[222 robot pontttpppiiiono0000.
def reset(self):
observation = super(). oro
[1], gelf.platfommssaarpppstttooo + self.platfom..siee
/ 2 + self. wiw..rroo.....k[[[..... htsp:a/iibass.pppsiiiiiiiititt/ positiion.inii((]]) rebot.
return observation
def step(self, action):
self distance to platform end self get distance to object platform_end observation, reward, terminated, truncated, info super() step action)
return (observation, reward, terminated, truncated, info)
def get_ task rewards (self, action)
new.ddistacee-te tetettopppplll.
wet.ttaneeto.plaaffm...mmm platform_end self Seff.get_distancee to objttt platform_end
survival = 1.0
reach-platffrmmeen
www.ffddstannce... caapnneettooplaffrmeeee.
ttwpaanceettolllaffmeeeeeeee / self
collision 11.0 if any((lenssel... w.t..eeetettttttoiintss ( robot. robot id, bodyB=obstacle id)) 0 for obstacle id in self.obstacle ids)) else 0.0
return ('survival': survival, 'reach platform end': reach-platffrmeend, 'collision': collision}
def get_ sen.teemminated(sset. action):
is_ fall = self. links| base Baaassss'aaeelosiiiiiiis. position self.
derrain_position22]
cs_collis = any((len(self..
robot
.robot id, bodyB=obstacle id)) 0 for obstacle id in self.
obstacle.ids). ids))
return is_ fall or collisioo
def get_success(selff): is_ is_oo__lattorm_en end = self gettddsttancee...eeeeeeeeeeeeeee to to_object (self. platform end id) self self.platfor size(0] / return is platform end
Task 13
import numpy np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2Env):
Navigate terrain with obstacles to reach a target zone.
Description:
The environment consists of start platform (3 3 x 0.5 m) and end platform (3 x 0.5 m) placed 30 meters apart.
64

---

Published as a conference paper at ICLR 2025
1
Figure 19: Task 13 of the OMNI-EPIC run presented in Section 5
The terrain between the platforms is filled with static obstacles such as rocks (cylinders) and trees (cylinders with a height 3 meters).
The obstacles are placed in grid pattern with enough space between them to ensure feasible path. The robot is initialized on the start platform, facing the end platform.
Success:
The task is successfully completed when the robot reaches the end platform
Rewards:
The robot receives reward for each time step it remains the platforms or the terrain, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the end platform, incentivizing swift movement towards the goal. The robot is penalized for collisions with obstacles.
Termination:
The task terminates if the robot falls off the start platform, the terrain, or the end platform.
def iselt
self pwpppplffassssee.
[3.0, 3.0, 0.5]
seatformmstart.posittt [0.0, 0.0, 0.0] self. self.platform.
iselffplattfor stsrt_ + self sreatformenddpositioo
30.0,
self.
self www.pooommtttttt.iiddo...oo. www..
[xtentext self. wwt.pplloommmszee0000 / 2, self self.plattorm.sizeii]] 2, self.
platform 2], postttonseef..patttoosttttrrpppo color=[0.8, 0.8, 0.8, 1.01)
self. gelf.platfore end id = self.cree. create box (ox half extents=( self self.platform size[0] / 2, self. size[1] / 2, self.
65

---

Published as a conference paper at ICLR 2025
platform_size(2) / 21, pot..toosssss....atto.....oopppp
color=[0.8, 0.8, 0.8, 1.0])
self
1 Lateralffriiiciinn00. ort.:/cccceeeoooee..seetttrreo.
restitution=O.5)
self = [strraain_siee==__ssfe.paaaaffon (self.plaaallaao end wtw.:r/mmmmmooooiiiooo.o. self.
p 0.1]
self
+ self.
terraan.ssiee00 /
self.
platform starttssatt... start_ positions
ceaf.platffrmmsizellee / self
terrain sizeniiiiee]]]]]____]_
self.terrain self.create box 0 half extensssttt= terrain 2, self. wer.aarrraniiiiiiiitt. 2, self.
terrain.sizei2]] 2], posttionssl...errainn.osstttt
color
=[0.5, 0.5, 0.5, 1.0]) sel self ss httngg/oymmnislooooooaaeeeeeeeer
linkIndex
=-1, LateralFFictionn0.., restitution==0.55
self self.walLssize [sseff.eerainnnsizzeo]) 0.1, 1.0] 1.
self www.aalllettpppiittiiioooo = [self.terrain 0 self.
[errain.possition[]] www..terrai...iiieeeeeeee size[]]
self.
terraann_tee22222eeggaaaaaae1112 self.
wall-
self.terrainn.psition]]]
terrain.size(21] 2 self..wall_size(22] 2]
sel
eelf.wall.riggttpositioo i|s@.ftterrainppsittiiiin
self.
terrain--osiiiooni]] self.terraiin.sizel]]]
self.
wall_size[1] selfftterainn.pssittonn2]] self.
Berrain_size(21 2 self.wall_size(22 / 2]
self self.walllleft.dd self.crratte.bo((mss==....
silf_extents=(seff wa wall_size[0] 2, self.wall_size(1) / 2, self welf.wall __zee_ / 2], postttonseef.aallleeee...iiiiii
color=(0.5, 0.5, 0.5,
1.0])
sel fflf. wa wall_ www.w / 2, self.wall.size(1] 2, self.walLssze(21 21. position=self wtt.eel...wwllLLllllle.. righ_riightpositionn, color=(0.5, 0.5, 0.5, 1.0]) self.nut obstacles = 10
@olf__extnts=ttee [self
self sllffobstacle radius = 0.5
self.obstacle height 1.0
self tei.ttreeeheight 3.0
self.obstacle-ids self.place.obstacles(
def create boxetf,, mass, half_extents, position, color):
cellisiio_shape.... self ohtappiioosstt...pp..rrttoooosss (shap
Pp..EMM BOX, halffxxeentsssalFF.xxeeett
GEOM_BOX, halffsxtentsssallffeeeeee,,, ortpr//eeeeatttattttttttenoooonn rgbacolor=color)
return oas.eeesccctteetiitoooooooaaaaao
aaccollssssshhphceeeecc.ooss..s. base//ssaaShhppenneeevvvvssasssp
def create cslin mass, radius, height, position, orientation
color):
collisiion_hhpeelie
-.P.GEOMMCYLINDEE
)
GEOM_CYLINDER, r hewgssshhhttttgghggeeeihhhhhhnt rgbacolor=color)
return ott.rr.....tttttttttttooooooaaas
baseVissuaSShhppelddeeevvvsssss id, basePositioo=poosition base@rrientationnooiieettttiii
def get_object_posseeeeeeeeeeeeeeeee pet.obbiett.positionnssfl object_id) :
return np. www.aarryylslll.....eeaaeeeeeeee oatpppaaaaaaaaae...eeeeeeeeeooen
[0]]
66

---

Published as a conference paper at ICLR 2025
def vet_aiie to object (bject_id);
object_position = _.get_object_posttiiii
robot_position =
Linksl bbase'
ppssition
return np. iiinalg.noomobb/........ horm(cbiect_pssition===:
position[
robot_position
def place obstacles(selff:
grid_ int htt../utcoobbsssss
x-positions = np. platform_sizele] platform end posiipooiiiio]--- o
+ self.
self.obbttalle_radius self.
-elf. size[0] / self.
obstacle radius, grid Size)
y positions = (tt.:oossssnnninspcceeeeffeeeeee [ self.
serraann_iizel11
www.obccleeeddiiio. self.
(errainn_osstionnl]]] self. terfttteeiiin obstacle_ grid_size)
self.
for x in x_positions:
for y in y-positions:
if np. (aadoom.rad()) < 0.5:
obstacle id
cyLinder radius= self celffoobttacle radius, height=self obltff.obttcceeee height self.terrain position [ttrrain + self terrain size[2] / + self. height / 21, orientation=(0.0 0.0, 0.0, 1.0), color=(0.5, 0.5, 0.5, 1.0])
position=(x,
else:
obstacle id = self. create wel.f.create.cylineeeee (massee radius= wwwtoclee position=(x, Cerrain_sizeli2 2 self.tree_heil
height=self. tree_ www.eteeeeeeigeeeeeeeeeee height, + self. 2] orientation= 0.0, 0.0, 1.0], color=(0.0, 0.5, 0.0, 1.01)
def reset(self):
observation = rr reset()
self. p. prreettssepiiiiiomooiiiiatttttrr robot. robot id,
self start- position self.platform start position
self self.platform.sizer links 'base' "oraee).oriettatioonn "'''assel.ppssiionnniitff] rebo.
[1], sel..ppatfoommstartt.oiiiiii
2
return observation
def step(self, action):
self. self.distance to platform..eee end self .elf.get.au destanne to. to
platform_end id)
observation, reward, terminated, truncated, info super () step( action) return (observation, reward, terminated, truncated, info)
def get task rewards (self, action):
new distance to platform end self. get distance to object platform end id)
survival = 1.0
reach-platform.ent end
to_ fo-pplatfommentt new.distance to.peetfoooaaatmmddd. / self.dt
collision -1.0 if
http://cseet...ppeeoottttttttttt robot. robot bodyB=obstacle self.obstacle_idst else 0.0
for obstacle id in
return ('survival's survival, 'reach platform_end end': reach paachhhpaafommmmmmmmmm end, 'collision': collision]
def get_terminated action): is_fallLoff = terrain-
links (aaiss[[bbar bbase J. position[]] < self.
67

---

Published as a conference paper at ICLR 2025
collisiio any((Len(self... pet.ppeeeeettttioiiissssolsssss robot
.robot_id, bodyB=obstacle id)) 0 for obstacle id in self. obstacle_idst return is_ off or is-colissoo
def get ssecessssss))): flrtform end self get distance to object (self.
is on on platform_end
) / return is.on_-latform_end
Task 14
Figure 20: Task of the OMNI-EPIC presented in Section
import numpy as np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2ENV):
Jump from static platform to static platform to cross lava.
Description:
The environment consists of start platform (3
x 0.5 m)
and an end platform (3 m x m x 0.5 m) placed 20 meters apart. The two platforms are connected by series of 5 static platforms 0.5 m) placed 3 meters apart.
(2 m 2
The platforms are placed above lava surface, and the robot must jump from one static platform to the next to reach the end platform
Success:
The task is successfully completed when the robot reaches the end platform.
Rewards:
The robot receives reward for each time step it remains the platforms, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the end platform, incentivizing swift movement towards the goal.
The robot is penalized for falling into the lava.
68

---

Published as a conference paper at ICLR 2025
Termination:
The task terminates inmediately if the robot falls into the lava or reaches the end platform.
def
self. lava_ 11000.0. 1000.0, 10.0]
[0.0, 0.0, 0.0]
self. self. box 0, half extents=(self,
2, self. lava size[2] / 2], positionsself.lavappssttioo color=[1.0, 0.3, 0.1, 1.0)) 0.5]
lava 2, self. size size[1] /
ott.:plaftommsttrr..sstttto--- [0.0, 0.0, self. ort.::.oot..........oooooottttoi self.Lava
self. Self.Lava.size(22 http://w..222i22222eetaatttttot. self.platform.starrt
self. platformmsttartt.dd self. box(mass=0000 half_extents= selt.plaffommstart..izeeo00////
self.plaffrm.startt.szzaa
sottioo / 2, sel..ppatfomm.sttrr-ssze]z] / 2], position=self.
starmmmmtttttttttttttiii color=(0... 0.8, 0.8, 1.0))
self platform end size [3.0, 3.0, 0.5]
self pwl.ppratoommmm end position = (self.platform start [o] +
20.0, self.platform pttpttfooomsstrtpppootottt (ttp:mmmmstatttpppiioolll]l self.
plattoom.starr.ppssttonn]]]]]
stt.ppatttommmmdddddeeeeerrrrrrr self.platform_end.e end_mm__en__iiieele 2,
half_extents=( size[2] / 2], position=self.
self. ptwtfoomeeee_ssieell
platform end position, color=[0.8, 0.8, 0.8, 1.0)) cenfformend__oosttion,
self num static.plattorm. = [2.0, 2.0, 0.5]
self. ptt.sstttti..laffom..ii...
self. www.ti..plaatommmmn...
for i in range static.platforms]
static_platformposiil =ssetfpuatfommssttttppsstttt = [self.
+ 1) 3.0, 2.2000a0...aaffmmmsttrrsssttoooo platform_ (troommsttart.ssttio11]]]]] static platform id self.creea (mass=00.0 half extents=[ self. static Bacilccplafforrooo size[0] / 2, self.
self.
static platform size[1] 2, self. platform size [2]
21, posttoonstaiiicpltrfooooooooo... : 0.8, 0.8, 1.0)) Sat.aaacchh.too.......ppmmpasaaa
color=[0.8,
def create create_box(self. mass, half postiinn color) stt.peeeeeiiisstt.....rrrreeeeeo
-p..EMM BOX, \
GEOM GOOM_BOX, halfesttentssshllleeertttss,, return seT.eppocrateedtttttoooooaasssss
rgbacolo==ccccol
def get ceeject_posiiionseeSe_eee id): return np. wrg.arrwyyet.....eessssstttoooor [0])
def get wet_anistnce_tt (tt.tanccettooooec((sseee.... object_id): ;
object_position self. rebot.obot.ro )
robot_position
return np. www.llannnnnnnnn.. hww../ooommoeett..osiii......
robot__ossition[222 :2])
def reset(self):
observation super().
r htt.pppreettsssttttoooodddoraaan roooo
self.pplaffom.stttrroopsssttotoo wtw.:/ssllllll.... wew..platooo....s.
www.tform.starr.posittttt
[1], selffpplttorm.sttrrtposttttooo + self.
69

---

Published as a conference paper at ICLR 2025
Slatform-start_1122 2 self. links "base'].
position_init(2)) self. robot. links bbsse' I.oiientation return observation
def step(self, action):
self distance to ho.pplatorm.eem end self distance to tt..ttttoooooooo. tse.... platform end id)
observation, reward, terminated, truncated, info super() step action)
return (observation, reward, terminated, truncated, info)
def get task rewards(self, action):
new distance to wwwpppllll..ooooo end = self self.get distance to object [latform_end_id)
on_platform = 1.0 if self robot. Links| base' baa.baaasssstttiliee possition[2] > self.
htt.foommsstarttpssttioooooooooo else 1.0 new
reach
sself.distance to tt_
wao-plaaformeee
/ self.dt
return { oon_ooofoorm' on plapafoorm, 'reach_.latfommeend""
reach paachppplafomeente
def get ter,,, action):
is in lava rolo..
robot. links|
( 'base "aase''.positions [2] < self.
[ava_positionn2] self lava_size[2] /
is_.on_platommmeenn self. delf.get_distance.tto ob obeettt
platform_end_id) self.platform_end size /
return is lava or is on platfoorm end
def get_ vet_successsselff: is_on_platform end self Seff.get_distance distance to to_ objoct (self. platform end id) < welffpplaffom..... end size / return is plaofopaaaoommee
Task 15
Figure 21: Task 15 the OMNI-EPIC run presented in Section
import numpy as np from oped. envs. r2d2. base import R2D2Env
70

---

Published as a conference paper at ICLR 2025
class Env (R2D2ENn))
Jmp from static platform to static platform while avoiding moving
obstacles.
Description:
0.5 m) and an end platform (3 m x m 0.5 m) placed 20 meters apart. The two platforms are connected by series of static platforms (2 m 2 x 0.5 m) placed 3 meters apart.
The environment consists of start platform (3
Between these static platforms, moving obstacles (cylinders with radius of 0.5 m and height of m) move along the y y-axis with sinusoidal motion. The ampli pude of the motion is 2 meters, and the period is 4 seconds. The motion of each obstacle is offset by 0.8 seconds from the previous one.
The platforms and obstacles are placed above a water surface, and the robot must jump from one static platform to the next while avoiding the moving obstacles to reach the end platform.
Success:
The task is successfully completed when the robot reaches the end platform
Rewards:
The robot receives reward for each time step it remains on the platforms, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the end platform, incentivizing swift movement towards the goal. The robot is penalized for collisions with moving obstacles.
Termination:
The task terminates immediately if the robot falls into the water, collides with an obstacle, or reaches the end platform.
def init (self):
super(). init ()
self.water size [1000.0, 1000.0, 10.0]
self wtt..wateerppsttion = [0.0, 0.0, 0.0, 0.0]
half-extents<(selff.
water-size(0] seie / 2, self. water-size [2]
2], positionssel...eterrssttttt
color=(0.0, 0.0, 1.0,
0.5]) self ptt.ppaafffmmstttttsieeee [3.0, 3.0, 0.5]
sel. [0.0, 0.0, self.water self.water-ppsitionn [2]
self. sesef..aterrsiieee2)22 welf. startt
2] box half extents(
self f
sett.plaffommmstartsszzeel///// self.platforn.starttsiiee
/ 2, ses.. 2], position=self. Ff
0.8, 0.8, 1.0)) self.plan platforr end size =3.0 3.0, 0.5]
self platform end position (self.llatfor start position[0] pttptformmstarrtppsiitooool.
20.0, self.platform.starre start position[1], self.
platform.sttrrt.pssittinn/
sel wtwppaaffrm..md.siel111111111111 2, haff-plaafrmmeedddiizee
half_extents=( 2], position=self.
2,
self self.platfommeen_siizelll
seeeformeen__postttii color=[0.8,
(olo: 0.8, 0.8, 1.0))
self. f.f..uum_ste num.sttaiic.llaffomss
platform size = [2.0, 2.0, 0.5]
www.stttpaaapttoommmmmmmmm ids
for i range(self. static platforms): static_platfommpooitttoo
(i 1) 3.0,
self.
platfomm_start_pssittinnn.
71

---

Published as a conference paper at ICLR 2025
=
bwwssss- hälf_extents=[
pul..sttaiccpllatfom..cm 2, self. staattttttc.ccccfomti size[
/ 2, self.
static.platfomm.szeele
/ 2], positionsstaticclllffom.omm 0.8, 0.8, 1.0])
color=[0.8,
self static ptaettiipppafffmm.tto ids appenosttaaicccpptattttota
self. moving obstacles 4
self. eelf.mviing_Oostacleeradduu = 0.5
self.movingg-bbttaciee..einnnn = 1.0
sell eef..nov.nng-ootteeleemmoo.tttd = 2.0
self. self.moving obsaalll
4.0
self.moving gelf.moving.obbttaccc/o abwtagoostttlleaaeeeeeeee phase offset 0.8
for www.miiggobbtttleeiiii.....e
self. self.moving wwwtaollsteccccc ids
i in rengeessel..nuu..oovnn..... Doving.obsttall_ppsittttt
1) 1.5, self www..ppaaaofm..sttateeeeeeeeeeee postiiiiiiti
self.platform.starle staffrrmssarr_ppssttoon2]] + self.
moving obstacle_height
moving iwvinggobstaciie
radius=
self moving abstacle_radiut height=self.
moving obstacle obighcle position-moving obstacle position color=[1.0, 0.0, 0.0, 1.0])
self moving obstacle ids append obstacle id)
def create_box mass, half_extents, posttion, color):
callision_shapeee
self htt.://..ereotcooooossssoossaaap
.... BOX, halffextenttshhalfffeettttt. extentss id htt.lpppp......ee..ccceee/aaaaae REE0M_BXX haa eeeeee return sef..pgcccccccctttttoooooaaaasrs
def create_cylinder(setf, mass, radius, height, position, collision_shape_ie att...p..arttttoolsoossaaeeeeeea G.GEMM CYLINDER, id sel.. oet.:ppp.rratttssssaaaaaaaaaaaaa
posiiiinoooooiiiionnn :
GEOM CYLINDER, radius:radius, radius-raaiuss henghuhhhghstnnggghhhehhhhhhhht return sar..pppcccaeetttttaaaaaaaaaaaas baggghaheeeiiiittgaooooooooooooi
rgbacolor=color)
base/issuahhhppphnnneevvvssseeee
tispoositioon-postttinn org.a//eeeeeeeieoeoio.ooooooooor
def fe object_id)
return np htt.:ttooooolaoooooooida:::aaa.. [0])
def get distance to object object id): object_position pelfr.geet.oeeetppoiiiirrroeeeee robot_position relo. Linkk .oosition
return np. winall.nlnnn. honnalggnnmmccbeettpsssttoooooo
robot_posittoon :2))
def reset(self):
observation super() reset()
time 0.
htt.::/wweeccasse....oooooooeoer htt.://ooooeootaaaioooeooerooooo selffpoatffommssttrtooossttolll
[1], self. selff.platommmstttttppssttttoo + self.
robo. links ('
position_init([]]. self. robot. links base oorietationn return observation
def step(self, action):
self www.daccesttooppooocc wo..ttamee.ttoplaafff...e.. platform_end self. platform_end
72

---

Published as a conference paper at ICLR 2025
observation, reward, terminated, truncated, info super step action)
self.time += self.df for i, booving_obstace id in enumerate obstacle
moving obstacle position postaicleposiiion
self objectt
moving obstacle id)
new moving costacle.ppsittiine
[moving.obstacle obstacle position self
[moving self.
* ninsin(2 * np.pi * (self. time self.
i self setT..nwiinn.o0sttceeeplleeeeefs
moving obstacle periiodd, moving obstacle position[2]] self resssttssPssitionnnooiiaaaaaaase (moving obstacle id, postaion [0.0, 0.0, 0.0, 1.0])
new moving obstacle position, return (observation, reward, terminated, truncated, info)
def get- flo-platformeend self distaddstaneeettt (t.. (self.
(self, action):
platform_end
on_platform = 1.0 if self. robot. links 'base base' wtt.aaaaaaeaaitiii.eeeoo posstiion[]] > self.
pplatformsstarreeeeepepprpeeassa positionn))) else 11.0
reach platform end (self.distance distance to platform end
new distance to platform end) / self.dt
P.getCCntaactPPintss collision with obstacle -10 if Len.self.
> or oot.eeeef..pppttttreeeeeoosooooo robot_ htt.://oooo.....ooooooooooooaaia .-P ....ecootactfonssooo//ssettroood
or (sn(f
bf..
0 or ( modyy==eeff.moving.otttaae ids
(bodyA=self..
[3])) > 0) else 0.0
return ('on_platform on pllptaro 'reach-platform.en
reach_ Baach_platfrrment "coltision.iithooossssi collision.with.oostacc
def get_ wet.tteeenaeeeeeee....... action):
is in water = self. water positiono) + [izer_
poosiio.position [2] < self.
size[2]
is platform end self distance to object (self.
platformLend seelf,platffm..end.sieel0000
co otl.osoooottttttttttteeeeeeerree wen.eeee...ettttttcoooooooooorrr
ottp://www.....oooo....o.o.ooaot bodyB =self. eself.moviiee htt..//wwonnnotttclcllllllll))(
robot__dd
0 or
oetcontaccoountttooooossee..oooo len(self..p.
bodyB=self. bocse,
((en(seiff-pppeeCCotatttol) bodyB=self moving obstacle ids
[3])) > 0)
return is_in_water or ps coltision..ithh.ossttll...
def get success(self):
is platform end self get distance to object (self. platform end id) self fll.pplaffomm end size(0] / return is=on_platform.em
Task 16
import numpy as np
from oped. envs. r2d2. base import R2D2EnV
class (R22222v)))
=**
Task: Jump over rolling logs on a bridge
Description:
73

---

Published as a conference paper at ICLR 2025
Figure 22: Task of the OMNI-EPIC run presented in Section 5
The envi ronment consists of wide static bridge that is 50 meters long and meters wide, elevated 10 meters above water surface. The robot starts at the beginning of the bridge, facing the positive x-axis.
-Logs (cylinders) with a radius of 0.5 meters are spawned 2 meters above the bridge and 10 meters ahead of the robot, rolling towards the robot with initial velocity of m/s along the negative x axis.
The robot must move forward on the bridge while jumping over the rolling logs to reach the end of the bridge.
Success:
The task is successfully completed when the robot reaches the end of the bridge.
Rewards:
The robot receives a survival reward for each time step it remains on the bridge, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the end of the bridge, incentivizing swift movement towards the goal.
The robot is penalized for colliding with the logs.
Termination:
The task terminates immediately if the robot falls off the bridge, collides with a log, or reaches the end of the bridge. =**
def (self):
(uper()..inite ()
welf.water.... www.wwwtrsiieeeee = [1000.0, 1000.0, 10.0]
self.wattr-posittion = (0.0, 0.0,
self.water.ia.
wt..craatebbb half_ extents=[self.
water_sizel0] 2, silf 2, self. watew_wizeeeeeeee [2]
21,
color= 0.0, 1.0,
0.51)
self. bridge length = 50.0
self.bridge width 5.0
74

---

Published as a conference paper at ICLR 2025
self wtlf.bridge..ossition 0.5
self. www.dbbdiiiiiigge
weelf.water.positteee
self.
bridge_length / 2, self. www.ar-ppssiiionill self.
water_posittonn2]]
self.water-sizze2) 2 + self.
brighe_height / /
self .tl.bbridgeiid self. crreate_box(maseeeeeee
half extents=[self.
bridge-length length / 2, self.bridge width / 2, self.bridge height color=(0.8, 8, 0.8, 0.8,
2], 1.01)
self. log-radius = 0.5
self. log height 2.0
self.log spawn distance 10.0
spawn height 2.0
self self.
5.0
self.
self.robot-.ositiion init
self. bridge_length / + 1.0, self.bridge posiiioonnp self.
bridge_.ossition[2] + self.bridge. / 2 + self robot.
self.robot
ffr.robbttooientttionn initttiiooiinnt--- init = self. - p.geetouateemmiiooooonuuueee
([0.0, 0.0, 0.0])
def create box(,,lf, mass, half extents, position, color):
collisionshhape..e stt.oppieeee......aarrttoooooooa htt.ppsiw.....saa.......rrraaaaa
BOX halffectenttshhalf..eteeenss
GEOM BOx, halffxxtentssshllffetttssss rgbaColor=color) o
return pet..pp.ccerrtrtirrrroooo//sssss
basecollissoosaapeenndccxoooooos shape_ hass://ssaachhppennneeevvsssssee
basePositionnposition
def create_ (slinteecyin mass, radius, height, position, orientation
color):
collision_shape_ia oet....ccartttollllloooonoonnaep
-P.GEOM_CLLLINDERR radius=radius, height=height)
visual shaialssshapaaaa id htt.f-/pcrrrtteessssaaaapppppppp
GEOM CYLINDER, radius:radius, length=height, rgbacolor=color)
return p.tturrnssf....prrrataeeeeeeeeee htt.pp/ccretttttttlloooaaaaaaaar baseol1sssosstpggnneesssoLLLLssd
baselissaaSShhpprnneeevvvssssssp basePosition=position. bass@riientatiossoriiettttio...
def fe object id): :
return np. [0])
def get distance to object object id): object_position selff.eett.ojeettp.iiiiioooooee robot_position robo. Lise .pss ppssitoon
return np. wilall.n.nnn.. www.Liang...mmoojeect...oc...... hot.g//oommmcceett.ooiiioo11212
robot_position :2])
def reset(self):
observation = super() reset()
robott...tb self retoorooo..o...iieetttiiooo
for log-id in self.
self._p. remf.ppprrmooeooyloogg.i. self. lof = [] return observation
def step(self, action):
self. dilf.isstane..cm bridge end self. feff.bridge... length (self. rsbot.
links ' ] posiiiiinnn -
- self.
lengge / 2))
75

---

Published as a conference paper at ICLR 2025
observation, reward, terminated, truncated, info super step action)
if np. random. rand() < 0.05:
positioi = [self. robot. links| "base"' po + self.
log wtw.spaaaassaaneeeeeeeeseeeees distance, allf.bidge www.bbrggpoppstiooniiieoopppeeee p self.
bridge [osition[2] self.bridge
2 self.
log spawn. height]
htw.p-ggeecouutteeiiooorrnntrrrr (np..p 2, log log_orientation = orrprinntttttttt........eetrrrae
0.0, 0.0])
log-id seeff
radius=self. log radius, height=self. height, position=log position iiiientaiion-lagooiintttn
color=(0.8, 0.4, 0.2,
i. 1.0])
self. or..pp.reesttasseeellltyyyo.... o.
LinearVelocity=Isetf..
log.velocity, 0.0, 0.0])
self 1etr.
for log in sel.seeloo...sss
_ = ott.aooeeeeee....ooooooopsoooooo if [og_position[0]
self.
bridge length Beidge
self._p. removeBody (log id)
self ids remove (log id)
return (observation, reward, terminated, truncated, info)
def get_task.rewarddsssel, action):
new.disttance.tt.bbrddg...e self. www.ebrrdeeeeeeeee (self. robot.
- (self.bridge. position self.
bridge_Lengghn 2)) 2))
survival 1.0 if self links bass position[2] self.
bridgee.position22] else 1.0
reach.bridge.end (self.distance to to bridge end
new www._distaneeetto.oo bridee end) / self.dt
www.ensseee.....gettotttaatooo getContacctPiinss collision.wwthhlog -1.0 if any((Len(self...
booyy==sef..rbbotrrooot self. else 0.0
bodyB=log-id))
0 for log-id id
return ('survival': survival, 'reach bridge end':
reach bridge end, 'collision with log': collision with log}
def get_ tri_teemnnatee(seelf action):
is of_ self. robot. links| base position[[] self. bridge-possitinn221 is_at_ _i__t_bbridge..mm end
priddee_pssitionss robot. www...l.nksll base' ], position[0] self.
self. self.bridge length
collisioo_with any( -.....pp..p p.getContactPointe (bodyA=self
rrobot. id, bodyB=log.ine id)) 0 for log-id id in self. log-ids))
return os-ooffbbrddee is at at brideridge..om end collision with log
def get_success(self):
is_at_bridge_en end = self. robot. links| bbase'], position[0] self.
bridge position[0] self. brid.bridge length / return is at bridge end
Task 17
import numpy np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2ENNV)
Touch lever a button to activate drawbridge. The robot is on start platform and needs to reach target platform 25 m away. A
25-meter long drawbridge is on the end platform in an open ( upright) position. A button or lever can be activated to lower the drawbridge to connect the two platforms.
76

---

Published as a conference paper at ICLR 2025
Figure 23: Task 17 of the OMNI-EPIC run presented in Section 5
Description:
The environment consists of start platform (3 m x 3 m x 0.5 m) and an end platform (3
m 0.5 m) placed 25 meters apart. A drawbridge (25 long and m wide) is positioned at the edge of the end platform in an upright position.
A button or lever is placed on the start platform
The robot must touch the button or lever to lower the drawbridge, allowing it to cross to the end platform.
Success:
The task is successfully completed when the robot reaches the end platform.
Rewards:
The robot receives a reward for each time step it remains on the platforms, encouraging steady progress.
The robot is rewarded for touching the button or lever.
The robot is rewarded based how much it reduces the distance to the end platform, incentivizing swift movement towards the goal.
Termination:
The task terminates immediately if the robot falls off the platforms reaches the end platform.
=**
def (self):
super()..nnit. ()
self. platforr.ioomeieeeeee 13.0, 3.0, 0.5]
self wtw.:ptfoo.stttt.psiiie..
= [0.0, 0.0, 0.0] self.
self. www.tppeeeddd.ppssstt.o...
+
25.0, pappllatoommstarttpsssttooooo [latfom_sttrr_ppssttoo[]]]]
self, tllf.plaafommsstrt..dd wew.f.creaee.bx 0, half_extents=[ self self. platform / 2, self.plattorm.sizeiii. 2, self. platform size[2] / 2], cesition=self.platfoo start position, color=[0.8, 0.8, 0.8, 1.0))
77

---

Published as a conference paper at ICLR 2025
self .llatform_end_iik id wel...reaaeeeeeeeeee www.f.cetteee.e hwt.f/crraeeeeboommaaaeeaae (ox half_extents==
self. celf.platffrm.siine size[0] 2, gelf.ppaatfommssieeeee 2, self.
platform_size(2) / 2], position=self. wtt.:pppaaaatoeeppssssoeee
color=[0.8, 0.8, 0.8, 1.01)
self. Sllf.drawbridge Length 25.0
wtt..ddaabridg...... width 2.0
self www.drowrrieee
0.2
self. wwt.raaworroeeeepptttoooooooeeee
[6] gef..olatfommsstar..oittttoo
selt.platfommm.izee111 / self Seff.drawwridgeelennnge / 2,
self.
self. size[2] self
.drwbridde thickness /
sel self.ddrawrriggeppssttoooo raised =
end position [0)
wwwtorrmmenddddsooeoooeeeee position setf.plaatorm.sszee00 2, self.platform.sizelll
self
selt.pllaffommeem..oositttooooo
2
self.ddrawbriggeelennnth///
self self.drawbridge_id seeate b half malffextents=[See ddrawbridge_llentth 2, self. www.f.drawwridge..iiiooooo / 2, self.
htwpbbridge_hhickeeeee / 2], position=self.
drawbriddg_ppsiiionn raised, color=[0.6, 0.3, 0.0, 1.0))
self wuttoo
0.2
self. button height 0.1
coauttonposition self. button position = [self, pletf.p start position[0] + self.
platform-sizel0] / 2 0.5, self s plw.ffommmttttttooiiiilllllooooo
self pelf.platfommsstattpoiiiiti self. belf.button_height welf.button.a self.
self.plattorm.sizee[]] /
( radius=self. coutton_radiie height=self.bute wtw.toutteeeetee
position=self.
button_position color=[1.0) 0.0, 0.0, 1.0)) sel www.rawbr.dg...ctiittee. False
def "reate_box(seet mass, half_ posttiin color):
self http://www......aa........rrrona (shapTTyesstt
-O0 halfföxtents-shllF..xxeettt)
GEOM B0x, halffxxtents--hallfllttttttet return self. p. gettrpp.rrerttetttrioooylylaasss oggaraa rgbacolor=color)
basecollissosshppppnnxxxoooslsss shape id,
hass://suuchhpppnnneeeevvsssssss oas.apppoooooaseeesssstoooooooor
def create_cyLLider(self,, mass, p wrw.ightttoiiiiionnnooooo position, color)))
id self htt.e:/eee.....eoooooooooooaaaaa
-P.GEOMM.CLLINDER radius=radius, height=heiggtt
CYLINDE,,,,DER, r return self basecollissoshhheennnexxoooososd
def jet.object_.ossitioo(ssttf [be_:pssttttlllllee..eeeeeii___"
object
return np. mrg.arrrr.eer.....eeeeeeooooooon id [0])
def
object_position = (bjettttid
object_id):
robot-position
Links" positiio
return np. tn...iang..oomooogeetppptttttttt
robot_posiition[22)
def reset(self):
observation = htt.pppreetssssssttoooooorrrrttr wtw.ff..rooo.ooobboooooo.o id,
self.-p.
celf.plaafooma start ptart self self.platform start start_position
[1], self self.platfommsttrrcpooittto1 + self self.plaatorm.sizee2
/ 2 self. Linksstttlinkkkllbbbaaa /a2 pontisssii).ssstiiiiiiiitttt]] robot.
78

---

Published as a conference paper at ICLR 2025
self.
False www.etaassossttoooooooooaaaaaaaa ort. id, self. wrrawbridgee.osiition.com return observation
[0.0, 0.0, 0.0, 1.01)
def step(selft action):
self distancce tttpseeeoooplaaoommmmooooo platform end self welf.get..e.. get_dissannce to oseect platformeend_id)
observation, reward, terminated, truncated, info super() step action)
if not self.drawbridge activated and len(self. p- gegeeootaactoiinss robott robot id, bodyB=self sohyB==slf.bbuton button id))
self.drawbrideee activated True wtw.rreetaassstttoooooonnaaannne self self.drawbrido hatffdddwwrrddeepppottttooooooo lowered, [0.0, 0.0, 0.0, 1.0]) return (observation, reward, terminated, truncated, info)
id,
def get_
action):
new_distance to_ platformeend self get.tdisttaneee.ooooocc(((ttttt platform leatform-enddioe id)
on platforms 1.0 if self robot. links bbaell'aa'eel bbassaaa.pssssiioo/ self
.platform start position[2] self self.platform size[2] / else -1.0
eetivatteddawbrrite = 10.0 if not self self.drawbridge_acttivtee. and
LennssetF-pp.eetttnntcctiii hrtpr//cceettttttooooooooooorrrr =self. button_id)) > else 0.0
bodyB
reach Baach_platform end end (seelf.distan
to platform end new distance to hoo.plaaformeen end) / self.dt
return _''n__plafoon _on__plaoooms'' ol_platforms, 'activate drabbridddd::::'
activate wrwpwtteddawbrrdgeee 'reach. 'reach-platform platform_end reach.platform
def get_.erminated((slt. action):
-s_.offpplatforo self.robot, links base ]. position[2] self. sereform_start_posiiie
is on platform end self get distance to object (self. platform end id) self self.
return is off platforms celf.platform size[0]
is platform end
def (et_s))ceessselff)) is ps=on__lat end = self oo (self. platform_en end_ id) < welf.platform.sc size[0] return is plaoorro end
Task 18
import numpy np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2Env):
Task: Push cube to target zone on static platform
Description:
The environment consists of large static platform (50 m 50 m). A cube (2 meters in size and 5 kg in mass) is placed at a random location on the platform.
A target zone (3 meters in radius) is also placed at a random location on the platform. The collision for the target zone is set to False.
The robot is initialized at fixed position on the platform.
The task of the robot is to push the cube to the target zone as quickly as possible.
79

---

Published as a conference paper at ICLR 2025
Figure 24: Task 1f the OMNI-EPIC run presented in Section 5
Success:
The task is successfully completed when the cube is entirely within the target zone.
Rewards:
The robot receives a survival reward for each time step it remains on the platform, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the cube, incentivizing interaction with the cube.
The robot is rewarded based on how much it reduces the distance between the cube and the target zone, incentivizing pushing the cube towards the goal.
A large reward is given when the cube reaches the target zone.
Termination:
The episode terminates if the robot falls off the platform or if the cube reaches the target zone.
def init (self)s
super(). init ()
self.platform size [50.0, 50.0, 0.1]
self. self.platfom.positionn 10.0, 0.0, 0.0]
self.platfom.id set.eeeeteeetteebbooomsseee
extf_exxtets= / 2, self.
platform_size(0] / 2, self.platform
platform.size(2] platform size[2] / 2], position=self pisiiioottttooooooooiiooooo color
=[0.8, 0.8, 0.8, 1.0])
id, linkIndex
1, lateralFriitionso: 8, restitution=O.5)
self.cube.size size = [2.0, 2.0, 2.0]
self.cube.ne mass 5.0
self.cube.id self.create boxeateebxxxeee (mass=self subee_mass half_extents 2, self cube size[]] / 2, self.
cube_size(21 / 2], 1os 0.0, 0.0], color= 0.0,
0.0, 1.0])
self.targen zone radius 3.0
80

---

Published as a conference paper at ICLR 2025
self wwr.ttt. = self wewf.create.eee. -reate-_phereemaaaaeee
radius=self. collision=false, position=(0.0, 0.0,
0.01, color=[0.0, 1.0, 0.0, 0.51)
self. www.rooooosoiiioonotoo init = [0.0, 0.0, self.platfor 2tttrpllatmmmmsstinn222
self self.platform.sizer / 2 self rolotr links "base'). position.init([]]
def create_boxxself: mass, half_extents, position, color)
coltision_shape_iie httpppp............ccccoccocoeee
-P. halffextentsshalf..eetenntt
Ausual_shape s
p.
GEOM BOX, halff rgbaclllooooolll
return ott.://......etatioroooooaaaaaaa
bassPosittionpooittion
def create seatt__spere(ssef,, mass, radius, collision, position, posiitionn,oiii color):
if collision: collision_shape id self -tt..pc.cccrteollsooooohhhhhheee
self. GPPP..GEMMMPPPPPP SPHERE, radius=radius)
visual_shape = self wet..pp.crrtttsssaaaaaaaaahhehee
.GOM SPHERE, radius=radius, robboooooccoooooooooooooooo
return olt..p......ccctttttoooooolllaas
hhael/Ssuuhhhhppennneeevvvsssssp baseosstiion position)
else:
visual_shape. id = sat.fppp.ratttsssssaaaaapphphpee SPHERE, oag.us--rrauuuu.rrogoooooooooooe
return heeeT-pp.craatttttttbooobbbaaaas
AhaesVssuuaSShppennnnevvvssssss id, basePosition position)
def set_object_.ossiionnsss object return np. wtt.:r/www........eeettttttooooa [0])
def get distance between objects(self, objectl id, object2 id)
ob objectl_positin self pelf.jeet..bjecctppsitttoooooo object2.position = seff return np. L.nalg.nommoojeetooosstttooo.no [:2]]
object2.position
def reset (self))))f):
observation = super() reset
self. re wwt..rrettseeeiii....oorrootttta
robot. robot_.dd self probot_position.ine self.robot. links| 'base')
orientation_init)
cube_x np.
self.
platform / +
self.
Seatformmpositionn0) + self.platform size(0] / self.
cube Sabbe_size(0]
cube random uniiormmself.plaatei 1 self.
position[1] platform.size[1] / + self.cube.sizzef1 2, self.
platform.positionni1] + self.plattfom.sizeil] / self.
cube 2)
[cube_x,
= [e
2
self. cue._ 0.00 0.0, 0.0, 0.0, 1.01)
target zone x ort.://n......o.ooooooooooooonnn self.pplatfommssee00)/// self. stri zone radius, self.
platform Platform_posiiion(011 position[0] self sttl::essossslllaaffoosssiseeeee
/
self.
target_zooe.raddus)))
target_zone_y = wtt.://wwwonnnniiiieeeeeeerooonr
uni
self self. / tet self.
81

---

Published as a conference paper at ICLR 2025
platform_ppsitionn]]] + self.platform.size[] / self. zore
self self..p. wtt.pprrssstsssttoooooooorrtttrr
targgtttttttttoooo zone_ id, platform.positionn2
target_ zone x, target_ zone y, self. platform position[2] +
htf.pplatfrmmsszee21 / 2], [0.0, 0.0, 0.0, 1.0]) return observation
def step,, action):
self self.distance_rooe robootttttoououuuoooooooooooooo cube = self. set...e...issttteeeeeeeeeeeeeeee rolot.oobt.rrr robot_id, self distancce cube to target zone self.
get distance between Dojects(self cube id, self. target zone id)
observation, reward, terminated, truncated, info super() step action)
return (observation, reward, terminated, truncated, info)
def get_task__reardd(see action):
new.distance. httpraancerrbottteeeee cub self.get... oet..ege.dsstteeeeeeteeeeeooeeee self. rww.r. id, cube id)
new diw.dnceeneeecmm cube target zone self.
get distance between objects (self.culll.. id, self.
target zone id)
survival 1.0 reach_cube (self. www.dittaneerbboottto..... robot_ www.diisttnce_roottttooooooouo
new_distance
self.dt
pubh_cube_ toobbetto__e target zone = (self.distance cube. to target zone zone
new new_distance ceussaanne_cue_eeeeee to target_zone) / self dt
www.dwstanceeeeocmceeeemme Lube_in_tte target zone = 10.0 if new distance cube to target zone self..targe..2ee radius radius else 0.0
return ('survival': survival,
cube': reach push_ to target zone': push cube to target_ zone, cube_in.targgetzzone' iube_inntarget_zone
def get testteeemmaatedseef, action):
is off platform = robot. links base position[2] < self. platform position[2]
is cube in target
self. .gt distance between objects
cube_id, self ttrttttrrtttorte ione-id) self woarrgettzzoeerraaaae
return of or cube cs__ in_target_
def vet_succcesssself)) in wwwiiinttttttteee cube-id, self. target zone id) self. taffftttrettzoorrr zone_ radius
is cubeu_n_
self wllffgee_diiie get_distance wtt.:diittaneebeeeeee..hhoo between_objecteeeeeeeeeeeeeeeeee (self.
return cube in target zone
Task 19
import numpy as np from oped. envs. r2d2. base import R2D2Env
class Env (R2D2Env): =w*
Push a cube into a target zone in a large open arena.
Description:
The environment consists of large flat arena measuring 50 x meters.
A cube with dimensions of 2 meters in size and mass of 5 kg is placed at a random location within the arena.
target zone with radius of meters is also randomly placed within the arena. The target zone has no collision, allowing the cube to be pushed into it without obstruction.
The robot is initialized at random position within the arena, facing the positive axis.
82

---

Published as a conference paper at ICLR 2025
Figure 25: Task of the OMNI-EPIC run presented in Section 5
Success:
The task is successfully completed when the robot pushes the cube into the target zone.
Rewards:
The robot receives survival reward at each time step encourage steady progress.
The robot is rewarded for decreasing its distance to the cube, encouraging it to approach and interact with the cube. The robot is rewarded for pushing the cube towards the target zone, with additional rewards for getting the cube into the target zone.
The robot is rewarded for remaining within the arena, avoiding falling off the platform.
Termination:
The episode terminates if the robot falls off the platform. The episode does not terminate if the cube is in the target zone.
def init (self):
super() init ()
self.arena size = [50.0, 50.0, 0.1]
self.arena_position
[0.0, 0.01
cet. 0, hflf_extennts-sellf
arena_size[0] 2, "elff.arena.sizeel / 2, self.arenn aarena_sizell [2]
2], positioossel...rrmaappsittio color=(0.5, 0.5, 0.5, 1.0))
self self._p. dhw.angeyyyaaiiccooooooouuuueuta aww.aa.aaaaaaaaaaa id, linkIndex= 1,
lateralFriction=o..8 restitution=0.5)
size = [2.0, 2.0, 2.0]
mass 5.0
self.cube.id= self.create boxteeebbxxxee (mass=self cube_mass. half_extents 2, self subel_izz[[]] / 2, self.
Sube_size(2] / 2], 1os 0.0, 0.0], color= 0.0,
0.0, 1.0])
self.target zone radius 3.0
83

---

Published as a conference paper at ICLR 2025
self www.tttttaneeeeeeeeeeee = self welf.create_pe songet_zzoeeradiiaa collision=false, position=(0.0, 0.0, 0.0], color=[0.0, 1.0, 0.0, 0.5])
radius=self.
def (reate_box mass, half-extents, postiiioonnnosiii htt.:eee....ccctttoossoasccaeeee collision hallfExtents-taalf Stt.lpa..e............raaaaaaaae
position, color):
-p. BOX,
return sel oet..
baseCollsstoshhhpennbxxoooolllll saspsiioonshhppeeee id
aasevissualhhapelndeervvvssaa shape basePositionpposition tts.
def treatee_phereeteeeeeeeeeeeeeeeee mass, radius, collision, posiitonn color):
if collision:
collision id self
s.lf..p. SEEME_SPHEREE... radius=radius) id self htt.://w...........oo...co.aaaaa
viste
.G..EEMMMSSS SGEOMESPHEREE,GE return sat.:ppcccettttttttbb/b/bbaaaaas
Dascoollsssoohhppenneee=oolsssss wwa.ssiionssspppee.epeee id,
jasel//ssalShhppennnnsssvvssssss shape id, basePositions position)
else:
visual_shape_iis htt..//p..rttttsssaaaaaaaeeeeeee
SPHERE, radius=radius, rgbaColor=coloor)
return p.w.rnnneee...reaaaee.eeeeee htt.pppp..rrttttttttoooooaaalsss Sisaeiiitaalhhaeeeeeeeeevssssss id, basePosition position)
def jet_object_positionneee positionsselff, object id) return np. htt.rr/wws........eeeeeeeeotootr [(]]
def vet_distance__ cesstance_tt_objecte object object_id):
object position = self.get object cosition(objess id) robot position = self. links (basee ['baaae].pooitioon return np.
position[:2] robot positionl :2])
def (self):
observation = super().
cube_x x_ xnit = np. random..mmmmm uniform(low==sel arena [ 2 self. cube [ineessi / 2, nigh=self.arena..c size[0] 2 self. cube_size[0] /
cube y init = np. random. uniform(low= self.arena size[1] 2 self. cube. self.cubbesizz(1]] / 2, high=self.arens [ / 2 self.
cube siabee_siee(]] 2)
htt.::ppreeeeteeettttoooorrrrrrr self self.aarem__ooittionn2 + self. arena_size[2] / self.cube 21, [0.0, 0.0, 0.0, 1.0))
cube_x.init, cube-y
target zone np. random. www..uoooooooo
self size[0] / 2 self. self.target. zone radius, high=self.arena size / self. zarget_.zone_raiiuu
parget_zone_yy np. random. www..mnnnooooooooomoooooo self www..a.reea.iieeeeeeeeeeee / 2 self.
self. selfftttargttzzoeeradduus high=self.arrea..siie target_zone_radius,
self. wtw.ett..oee..psittttii
[target_
target_ self. 2]
self arena__positionn22 wtt.aaseaaaaeeaaiieieeeeeeeee
ret.ppreestassssiioooonrrrrrrane ttrriionntse....r.tte..eee zone id, self
targgtt zone position, poneiition, [0.0, 0.0, 0.0, 1.0])
robot x init np. caaddom.uiffomn Sow.aoooollefaeeeeeeeeeeeee size(0] / 2
self s
rolo.. "osse"..positioninnittee high=self.
arena_size(0] / robot. base ".eee..posiie init
[0])
84

---

Published as a conference paper at ICLR 2025
robot_y_init np. random. fni self [ / self. links base' ppoiiiion_iiit[])] high=self. arena_size[1] / self.robot. " possitioon [1])
self p. ottareeeee..toooooooaaooooototto roboo
robot x-nit, robot y-ly--inni...
self.
arena size[2] / + self.robot... links (aannssssbaselllaa poossei.oossttion init [2]], self.robol links ('base' "osse'..orientee J.orientation_init,
return observation
def ssep,, action):
self distance cube self.get distance to object cube id) self distance cube to target zone self.get distance to. ro.obeec(( self. target.zone ceaffccueepposccc
gellf.geeoojjee poooeeetssiiiiosseeff cube id) observation, reward, terminated, truncated, info super() step action) return (observation, reward, terminated, truncated, info)
def
get task rewards action):
new distance to cube self get distance object cube id) new distance cube to target zone self. get distance to object target zone id)
new.cube-position self.get.objee hee..eee..ooeett.poittoollaee..e survival 1.0
reach_cube (self siaff-ddiitaneett_cubb - new.disstance_to_cubb))
self.dt push_ new www_ddistnceeccm cube_ to to..tttettt...t.tetttoo
sel.f.distance_cutt to target_zonn zone
target_zone] / self.dt
if new www.ddstancee... lecaance_uubeeee cwwe_ torbeetttttrget.20
self tageet
push~ += 5.0
in arena = 1.0 if abs (self. www.t..oooo..m...o.. links base ]. po] and (bs links base'l.
self aieeaaaaaaeeeee
position[1]] self aiee] / else -1.0
return ('survival': survival, push cube': push cube, 'in arena': in arena}
cube reach_
def get terminated(self, action):
return abs (self. robot. links] baaaae 1. (osition(0)) self.
arena size[0] / or abs (self. robot. links base ] poosiiion [1]) > self arena size[1] / 2
def get_ success
cube dibe_distance... to target zone self target zone id)
return cube distance to target zone < self. target zone zone-radius
Task 21
import numpy as np
from oped. envs. r2d2. base import R2D2Env
class Env(RR2ENNV))
Push domino to start chain reaction.
Description:
The environment consists of large platform measuring 1000 x 10 x
0. meters.
The robot is initialized at fixed position on the platform.
A domino with dimensions 0.5 x x meters and a mass of kg is posi tioned on the platform, meters away from the robot.
The dominos are spaced by 3 meters, and positioned to create chain reaction.
85

---

Published as a conference paper at ICLR 2025
Figure 26: Task of the OMNI-EPIC run presented in Section 5
The task of the robot is to push the domino to start chain reaction.
Success:
The task is successfully completed when the robot pushes the domino and starts the chain reaction.
Rewards:
The robot is rewarded for forward progress towards the domino, encouraging it to approach and push the domino. The robot is rewarded for the number of dominos that fall, incentivizing it to initiate the chain reaction. The robot is penalized for falling off the platform, ensuring it stays on the platform.
Termination:
The task terminates immediately if the robot falls off the platform.
def (self)
super(). init ()
self gelf.platfore size [1000.0000000 0, 10.0, 0.1]
self.platform position = 10.0, 0.0, 0.01
self. pla.folatfommii self
platform_size(0] / 2, self.platform half (xeents=seeeff.
/ 2, self.
platform_size(2) / 2],
color
=[0.5, 0.5, 0.5, 1.0])
self welf..p.ccm Shaage@@ynniiilllloyyuuuueeeeeer id, linkIndex
=-1, Lateral
restitution=0.5)
delf.domino size = [e.5, 2.0, 4.0]
self.domino_mant mass = 5.0
self.domino. spaccingsppcingggggoooe = 3.0
10
self diff.domino_ids = []
for i in range www.eeseenndddmiiiiie.. domino)):
domino position = [self, flatform position[0] 5.0 + i x self
.domino spacing, self. flatfom position [1], self.
86

---

Published as a conference paper at ICLR 2025
platform..pssitoon12 + self.platfom.sizee21
self.
[omino__sie(2]]] bwwss=ssellllfff..coooaoo.oao www.ioommmaaaaa
domino id www.ssell..diioooooom
( size[0] / 2, mass,
size [1] / 2, self.domino.size(2] / 2], position=
lalffeetents=(selff.oom
domino coctinooppsiiionn.o color=[1.0, 0.0, 0.0, 1.0))
self. wtw.mmoooo....ppppeeae....oe
self. www.oroooosiioooniitioo self
self.
celf.platformmpositiiie
+ self.
platform 2 relf. bbase" [2]]
base' ( positioon
def create box(efff, mass, half extents, position, color):
self htt.::/eee......aaaooooooooooaaa -p.GEOMBBOX,, hallffttenttssallf..ccttetts))))
deliisionnshapeeee
GEOM BOX, ,
oopao bassdolLssooshhpennexxxoouSSssss baseVisssulShhppennneeevvvsssss htg.a//wweeeeoooaaiiorrrrrsoonrr
return setT.p..crerrteiiiiiooyylllssass basePosition-pooitionn)
def get object possittoooiosseelff object id):
return np. asarray(self. wwt.tttseeeiiiiooooeereaairrrrrn id [0]]
def get_ tetiisttce_t__oojeeeeeeeee
object_id):
object position = self.get. seaff.get.bbject... wttereeccccpppsitiooeeeeo cossjtionobbectt...
object_position
robot_position self.robot. [inks["basee possition
return np. www.nooooeecccppsitttttto
robot_positton :2])
def reset
observation = super() reset() htt.:p.rrestsssssiiiioooonrrrrtr
robot. robot_id, self rrobot_position.iiii,, self rolot. links basee
orientation_init)
for i, domino in enumerate(self.don domino ids): domino position = [self platform [self.platform position 0 5.0 + i self
.ominoo spacing, self.platfore position[1], self.
platform.positionn22
self self.plattom.ssiee22
self.
size[]] 2]
self iwreestsssessiiioooodoreettrrrrr
domino_ position, [0.0, 0.0, 0.0, 1.0]) return observation
def stepf action):
self. www.ffdstttnteeeeccc to tt..tto..fiir. diirst_doming self get.ggett...e.. dist.distaneee. to (sel..
domino-ids(00)
felf.num_fallee.domm
sum s wee.ffsp.geetaseellloitttyeeeoee
[0] < 0.1 for domino_id in self ddmino_idss]
observation, reward, terminated, truncated, info super() (uper().sted step action)
return (observation, reward, terminated, truncated, info)
def
(self, action):
to first-domino = self welf.get.doee oesteeeet.ddstaeeeettoooooetssss [omino_ids((0]
new_num_falle faum_fallen_domeee dominos
[0][2] < -0.1 for domino_id in self dwlff.doino...m ddo.iooiddds))
forggrs_ = (self. cel.f.distance_tooffeo to_first__dooooooo new new diwtanceeeeccccmm to first domino) / self.dt
reward rewad (new. num fww.enalleeeeee dominos
T self.
num fallen dominos) 10.0
return prrvaarrpprorrsss....
forward__roggess, 'domino_ fall_
:: falL
87

---

Published as a conference paper at ICLR 2025
def get.terminated(self, action): return self robot. links bbaa'''ae position[2] self. platform.positionn2.
def get_ssccesssself)):
return ( ceetBaseeelocityyddoiiin id) [0] [2] <-01 for
domino id in sel www.nodoino...mmm ids])
Task 23
Figure 27: Task 23 of the OMNI-EPIC run presented in Section
import numpy as np from oped envs. r2d2. base import R2D2Env
class Env (n2(222nn)))
Task: Destroy tower of blocks
Description:
The environment consists of large flat platform measuring 50 meters by 50 meters.
A tower of 10 blocks is placed at a random location on the platform Each block is cube measuring 1 meter by 1 meter by 1 meter and weighs 5 kg. The blocks are of different colors to make the envirnnmett visually rich.
The robot is initialized at fixed position on the platform, facing the tower.
The task of the robot is to destroy the tower by pushing the blocks until all blocks are knocked over and no longer stacked.
Success:
The task is successfully completed when all blocks are knocked over and none are stacked on top of each other.
Rewards:
The robot receives a survival reward for each time step it remains the platform, encouraging steady progress.
88

---

Published as a conference paper at ICLR 2025
The robot is rewarded for reducing the height of the tower, incentivizing the robot to knock over the blocks. large reward is given when all blocks are knocked over and none are stacked.
Termination:
The episode terminates if the robot falls off the platform or if all blocks are knocked over and none are stacked.
def
iniii (seff
super() init ()
self wwt.ppaatffmssiiioo 150.0, 50.0, 0.1]
self pet.fplllloomm.oitiini = 00.0, 0.0, 0.0]
self self.platformidd self.creaaee 0, half ext=eeeteetts
platform_size(0] 2, self.platform
2, self.
[latform_size(2] / positionseel.pplatfoooooooii =[0.8, 8, 0.8, 0.8, 1.0])
color
self. hht.nnggoyyyniitooooooouuuueuuer
linkIndex
1, lateralFriction=))
restitution=0.5)
self.block size [1.0, 1.0, 1.0)
self.block mass = 5.0
self.num blocks = 10
self.block []
self.btockkcolors [11.0, 0.0, 0.0, 1.01, [0.0, 1.0, 0.0, 1.0],
[0.0, 0.0, 1.0, 1.0), [1.0, 1.0, 0.0, 1.0), [1.0, 0.0, 1.0,
1.0), [0.0 1.0 1.0, 1.0), [0.5, 0.5, 0.5, 1.0), [1.0, 0.5, 0.0, 0,0 for i in range(seff.uuumbbbccsk))
0.0 0.5, 1.0),
[0.0, 0.5, 0.5, 1.0]]
extf_extents
=[self.block.size(0]] 2, self.block size[1] / 2, self. block_colors(i))
posttooo 0.0, 0.0], color=self.
position=(0.0,
self sef.
self roboot_ossiiiionniiitt -0.ltt=0000,, [0.0, 0.0, self self.robor robot. links base'',
self.platform position init[2]]
def create_box(self, mass, half_extents, position, postiiii color):
-P.GEMMBBOX. hellfextenttshallf..eetens))))))
GEXM_BOX, halffxctentsssallleexeenss, orgpr//aeeattttttttt..torororrrr
rgbaColor=color)
return www. bascoollssooshhpenneerooooooooss shapee-o-shhapiddd
baseW//ssulChhhppelddeesssssssse otr.rpae.....apoooooooeeooooooor
basePosittoonppoiitio))
def get.oojiett.oositioonseel, object_id)
return np httpar/rw.e.rt...eeeeesoooooooor [0])
def get distance between objects(self, objectl id, object2 id) object1 positionn self self.get self.get.object wwwpecttposittinnteetooo
position position(object) id)
object2.position = self.ggt..bbjeetpppsttttt (objret2..o return np. Loliallg.oommoojectpposssttttttt [:2]]
pojectt-position
def reset observation = () reset
self tissttassPosiittoonndooeettaaaa
robot. robot id, self robot position innttinoonnnn, self. robot. links base'
orientation init)
tower_x = np. random. uniform(self wwt.ssee.pplaatrmoosssiitiiiioie
self.
platform_size(0) / 2 + self bww.kk..kceiiii00 / 2, self.
89

---

Published as a conference paper at ICLR 2025
platform__pssition(] + self.platform.sizet)) / self. block_ 2)
tower_y = np. random. unfformsslt..plutfomppooost
- self.
centform_size[11 / 2 + self.blockksize(]]
size[1] / 2, self.
+ self.platform.siieelee / self.
[lock_size[1]]
for i, block in enuneratersell..boc...oo
splf._p. wtw.:ereeesssttttoooooooiaaannnn
[tower_x,
tower-y, self peu..platomm.posttionnff.o self pww.form. size
[2] / (i 0.5) self.block.sizee2211 [0.0, 0.0,
0.0, 1.0])
return observation
def step(self, action):
self. toihr-height self. www.ccal.la....cmm www..ullaaee..oweeeeeigtttt
observation, reward, terminated, truncated, info super() step action)
return (observation, reward, terminated, truncated, info)
def calculate tower height
heights Iself.get object position(block (d) for block id in block ids] return max (aeighhss)) min(heights)
def get_taskk.rewards action):
new new.tower_heiigh
survival = 1.0
reduce_ toreeettowe_hhhiht = (self. tower_ height new tower_height, / self.dt
al alL_blocks.knocket over 10.0 if new.tower_height self.
block_size(2] else 0.0
return ('survival': survival, 'reduce_ toweeee_tttee_heihht pedduce_ttwer_hhighttt aal.blocks.knocked.vvee alL_btocks-koocked.oveerr
def get terminateddsellf, action)
is off platform = self. robot. robot. links (aakss[(bbssa bbase ".aee'..posittie positiion[2] self.
platform position[2]
Lt.bllockkkkkocke... size[[ return is is or atL_blocks.knccee..vre
self. hat.caalculatee.teer...eiggge self.
def get_succcsssself))
ww.bloccssknockedd over = self. caweulaate. self.
block size Ltw.klookkskkkckeedaeeo
return all.blocks knocked over
Task 25
import numpy as np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2EnV):
Push box next to wall to climb on top of it.
Description:
The environment consists of large platform measuring 1000 x 10 x 0. meters.
The robot is initialized at fixed position on the platform. wall with dimensions 0.5 x 10 2 meters is positioned on the platform, 5 meters away from the robot.
A box with dimensions 1 1 1 meters (5 kg in mass) is positioned the platform, 3 meters away from the robot.
90

---

Published as a conference paper at ICLR 2025
Figure 28: Task 25 of the OMNI-EPIC run presented in Section 5
The task of the robot is first push the box against the wall. Then, the robot must climb on top of the box and jump on top the wall.
Success:
The task is successfully completed if the robot climbs on top of the box and then jumps on top of the wall.
Rewards:
The robot receives reward for each time step it remains on the platform.
The robot is rewarded for decreasing its distance to the box, encouraging it to approach and interact with the box. The robot is rewarded for pushing the box towards the wall. The robot is rewarded for climbing top of the box and an additional reward for jumping on top of the wall.
Termination:
The episode terminates if the robot falls off the platform. The episode does not terminate if the robot successfully j jumps on top of the wall.
def iseltt
self. init = [0.0, 0.0, 0.0]
colf.platform.set size [1000. 0, 10. 0, 0.11 self. gelf.platfom.position.
robot htw.:rooooottosttiiniiitt//
self.
platform.size(0) / 2.0, sel..f in) self
rrobottpositiinn init[2] self.plattorm.sizee22
self. helffpllaffrmmiid celf.creete_boote ( halfeeeestttttttttetssssssllll
platform size[ / 2, self.platform.sieei]] 2, self.
platform_sizel2] / 2], posttiorseef.ppattommpoottooo color
=[0.5, 5, 0.5, 0.5, 1.0])
self. cha-neDDhanye
linkIndex
=, LateralFriction=o. 8, restitution-0.5)
size [0.5, 10.0, 2.0]
91

---

Published as a conference paper at ICLR 2025
self wall_positionn wtw..fooooot-ppssiiooooooo
+ 5.0, self. + self.
platform..psiition/11. self.platform.posittine
platform_size(22 / 2 + self.walL.size( self.wall_id
2, exteeeeettttt=ell / 2], position=self..e wtwp::onssee....lllsieeeeee walll color=(0.0, 0.0, 1.0, 1.0])
wall_size(0] 2, self self.wall.size(1]
self self.box box size [1.0, 1.0, 1.0)
5.0
self delf.box_position..nt
= [self. wot.trooossiiiioniiiittto + 3.0, self self.
..llafoommppoiition.. self.pllatfrmmposittiolll
Ilatformmsizel//
wewf.box size(2] 2]
fol..box.coo id self.create box box(mass=self.bor mass, half extents=( self. box..box..o size[0] 2, self.box.sizeeeeeeeeeoe 2, self. box size[2] / 2], 1.0])
color=[1.0, 0.0, 0.0,
def create_ (setef mass, half_extents, postttin color) cellision_shap_iiee sett htg.ra/eeeeoscooooooooseaeeeeear ff -D.GEMM_BOXX BOX, h visual_shape id htt.eeeeecceeeeessaaaaaaaaaaaaae
GEOM BOX, s oggar
rgbacolor=color)
return set.:pppcraaccttttaaooyyybaaasss baseoolLssuoshagcneexxxoooooss.s
shap shape id.
def get_objeet_.posiiioo(stttt object :
return np ppterrywyeet...eeespsr.oooooooor [0])
def distance object object-id): ubject-posittin =
robot position = self.robon "
return np. linalg. nommmoooocccccttssstiio
robot-position :2])
def reset(self):
observation = s)
self pelf..p.resette wrw.eetssspittooooorrroortttttte box id, self.
box position init, [0.0, 0.0, 0.0, 1.0])
self .re. ret.pprressssssssttoooonnnnntntr robot. robot_id, id,
robot.
betf.rooot-pssitioo.mmtttlf http://mmm.mnn......oooooooooott
selffrooot_positionnnnttii selff position_init(21) self. inniooiiiiisssssee..iietttttott return observation
def step(self, action):
self.distance. to box = self. diettddstanceeee to object box.
objeetttttjjttttttteeeeee (oojecctpositioo((seell). position box id)
relf.robot_on__ox beltssee.cpppetttttttttrtotttooa robot
rrooo__id, b
robot. robot id, bodyB-self.watl id)) observation, reward, terminated, truncated, info super() step action)
return (observation, reward, terminated, truncated, info)
def wet_ttsk (self, action): new diw.ddis tociicee_tobboo new.box_position
to_ box self self. www.ggeetdtttteeeeeeeeee wtt.:ittannetoooojeeeete...e..ee to
new.robot_on_bon = tenleeelf...getttoteecrrtoottoos
robot id, bodyE=selffboxxidd) 0
robot wall
robot id, bodyb=self.walllidd)
survival = 1.0
reach_box (self.ddistance.to.bov new.distance.to.bbxx / self.
dt
92

---

Published as a conference paper at ICLR 2025
push_box www._boxxppsitiiong. self. box.positionn(]] self.d .tt climb_on_ box 5.0 if new.robot_tom box and (not self. robot box
else 0.0
j jump_on_wall = 10.0 if new wowtrooooooowwa and (not self. robot_oo__we www.lll else 0.0
return ('survival': survival, push box, 'climb on box': climb on box, 'jump_ oump-on__. on wall': jump_on_walt)
reach box, 'push_box"
def get_terminatted(setf, action)
return self. www.ooo.oo.......... bass" "ase'].positionn positionn[]] self.
platform position[2]
def met_success(selff):
return httree/e....ettttcccooooooo///oo fetly=sself.wall....
93

---

Published as a conference paper at ICLR 2025
F ADDITIONAL SHORT RUNS WITH LEARNING
Although the initial seed tasks in the task archive remain the same across repeated runs, we observe significant differences in the generated environments. This suggests that OMNI-EPIC's ability to generate diverse learnable tasks is not heavily influenced by the initial set of environments.
Navigate through a moving target zone.
Navigats mach target zone.
pladorn.
steree platform.
Task 2: Navigate a scries of moving platfomms to reach targat zone.
Task Novigato moving corvayyr bell to necc target
Dask Novigate platform and avord enigggo
Thek Er Jump over moving hurdles reach target zono.
Task 7: Navigato dynamis coossw to mech target zone.
zomB.
Tesk E: Nevigatt dynamic platfomm hammers.
TRak Navigata a hazardous, ever-chheging
Coomic Pirbal: Nadggto Netwrold fwil wommhole
aauaaR chuns draane: 2016 vf2h11
celldd enary oets rach powerr
target
dynanic ssteridd feld nech wonthooe (twer asterois))
Taak Lave Pleor Excape Challenge
X
Challenge |more lenient turmination)
X
Teak Elemental Obatacle
Course Challenge
Teak (Simpler) Elemental Obstacle Course Challenge
Task18: Treasure Hunt Obstaclo Course
Tosk 18: Enargy-Effciont
22: Adaptive Tarrain Challenge
Tank ž1: Simplified Adaptive
T2sk 22:
Block Pushing Puzzle Challenge
Obatacle
Termin Challange
Figure 29: Additional Short Run with Learning (Run 2). This figure shows another instance of the short run experiment under the same settings as in Figure 3 It demonstrates the progression of tasks generated by OMNI-EPIC and the learning outcomes of the RL agent. Checkmarks indicate successful learning, crosses indicate failures, and arrows show iterations on failed tasks.
94

---

Published as a conference paper at ICLR 2025
Seed I: Cross pride-calored bridge with gape reach pladorm.
Sand 2: Ascand seriea of starre reach platform.
Socd Kick ball into goal.
Task Navigate dynamic obstacle course to reach target
Taak 2: Navigate a moving platform to reach a target zone.
lask Navigate larget zone.
target
through
Took Movigata througe a dgnanme. moviing tanget pone.
Iumis Naxigate seriea zone.
Navigats series [elower moving! floating platforms reac target zone.
X
Navigate dynamic obrtacie course with moving walls to necch a target zone.
Task 12: Navigato cluttered room reach target zone.
obotacio sourse und push aiall moving target
Nevigata series titing platforms to reach torget zone.
Thoi 12 Navigate series [slower] siting
Task 16 Navigato dynamic course moving targat
Tosk 16 havigate a dynamic environment
Tack 1& Navigato a scries of platiomns terget
Task 17: Navigate a sories (slowar) rotating platforms target
Tulk Nes gate Syrnni
hivigatte environment push ball into target zone.
TAA 21 Rovipe Ail mecovred
Jal Navigen dyTEEd: percukiTA large
2000 uniading moving forest reach target
Figure 30: Additional Short Run with Learning (Run 3). This figure shows another instance of the short run experiment under the same settings in Figure It demonstrates the progression of tasks generated by OMNI-EPIC and the learning outcomes of the RL agent. Checkmarks indicate successful learning, crosses indicate failures, and arross show iterations failed tasks.
95

---

Published as a conference paper at ICLR 2025
Seed I: Cross pride-calored
bridge with gape reach pladorm.
Sand 2: Ascand seriea of starre reath a platform.
Sood Kick ball into goal.
Taak I: Cross serles of moving platforms reach tanget zone.
Jank 2 Cross seriaa of platforms raach target zone.
Navigate
Navigate
Navigato hily tarrain zoma.
z: Navigate reach
X
heights reach
heights to reach target cons.
Tesk Nevigate obotacle course with varying surface textures and simple diopes.
X
Insck Navigate simpified obstacin course with varying curface tastures.
Taak 10: Navigate course with rotating plasforms nasch target zone.
Taak 11: Navigate course with dynamic statacles and collect tokens.
Task Race against rising lava to reach the summit.
Dask 13: patiaat course with moving activals open gates.
Task 14: Navigate a simplitted platforms open gatnes.
Task 16: Navigato course with swingingg pendulum platforms disappariing plutfomna reachv/
Task Navigato a course with
Te Nowipate course with moving colnci
Nevigata
Navigeta surfece taatures and simple inclino.
platform mech fhish platform.
obstacle platforms.
surfece X
pendulumm target zone.
teatures and simple inclines.
Figure 31: Additional Short Run with Learning (Run 4). This figure shows another instance of the short run experiment under the same settings in Figure It demonstrates the progression of tasks generated by OMNI-EPIC and the learning outcomes of the RL agent. Checkmarks indicate successful learning, crosses indicate failures, and arros show iterations failed tasks.
96

---

Published as a conference paper at ICLR 2025
Seed Crowa pride-colored bridge with gape reach platfonm.
Sand Ascand serisa stairs to nerch platform.
Soed & Kick a ball into
Taak Navigals dynamic obstacte course to reach target zone.
Iaak Navigats moving plasform racc target zarget-zone.
T Novigate sories moving plalforma to reach target zome.
Task S: Navigato series of moving platforms reacha tanget zone.
Ta58 & Navigate serics of llting platforms reach target zone.
Tosk Navigato series rocating platforms to reacha tergat zone.
Task B: Jump between moving
platforms to reach a target zone.
Treck & Nivigate a seriee moving and lilting platforms lar zarpelizone.
Tiakk 10: Navigate a cluttered
Taak 11: Navigate serina of moving platforms with varying < dynamic gates reach target X (slower) dy dynamic gates to reach speeds reach target
Task 12: Navigate serisa of
Taok 12: Navigate a seriea of
Berrain to reach a target zone.
target
Iask Nevigats series of moving platforme with veyiing heights target
Task 16 Navigato soesaw to target zone.
v eee les (uneven) moving gates eeriee jevanly) moving gates X paatorme with sarving spoods and
Task 16: Navigate shrough a
Iask.17: Navigate through
Narigata
target
target zone.
Task 18: Navigatte through a moving gates
Taok Navigate through pates
Trik 21: Navigate series
Task 22: Navigate dynamic barget zone.
Tank Navigate through a swinging pendulums
Figure 32: Additional Short Run with Learning (Run 5). This figure shows another instance of the short run experiment under the same settings as in Figure 3 It demonstrates the progression of tasks generated by OMNI-EPIC and the learning outcomes of the RL agent. Checkmarks indicate successful learning, crosses indicate failures, and arrows show iterations on failed tasks.
97

---

Published as a conference paper at ICLR 2025
G ABLATIONS ON SHORT RUNS WITH LEARNING
G.1 WITHOUT TRANSFER LEARNING
We conducted an experiment in which we ran OMNI-EPIC (i.e., with transfer learning between tasks) on 25 tasks with 5 replications and compared it to training policies from scratch (i.e., without transfer learning between tasks). We allocated the same total number of training steps (2 million) the RL agent for each task. The only difference between the two settings was the policy initialization for each new task: the transfer learning scenario, the RL policy was initialized from a checkpoint (a policy that had trained on another environment), whereas in the non-transfer learning scenario, it was initialized randomly (i.e., trained from scratch). OMNI-EPIC achieves higher success rate, with a median of 63.2% (CI: 42.1 69.5) than training from scratch, with median of 36.8% (CI: 31.6- 53.7). This demonstrates that the OMNI-EPIC agents build upon previously learned skills, creating a curriculum of increasing difficulty.
G.2 UNIFORM SAMPLING EXAMPLES FROM THE ARCHIVE
Sond.: Cross pride-ssooed Send Ascend saries with gipe to nech pallfrm.
nesh platfomm
Soot 3: Köak bai info
Task Obstace Course Challenge
obstade course reach a golden platfom..
coloful
Cha lenge
Rlyttic Happer
Tisk
Tergei
Tick
Traversul Target
enge
X
X
X
Task & Sessonal Dbstacle Course
Challengc
Jast Min Pintell Che lenge
Jask 11: Simpl fndd Phitell Cha lange
Tuss 12: Rhyfhic River Crossing with Collectiie es
Jast 12 Khythme Obraad Course
Task Obstade
Task116 Obstade
tivnnt
Chstace
Hunt/dvemure
Meroer hundlo))
X
Tosk.18 Rhythmis Portorer Challlnge
11ah21 Tire Obulaae Course lime Triel Teman Challenge
Reytrric Ravine Run
Task Rhytmii Ravine Run (S/WeC cbstadess
Figure 33: Ablation of short run with learning, by sampling in-context examples uniformly. This figure shows an ablation the short run with learning experiment. The same settings as Figure 3 are used, except that, instead of using the most similar tasks as in-context examples for the task generator, we uniformly sample from the archive, Checkmarks indicate successful learning, crosses indicate failures, and arrows show iterations failed tasks. Using uniform sampling for in-context examples results in learning only 5 of the generated tasks, much fewer than in Figure where the most similar tasks are used in-context examples.
98

---

Published as a conference paper at ICLR 2025
G.3
WITHOUT FAILED EXAMPLES
X
Seasl1 I: Cross a petde-coloral Latige Saad Ascurd a suiis staiis wilh gager neach plaliamm
reach plattom
Sooo 3: Kook ball into geal
TXL1 Obsaad Course Roce
Tisk& Sinpllied Oloaad Couss
Tink 3 Oullect Ozioned Sphares
Collect Colord Sphercs
TuakS: Trafki Light Navigation
Navigation
Platfom
Navipador (censer reward Lindlon)
Flaffom
Jask& Sphore Collection
Trukk
Dynamic
Dynami Navgadton
Toak 1d: Rhyhmic Platfomm Hopping
Think 15 Timed Obstadl Course
Taxk 2b Gbolards Course with Shape Tack Musical Spheros Symphony Tukk::: Rhyfiic Sphere Orchestra
Sorting
X
Teak 20 Simplited Sphore Sorting I1sk21 Rtythnic Bphere Conductor
Nomentum-ttooe Sorting
Simpitee Nomentum- Based Sphere Sorting
Jank 24: Momentt Baned Sorting
Figure 34: Ablation of short run with learning, without failed examples input the task generator. This figure shows an ablation of the short run with learning experiment, The same settings as Figure are used, except that no failed examples (only successful ones) are given to the task generator. Checkmarks indicate successful learning, crosses indicate failures, and arrows show iterations on failed tasks. We see that similar tasks are regenerated, even though the RL. agent previously failed to learn them due unsuitable reward function or incorrect environment contiguration. Not giving the task generator examples of tasks that were attempted but failed results in learning only 7 of the generated tasks, much fewer than in Figure where both successful and failed examples are used as input.
99

---

Published as a conference paper at ICLR 2025
G.4
WITHOUT LEARNING PROGRESS
onige
Boweing
Tric 3 Laser Maze Escape
Tiskd: Laser Muza Excape Ounger im!)
Think Elemenaal Or Collectoon
Jicks Blmmana Orb Colladion Jeeosr perally
Tark 7: Domino Raily Challenge
Domino Raily Challenge (fweer dominos)
Inski Menta Wiaard Challenge
Tisk Oostaole
Challonge
Elmeeaal
Challenge
Chaikenge (longe lmm,, feew: hazart))
Chellenge
Task Simpl tned Gosme Pinbal
Task Muscaa Platorme Challenge
Tisk goal
Rlythr Challange
Nous 18 Rhytim Recar Chalenge (ungee Iim kril)
Tak 12 Rtythmi: Patlormrr Chiilenge
20: Chi lange (Ranie taxk just masching gow).
Tock 21: Elamrnnall
Juak 22 Elumrrn Clox lenge (Jorge lime lms))
Challenge
Figure 35: Ablation short run with learning, without learning progress notions in the task generator's prompt. This figure shows an ablation of the short run with learning experiment. The same settings Figure 3 are used, except that the task generator's prompt does not include any notion of learning progress. Checkmarks indicate successful learning, crosses indicate failures, and arrows show iterations on failed tasks. Without incorporating a notion of learning progress the task generator's prompt, tasks successfully learned, which is much fewer than in Figure where the task generator's prompt includes both notions of interestingness and learning progress.
System Prompt:
You are an expert in reinforcement learning. Your goal is to help a robot master diverse set of interesting tasks in simulation using PyBullet. You will be provided with the list of tasks that the robot has successfully learned, along with their corresponding environment code, and the list of tasks that the robot has attempted but failed to learn, along with their corresponding environment code. Your objective is to decide the next task for the robot, selecting one that is interesting and novel.
Instructions:
The next task should be interesting:
Novel and creative compared to the tasks the robot has already tried.
Useful according to humans.
Design rich environments with large number of diverse objects and terrains, and with clear task for the robot to execute. fun or engaging to watch. You can draw inspiration from real-world tasks or video games. Be creative! Be specific in the task description:
The task should
State clearly what the task of the robot is.
100

---

Published as a conference paper at ICLR 2025
Define clearly what the success condition is. Define clearly what are the different reward and penalty components
Deefine clearly what the termination conditions are. If the reward components include a survival reward, ensure the episode only terminates when the agent fails the task.
The task should not take too long to complete.
The robot can push objects around but lacks the ability to grab, pick up, carry, or stack objects. Don 't suggest tasks that involve these skills.
Don't suggest tasks that require the robot to navigate through maze.
If the task involves navigating terrain with obstacles, make sure that the robot can not go around the obstacles.
If the task involves a target zone, make sure that the collision of the
target zone is set to False.
Return only the task description, not the environment code.
Ensure that the designs pose no harm to humans and align with human
values and ethics.
Robot description: (ROBOT_DESC)
Desired format: Reasoning for what the next task should be: <reasoning>
Next task description: \"\"\"
<task description "\
101

---

Published as a conference paper at ICLR 2025
H HUMAN EVALUATION SETUP
To evaluate the alignment between human judgment and the automated success detector, we conducted study with 50 participants. The goal success detector's assessments of task completion by the robot. Each participant reviewed videos of robot attempting various tasks, alongside the corresponding task descriptions. Below are the detailed instructions and setup used for the evaluation.
assess whether human evaluators agreed with the
Taak:
Dinc oe budgg gras sorear spiidlure
description.
Description.
Respond:
Completed" Complete"
Did robot successfully complete tack?
shew your padgmant sorety on lank descripion and vndee. dbjective honest.
Cocodde the out diampaos adove www. exiclttoos
Coample:
Task: "Cres bridge with moving segments to piathorm" if the robot reaches the platform without falling, answeet "Successfulty Campleted":
the robot falls off bridge before reashing the platform, answer Complete".
Figure 36: Human evaluation setup. (Left) Instructions given participants. (Right) Annotation interface used by participants to annotate the data.
The human evaluation study was conducted using the cloudresearch, com platform. Participants were compensated S0.25 for each response (estimated hourly rate of $15.00/hour, as each response took less than minute), ensuring fair and ethical payment for their time and effort. The study aimed to collect accurate and unbiased assessments to compare with the success detector's automated evaluations.
102

---

Published as a conference paper at ICLR 2025
1 SUPPLEMENTARY MATERIALS FOR QUANTITATIVE RESULTS
Archive
Auchive archive
Dimerspon
Figure 37: Archive diversity results. Archive diversity plots of long runs with simulated learning of OMNI-EPIC and the controls. Fewer tasks fall into the same discretized cells for OMNI- EPIC than OMNI-EPIC Learning Progress only OMNI-EPIC w/o archive. The substantial difference between the left and center plots is more easily observed in Figure
0.30
3.3
4.5
842
Figure 38: Cell coverage of archive diversity plots with different discretization levels for long runs with simulated learning. This figure similar to Figure which uses an archive discretization level of 50, but here we present cell coverage results for archive diversity plots with discretization levels of [10, 20, 30, 40] across methods on long runs with simulated learning. OMNI-EPIC consistently achieves significantly higher cell coverage compared to the controls, even across different archive discretization levels (p 0.05, Mann- Whinnee U test).
Aushile
Arthiie
0.12
0.13
0.09
0.000
0.115
OMAIFRIC
kuniing progris only
ASchive
OMAI-RR
DON
DARI
Figure 39: Cell coverage archive diversity plots with different discretization levels for short with learning. This figure is similar to Figure which uses archive discretization level of 50 for long run with simulated learning, but here we present cell coverage results for archive diversity plots with discretization levels of [10, 20] across methods short runs with learning. While similar trends as Figure 4 and Figure 38 the differences between methods are not always statistically significant (not all p 0.05, Mann-
Mann-Whitney Whitney U test). This is due the shorter training runs, the effects of OMNI-EPIC become more pronounced over longer runs.
103

---

Published as a conference paper at ICLR 2025
OMNI-EPIC
ANNECS-OMNI
ANNECS
Suczesshilly Learned Archive Stre
Successtuly Leamed Archive Size
Figure 40: (Left) ANNECS and (Right) ANNECS-OMNI results. Short with RL training by OMNI-EPIC and the controls are repeated five times. Darker lines are median values, shaded regions 95% confidence intervals. OMNI-EPIC significantly outperforms the controls both metrics. There is no difference between ANNECS and ANNECS- for OMNI-EPIC, indicating that all tasks learned by OMNI-EPIC considered interesting.
100
CNN--PIC
Tranned from scratch
Percentage Thoks
2.5
5.0
7.5
10.0
12.5
15.0
17.5
Tutal Number Traiiinss (Sucoessful Raled)
Figure 41: Percentage of tasks learned over attempted tasks in short runs with learning experi- ments. Attempted tasks include all tasks that attempted with RL training. Short runs with RL training by OMNI-EPIC are repeated five times. The blue line represents OMNI-EPIC, while the green line represents using the same generated tasks from OMNI-EPIC and training each task from scratch. The darker lines indicate median values, and the shaded regions represent 95% confidence intervals. The decreasing percentage of learned tasks, along with the widening gap between tasks learned by OMNI-EPIC and those trained from scratch, shows that OMNI-EPIC generates increas- ingly difficult tasks over time.
104

---

Published as a conference paper at ICLR 2025
J SELECTING THE MOST SIMILAR TASKS
To generate new tasks that are both relevant and challenging, is essential to select tasks from the archive as reference. This selection process ensures that the generated tasks build upon the agent's prior knowledge and skills. OMNI-EPIC's process of selecting the most similar tasks, given query task, takes inspiration from Lewis al. 2020). The first step is to embed all tasks (both their natural language descriptions and environment code) within the archive into high- sigh-dimensional vector space using a pretrained language encoder. This embedding allows us to represent each task as dense vector that captures its semantic meaning and characteristics. Once all tasks are embedded, we can efficiently compare the similarity between tasks using cosine similarity. For a given query task, we retrieve predefined number of tasks from the archive that exhibit the closest cosine similarity to the query's ' embedding, ensuring that the selected tasks are the most relevant and similar to the current learning context.
potential limitation our approach the possibility of cyclic behavior where the task generator alternates between generating task types that consistently rejected due their similarity to existing tasks the archive. While this scenario unlikely in practice due to the stochastic nature of task generation and the diversity of the environment distribution, it cannot be completely ruled out. Furthermore, as context lengths increase with advancements in FM capabilities, the likelihood of such cyclic behavior would diminish even more.
105

---

Published as a conference paper at ICLR 2025
K HYPERPARAMETERS
Table OMNI-EPIC hyperparameters
Component
Parameter
Value
Max number of iterations
General
OpenAl
Embedding method
(exx-embeeding-3--mallll
Number of learned examples Number of failed examples Client
Task Generator
Anthropic
Claude 3 Opus
Model
(claude-3-puu-22244229)
Temperature
Number of few-shot examples
5
Client
Anthropic
Environment Generator
Claude 3 Opus
Model
(chudd-3-opus-22224229)
Temperature
0
Number of similar tasks
10
Client
OpenAl
Post-Generation Mol
GPTT-40
Model
(spt-4o-2024405-133
Temperature
0
Client
OpenAl
Success Detector
GPT-4o
Model
(22t-40-2022-05--13
Temperature
0
Max number of iterations
1
Number of few-shot examples Client
5
Task Reflection
OpenAl
GPT-4o
Model
(gpt-4o-2024-05-13)
Temperature
0
Table 2: DreamerV3 hyperparameters
Parameter
Value
Total time steps
2 10
Replay buffer size
106
Batch size
16
Batch length
64
Discount factor
0.997
Learning rate
3 10-4
Compute Resources For each task, the R2D2 agent is trained for approximately 1 hour using 2 NVIDIA RTX 6000 Ada Generation GPUs and 32 CPU cores.
106

---

Published as a conference paper at ICLR 2025
L PROMPTS
L.1 TASK GENERATOR
System Prompt:
You are an expert in curriculum learning and reinforcement learning. Your goal is to help robot master diverse set of interesting tasks in simulation using PyBullet. You will be provided with the list of tasks that the robot has successfully learned, along with their corresponding environment code, and the list of tasks that the robot has attempted but failed to learn, along with their corresponding envi monment code. Your objective is to decide the next task for the robot, selecting one that will maximize learning effectiveness based on its past successes and failures.
Instructions:
The next task should be learnable:
Not too difficult for the robot to learn given its current skill
set.
Realistic for the robot based on its description.
Possible to complete in simulation in PyBullet.
The next task should be interesting, i.e., either:
Novel compared to the tasks the robot has already learned. You can
either add complexity gradually on an existing task or design radically novel task from scratch.
Useful according to humans, making it worth learning.
Creative or surprising.
Optionally, the task can be fun and engaging to watch.
Be specific in the task description:
State clearly what the task of the robot is.
If the task involves objects, be specific about their positions and orientations relative to the robot. Be careful to avoid
collisions between objects or with the robot when you decide on the initial positions.
If the task involves dynamically moving objects, be specific about their movement.
You can draw inspiration from real-world tasks or video games. Be creative!
The task should not take too long to complete.
The robot can push objects around but lacks the ability to grab, pick
up, carry, or stack objects. Don 't suggest tasks that involve these skills.
Don't suggest tasks that require the robot to navigate through maze. Return only the task description, not the environment code. Ensure that the designs pose no harm to humans and align with human values and ethics.
Robot description:
(ROBOT DESC)
Desired format: Reasoning for what the next task should be: <reasoning>
Next task description: .na
<task description
User Prompt:
Environment code examples:
(ENV CODES EXAMPLE}
107

---

Published as a conference paper at ICLR 2025
Learned tasks and environment code: LENV_CDDES_LEARNED)
Failed tasks and envi ronment code: {ENV. CODES }
L.2 ENVIRONMENT GENERATOR
System Prompt:
You are an expert Python programming and reinforcement learning. Your goal is to implement an environment in PyBullet specifically designed to train a robot for a given task. You will be provided with the task description and with pairs of task description and environment code. Your objective is to write environment code that rigorously aligns with the task description, helping the robot learn the task as effectively as possible.
Instructions:
Write code wi without using placeholders. Don't change the import statements.
For each object, always define its size first, and ensure the object's initial position is set relative to the platform it starts on or any other object, as demonstrated in the provided environment code
examples. For example, if an object is initialized on the ground, self. define its position as: [self. wtw.ffpllafomm.sstiionffffoooooo + self.platform..5 size platform_positiontll
[2] 2 Ensure the robot's initial position is set relative to the platform it starts on, as demonstrated in the provided environment code examples. For example, if the robot starts on a platform, its initial position position[0], self.platform position should be set to position[2] self.platform size[2] 2 self.
2].
[1], self plaffor robot. links(" "asst"" .pssition init[2]].
If the task involves navigating a terrain with obstacles, make sure that the robot cannot go around the obstacles.
Implement the methods 'Evv. reset()', 'Env. step()', 'Env. (Cwceest You
(et__tas__ewwrrs(() 'Env get_ terminated can implement additional methods if needed.
SEnv.get_tastt task rewards()* returns a dictionary with the different reward components to help the robot learn the task. You should implement dense reward components that are easy to optimize and defined in the range 10. to 10. ()riinneii))))) returns boolean that indicates whether the episode is terminated.
"Env.get__sun ()ccess)) returns boolean that indicates whether the task is successfully completed.
Robot description:
{ROBOT DESC)
Desired format:
Environment code:
'python <environment code>
User Prompt:
Pairs of task description and environment code: {ENV CODES EXAMPLE}
108

---

Published as a conference paper at ICLR 2025
Task description: (TASK_DESC)
L.3 ENVIRONMENT GENERATOR REFLECTION
System Prompt:
You are an expert in Python programming and reinforcement learning. Your goal is to implement an environment in PyBullet specifically designed to train a robot for a given task. You will be provided with environment code examples, with environment code that returns error when executed and with the specific error that encountered. Your objective is to reason about the error and provide a new, improved environment code with no error.
Instructions:
Write code without using placeholders.
Don't change the import stateme nts.
For each object, always define its size first, and ensure the object's initial position is set relative to the platform it starts other object, as demonstrated in the provided environment code examples. For example, if an object is initialized on the ground, define its position as: [self wwwtfoom platform position[1), self.platform wtw.ppaafooooosssioo12222.2 + self. self.platform size [2] / 2 self. size[2] / 2].
any
self.
Ensure the robot's initial position is set relative to the platform it starts on, as demonstrated in the provided environment code examples. For example, if the robot starts on a platform, its initial position should be set to wtw.::plaatoooosssiiooioio [1], self plattfo...siiiiiii222) robot. "
self.platform. posittion self stt.pppotise.........armmmssss.
2 self.
If the task involves navigating a terrain with obstacles, make sure that the robot cannot go around the obstacles. Implement the methods reset()) get task rewards()), 'Env .Env.get terminated()', 'Env.get success() () You can implement additional methods if needed. wew.tetttttkeeaarrereeaa
SEnvss)ep()', 'Env.
returns a dictionary with the different reward components to help the robot learn the task. You should implement dense reward components that are easy to optimize and defined in the range -10. to 10.
'Env. (erminter)))))e())) returns a boolean that indicates whether the episode is terminated.
Env www.gget_succoe ()ccess()) returns boolean that indicates whether the task is successfully completed.
Robot description:
[ROBOT_DESC]
Desired format: How to solve the error: <reasoning>
Environment code:
.' 'pyyhhn <environment code>
User Prompt:
Environment code examples: WWW.CCOEES_EEMPPEEES
code with error:
109

---

Published as a conference paper at ICLR 2025
{ENV_CODE}
Error: {ERROR)
L.4 POST-GENERATION MODEL OF INTERESTINGNESS
System Prompt:
You are an expert in curriculum learning and reinforcement learning. Your goal is to help robot master diverse set of interesting tasks in simulation using PyBullet. You will be provided with list of old tasks and with a new task. Your objective is to determine whether the new task is interesting or not.
The new task
be considered interesting if one of the following is true, the new task is:
Novel compared to the old tasks, to build a diverse skill set. Creative or surprising. Fun engaging to watch.
Not easy for the robot learn given its current skill set, progressing toward more complex challenges.
Useful according to humans, making it worth learning.
Robot description:
(ROBOT_DESCC DESC)
Desired format: Reasoning for why the new task is interesting or not: <reasoning>
Is the new task interesting?: <Yes/No>
User Prompt:
Old tasks: {ENV CODES EXAMPLE}
New task: (ONV_CODE)
110

---

Published as a conference paper at ICLR 2025
M FEW-SHOT EXAMPLES
import numpy as np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2EnV):
Cross pride colored bridge to reach platform.
Description:
A start platform and an end platform (each m in size and 0.5 m in thickness) are placed 30 m apart.
The two platforms are connected by bridge (2 wide) divided in multiple segments. Each segment has a different color
corresponding to the pride colors.
The robot is initialized on the start platform.
The task of the robot is to cross the bridge to reach the end platform as fast as possible.
Success:
The task is successfully completed when the robot reaches the end platform
Rewards:
To help the robot complete the task:
The robot receives reward for each time step it remains the bridge or platforms, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the end platform, incentivizing swift movement towards the goal.
Termination:
The task terminates immediately if the robot falls off the start platform, any segment of the bridge, or the end platform. =**
def (sef): super init ()
# Init start platform
self platform_siie [3., 3., 0.5]
self hatf.plaatrommeen_psiittio = [0., 0., 0.]
self. www.:ooomm..taar.poottttto.
start_ + self.
30., pet..plaffomm.statt.psitttiioooo plaffom_starrt_pssiiinn]]]]
self pllatfom.sttrrt.dd self.crraaeebbx((asss) half_ [ self self.platfommssiee(0 / 2, self.platform size[]] / 2, self.
platform sirform 2],
platform start position,
color= 0.8, 0.8, 1.1)
self. self.platform end id self.create box half extents= self self.
/ 2, wel.eel..pp wwa.ppollooooiieeteeeeee 2, self.
[latform_size(2] / 2], position=self. wtw.taareeetoeeeeeesiiiiiii color=[0.8, 0.8, 0.8, 1.1)
Init bridge welf.bridge_eer
self fllf.platform. end positionnnnn[0] - self. slatform_start_posst bridddddi 2.
self.platform.st
cride-ccoloss
[1.0, 0.0, 0.0,
1.
Red
[1.0, 0.5, 0.0,
1.1,
Orange
[1.0 1.0 0.0,
1.1, Yellow
[0.0, 0.5, 0.0,
1.],
Green
[0.0, 0.0, 1.0,
1.1, # Blue
111

---

Published as a conference paper at ICLR 2025
10.7, 0.0, 1.0, 1.1,
Violet
# Segment length num num.colors segment size self self.bridge.-engieee / num colors
# Create segments
for i, color in enumeratee(price.coltrr
segment_id self create_bbxx half_ Solfeextent=-[
segment size / 2, self.bridge width / 2, self.
platform size / 2], position=sself. platform start position[0] self. { / 2 +
segment-size 2 + segment-size stra
$isssesss,isss,eeeeeeeeeee,e,,, self.
[2]], color=color) self self._p.c LateralFriction-l) 8, restitution=0..5
htga://mmmccctoooniuuueeeeeeeeno linkIndex=-1,
def create createeebo(((ee box mass, half extents, position, color)
id self s (hapssssssssspppsspppeeeeeeeeee P.G.M BOX, malfExxtents=halfffeeeeo extents) visual htt.://eeaeaaeeeaaneaeeeeeeeeeea
collision htt.aa/aaeaaa...aaaatataaaaaaaaa
GEOM_BOX,  return stt.ee/cc.ccccccttaaaaa/aaaaaass hasesissaaShhpppcnneeeessssssssp oasappooooopssPsssttttoooossttoo
id
p.
def get_ www.ooeceppsiitiont.oo
object id):
return np. pat.prrr.........tsseeottoooooor [0])
def vet_distaace_ttooojectt [oj
object_id):
object-position
robot_position relo. (inbott .position
return np. w.lalin...... norbooject position[:2] robot position| [:2])
def reset(self):
observation super() ()
# Reset robot position on start platform
..p. reeeaassssttooooorrerritttttaoss robot. robot_.dd id,
silaatormmstart.ppoo + self. www.fpplaffom...... platform_
[1], hat..plaffomm.start.osittt
/ 2 + rebot. ("bsse"] "nt.rooot.. init(2)), rebo..obot.
Links["based oor.eeoooooiiiiioooo init)
return observation
def (self, action):
# Before taking action
self distance to platform end self.get gelf.get..n distance to object platform end id)
observation, reward, terminated, truncated, info ()per()... step action)
return observation, reward, terminated, truncated, info
def get
action):
# After taking action new distance to platform end self get distance object (self. platformlend_id)
# Survival
112

---

Published as a conference paper at ICLR 2025
survival 1.
Reach end platform Beoch_platforge end = www..acsstceecccccmc to platformeend new distance to platform / self.dt
return ("survival": survival, "reach platform reach-platformeend)
def get_terminateddselff action):
# Terminate if fall off return self. "inks("base"),pp position self. platform start position[2]
def get-successsself) Success if reach end platform de.on_plaatoo end self welf.get_aee lel.gee_ddiii distaniitnncee.. to_ object (self.
platform_ent ind_ < self. www.tpll..fosssioommomooo.m / 2 return plaofo-lato
import numpy as np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2EnV):
Cross over lava on boat to reach target zone.
Description:
The lava is simulated with an orange, 10 x 10 m heightfield. There are two platforms on either side of the lava, each measuring x 10 One serves as the start platform and the other as the end platform.
The boat is a box with dimensions 3 meters in length, meters in width, and 0.2 meters in height. It is initialized next to the start platform at a random y y-position.
The boat has button that, when pressed, activates the boat to move over the lava a speed of meters per second.
The end platform has a target zone indicated by a green, transparent sphere.
The robot's task is to jump onto the boat from the start platform, press the button to activate the boat, and travel across the lava to reach the end platform. The robot must then enter the target zone to complete the task.
Success:
The task is successfully completed when the robot enters the target zone on the end platform.
Rewards:
To guide the robot to complete the task:
The robot receives a reward for each time step it remains active and does not fall off or touch the lava.
The robot is rewarded for making progress towards pressing the button the boat.
Additional rewards are given for progressing towards the target zone, with a significant bonus for entering the target zone.
Termination:
The task terminates immediately if the robot falls off the platform or the boat, or if it touches the simulated lava.
def init (self):
113

---

Published as a conference paper at ICLR 2025
super() (nar()).itttt00
Init lava
self. lava_ = [10., 10.]
self lava-height = 0.1
self postitio [0., 0., 0.]
self.lava.id self. heightfield(
size=self..tavasizz.
# create small bumps to create
fluid-like surface
position=self. lava l
resolution=20, # number of points per meter repeats=2,
self self._p p. cht.nngeeissuaahhaappeooooiituue Ltw.uuueeueee-see..aa......aa linkIndex
1, rgbaColor=(1., 0.3, 0.1, 1.1) # change to lava color
Init platforms
self. www.-pllttfmsizoeo 15 self. lava_ 1.1
self self.platform.stee position [ self.
lava size[0] 2 self wttppplaffo..ssieeeee. / 2, self.
lava position p positionn0 + self.
position
self. pla.fol end end =elf.platfomm.siee(( 2, self.
lava_size(0] (ttp:iiieelllllesttlltllaaaaaaa lava_position selff.lwa.positiion]]]]]]
self, platfomm_start_id set.rrredss.....rrre...ooomssaas
ollf_extent==[ / 2, self.plattorrm.s.. size[1] 2, self.
self.platform
platform-size(2] 2], position-self.llttoome startrtssstt_tprtt position
color=[0.3, 0.3, 0.3, 1.1)
id Self..create.box(massseeee
half_extents=t
self.plattorm.size size[0] / self.platform.sizeeteee / 2, self.
platform platform.size(2] 2], posittonnself..llatffm.eeee ootpaatttoormeeneposttioooooooo
color=(0.3, 0.3, 0.3, 1.1)
self stt..cchhhgooniiicoollliiiaetlte
self. p. www.hhgwwwyiiiiitoooooooottettee restiuutioo
1inkIndex= 1
end id, linkIndex LateralFriction-o..1 restitution=0.5)
Init boat self.boat boat [3., 2., 0.2]
[self. dava_position self. lava_ self. wta.fboottposiiioniiii biaell 2, self. lava position[1], self.
[0] / 2
boat_size(2] / 2]
self bolf.boat_speed 3.
self.boat.id self.create www.ereeeeeeeeeeeeeeeeeeeeo (masssO:.. half extents=(self. boat size_ / 2, Stlf.boat-sizeill / 2, www..beat.sft...eeee
boat_sizee /
2], pfwition=self..focoo..oooo wttpseelf...aatttstiiiooo inii,, init, color= 0.8, 0.8,
1.1) sel ... htwppcccanngyyyynno.oooooooooneo bbatt.id linkIndex= 1, lateralFriction-0..8 restitution=0.5)
Init button
self www.fbutton..o radius 0.25
self. buthon.eeiih 0.25
self Nalf.button_postiinn.ii iniiiinitttiiii
coteat_positioonnnne + self. foat_position_inittee init[2] lavaa__osition1]] self.
boat_size(0] / 4,
helf.boat_sizelt button_height / 2] put button on the right side of the boat
2 self.
self delf.button_ic id self create. (linderrmaas=0.... radius=self. button button_radius radius, height=self bugtttttelf.buutonnn button height, position=self. button position init, color=(0., 0.5, 0., 1.
# Init target zone self targett radius = 1.5
114

---

Published as a conference paper at ICLR 2025
self wwr.ett = self wewf.create.eee. c radius=self.
zong s collision=false, position=(self.
platform_end plattorm_end_postttonnl))) .platform wttprmmend_oosition.. position[2] + self.platform.sizeel 2], color
wtw.plaaformmeen_poiitttitt self
=[0., 1., 0., 0.5])
self ellf.objects. on boat dellfbutton id]
def create_box(selff, mass, half_extents, postttt color):
self oat.daoeeeeee...ctttttttooaaaaaa (htprSSSaapssppppeeeeseeeeer P...EMM BOX, s iisual_shape id sel att.aaaae...caraaaaaaaaaaaaaaaaa GEOM B0X, hal.ffxteents-hall..tteeets, rgbaColor=color) obbaaaaaeottttttt..o.rrrrrrorrrr
return oat.a//....cceetttttttoooooo/sss
basePositiooFpooition) http://www..s.ooooooooooooooooor BaselissaaShhppernneeevvsssssssp
def creatte.cyLinder(sell,, mass, radius, height, position, color):
collision_shape__e
-...GEOM radius=radius, heighthhggththttthhhhhhhhhhhh
GEOM CYLINDER, radius=radius, oet oet.rpp.creetessssaaaaaaceppppee
return per.eee.....rrrttttttoooooaaaaaa rgbacolor=colon baseellLssoffoennneeexxcsssssssd baselissashhhpppnneeevvvsaaaaabu
def create sphere mass, radius, collision, position, ,ostiioon,, color):
if collision: collision id self wtt.p.rcttttelllllloooaaacacheee
GEOM_ http:p/..aaaaoaeeee....aaaaaaaar
radius-radius)
= sel GEOM SPHERE, ( return oat.ai/.....cctettttooooooaaaaas
basel//ssuSShhppenneeevvvsss.slp basePositions position)
else:
visual shape id self p...f.p.c..r htt.://reeeeiiiaaaaaaaaaaaaaaeee SEEOM_SPHERE, radius-radius, robcoooooooooooooooooooooor return hat.ee/.....r.atttttoooooaaaaaas DaseVVssuucChhpeeneeeevvvvsssss id, basePosition position)
def hreate_hhiihhffeedd(slll size, height_max, position, resolution, repeats=2):
neightfield_datte = np. random. uniform(low=0... high=height.ne max,
size=(int * resolution repeats), int(size[1] resolution repeats)))
heightfield__dta- = np.reppad (np. (ereaaarc...rpeettteeggtttee.dd.
repeats,
axis=0), repeats, axis=1)
mesh scale - [1/resolution, 1/resolution, 1.] cetpittfield.ooolliiiiooo..o heightfield collision shape id self. p.createcolllisionhhapee
shapeType==elff.ppGGGe HEIGHTFIELD,
EEENGEGEFEEDD..EEEEEEEEEEEEEEEE meshscale--mehhssaalee,
heigghffieedoptaaeeeeiiiffeeeeee reshapel 1),
shapeeeeeaaaa_e muw.leeihhhfaaccccccummseeeeeeee shape
return setf.pp.crartehutttDooyyllasaaa collision shape id,
iisePositionn=posiiiiooooo pospiiiooonoiiiiii/iiiio position[]] position[2] +
mesh scale[2] height max 2])
def get_ohject_possiioonnssll object_id):
115

---

Published as a conference paper at ICLR 2025
return np. hra.rrrwyerr....eesssseeooooooor [00]
def det_distanceeeo to object object_ id):
= self.get..bjfect. seff.geet.bjjec.ppsiiiooofffffe
position(objeet_ti
robot_position rolot. Linboot.iinss""baabea
robot cobot_position| return np. linalg. wwt.ooooecotoooosiiiiiiii
def reset(self):
observation super() reset()
Reset boat position
y moat_y-init = np. www.oounniooooooossseeleeee lava size[1] / 2 + self
bboat_size_1] / 2, high=self Lava_ / self.
boat_size(1] 2) # randomize position
self www.reeeetaseessiteeee itw.sooosstttonnooeeenattttioioo (se.fiioosse...attti...... [self.
boat_position inioonnni((]]] boat_y-init, self.boat.position.iee
[21]. [0., 0., 0., 1.])
# Reset button position self
button id, [self. button position init[0], boat y init, self. button position init[2]], [0., 0., 0., 1.1)
Reset target zone target_zone_y. np. random. weandom.uniform(LLl
self lava_size[1] pone_radive high=self. laatseel..laaasiiieeeel
/ self.
ponet__zone_rddiia
# randomize y position
self targget
leeef..paaffomm..ccc.. end _ position[0),
target- self self.platform. end positionn)) + self.
platform 2]
sel r .targett_zoe_ppoiittin [0., 0., 0., 1.1)
barget_zone id, self
Reset robot position self. p. prr.eetttsessss..oooooooonnttttt roottaaiiiiieee..e..oo........ robot.robbot id,
self. pll.fplatform start positionn),,, self.platfor start position
[1], self. start position[2] + self wta.fppllffrr.ssssc size[2]
2 self. robot. links ("anse""baee" "osi"aaaa".ipiiiiniiiitttt. positiin robot.
links["basee "rriss"".oieenattiion.n
return observation
def step(self, action):
# Before taking action
self. wwlff.iittnccee.m to button = self fll..c. www.gge....... distancee to to.jecttte..... button-id)
wtfffddittaceettotttt
oeff
target_zone_iue
toaas_touchhd_plaa end
bodyA=self. robot. robot id, bodyB=self platform folyt=-ee..pllaffrmm end id))
observation, reward, terminated, truncated, info super() step action)
# Check if button is pressed contact_points htwp:=sell..uutoo...m robot_id, bodyB=self www.bbtttonno.. id) button_pressed len (en(ontactt.points) 0
if button_ pressed:
Move boat and everything on boat forward
for body_id in iself.boat_in. self.objects_.nn_batt body_position self. ttw.:ooojec_pssiiiooooooooool
116

---

Published as a conference paper at ICLR 2025
www.objecttposittin body_position pp.array([self. pppat_peet self.dt, 0., 0.])
self.-p. www.eeeeaeeeosittooonnniieeeeeee www..object_posittie [o., 0., 0., 1.1)
(body-id,,
return observation, reward, terminated, truncated, info
def get_ rewardss action):
# After taking action
new-distance_ to-butonn self got. button id)
new distance target zone self. get distance object(self target zone id)
# Survival survival 1.
# Reach button
reach_button = www.:diinncttoooutttteioeee new.ddistance..tobbutooo self.dt
# Reach target zone reach target zone
to target zone
new tociice__ttttt target-zone) / self
if reach_target_zone 5.
self. tolf. radius:
return ("survival": survival, breach_buttons reach_button, reach_ target_zone)
def get- (sefterriinated action):
# Terminate if touch lava
contact-points se.......gerrotttttttttttttoooos robot. robot_id, bodyB=self lava_id) ts_thcching_lave len (contact_pointts
# Terminate if fall off is fall off self. robot. Links["base"") possitionn)) self. platform_start_ppsiitt positionn2]]] return ts.touching lava or is_ oall
def success
# Success if stand in the target zone distance to toweettttttrgtoooooeoooeo = self self. target zone id)
return distance to target zone zone < self. wargtarret...mm zone radius
import numpy as np
from oped. envs. r2d2. base import R2D2Env
class Env (R2D2Env):
=am
Descend series of stairs to reach the ground.
Description:
The environment consists of a ground platform (1000 m x 10 m x 10 and a set of 10 steps.
Each step has dimensions of in length, 10 in width, and 0.2 height
The steps are positioned to form a descending staircase starting from an initial height, with each subsequent step lower than the previous one.
The robot is initialized at the top of the stairs.
117

---

Published as a conference paper at ICLR 2025
Success:
The task is completed when the robot successfully descends the stairs and touches the ground platform.
Rewards:
The help the robot complete the task:
The robot is rewarded for survival at each time step. The robot is rewarded for forward velocity, incentivizing it to move down the stairs.
Termination:
The task terminates immediately if the robot falls off the stairs or the ground platform.
def (self)) (uper().-.init__) ()
Init ground
self self.ground size [1000., 10., 10.]
self.ground position = [0., 0., 0.1
id self. box omass=00. half extents=[self.
ground_ 2, self self.ggroun_ssie(1 / 2, self.ground grwundrouuddzzz size [2] 21, positionset..grrnnd_posttto color=[0.5 5, 0.5, 0.5, 1.]] te ground id, LinkIndex= 1,
self lateralfriction=0.6 restitution-0.5)
Init stairs self.num steps 10
self. size = [1.0, 10 0.2]
self eelf.step.pssitonnini = - www.ndd.ossiiiioon00.. self.
ground_positionn11], (etp:i1ss___sffggeeeeeeee colf.ground_position/o + self.
self.step_size(2] self.create stairs dswnp size=self. size,
ground_size[2] / / s
step position mnt-=eelf.ster position init, num steps=self. num steps)
def create_box(selle mass, half_extents, posttettttoossiiiooooooooo color):
sellisioonshape_iio = self (tt.eeeee..etetooooooooooeeeeeee
GE. BOX, hallffxtents=hhafff.xttettt
id httppppp..eeggt..cceeeeeaaaaaaaa
GEOM_BOX. wtw.eattetsssall...ttttttsseooos rggcooooo return set.rp...pccceettttaaooooossssss
basel//sssasShhapeendeexvvvssss id, baseposittion=posittion) htg.aaaahhh.haaaaaaaaoootooootot
def create_staairs.ddnn(selll step_size, step.position_init, num_steps
):
color 1 = np.array( 0., 0.
color
for i np.array([0., 0., 1.1)
step-position steps):
+ step_size(0), step_positionniiitffi init step-position
step_size[2]]
interpolation
(num_steps 1)
step_color (1 interpolation) color_1 interpolation color-2 # shade steps for visualization self. box (oxmss==0.,
extentsstttst size[0] 2, step size[1] / 2, step size[2] 2], position= step position, Mtor=nn..ppentt step color, 1.))
def reset(self):
observation = super () reset()
118

---

Published as a conference paper at ICLR 2025
# Reset robot position at the top of the stairs
self ressttsseooiionnnrrriinaaaaanni
robot. robot_id.
self. sttp_ppssiiiionntooo step_pooitiin_
self.step_pont per.sstttppsiiionniiiieeeee initoon self. + self. self.step.sizee2]] size[2] 2 + self. robot. self.robot. links "baase"]
links "aaikss("ase"]ppsssiii wossition. orientation init)
return observation
def step,self, action):
Before taking action
self. self. position self. robot. (Alasee" l. position
observation, reward, terminated, truncated, info super() step action)
return observation, reward, terminated, truncated, info
def get tasttaaewaaaaaaaaeeeeeeae (self, action):
After taking action new wow.tissiionn = self. robot. links ["base"] position
# Survival survival 1.
# Forward velocity
vorwari = (new now.pposition[] self position(0)) self.dt
return ("survival": survival, "forward..elocityy:
forward
def get_terminatedd(slff action):
# Terminate if fall off return rebot links (iinks("base") "base"" positioon [""bass"..positioo ground position[2]
self.
def get success
# Success if reach end stairs and touch ground
contact_ points self robot_id, see--....ttttttttcocoooo (boodyA=see (bodyA=self. robot. os_ ground = len contactt_oointttttt > return is grouud
import numpy as np
from oped. envs. r2d2. base import R2D2Env
class Env (RV2D2Ennv)
Activate lever to open door and move through the door.
Description:
The environment consists of large platform measuring 1000 x 10
0.1 meters.
The robot is initialized at fixed position on the platform.
door with dimensions 0.5 x 2 x 2 meters is positioned on the platform, 5 aways from the robot, initially closed.
The door is flanked by walls to prevent the robot from bypassing it
lever is placed on the platform, meters to the left of the door
The task of the robot is to move to the lever, activate it to open the door, and then pass through the door.
119

---

Published as a conference paper at ICLR 2025
Success:
The task is successfully completed if the robot passes through the door and moves more than 10 m beyond the initial position.
Rewards:
To guide the robot to complete the task:
The robot receives a survival reward at each time step.
The robot is rewarded for decreasing its distance to the lever.
The robot receives bonus rewards for activating the lever to open
the door.
Once the door is open, the robot is rewarded for moving forward.
Termination:
The task terminates immediately if the robot falls off the stairs or the ground platform.
==.
def -seltt super(). init ()
self. robot position init = [0., 0., 0.]
# Init platform
self platform_size [1000., 10. 0.11 self. platformmposiiioo
[self. [onoot_ppssttiioniini][]]]]]]]] + self.
platform size[[] / 2 2., robot_ wwwppiioniinitttteoooooooppppppe by m to avoid off edge or edge placement
wwtp:ppppisiiiiiiiiiiiiiiilillie self.
self.platform.sizel2] 2] # offset
self. plattormm id Slatform-size(e / 2, self.platform.sizet1] 2, self.
hal._extentss=sself.
platform size[[] / 2], position-self.plattors position, color =10.5, 0.5, 0.5, 1.])
sel self._p. cs htwp:///w......oooo...ooooeaaaaa
linkIndex
1, Lateralfriction=D..B, restitution=0.5)
# Init door self.dom size [0.5, 2., 2.]
Ail..door_positionnini. [self. ro 5., self self.platform.posittinn22 self.
-pllatfommpposttion]]]
platform_size(2] 2 self.door.sizee22/ 2] @elf.door_id
half_extents=
door_size(0] 2, self.door.size(i] 2, self.door_sizers / color=(1., 0., 0., 1.1)
21, self self.door-open False
Init wall self wall_wizee [self door_sizee()), (self.platfomm_sizel) self.door-size(1]
2, [olf.doorrssze(2]] # walls plus door span the full platform to prevent robot to go around self.crean create box half sttents=(seff..all size[0] 2, Colf.wall 2], (sel
self. wtl.f..wllsiize(1 door pww.:rppsiiiooooo init[0], self.door l init[]] self. ot
plarronmooosttoo222=effrrppooooo self.
/ self. self.
# left section
sel hrtrreeeeeoornnssseeenanaaaatttr
2,
self. self.wall.sizeel11/3 self.wall.izzee22 /1, p pssi .ddoo_ppsiitionniiittt))
door dior_s self.borr.psiiionn.mm
self. self.
platform 2 helf.ppattorm.sizzee 2 self. wall / 2], color=(0., 0., 1., 1.1) # right section
# Init lever
self lever- radius 0.05
120

---

Published as a conference paper at ICLR 2025
self Lever-height 0.5
lever-position [self door -ot.-ddor--poiiion iepiitiooninit(00 2., size self celf.ppaaffom_siii
[1], self. wolaatorm_poosteeee Delf.lever_heer Lever_height / 21 two meters to the left of the door on the platform
/ +
self. lever-er self.creat (ylinder((mas=-0) radius=self. radius, height=self. -iver_height, position= Lever-position, color=(0.5, 0.25, 0., 1.
def create_box(selff mass, half_extents, position, color):
collision id self s (shapeTSssppssspppeeeesssllll p...EMM BOX, hflfExtentts=half extents)
visual hawpa/aeaeaaietttteeettttettrrra G return self seT..p/pcrraccchtttoooo/////ssss
def create_cylliderrls
mass, radiaaddusssaa o postinn,
position, color):
collision id self oat.:ppacaatttttoloosooocceeeeee
GEOM. CYLINDER, radiussrrriuss higgggihhttttttihggggeehihhhhhie visual-shapp id = aet..pppcrrateessaaaaaaaaaaaaaaa
GEOM CYLINDER, raduu Lengghhiui,,..eenggeeeeeeigggg, ht
oag.g:hhhheaiigg..oooooooooooooo rgbaColor=color)
return wwt.ee/.....rtttttttiiollaaaaaaa
def wtw.bbecct.ossiionssstteeee object id) i
return np ppp.aaarraalsslff.p..eeeeeaaee httpaa/wwwe.t....aaaaaaaatooooer [0])
def dis_anie to object object :ibect_dd): object_position = self oet.eee.ooeeec.pssiitioorieeeeee
robot_position self.robon Links"""ae"" .pssiiiioooooiiii
.position
return np. linalg. nomm(chject_possttinn
robot_position| :2])
def reset(self):
observation super() reset()
Reset door self. ddoor_open = False self door-position.init, [0., 0., 0., 1.1)
(ww.sssef..oo.dd..... self.
# Reset robot position
self htt.ppppeeetttsttiioooooooooooaa rrotriiooonsee..roor......... robot... id,
self. robot ]osition_inii[[]] self. renf.roo itntottooosiiiiiinnnnntooooo iintittiionnnn]]]_.. self. Celffrobot_positionniiiie
rebot (basee))
position_init([]1, self. robot. [inks["basee "rrse"".orietttiionnnnn
return observation
def step(self, action):
# Before taking action
self selffposition self.robot. links "basee" "basss""aa"""""ssiiiii position
self.distance..t... lever self selffget-ddil distance to_oojecttssel lever_id)
observation, reward, terminated, truncated, info super step action)
contact points self. tettteeaaeeeeeee.ggeeeo..eee.. oogettontaatooiittttboooossssss robot. robot_id, bodyB=self. lever__id
if len (ontcaaacttpoointss > 0 and not @elf.door_opent self delf.door_open True
121

---

Published as a conference paper at ICLR 2025
self . wwt.reetsssssettoooooooooaaaaaae [self.
door 1ntpiittioninn((00) self.door http.doorrppsitioonnni////// init[l] self. door_size[1], wwtorrposttionninii(]]]]] [0., 0., 0., 1.1)
return observation, reward, terminated, truncated, info
def get_ rewards
action):
# After taking action
new-position rolf. links
new distance to lever self. get distance to object lever id
# Survival survival 1.
Reach lever if not self.door_opp... and het.e:pp.eeeccnacttooonstttooaao .robot, robot_id, bodyB=self. lever_id)) == 0:
reach-levers = iistt.distance to lever distance to lever
/ self.dt
elif not self. open and len(self. p.ceel.....pepeeeee...m g.getContacPPonnts (bodyA=
self. robot. robot id, bodyB=self. lever id)) > 0:
reach_lever = 10.
else:
reach_lever 0.
# Forward velocity
if self.door open:
forward_velocie (new.position[0] self.position(O)) self.dt
else:
foroard 0.
return ("survival":
survival, "reach_ "reach_lever"; reach_lever,
forward velocity": forward velocity)
def get terminated(self, action):
# Terminate if fall off
return gelf.robon links links("base") "base" "osse"].position() < self.
platform.posiitonn12
def get_ (scccsss
# Success if pass through door
return self. robot. links fonks("base" basse" "asse"..posiii posiition()] 10.
122

---

Published as a conference paper at ICLR 2025
N
TASK DESCRIPTION SEEDS
Cross pride colored bridge with gaps to reach platform.
Description:
A start platform and an end platform (each m in size and 0.5 m in thickness) are placed 50 apart.
The two platforms are connected by bridge (2 wide) divided in multiple segments. Each segment has different color corresponding to the pride colors.
The segments are separated by gaps measuring 2 m.
The robot is initialized
the start platform.
The task of the robot is to cross the bridge to reach the end platform fast as possible.
Success:
The task is successfully completed when the robot reaches the end platform.
Rewards:
To help the robot complete the task:
The robot receives reward for each time step it remains standing on the bridge or platforms, encouraging steady progress.
The robot is rewarded based on how much it reduces the distance to the end platform, incentivizing swift movement towards the goal.
Termination:
The task terminates immediately if the robot falls off the start platform any segment of the bridge, or the end platform.
Ascend series of stairs to reach a platform.
Description:
The environment consists of ground platform (1000 10 10 m) and a set of 10 steps.
Each step has dimensions of m in length, 10 m in width, and 0.2 in height.
The steps are positioned to form an ascending staircase, with each subsequent step higher than the previous one.
The robot is initialized on the ground at the bottom of the stairs.
Success:
The task is completed when the robot successfully ascends the stairs and reaches the top platform.
Rewards:
To help the robot complete the task:
The robot is rewarded for survival at each time step. The robot is rewarded for forward velocity, incentivizing it to move up the stairs.
Termination:
The task terminates immediately if the robot falls off the stairs or the top platform.
Kick a ball into a goal.
Description:
The environment consists of large flat ground measuring 1000 1000 10 meters.
A ball with a radius of 0.5 meters is placed randomly on the ground. The goal is defined by two goal posts, each 2 meters high and placed 3 meters apart, forming goal area.
123

---

Published as a conference paper at ICLR 2025
The robot is initialized at fixed position on the ground.
The task of the robot is to move across the ground, reach the ball, and kick it into the goal.
Success:
The task is successfully completed if the robot kicks the ball that it passes between the two goal posts.
Rewards:
To help the robot complete the task;
The robot is rewarded for survival at each time step.
The robot is rewarded for decreasing its distance the ball.
The robot is rewarded for kicking the ball towards the goal, with additional rewards for successfully kicking the ball into the goal.
Termination:
The task does not have a specific termination condition.
124

---

Published as a conference paper at ICLR 2025
O DISCUSSION ON TRAINING GENERALISTS
Previous work (Adaptive Agent Team et al. 2023) has shown that training a large model on a suffi- ciently diverse of tasks can produce generalist agent capable generalizing to new tasks within that distribution. However, generating such a distribution is both time-consuming and challenging. Furthermore, to achieve efficiency, it is crucial automatically create curriculum within this distribution. Since these are the main challenges, our results section focuses on demonstrating that OMNI-EPIC addresses both issues effectively (Section 5). While our academic lab lacks the resources to train generalist agents on a large set of OMNI-EPIC - tasks, our findings suggest that it should be feasible to so, as OMNI-EPIC generates a progressively complex and diverse curriculum. Nonetheless, acknowledge that training such agents would still be significant endeavor, as demonstrated by Adaptive Agent Team al. (202)), which required an extensive distribution of environments, considerable eftort, and substantial computational resources.
125

---

Published as a conference paper at ICLR 2025
P EXPERIMENTS ON DIFFERENT ROBOTS AND ACTION SPACES
OMNI-EPIC is designed to accommodate a wide range of robotic systems, regardless the robot type action space. To demonstrate the flexibility of our approach, train an Ant robot on generated tasks using OMNI-EPIC. The Ant robot is 3D quadruped consisting of a torso with free rotational movement and four legs, each composed of two segments (Towers et al. 2024). Apart from the robot type and action space, all settings, including the observation space and RL algorithm, are kept consistent with those used for the R2D2 robot. The Ant robot's 's action space defined as continuous 8-dimensional vector, with each element bounded between -1 and 1.
Cross colored bridge with gaps reach plattorm.
Cross over lava on boat to reach target zone.
top rolling eylinder maintaining balance on
Cross a bridge with hurdies to reach platform.
Figure 42: Ant robot successfully completing different generated tasks. These examples highlight OMNI-EPIC's ability to train various robot types and operate across different action spaces.
126


Reviews:
[{"review_id": "AoawghRgsf", "reviewer": "Reviewer_kcfh", "summary": "The paper presents OMNI-EPIC, a system to design a large library of environments for reinforcement learning agents. OMNI-EPIC relies on Foundation Models, both large language models and vision language models. The system uses a pipeline where problems are specified in natural language, and then it uses an LLM as a synthesizer to generate programs encoding the environments. The environments are also evaluated with another LLM with respect to their interestingness; environments that aren't \"interesting\" are discarded. Finally, the system also has a module that checks whether a task was solved or not.\n\nA few results of the system show that it is able to generate a diverse set of environments.", "questions": "1. Is there a way you could have OMNI-EPIC design simpler problems for which you could thoroughly evaluate RL agents? Even with the amount of computational resources you have available in your research lab?\n2. How can you verify whether OMNI-EPIC is solving a retrieval problem or an environment-generation problem?\n3. Why is Darwin-completeness defined as the potential to generate any learning environment? How is Darwin's theory of evolution related to the ability to generate learning environments?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "The paper attempts to tackle the ambitious problem of continually generating a diverse set of problems, from which an RL agent could learn. The idea of leveraging foundation models for this task is a promising idea. These models contain what one might call \"general knowledge\" that can be helpful in tackling this problem.", "weaknesses": "There are two key weaknesses with the paper that justifies my scores and recommendation. \n\n**Scholarship**\n\nFirst, the paper lacks scholarship, in the sense that the writing is sloppy (you read the sentences and say 'yes, I see what you mean, but what you wrote isn't technically correct.') and it lacks comparison with a very large body of works devoted to generating content in simulated environments: procedural content generation. I will give concrete examples and suggestions next.\n\nThe introduction of the paper talks about the grandiose plan of open-endedness, where systems would be able to come up with any computable environment. However, the paper is about generating environments in a restricted environment, with one specific simulator and small and discrete action spaces. It is not clear how the constrained experiments performed in this paper contribute to the grand goals of open-ended learning.\n\nThe environments that OMNI-EPIC generates could just be environments that humans implemented, made available online in GitHub, and an LLM was trained on. In this case, OMNI-EPIC is working as a retrieval system that can create different environments within a single codebase. While I see value in such a retrieval system, the paper is not about it. The paper doesn't even consider the possibility that the LLMs are simply retrieving knowledge and replicating it. From what I understand, this doesn't align with the open-ended goal of being... open-ended. A retrieval system will be bounded by whatever we created and made available online. The paper would have to go through a deep re-write to adjust these claims or to perform the proper evaluations to address these concerns. \n\nHere are three examples of sentences that are sloppy.\n\n1. *Because the model has distilled a sense of interesting from reading the internet.* The model does not read; we should use the correct technical terms. The model is trained on Internet data.\n2. *Since code is Turing complete.* This is not true. You could have a language that is not Turing complete. The authors might be referring to commonly used general-purpose languages that are Turing complete. \n3. *While our academic lab lacks the resources to train generalist agents on a large set of OMNI-EPIC-generated tasks, our findings suggest that this would be straightforward.* While I appreciate the use of the word \"suggest,\" I don't see how we can conclude that learning a generalist agent is straightforward. Training RL agents is tricky (catastrophic forgetting and saturated neural networks are examples of issues one might come across). Evidence suggests that doing this in dozens of environments cannot be straightforward.\n\nOverall, the paper currently reads like an informal report, which makes it difficult to pinpoint the problem that is being solved and the claims that are being made.\n\nFinally, OMNI-EPIC is closely related to procedural content generation, an active research area in the computer games community. See the book *Procedural Content Generation in Games* by Noor Shaker, Julian Togelius, and Mark J. Nelson for an outdated overview.\n\n**Evaluation**\n\nThe experiments are unfinished, as the environments generated with OMNI-EPIC are not used to actually evaluate RL algorithms and how such environments can potentialize them. The evaluation focuses on metrics of diversity of the environments used, instead of focusing on the actual goal of improving learning. If the goal is to learn a generalist agent (is that true?), then why not evaluate it that way? The current evaluation serves as a proxy for the actual evaluation, and it is not clear how good of a proxy this is.\n\nThe paper justifies why the evaluation is restricted by saying they do not have enough computational resources. While I sympathize with the issue, I cannot recommend acceptance of a paper that doesn't evaluate on the problem it is trying to solve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3rSLOqngB7", "reviewer": "Reviewer_Cvga", "summary": "This paper uses an LLM to generate environment code for an RL agent. This is combined with another foundation model that serves as a proxy for human models of interestingness. Together, the first LLM generates code whereas the second guides this process towards novel and interesting tasks. Theoretically, treating the environment as code is a much less constrained way to generate environments compared to much of the prior work. The paper also provides two experiments, one with simulated learning to focus on the environment generation, and another with actual RL agent training to identify how these two components interact.", "questions": "- Is the code open source?\n- To what extent will this be limited by the dataset of the coding LLM? Will there ever be truly novel environments generated?\n- It is a bit unclear to me, but when starting the process, do you first sample a random task from the archive, and then use this as the reference task to choose the N most similar ones.\n- Consider the following case:\n\t- You have 10 tasks in the archive, say 5 of type A and type B. Type A tasks are all somewhat similar to each other, but different to type B. As an example, type A tasks involve kicking a ball and type B involves crossing a bridge. Suppose the number of tasks given to the LLM if 5, say.\n\t- Would the following cyclic behaviour be possible? Please comment/explain how or how not, and what possible solutions there are.\n\t\t- The task generator receives 5 tasks of type A, and is asked to generate a novel and new task. Then it does this by generating a type B task. However, the post generation MOI rejects this because the archive already has many of these, so the task is not added. Whenever the randomly-chosen task is type A, this happens, and the opposite for type B.\n\t\t- If this happens, it seems like the process will just remain stuck and never generate anything new.\n- Would there be axes of variation that are not explored at all by an LLM, potentially due to the bias in its data, or by virtue of how it predicts the next token? Concretely, suppose you apply this to a more general environment representation (where the LLM can change everything), and it happens to generate a physics-based R2D2 environment. Then, when generating new tasks, will it ever do something very different, like generating a game like Pong (different controls, doesn't use a robot anymore, etc.) or will it continuously generate variations of R2D2 physics tasks? Please elaborate on this", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- I agree, using code as an environment representation is indeed much more expressive than changing hand-designed features within a very constrained environment, as prior work does.\n- It is great that this paper does provide a full pipeline that theoretically can be run for a long time to continuously generate novel tasks.\n\nI am not giving it a lower score because I think the idea of using code as an environment representation is a good idea that future work can build on. Furthermore, demonstrating a full end-to-end pipeline is useful.\nI am not giving it a higher score because I think the results are not that compelling. That is not to say that this approach does not hold promise---and I hope future work/the authors build on this---but the outcomes in this work are not that open ended.", "weaknesses": "While code as a representation is expressive in theory, all of the different environments seem quite similar. This is partly due to the agent in all of these tasks being the same.\n\t- However, if you remove the restriction of using this particular robot in a particular simulator, then the task may become too difficult for current LLMs, or you could end up generating only a certain type of task. \n- It feels like the claims made are that we can express any possible task, and that, when run long enough, we will eventually obtain a very large number of semantically diverse environments; however, the results do not seem to substantiate this claim. \n- Figure 2 is very hard to understand and parse, is there a better way to show the same information?\n- This sentence is a bit hard to parse/can be rewritten\n> Because the model has distilled a sense of interesting from reading the internet, it has a model ofi nterestingness (MoI) that emulates the human capacity for nuanced judgments of interestingness in open-ended learning (Zhang et al., 2023), which it brings to bear when generating tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "avUxN9n5Fd", "reviewer": "Reviewer_PisJ", "summary": "The paper introduces OMNI-EPIC, an open-ended system that generates novel and learnable environments and tasks for RL agents. The method leverages current language and vision foundation models to generate the environments (through code) and filter the novel and learnable ones. The authors show how the system generates diverse, interesting, and learnable tasks for RL agents, starting from a very small set of initial environments. Experiments show the potential of the system to produce endless automatic curricula for increasingly complex agents.", "questions": "**Q1:** Do you have any result or hypothesis on how much the initial set of environments in the task archive conditions the generation of new environments?\n\n**Q2:** How does the task similarity method scale with the number of tasks? If the idea is to endlessly generate tasks, this similarity method should be very computationally efficient.\n\n**Q3:** Does the simplified action space limit the diversity and complexity of the generated environments? \n\n**Q4:** L283 states \"We employ a success detector, instantiated as an LLM or VLM\", later L304-305 mentions \"Although our preliminary testing found that current VLMs are not yet accurate enough to be used as success detectors\". Given the second sentence, I find very surprising the usage of VLMs as success detectors in  OMNI-EPIC. Why does OMNI-EPIC employ VLMs for this purpose even though the authors mention that VLMs are still not ready for the task?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well written and the ideas are mostly clearly transmitted. The figures are clean and support the text, helping to transmit the paper's main ideas.\n\n-  I think that the contribution is substantial. The fact that OMNI-EPIC can successfully generate diverse, novel, and learnable environments through code (Python+PyBullet) constitutes a valuable opportunity for the open-endeness community to analyze these systems and advance the field. \n\n- Moreover, OMNI-EPIC successfully employs foundation models to leverage notions of human interestingness. Generating novel, but interesting environments (that are relevant to humans) is a key challenge in unsupervised environment design and the open-endedness field. Although previous works have pointed to the possibility of using foundation models for this purpose [1], this work is an excellent example of how foundation models can be successfully used for this purpose, which is also a valuable contribution by itself. \n\n[1] Hughes, Edward, et al. \"Position: Open-Endedness is Essential for Artificial Superhuman Intelligence\", ICML24.", "weaknesses": "**W1:** One of my main concerns with this work is the gap between the claims made and the actual demonstrations in the paper. \nAuthors claim multiple times that OMNI-EPIC can generate an endless stream of environments (e.g., L082, L151, L424). See line 424: *\"[...] ensuring a progressive, never-ending curriculum for training\"*, note that this is a **very** strong claim. However, the experimental results are far from demonstrating this capability. Experiments show the generation of 200 environments, but considering that all are solvable (which I consider a very strong assumption). When considering the performance of RL agents in the generated environments, the authors only show a sequence of 22 tasks.  These are some suggestions to address the issue:\n\n- Use simpler alternatives to Dreamer V3 (e.g., PPO with simpler NN architectures) to reduce the computational cost of training the agent and demonstrate that the novelty and learnability of the tasks generated by OMNNI-EPIC hold for longer environment generation steps.\n\n- Reduce the claims made in the paper. Under the current experimental results, there's no real evidence that OMNI-EPIC could continue to generate novel and learnable environments for very large runs.\n\n**W2:** Although the performance of current LLMs is excellent for many coding and text generation tasks, these also present some limitations, which some relate to difficulties in generating novelty. The fact that OMNI-EPIC completely relies on these models, makes me believe that the presented method would especially suffer from the mentioned novelty issue. This weakness is related to the previous, as no evidence is shown on the capability of OMNI-EPIC to  \"endlessly\" generate novel and solvable environments.\n\n**W3:** As many deep NN models, LLMs also suffer from brittleness. As OMNI-EPIC relies on multiple LLMs working together, this makes me believe that the presented system can greatly suffer from this brittleness issue, as there is no evidence for the contrary. Further experiments on this topic would greatly help to analyze the actual capabilities and limitations of the proposed method. What is the ratio between successful and failed attempts at generating working/solvable environments? This is only an idea, but many similar experiments could be included that would significantly improve the soundness of the paper.\n\n**Minor issues:** \n- Authors mention multiple times that \"code is Turing complete\" (e.g., L93, L520). This is very vague, please be rigorous, as many types of \"code\" (i.e., programming languages) exist but not all are Turing complete. \n- Python being Turing complete does not guarantee that a foundation model (and by consequence, OMNI-EPIC) can generate any computable environment. There would be many computable environments that would have zero probability of being generated by a specific foundation model. Present rigorous arguments or remove/modify this claim.\n- I think that the \"cell coverage of archive diversity\" metric is not explained in the main text. I think that the overall clarity of Section 6 could be improved.\n- Check the coherence of the references. For example, some references include the proceedings with different styles: some use \"In Proceedings [...]\", others \"In [...]\", and others just directly mention the specific proceeding.\n\nI want to emphasize that I think that the contribution of the paper is valuable to the field and that I'm willing to update my score if the mentioned issues are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7QYNbFTCXd", "reviewer": "Reviewer_xWEx", "summary": "OMNI-EPIC is a novel framework that aims to create truly open-ended learning environments by combining foundation models with reinforcement learning. The system generates new tasks (descriptions + code) with potential applications for open-ended learning.\n\nOverall I really like the ambition of this paper, and I think it would be good to show new directions to the community. It's tackling a very important problem and proposes a sensible approach to tackle it.  This said, I think the claims made in the paper are strongly exaggerated and mostly unsupported. Below I will list ideas of follow-up experiments that could help support the claims. \n\nI feel like the computational constraints of the project associated to its ambition caused a necessary tradeoff between running the main experiments of this paper and running the sufficient set of side experiments (variations, ablations, baselines). \n\nI will assign a score of 6 as I think this paper is already valuable in its current state (if the claims were to be toned down). I'm willing to raise the score if new experiments are shown to support the claims.\n\nI updated my score in light of the discussion (see below)", "questions": "This section includes questions and comments. \n\nThe \"never-ending curriculum\" claim:\n* The approach does not train a single generalist agent to solve all tasks, why not? Was it tried and did it fail? I assume there would be issues with continual learning / catastrophic forgetting, did the authors find this was a problem?\n* Is there transfer learning across tasks? Here, there is a different agent for each new task, but an agent for a new task is initialized to the weights of the agent for the nearest solved task. This supposedly enables some form of transfer learning, but I don't think this is empirically tested anywhere? \n  * Do agents initialized this way empirically learn faster than agents trained from scratch?\n  * Is training an agent in a given task generally faster when this task is invented later, than earlier? When invented later, there should be a closer nearest neighbor and if there is transfer learning (previous point), then agents should learn faster and faster as the run progresses.  This would indicate that the overall agent (collection of agents) is actually learning something on the way (getting better at achieving tasks). \n* To show evidence for a curriculum effect driven by task generation, one would also need to show that the difficulty of tasks is increasing with time, i.e. by showing that the success rate of agents trained from scratch is decreasing as the experiment advances. In POET for instance, the latest tasks are shown to be out of reach for agents trained from scratch, is this the case here? Showing this would be strong evidence to support the \"curriculum\" claim (not sure about the \"never-ending\" bit).\n* The claim that the task generator optimizes for learning progress is also not empirically verified, is it really shifting the distribution towards harder tasks as the experiment advances (point above)? Here one would need to show a baseline that doesn't leverage the success/failure feedback or simply is not prompted to generate tasks of intermediate difficulty and compare the relative abilities of the two algorithms to solve new tasks in the end (by taking the nearest neighbors in the respective archive and training from them). Is the \"learning progress\" bit actually helping training more competent agents?\n* Since the agents don't seem to be trained again on past tasks, the archive does not contain any information about the current capabilities of the meta agent (population of agents). As a result, the task generator cannot generate \"tasks of intermediate difficulty\" for a progressing agent (it doesn't get information about the current agent). It can only generate tasks of intermediate difficulty for the \"average\" of all agents since the start of the experiment. \n* Without evidence for transfer learning across tasks, agent (population) improvement over time and task difficulty increase over time, the claim of a \"never-ending curriculum learning\" is not supported and should thus be removed.\n\nMeasuring success / performance\n* Success is measured by a VLM, but this VLM is not validated. Does it agree with people? I see human judgements in the video on the website, is there an evaluation of how good the VLM is using these ground truth feedbacks?\n* If this signal is not reliable (as suggested early in the paper), then the final metric measuring the number of different tasks solved is also not reliable. Having an unreliable fitness metric is fine during training, but becomes a problem when it's used for evaluation and algorithms comparison.\n* The \"long run without learning\" experiment is interesting to get a sense of the diversity of the tasks, but it would be good to know how many of these tasks are actually learnable? I feel like there is a high chance that while the code is executable, the task might be purely impossible to solve. Maybe this could be obtained by subsampling generated tasks and estimating the proportion of tasks that can be solved? The second experiment does test for solvability (through learning and the solved/failed label), but it would be interesting to know if the in-context learning of the task generator (prompted to generate tasks with intermediate difficulty) is actually working, ie if the ratio of solvable tasks is higher in the second versus the first condition?\n\n\nMeasuring novelty, interestingness, diversity:\n* The diversity metric in Figure 4 is computed on the simulated learning variants, which means that it's not filtered for solvable tasks and it's not clear which of these are actually solvable. I feel like we care more about the diversity of solved tasks (so the diversity of learned behaviors).\n* Is the PCA computed over all tasks from all algorithms? The discretization size is very important here and I'm not sure these results would hold for different step sizes?\n* Wouldn't sampling in context examples uniformly in the archive generate even more diversity by increasing the possibilities for recombinations?\n* Is the MoI really doing its job? In Figure 3, tasks are considered novel enough if they just change the surface form of the goal, eg \"jumping across platform\" is considered new if it's above water, or above lava, although these don't change anything to the task, only the color of the background. More generally, this model is not validated against human judgements but is here also used for algorithm evaluations. \n\nOpen-endedness claim:\n* As far as I understand, this claim is only supported by the graph in Figure 4. I appreciate the extra criterion on whether a new problem is \"interesting\" but this decision mechanism is probably quite unstable, e.g. the order in which tasks are presented probably influences the judgment of whether a new task is deemed novel. \n* Stating that we don't see it plateauing considering there are only 13 environments generated is true but also not a strong marker for open-endedness and shouldn't be used to make this claim.\n* Support for actual curriculum learning would help support the claim for open-endedness, although the \"never-ending\" aspect would probably require much longer runs of the algorithm.\n\nRelated Work: The related work mentions key research in environment design but does not acknowledge the older and wider field of intrinsic motivations, which I feel is foundational for the presented work. \n\n* Learning progress has an older history than Kanitscheider et al. LP in prediction networks dates from Schmidhuber's 91 work on curiosity (see history in https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=5508364), and was formalized in 2007 by Oudeyer and Kaplan (https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=4141061, https://pmc.ncbi.nlm.nih.gov/articles/PMC2533589/)\nAround 2013 Oudeyer's lab has been using the \"competence progress\" form, measuring progress in goal-achievement / task completion to drive exploration and skill learning (https://www.sciencedirect.com/science/article/pii/S0921889012000644 and others in 2013-2014). Since 2018, it's been using with intrinsically motivated deep RL approaches too (http://proceedings.mlr.press/v97/colas19a/colas19a.pdf, https://proceedings.neurips.cc/paper_files/paper/2019/file/b6f97e6f0fd175613910d613d574d0cb-Paper.pdf)\n* \"Focusing on interesting tasks\" is basically what intrinsic motivations are about. In Oudeyer and Kaplan's initial framework on IM, there is already the \"competence-based IM\" aiming at selecting the most useful goals for exploration and skill learning (https://pmc.ncbi.nlm.nih.gov/articles/PMC2533589/). Since 2018, it's been combined with deep RL and there is now a whole field of methods focusing on that (see review in https://arxiv.org/pdf/2012.09830). \n* VLM success detectors: https://arxiv.org/pdf/2303.07280, I don't think Christiano 2017 was a VLM? There is no language there and the model is for a single task.\n\n\n\nClarification questions and other suggestions:\n* Do the authors plan on releasing the code? This approach has many components and many parameters, probably making it hard to reproduce. Releasing the code would ensure reproducibility.\n* \"Previous attempts have resulted in pathologies when optimizing against definitions and quantifications of interestingness\" → which ones? How is the current approach overcoming these issues?\n* Are the task embeddings computed from the linguistic descriptions? the code? the agent trajectories? combinations of these?\n* It's not so clear what happens to tasks who keep on failing to compile, are they added as failed to archive? If not added to the archive, how is this information routed back to the task generator? If added as failed, shouldn't we differentiate \"failed because of compilation\" and \"failed because the agent couldn't solve it\"?\n* It would be good to show somewhere the success rates at various stages, e.g. the % of tasks that are successfully turned into executable environment code, the % of these that are accepted by the post-hoc MOI checker, the % of the latter that are successfully solved. In the long run without training, how many of the 200 iterations led to executable environment code? And how is executability checked here? By running a random policy maybe?\n* I think it would generally be good to conduct experiments for variants/ablations of the task generator:\n  * does the LP prompt actually generate tasks more tailored to the agent?\n  * does combining it with success/failure actually help generate tasks that are more solvable and not trivial?\n  * does conditioning on nearby tasks help the LP modeling and the effects above compared to sampling tasks uniformly from the archive?\n\n\nTypos\n* L17: \"narrow distributions of environment\" → +s\n* L175: \"OMNI-EPIC continuously generate and solve new, interesting tasks in simulation.\" → generates, solves", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "* The idea is original and forward-looking: this is making first steps towards the future of autotelic learning / open-ended learning by extending the space of tasks an agent can generate to drive its own learning.\n* The paper is well written and well motivated, the methods are well described with many details in the appendix.\n* Generated tasks are fun and I appreciate the accompanying website showing task descriptions and codes.", "weaknesses": "* The paper doesn't make an explicit list of contributions. Is it about showing open-ended learning? Is it about extending task generation from goals and levels to full environments? Is it about this and tailoring task distributions to the agent's learning capabilities? I would appreciate a clear statement about the contributions of the paper.\n* The paper makes a lot of strong claims about interestingness, diversity, curriculum learning and open-endedness, but I would argue that the experiments don't support most of them (see more below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)