Environment Curriculum Generation via Large Language Models
Authors: William Liang
Sam Wang
Hung-Ju Wang
Osbert Bastani
Dinesh Jayaraman
Yecheng Jason Ma
Venue: CoRL 2024 (2024)
Keywords: Large Language Models
Environment Curriculum
Quadrupeds
Sim-To-Real Reinforcement Learning
TL;DR: We use LLMs to automatically design the sim-to-real RL environment curriculum for training a quadruped to acquire parkour skills.

Abstract: Recent work has demonstrated that a promising strategy for teaching robots a wide range of complex skills is by training them on a curriculum of progressively more challenging environments. However, developing an effective curriculum of environment distributions currently requires significant expertise, which must be repeated for every new domain. Our key insight is that environments are often naturally represented as code. Thus, we probe whether effective environment curriculum design can be achieved and automated via code generation by large language models (LLM). In this paper, we introduce Eurekaverse, an unsupervised environment design algorithm that uses LLMs to sample progressively more challenging, diverse, and learnable environments for skill training. We validate Eurekaverse's effectiveness in the domain of quadrupedal parkour learning, in which a quadruped robot must traverse through a variety of obstacle courses. The automatic curriculum designed by Eurekaverse enables gradual learning of complex parkour skills in simulation and can successfully transfer to the real-world, outperforming manual training courses designed by humans.

Environment Curriculum Generation via Large Language Models
William Liang, Sam Wang, Hung-Ju Wang, Osbert Bastani, Dinesh Jayaraman', Yecheng Jason Ma GRASP Laboratory, University of Pennsylvania w wihliangsseas, wpw.n. edu
http:://eurrka--reeeachh http:t///arrree.....rccccch..hae wtt.hubii/euureaaverss.
Abstract: Recent work has demonstrated that promising strategy for teaching robots wide range of complex skills is by training them on curriculum of progressively more challenging environments. However, developing an effective curriculum of environment distributions currently requires significant expertise, which must be repeated for every new domain. Our key insight that environ- ments are often naturally represented as code. Thus, we probe whether effective environment curriculum design can be achieved and automated via code generation by large language models (LLLM). In this paper, introduce Furekaverss rse, unsupervised environment design algorithm that uses LLMs to sample progres- sively more challenging, diverse, and learnable environments for skill training. We validate Eu rekaverse's effectiveness in the domain quadrupedal parkour learning, in which a quadruped robot must traverse through a variety of obstacle courses. The automatic curriculum designed by Eurekaverse enables gradual learning of complex parkour skills in simulation and can successfully transfer the real-world, outperforming manual training courses designed by humans.
Keywords: LLMs, Curriculum Learning, Environment Design, Quadrupeds, Sim- To-Real RL
Box Climbing
Gap Crossing
Forward Ramp
Staircase
Success rate
80%
60%-
80%
80%
40%
40%
40%
40%
20%
20%
20%
70 0%
30
50
10
20
Box Height (cm)
Gap Distance (cm)
Ramp Angle
Number of Steps
Human-Designed
Eurekaverse
Figure Policies trained on Eurekaverse parkour courses significantly outperform those trained on Euman-Des gned across four distinct real-world tasks, each with a variety of difficulties.
'Equal Advising
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

x
Initial Prompt
Environment Code
Policy Training
Robot Parkour
Best
Policy
Policy Evaluation
i
Coding LLM
Evolution
Policies
Eurekaverse
Prompt
Figure 2: Eurckaverse automatically learns complex skills by performing ugent-environment co-evolution, which iterates between evolutionary environment generation and population- based policy training and evaluation.
Introduction
is often hypothesized that the evolution of intelligent motor control in biology driven by the need habituate in and adapt to varied environment conditions [1, 2]. Inspired by this observation, training robot policies over a curriculum of environments has been shown to be effective strategy for the acquisition of complex robot skills [3-7]. Designing a useful curriculum of environments is, however, time-consuming and requires domain-specific knowledge to work well [8]. In limited cases, when the environment variations can be succinctly represented by scalar parameters [9, 10], unsupervised environment design methods have shown capability in automatically generating progressively more challenging tasks in low-dimensional simulation domains [9, 11, 12]. However, whether these methods can scale up to challenging real-world robotic tasks is an open question.
Our key insight is that, in many practical robotics scenarios, environment variations can be represented by diverse programs. This allows us to solve environment design using large language models (LLMM) since they have been shown to excel at code generation in diverse domains [13-16]. We introduce Eurekaverse, a language model powered environment curriculum design algorithm that can generate and evolve environments in code space. technical level, Eurekaverse instructs an LLM generate novel environments to teach the target task, trains policies for each environment, and evaluates the best-performing policies to inform the LLM when it generates the next round of environments. Given that large language models (LLMs) have exhibited strong code generation and self-improving capabilities other domains [17-19], we hypothesize that using LLMs for environment curriculum design could also enable learning complex robot skills. In particular, we validate Eurekaverse e's efficacy on the task of quadrupedal parkour [20, 21], in which a quadruped robot is tasked with traversing diverse and unknown challenging courses with varied terrains and obstacles such as gaps, hurdles, boxes, and ramps. In order to train a policy deployable in the real world, prior parkour methods carefully designed training courses in simulation that enable real-world policy transfer [20, 21]. Because parkour courses represented as programs that specify detailed course terrain and geometry (see Fig. 3), we posit that parkour is an ideal testhed for testing Eurekaves reee''sss 's capability in handling the complexity of evolving environment programs. Parkour involves a rich variety of challenging locomotion skills so our results could also be of independent interest for scaling up locomotion learning.
We demonstrate that the automatically generated curriculum of environments from Eurekaverse leads to policy training that continuously improves over time without plateauing; in contrast, baselines that only train over fixed set of terrains, designed by either humans or an LLM, tend to overfit on these training terrains, resulting in worse generalization to test simulation and real-world courses. Specifically, on a set of 20 held-out simulation test courses that are carefully designed to holistically evaluate robot parkour skills, Burekaverse-traine policies outperforms various baselines and ablations. Furthermore, on several real-world courses, Eurekaver seveerse-trinng se-trained policies successfully transfer and exhibit more robust and adaptive behavior than policies trained using limited manually designed training courses.

---

In summary, our contributions are:
1. Eurekaverse, an I.L.-based unsupervised environment design algorithm that can automatically generate curriculums of environment programs for robot skill learning.
2. Extensive simulation and real-world validation of Burekaverse on quadrupedal robot parkour.
2 Related Work
Large Language Models for Robotics. Recent work has demonstrated that LLMs can be used in robotics domains in various ways. They have demonstrated capabilities as high-level semantic planners [22-34] and control policies [35-37]. Leveraging the code generation capability of LLMs, recent work has also explored using L.L.s to guide policy learning via reward function design [38- 41] and environment design [42-44]. In this latter category, prior work shows that LLMs can generate varied environments for top-down pick-and- place tasks that enable semantic generalization [42] or diverse tasks across different embodiments without clear intra-task dependencies [43]. In contrast, our work is the first to demonstrate how LLMs can be used in an evolutionary approach to generate full environment curricula that can guide the learning of complex physical skills such as parkour. Finally, Ma et al. [44] proposes using LLMs to automatically sample physics parameter randomization for sim-to-real policy learning [45-47]; physies randomization is a restricted class of environment design, involving selecting a value range for each physics parameter in an enumerated list. Our domain specification here is much larger, more complex, structured: it is full-fledged program. In our specific case study of robot parkour, this program specifies out of a very large space of environment geometries.
Environment Design and Curriculum Learning. Beyond using LLMs for environment program generation, environment design and curriculum learning have been extensively studied in prior literature. Prior works have considered framing environment design as multi-agent game between the environment generator and the policy [9-12, 48- 48-51]; however, these methods require manually designed environment generators and can handle only small space of environment variations (e.g.. layouts in 2D maze), limiting their scalability to challenging problems such as parkour that has large and complex design space. Curriculum learning has been used in robotics to learn complex skills [5, 52-56], but prior works use manually designed curricula or confined environment variation space that require extensive domain expertise and tuning. Our work demonstrates how scale up adaptive environment curriculum for challenging physical skills by using LLMs to automate environment generation and modification.
Problem Setting
In this section, we formalize the unsupervised environment design setting as introduced in [9]. First, we model a fully specified enviromment as a Partially Observable Markov Decision Process (POMDP). Here, an environment is tuple M = (A, R, O, T, 7),, where A is the space of actions, S the set of states, O is the set of observations, R: A R is the reward function, T S x A - (SS) the transition dynamics function, I Given M, the goal of reinforcement learning is to learn policy S A(A) that maximizes the cumulative discounted sum of rewards: V (M)) = E E..7 ET o a.)..
O the emission function, and 7 is the discount factor.
Then, POMDP template is tuple M = (A, R, R,O, where θ represents the space of environment variations that can be incorporated into the transition dynamics T S x A x e + (SS). While environment configuration can include parameter variations such as simulation physics values [44], in this work, we focus on semantic and geometric variations (c.g., different parkour terrains) that are represented as programs. Given this, an environment curriculum can be parameterized by sequence of programs Ø = (OI...., 0T) E O",, Then, in unsupervised environment design (UED), we are interested in environment generator A II - θ that generate curriculum of environments 0 for the policy E 1l to continue learning, The goal of UED is to
3

---

You are reinforcement learning engineer trying functions for quadzuped robot parkour task...
set, set_ vet_terrain (length, width, difficulty, ...:
platforn_lengtr 0.2 difficulty platform_widh no. randon, uniform(122, 1.6)
System Prompt
def set set__errteeeeeeeeeeee tergtin width, difficulty, ...): height_ = (0.engt width)]
def add add_platfozn(startt end_x, mid_y, height);
goals (BB 2))
def add_ramp(start_ end_x, mid_y, ...):
plat_length 1.0 0.3 difficulty
for in zange(6):
def add_platen aad__platfom(....:
platfozm_heiie add_ add_plattoomccur.k cuz_x + ...)
apo.zandom.uniioze.
for i in range(6): Set up 6 platforns dx, dy waw.ddo..ancinn((...) add_plas rib_ppaatforn....)
1 0: add_ rod_ram cuz_x x ...) else: cuz_x += gap_length
* Put geal in the center of the platform goals[ā+1] =
LLM Output
Environment Rendering
(platform_leng) dx) 2,
mid_y dy
return height_fieldd goals
In-context Example
Figure 3: Our prompt and in-context example (blue), example LLM response (purple), and its visualization. In the rendering, large red dots indicate goals, and the blue dot is the current goal; small dots indicate heading command (direction the goal).
learn a policy that can zero-shot generalize to unseen environments, maximizing E, V (T) for a set of unseen environments { (Mes, parameterized by O = {0 Therefore, must be able to generate both learnable and useful environments for policy learning in order to maximize generalization capability.
In this work, we investigate whether LLMs can be an effective choice of A for designing robot parkour courses for quadrupedal robots, where θ can be thought of as the space of programs that captures different terrain configurations
4 Methods
At high level, Eurekave proceeds as follows. First, it uses the L.M generate initial of environments train Then, it uses process call agent-environment co-evolution, where it iteratively uses reinforcement learning (RL) to train agents on the current set of environments, followed by updating the set of environments so they continue to challenge the best current policy without being too difficult for learning. Intuitively, as the best current policy learns to act more effectively, the environments are evolved so they can more effectively improve this policy. This strategy resembles a traditional curriculum learning pipeline, with the key difference being that the curriculum itself generated automatically by the LLM.
Throughout this section, we ground the exposition in the context of quadruped parkour to concretize the method discussion; we note that the general algorithmic principle is broadly applicable to other environment design problems. Implementation details and pseudocode are in the Appendix.
4.1 Initial Environment Generation
In order to design an effective environment curriculum, we require a mechanism for generating an initial set of diverse environments. We first query the LLM, given a description of the task and a single in-context example environment program; then, the LLM responds with an environment program 0. To obtain a diverse of training environments, we can sample the LLM multiple times with nonzero temperature. In particular, given in-context example Onnoomext, we sample environments
8g A (Omcontex).
1,..,,.NJ.. NJ,,
where N is the number of RL agents and J is the environments per agent (see Sec. 4.2 for details).

---

In quadruped parkour, this configuration 8 is an obstacle course terrain, specified as a height field 2D grid H that defines the height of coordinates along the ground plane-and list of goal coordinates C (I1, .................... (Is, Us). Both H and C are implemented as Python code and defined via for loops and Numpy operations. Due to this programmatic format, our generation procedure also includes a simple validity check for maximum height and maximum height difference thresholds to filter out obviously impossible incorrect terrains; details are in Appendix. See Figure 3 for the generation prompt, in-context example, LLM output, and environment rendering.
4.2 Co-Evolution of Agents and Environments
The initial environments can already be used for training RL agents. However, a key issue is that the LLM has zero knowledge about the RL agent's capabilities and limitations, so they may not be very useful for training. Furthermore, they are sampled i.idd., so they are unlikely to form a progressively more challenging curriculum. Indeed, as we show in our experiments, a single policy trained on these environments sub-optimal, as much of the training budget likely spent on environments that are too difficult or too easy for the agent's current capabilities.
To address this issue, we propose a co-evolution approach of environments and agents. Specifically, this process alternates between training RL agents to improve on the current set of environments, and using the LLM to evolve the set of environments to further improve the agent. This alternating strategy naturally leads the environments to form a curriculum for the corresponding RL agents. We describe both of these steps in more detail below.
Policy evolution. At the beginning of each iteration, use RL to train a population of N agents {I' { each on its own training library of J environments {0} J {0]))=1 this training hap- pens independently and in parallel. Then, we evaluate each agent on the union of all environments Oproxy Ur(0)) ) across all iterations, and select the best-performing policy x best In the first iter- ation, the RL agents are randomly initialized; in the subsequent iterations, they are initialized to bestt from the previous iteration. Thus, the policies progressively become more effective across iterations.
Intuitively, maintaining a population of agents increases the likelihood that some agents are trained on useful environments and not affected by bad environments; then, evaluating policies on the union of all environments selects the one that generalizes best to unseen environments.
Environment evolution. In the first iteration, the initial environments generated as described in Sec. 4.1 are randomly split among the N RL agents. In subsequent iterations, we create new environments by evolving those that were used train Thest [()re)] Jj=1- environments are known to be effective for training. We then use the LLM to evolve them: for each 0j in this set, we provide the LLM 0, and ask it produce variation on that environment:
) Since 7 hest is effective, these
A (0j. Omncontext)
Here, 0, serves as a previous LLM response, and our LLM prompt contains additional information including environment statistics (e.g. maximum terrain height difference) and policy training statisties (e.g. reward terms and success rate) as well as an in-context evolution example. See Appendix for prompt and details. We perform this procedure independently N times, which produces a new set of training libraries (0)) 1,, I,...,N N for the N policy training runs in the following iteration.
5 Experiments
5.1 Experimental Setup
Simulation Training. We adopt the simulation framework from Cheng et al. [21], which trains parkour policies for the Unitree Gol robot on obstacle course terrains defined by a height field and goal coordinates. During training, terrains are instantiated multiple difficulties affecting obstacle dimensions and spacing. Policies are initialized low difficulties, and they promoted demoted to higher or lower difficulties depending on the number of goals reached. This procedure gradually
5

---

introduces the policy to harder obstacles and ensures that the thousands of parallel training robots cover a wide, diverse set of courses to avoid overfitting behaviors at local minima.
Using PPO [57], we train a teacher policy that takes in privileged scandot sensing (i.e., terrain heights specific positions around the robot). To deploy, we distill the teacher into student that receives depth frames from a front-facing camera. All methods and ablations use the same reward and training setup. Additional training details are in the Appendix.
Algorithm Details. Eurekaverse uses GPT-4o as the LLM, and we run 5 iterations of generation, each with 8 parallel policy training runs of 2000 steps. To teach our policy stable walking gait (which requires reward terms not relevant to parkour, e.g., torso orientation), we first pre-train simple walking policy with 1000 policy update steps. One full run of our method takes around 24 hours on 8 A6000 GPUs, each with 48 GB VRAM, and incurs an OpenAI API cost of around $15.
Methods. We compare Eurekaverse with the Human-Desi gned environments from Cheng et al. [21], which were also designed for a generalist quadruped parkour agent. We emphasize that this comparison does not take into account the time and training steps required to first design Human-Des? gned optimally, whereas Eurekaverse must perform both environment design and policy training within the allocated steps. Additionally, compare Random baseline that places randomly positioned ramps and boxes; these obstacles are sized randomly between half and double the bounding box of the quadruped, making them feasible for a well-trained policy. Finally, we consider an Oracle policy trained directly on the testing environment. Random serves as whether environment design is actually necessary- and Oracle serves as an upper bound we have access the true testing environment and can directly train on it.
lower
Ablations. We also compare against various ablations to probe the importance of our algorithm components. First, we run variant that always asks the LLM to make the environments more chal- lenging, without providing policy performance feedback (No Feedback). Next, we consider training a policy for the same number of steps on only the initial set of LLM-generated environments (Initial Envs), the set environments the final iteration (Final Envs), and set of environments where each terrain is generated sequentially, conditioned on mreviousl-ggeneet terrains, to maximize diversity (Diversity Only). Finally, we train a policy on the in-context example provided to the LLM (LLM Example).
5.2 Simulated Parkour Benchmark
To compare the zero-shot generalization performance of different policies in simulation, we design parkour benchmark suite consisting of 20 diverse parkour tasks as our testing environments. These tasks are variations of real world obstacles seen in prior robot parkour learning works, such as box climbing, forward ramps, sideways ramps, A-frames, box jumps, stepping stones, staircases, narrow passages, agility poles, and balance beam [21, 58-63J. Note that these benchmark environments independent from the numan-Desiion generation, holistically testing all methods' generalization capability. For each of the 20 tasks, we instantiate 10 versions scaled by difficulty. The benchmark is visualized in the Appendix.
ones and not revealed to the LLM for environment
5.3 Simulation Experiments Results
In Figure 4 (left), we report the benchmark performance of Eurekaverse, Human Hwwan-Desiignneeeee Designed, and Random across training steps as well as Oracle' e's best performance. Figure 4 (right), we compare our method's final performance with that of various ablations. Additionally, in Figure we compare Eurekaverse" iterations with Human-Des gned on each benchmark obstacle.
Environment design is necessary for parkour. First, we see that the Random baseline does not increase performance on the benchmark, indicating fails to learn in randomly-generated environments; thus, careful environment design is needed to learn complex quadruped parkour skills.
Eurekaverse outperforms human-designed environments. On average, method's final policy achieves nearly 2 additional goals over the policy trained on human-designed environments.

---

Method Experiments
Method Ablations
Mean Benchimad Goals Reached
*
2-
Ok
Ours
Liversity
Number Parallel Steps
Example
Frus
Fendback
Only
Ours
HumannDesigged
Random
Oracle
Figure 4: Comparing sim benchmark performance training steps for Eurekaverse and baselines (left), and final benchmark performance for Burekaverse and ablations (right). The training curve for ablations in Appendix. Experiments are run over seeds.
Climb On And Box
Forward Ramp Lips
Sphere Bump Lips
Forward Remp No Lips
Flush A Freme
Number Benchmork Stals Reached
Sphere Bump
Box jump Even
Jump Uneven
jump
Bump jump
Sideways Ramp
Stepping Stones Cylinder
Stepping Stones Rectangle
Staircase Welking
Staircase Walking Narrow
Staircase Climbing
Staircase Spiral
Squeeze
Agility Poles
Balance Beam
Difficulty Level
Human-Designed
Figure 5: Comparison of Eurekaverse's iterations against Human Ceaman-Desiiee ined, visualized per obstacle type and over each difficulty (casiest to hardest) on benchmark. Higher area under the curve is better.
The latter lears quickly but plateaus; intuitively, though well-designed, the relatively small number of distinct terrains limits the capacity for generalization. On the other hand, while our policy learns slower in the first few iterations due to imperfect initial environments, it exhibits a stronger upward trend since the environments are iteratively adapted to the current best policy, allowing the policy to continuously improve. Similarly, we see in Figure (right) that Eurekaverse surpasses LLM Example, the caraand in-context example for the LLM queries; while the well-designed example teaches the policy succeed in specific subset of benchmark tasks, the LLM creatively construct more diverse array of environments, thereby teaching more generalist and performant policy.
it to
Eurekaverse nearly matches the oracle. After iterations of design, our policy nears the performance of Oracle Figure 4, despite not receiving any information about the benchmark.
Eurekaverse generates curriculum for continuous learning. From Figure 4 (right), that Initial Envs and Final Envs achieve subpar results. Thus, neither teaches the policy complete set of parkour skills; the former was generated without information on the policy's 's training progress, and the latter is tuned for policy that has already learned skills previous iterations, making unsuitable for a beginner policy. Additionally, Diversity Only underperforms, showing that diverse environments alone, without an adaptive curriculum, are insufficient for optimal performance. No Feedback is also inferior; though improves similarly to Eurekaverse in the initial iterations. policy performance degrades the evolved terrains fail to match the policy's capabilities (see Appendix). Thus, curriculum tailored to policy training is necessary, rather than one that simply

---

increases difficulty. By building a curriculum that bridges the initial and final environments while increasing diversity and taking into account policy feedback, Eurekaverse trains a policy that gradually improves over each iteration; we see evidence this in Figure 5, where our policy generally improves at each benchmark obstacle type.
5.4 Real- World Experiments
Robot Deployment Details. We deploy on the Unitree Gol, quadrupedal robot with 12 degrees of freedom. Note that the Gol weighs 12kg and is equipped with motors that have maximum torque of 23.70 Nm, which is significantly lower than prior work [20, 21] with the A1, whose motors reach 33,50 Nm and also weighs 12kg. Additional details are in Appendix.
Real- World Tasks. Figure shows Eurekaverse's and Deraaan-Deiinnnn box climbing, gap crossing, forward ramp, and stairs tasks, four representative tasks in our simulation benchmark as well prior works [20, 21, 59, 60]. Box climbing involves getting on top of box of varying heights, and gap crossing involves jumping from one platform to another. In stairs, the robot must traverse as many steps of stairs as possible, and in forward ramp, the robot must step onto and go across a ramp, with difficulty determining the ramp angle.
performance on
We deploy distilled policies from Eurekaverse and Human-Des gned and compare their success rates across the four tasks, each of which vary in difficulty corresponding to the key obstacle property. In box climbing, gap crossing, and forward ramp, perform 10 trials for each difficulty; in stairs, we perform 10 trials total and measure the number of steps traversed. For each obstacle, we initialize the robot at preset locations with varying distance to the obstacle.
Results.
In all tasks, Eurekaverse generally outperforms Human-Dest gned across difficulties, succeed- ing at jumps up to 75cm (above the Gol length), climbing up over 50em (above the Gol height), walking up 30 degree ramp, and traversing the entire staircase. Our policy is also much more safe and stable than Sluman-Designe in the forward ramp task, DemmannDeeignee incurred physical damages on our robot hardware, so we refrained from completing the remaining trials (dotted line in Figure 1).
which often trips controller's motor power protection fault;
We also observe Eurekaverse exhibiting recovery behavior, where the policy quickly reacts to slipping or misplaced feet; for example, if it misses a jump, it quickly kicks up its hind legs, allowing the front feet grab more stable footing and climb up. Moreover, in the staircase trials, the policy is robust to seeing chairs and railings on either side, despite never seeing similar features in simulation. We hypothesize that this behavior is the result of two mechanisms: during training, policies fail but have a chance to recover before the environment resets, and during depth distillation, random blackout and noise augmentation increases robustness against out-of out-of-disttibution depth readings.
Conclusion
We have presented Eurekaverse, a novel method for automated environment curriculum design with large language models (LLMs). Eurekaverse generates effective environment programs that adapts the policy's current training progress, enabling it to learn new skills and hone existing ones. We validate our method quadruped parkour, where our policy outperforms prior human- designed ones in both zero-shot simulation and real environments. We believe that Eurekaverse demonstrates the potential for LLMs to generate an infinite variation of environments, enabling continuous policy learning and serving as path toward truly open-ended, generalist robot agents.
Limitations. Our system requires moderate amount of LLM samples to consistently generate and select valuable environments; important future direction necessary for scaling up is increasing sample efficiency, for example by fine-tuning the LLM. Additionally, our evolution prompt only uses textual feedback. A potential extension can additionally provide environment visualizations, leveraging recent multimodal foundation models to potentially perform stronger spatial reasoning.

---

6.1 Acknowledgements
This project was supported in part by NSF CAREER Award 2239301, NSF Award 2331783, and ONR award N00014-22-12677 12677. We would like to thank Ge Yang, Gabriel Margolis, Alan Yu, and Yajvan Ravan for their insights and help in setting up and tuning sim-to-real transfer and deployment infrastructure. We would also like thank Unitree Roboties (Wen Shuo and Tony Yang) for their support in repairing our robot.
References
[1] D. Floreano, Husbands, and Nolfi. Evolutionary robotics. Handbook of robotics, 2008.
[2] A. Gupta, S. Savarese, S. Ganguli, and L. Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1))5721, 2021.
[3] T. Bansal, J. Pachocki, S. Sidor, I. Sutskever, and I. Mordatch. Emergent complexity via multi-agent competition, 2018.
[4] B. Baker, I. Kanitscheider, Markov, Y. Wu, G. Powell, B. McGrew, and Mordatch. Emergent tool from multi-agent autocurricula, 2020.
[5] Z. Tang, D. Kim, and S. Ha. Learning agile motor skills on quadrupedal robots using curriculum learning. International Conference on Robot Intelligence Technology and Applications, volume 3, 2021.
[6] B. Tidd, N. Hudson, and A. Cosgun. Guided curriculum learning for walking over complex terrain, 2021.
[7] L. Marzari, D. Corsi, E. Marchesini, and A. Farinelli. Curriculum learning for safe mapless navigation. In Proceedings of the 37th ACM/SIGAPP Symposium Applied Computing, SAC '22. ACM, Apr. 2022. 1145/34773 4. 11 33071827
URL http: //x doi. ottp:///.....o.......//o///o/1/. wwg/oo.....ooooooooooooooo.....l
[8] H. Duan, A. Malik, M. S. Gadde, J. Dao, A. Fern, and J. Hurst. Learning dynamic bipedal walking across stepping stones, 2022.
[9] M. Dennis, Jaques, Vinitsky, A. Bayen, Russell, Critch, and Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 13:19, 2020.
[10] M. Jiang, M. Dennis, J. Parker-Holder, J. Foerster, E. Grefenstette, and T. Rocktäschel. Replay- guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1884-1897, 2021.
[11] M. Jiang, E. Grefenstette, and T. Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pages 4950. PMLR, 2021.
[12] R. Wang, J. Lehman, J. Clune, and K. O. Stanley, Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint
2019.
[13] B. Rozière, J. Gehring, Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve, Code Ilama: Open foundation models for code, 2024.
[14] Y. Li, D. Choi, Chung, N. Kushman, J. Schrittwieser, R. Leblond, Eccles, Keeling, F. Gimeno, A. Dal Lago, T. Hubert, P. Choy, C. de Masson d' Aututme I. Babuschkin, X. Chen, P.-
S. H.ang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. Sutherland Robson,

---

P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals. Competition-leve code generation with alphacode. Science, 378(6624)) Dec. 2022. ISSN 1095-9203. doi: 126/ science.abq 158. URL
doi. org/ 10..011.01111 1126/ science. 1
[15] E. Nijkamp, B. Pang. H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model code with multi-turn program synthesis, 2023.
[16] A. Badagabettu, S. S. Yarlagadda, and A. B. Farimani. Query2cad: Generating cad models using natural language queries, 2024.
[17] A. Patel, M. Hofmarcher, C. Leoveanu-Condrss M.-C. Dinu, C. Callison- and Hochre- iter. Large language models can self-improve at web agent tasks, 2024.
[18] Chen, Y. Deng, H. Yuan, Ji, and Q. Gu. Self-play fine-tuning converts weak language models strong language models, 2024.
[19] Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han. Large language models can self-improve, 2022.
[20] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning, arXiv preprint arXiv:2309.05665 2023.
[21] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint arXiv:2309. 14341, 2023.
[22] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169/ 2022.
[23] W. Huang, F. Xia, D. Shah, D. Driess, A. Zeng, Y. Lu, P. Florence, I. Mordatch, S. Levine, K. Hausman, et al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprint urXiv:2303.0085: 2023.
[24] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 11523-11530. IEEE, 2023.
[25] J. Zhang, J. Zhang, K. Pertsch, 7. Liu, X. Ren, M. Chang, S.-H. Sun, and Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. arXiv preprint 1011:2310.10021, 2023.
[26] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. 7eng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9490-9500. IEEE, 2023.
[27] H. Wang, G. Gonzalez-Pumariega, Y. Sharma, and S. Choudhury. Demo2code: From sum- marizing demonstrations to synthesizing code via extended chain of-thuugtt. arXiv preprint arXiv:2305.16744 16744, 2023.
[28] Huang, Z. Jiang, H. Dong. Qiao, Gao, and H. Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305.11176. 2023.
[29] G. Wang, Y. Xie, Y. Jiang, Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models, arXiv preprint arXiv:2305.1622 2023.
10

---

[30] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone. Lim+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304. 1477, 2023.
[31] T. Silver, S. Dan, K. Srinivas, J. B. Tenenbaum, L. P. Kaelbling, and M. Katz. Generalized plan- ning in pddl domains with pretrained large language models. arXiv preprint arXiv:2305. 1014, 2023.
[32] Y. Ding, X. Zhang, C. Paxton, and S. Zhang. Task and motion planning with large language models for object rearrangement. arXiv preprint arXiv:2303.06247, 2023.
[33] K. Lin, C. Agia, Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions feasible plans. arXiv preprint arXiv:2303.12153, 2023.
[34] Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh. Translating natural language to planning goals with Harge-langagge models. arXiv preprint arXiv:23022.51128, 2023.
[35] A. Brohan, N. Brown, J. Carbajal, Y. Chehotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-actinn models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 2023.
[36] A. Szot, M. Schwarzer, H. Agrawal, B. Mazoure, W. Talbott, K. Metcalf, N. Mackraz, D. Hjelm, and A. Toshev. Large language models as generalizable policies for embodied tasks. arXiv preprint 2rX11:2310.17722, 2023.
[37] Y. Tang, W. Yu, J. Tan, H. Zen, A. Faust, and T. Harada. Saytap: Language to quadrupedal locomotion. arXiv preprint arXiv:2306.07580, 2023.
[38] H. Ha, P. Florence, and Song. Scaling up and distilling down: Language-guided robot skill acquisition. arXiv preprint arXiv:2307.14533 2023.
[39] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, Erez, L. Hasenclever, Humplik, et Language rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08647 2023.
[40] T. Xie, S. Zhao, C. H. Wu, Y. Liu, Q. Luo, V. Zhong, Y. Yang, and T. Yu. Text2reward: Automated dense reward function generation for reinforcement learning. arXiv preprint Stti:2309.11489) 2023.
[41] Y. J. Ma. W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Eureka: Human- level reward design via coding large language models. arXiv preprint ar arXiv:2310.12911, 2023.
[42] L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang. Gensim: Generating robotic simulation tasks via large language models. arXiv preprint 1rXiv22310.0131, 2023.
[43] Y. Wang, Z. Xian, F. Chen, T.-H. Wang. Y. Wang. K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:231..0144 2023.
[44] Y. J. Ma, W. Liang, H. Wang. Wang, Zhu, Fan, Bastani, and Jayaraman. Dreureka: Language model guided sim-to-real transfer. In Roboties: Science and Systems (RSS), 2024.
[45] Tobin, R. Fong, A. Ray, Schneider, W. Zaremba, and Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RS/ international conference intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
11

---

[46] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
[47] Muratore, Ramos, G. Turk, W. Yu, M. Gienger, and Peters. Robot learning from randomized simulations: A review. Frontiers Robotics and Al, page 31, 2022.
[48] R. Wang. J. Lehman, A. Rawal, J. Zhi, Y. Li, J. Clune, and K. Stanley. Enhanced poet: Open- ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning, pages 9940-9950 PMLR, 2020.
[49] Parker-Holder, M. Jiang, M. Dennis, M. Samvelyan, Foerster, Grefenstette, and
T. Rocktäschel. Evolving curricula with regret-based environment design. In International Conference Machine Learning, pages 17473-17498. PMLR. 2022.
[50] V. Bhatt, B. Tjanaka, M. Fontaine, and S. Nikolaidis. Deep surrogate assisted generation of environments. Advances in Neural Information Processing Systems, 35:37762-37777, 2022.
[51] Mediratta, M. Jiang, Parker- Holder, M. Dennis, E. Vinitsky, and Rocktäschel. Stabilizing unsupervised environment design with learned adversary. arXiv preprint arXiv:2308. 10797, 2023.
[52] C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning, pages 482-495. PMLR, 2017.
[53] C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528. PMLR, 2018.
[54] G. B. Margolis, G. Yang, K. Paigwar. T. Chen, and P. Agrawal. Rapid locomotion via reinforce- ment learning, arXiv preprint arXiv:2205.0288 2022.
[55] Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, McGrew, Petron, Paino, M. Plappert, Powell, R. Ribas, al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 2019.
(56) F. Kaufmann, L.. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982- 987, 2023. doi: 10. 1038/00311111 586-023-06419-4, s41586- 02:30223--2 6661-49444
URL
[57] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: trXiv:1707.06347 2017.
[58] R. Yang, G. Yang, and X. Wang. Neural volumetric memory for visual locomotion control. In Conference on Computer Vision and Pattern Recognition 2023, 2023. URL https:/ openreview. n forum deruumidd=yye id=JJYWCCcmDDS.
[59] K. Caluwaerts, A. Iscen, C. Kew, W. Yu, T. Zhang, D. Freeman, K.-H. Lee, L. Lee, S. Sal- iceti, V. Zhuang, N. Batchelor, S. Bohez, F. Casarini, J. E. Chen, O. Cortes, E. Coumans, A. Dostmohamed, G. Dulac-Arold, A. Escontrela, E. Frey, R. Hafner, D. Jain, B. Jyenis, Y. Kuang, E. Lee, L. Luu, O. Nachum, K. Oslund, J. Powell, D. Reyes, F. Romano, F. Sadeghi, R. Sloat, B. Tabanpour, D. Zheng, M. Neunert, R. Hadsell, N. Heess, Nori, J. Seto, C. Parada, Benchmarking animal-level agility with
V. Sindhwani, V. Vanhoucke, and Tan. Barkour:
quadruped robots, 2023.
[60] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision, 2022.
12

---

[61] D. Hoeller, N. Rudin, D. Sako, and M. Hutter. Anymal parkour: Learning agile navigation for quadrupedal robots. Science Robotics, 988)eaddi7566, 2024. adi7566. URL https:/ http:::/wwww.sceenee. /tw.scc.eece..or//o///aaaaa/ee// http:////..../////ooooo..c. adi 6.66
[62] A. Loquercio, A. Kumar, and J. Malik. Learning visual locomotion with cross-modal sunperiooissuprriissson In IEEE International Conference on Robolics and Aulomation (ICRA), pages 7295-7302. IEEE, 2023.
[63] Y. Cheng, H. Liu, G. Pan, L. Yc, H. Liu, and B. Liang. Quadruped robot traversing 3d complex environments. In arXiv preprint aBXiv:TBD, 2024.
[64] Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: Learning unified policy for manipulation and locomotion, 2022.
[65] M. Lopes, T. Lang, M. Toussaint, and P.-Y. Oudeyer. Exploration in model-based reinforcement learning by empirically estimating learning progress, Advances neural information processing systems, 25, 2012.
[66] I. Kanitscheider, J. Huizinga, D. Farhi, W. H. Guss, B. Houghton, R. Sampedro, P. Zhokhov, B. Baker, A. Ecoffet, J. Tang, O. Klimov, and J. Clune. Multi-task curriculum learning in complex, visual, hard-exploration domain: Minecraft, 2021.
[67] S. Forestier, R. Portelas, Y. Mollard, and P.-Y. Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. Journal of Machine Learning Research, 23(152): 1-41, 2022.
13

---

A Appendix
A.1 Algorithm Details
We provide additional implementation details for Eurekaverse below. We first describe the core algorithm loop in pseudocode. Next, describe "soft" policy selection, more robust way select the best policy using evaluation the proxy (all generated training) environments.
Algorithm Eurekaverse
1: Require: RL algorithm A, coding LLM LLM, performance criteria F, LP transformation G
2: // Generate initial environments
3: 83 ALLM (Omnomtexs)
4: // Run iterationa of co-evolution
5: for generation iteration / = to T do
6: // Train multiple policies environments
7:
for
to M do T; = ((1)))=2)
8:
9:
// Const ruct evaluation envirenment and select best policy
10:
Praxy = U: 03(0) across iterations
11:
Tbest arg max=, F Preay)
// Evolve ve envi ronments
13: Oo ~ Af1M (0)rm)
14:
Output: best final policy π
best
Soft Selection. During co-evolution, we select the best policy π best via performance all generated environments proxy, which tests the policies' generalization and serves proxy for the true testing environment. Empirically, we find that this proxy accurately orders the policies, where the best policy in Praxy also performs the best in our simulation benchmark. Nonetheless, there are rare cases where the best-performing benchmark policy is not the best one chosen by the proxy. To increase robustness against these inaccuracies, we "softly" select the best policy during each iteration of co-evolution: the selected policy is chosen randomly, where the best policy has probability P1, second-best has probability P2 Pl, and so on. Each agent in the next iteration performs this selection independently (so that N are initialized the best policy, Np2 are initialized to the second-best, etc.). In practice, select P1 0.75, P2 0.25, P3. ........ PN 0.
A.2 Prompts and Examples
are a reinforcement learning engineer trying to write environmens functions as effectively possible for quadruped robot parkour task, Please keep in mind that this robot will trainad stmulation and daployed in real obstacle course, 50 want the obstacles to be realistic and challenging.
so, you should perform
following steps:
1. Cazefully read the preblem statement, specifications, and tips below.
Explain what your obstacle course will look like. Keep
your should be relative to the quadruped's siza, and the entire cbstacle should fit within the terrain bounds. You should be creative when designing your obstacles, and You may drew inspiration from features
in dog packs, playgrounde, and urban environmente. 3. Write function that generazes the obstacle course according to your plan.
Prcblen Statement:
are given quadruped robct
to navigate through an cbstacle course The floor of the course is represented as 2n numpy array, height_field, where each element represents the height of the ground at that point, in meters. To direct the resca in order. The geal
space.
quadruped, the course has goals that the quadruped locations should be stered in a 20 numpy array, geals, as ix, y) indices in hield.
To create your course, you will write function set pes_terrainnlengtt width, field_resclution returns the height_field and geals arrays. The function takes in the length and widtr of the course, in meters, the scale of crantisation for the obstacle course, in meters, and the difficulty of the obstacle course, a float between (easiest) and 1 (hardest) inclusive. Please follew the template below write your function:
difficulty! that creates
"pyyhhrr.

---

import numpy np import random
def set (Lenggtrain width, field_casolution, difficulty): """pescriptio your course the tested skill,
def iLto_idx (m)
"*"Converts
quantized indices. **
return np, round in / field_reso uon) yastype (np.1nt16) 1" not risinatance (n, (ist) in m]
isinscance tuple)) else [zound(1 resoltton))))
height_ ne.2eros im_co_idx (Length), m_to (dx (width)) goals sp.zeros 21)
Your code here
return height__reidd goals
You any function from the numpy and rendom libraries well any in-buils Python functions. please write everything as Python code and annotate your code with commente, iroluding ona-lire docstring after the function dafinition that summarizes your obstacle course.
rovment imeolcicaiions
1. height_field quantizes the ground glane into a grid of points, resoluueno 0.05, in y) which represents the distance
is the quantization of the ground's betwaen adjacent points in the height_ field array. helght_rielo.aaapp
fie e_resolution, 1 / field_ = (240, 80). Please make suze that your obstaule course apans exactly
ground
meters,
dimensions.
3. The quadruped's standing size (length, width, height) is 0.645 0.28 x 0.4 meters. Keep these dimensions in mind when designing the size your obstacles. (1, width
4. The quadruped will spaın with its center (x, placa obstacles at indices with x >= field_resolution to avoid the quadruped apawning inside obstaole.
meters.
5. Please make sure that your obstacles have a width of at leas: i meter. Hlwever, in rare cases, will
narrow obstacles snaller, as long the quadruped is not expected to climb
length
down.
6. The goal coordinates should be within the bounds of the course. Even if you have fewer than all goal positicns in the goals array.
8 chstacles, you
1. The quadruped will be instructed to travel in a straight line from ane goal to the rext. Thus, you want
quadruped Lurn, you 8. Your course should test particuler one of the robot's skills, for example by repeating single type obstacle,
place goal
tuzming point.
course relatively consistent throughout. There will other coursea to test other skil is and different cbstacles.
The heights in field_height can be negative. the quadruped's spawning area will always be meters,
nagative heights create pits gapa in the cerrain, Uae this to force the quadzuped to walx or jump across the tops of your obstscles without climbing up
height
you
10. since each index in the terrain has one height value, it is impossible to create overhangs your course.
tunnels.
try to create thase
Tips:
your function. only write the function definition the code inaide 't. 2. To broadcast 1D array to 20 ariay (or 20 slicet, you mus: first add new axis to the
array.
3. You should not add noise directly to height_ field. sandom naisy tesrain is not necessary not want as an obstacle.
for our task, and
4. When slicing into beight_[ield, make suze to convert from mecers quantized units beforehand. For example, a metar alice looks like [x (dx m_x y m_to_ida (tt__id(() m_to_idx(11]
5. If you write helper functiens, please make them nested functions of se: se:_terzein. Your response
self-centained function definition following our templete.
Listing 1: System Prompt. In the function template, we provide a conversion function m to idxx to simplify the conversion from coordinates, in meters, to the index in the array.
The following is an example a terrain generation fonction, Please zeference the example provided, but make your terrain different.
import nampy
def set set_terrain ength, width, pelldresolution difficulty): """Mlliiple platforms Lzaversing a pit for the zobot Lo climb on and junp across,
def (dx
*#"Converts guantized indices. *
15

---

ap.reundli ficld_resolution) (np. if (isinstance (a, List) isinacanca (m, tuplel) clse [round (1 fieldd_esslluionn) for in m)
height_ffeeld zeroa in__ ( (length), (dto_idx(iidth)) goals ne.zeroa (8, 21)
Set up platform dimensiens make the platform height
minimum difficulty the quadruped learn
climb up
plat.oos__ennghh
difficulty
platfom_lenggth m_to_ldx (alattotoooooeemee platfors_width np.random. formol 1.6)
platfemmwvitth: a_te_idx (attpppaooorwwiittt
platfom_height__ma. blatfor__heihht.ma 0.0 0.2 difficulty, 0.05 + 0.25
difficulty
gap_length 0.1 0,7 circiculty gep_length (Lto__dx(fep_eeggg
mid_y iLto_idx(widthi
def add_platform(otrtt..t end_x, mid_y):
half_width platform_width x1, x2 atart_x, end_x
yi, y2 midLy half_width, mióLy widch betffom__neggh height_ yl:y2] piatform_height
plactorn_height_me
dx_min, mx_ = -0.1, 3.1 dx_min, dy_min, dy_max - 0.4, 3.4 dy_min, dy_max muto [_te_idk(dd__mnn) (dy_min),
n_to_ldx (dx_nin), m_to_idx
Set spawn area to flat ground spavr_length n_to_idx(2) beight_fit field[(:spamn langth, :) first goal spawn
goals|0; = [spewn_length nLtc_idx((0.0 midLy]
Set remcining area to be pit Ne do this to force the robot to jump from platform to platferm Otherwise, the robct just jump down and climb back up hef
!! -1.0
cur_x spasn_Length
zangel6)
up platforms
dx = mp..on.,om randint (dx_min, 6x_max) dy = reg.random. (dy_min, dy_max) add_plaatformeurr. eur_x platformllengt mid_y
Put goal goala(i+1] = [eur_x + (platfom__ength dx) / 2, mid_y dy]
platform
# Add cur_x += platforn_length dx gap_length
Add final goal behind the last corm, fill in the zemaining gap goals|-i] [cur_x auto_idx 50,5), mid_yl
deights_feeddeeur___
return beight_rieid, goala
Listing Initial Example. This example demonstrates main structure a terrain function, including helper functions and for loop over multiple obstacles.
trained a quadruped pelicy perform parkour on chstacle ecurse created by generation oode above (aa well as others), and we tracked the va values of individuzl reward componenta as well as other netries auch as the number of goals reached fout of 8), episode length, and the rate edge vielstiens Ifeet too close edge obstacles): <LNSERE FOLLCY SLArISTICS HERE>
We have slso computed statistics fer the terrein height of the direcs path between goals acress multiple difficultiss. Nate that approximates the locations an optimal quadruped would traverse. The itatistios de not reflect the actual path Laken by the quadcuped policy, nor does it include the height gaps, pits, and other obstacles that the quadruped would <INSERE TERRAIN STATISTICS SERE>
hauristical shortest-path between goals
16

---

Please cerefully analyze the statisties above and provide
improved generation function. You should pay ottention to which parta of the course the quadruped scccassfully learned balance difficulty feasibility for the quadraped rabot, allowing it to learn and perform better. thus, if certain obstacle goal, you should consider changing or cemoving it, You should also follow the guidelines below:
which parta striggled
goal your
the robot is gecting stuck
the number of reached goals is over 808, please create a harder course while ensuring obstacles,
that it is feasible and safe for real robot. incresaing the complexity of the course, and increasing ng the difficul ty of exlating junping yaps wider.
should consider adding
climbing obstacles If the number of reached goala la below 204, p please create an easier course by decreasing the difficulty existing obstacles or simplifying the course layout. For example, you junping geps asrrower. Please also double check
obstacles.
example, you
make elirbing obstacles shorter that the course obstacles are fair and feasible for the quadruped.
otherwise, please
with the difficulty but different obstacles. If tae quadruped seens stuck on certain obstacle, please change it
variation the
Again, please be creative when designing your course as we want to provide diverse set of ssaining eaviroaments for the quadruped. Hlere list of the courses and skills that the quzdruped was already trained on. You may use then as inspiration, but please make yours la different:
<INSERT TERRAIN DESCRIPTIONS HERE>
Please use the template for the course generation function and provide detailed reasoning of the changes you made. The function signature should zemain the same.
Listing 3: Evolution Prompt. Here, insert policy training statistics including reward term values, episode length, number of goals reached, and number of edge violations (feet too close to edge) before and after training. We also compute height field (terrain) statisties, including maximum value, maximum difference between consecutive indices, and standard deviation. Finally, we provide the LLM with doestrings of previous terrains used for training.
The following is an example of an initial terrain generation function.
(SAMB INITIAL EXAMPLE)
And the lowlowing is an example of new, improved terrain genaration on functioon that has more complex obatacles. You can reference the example provided, but please make your tecrain illustrate haw terraing ba made complex,
diffarent. This example you should tune the difficulty according to the previous mott.ss.
import numpy 25 np
def set ter_ateriin (length, width, field_serolution, difficulty): *" sidexays- rampa traversing for the robot flbmbb and jump across.
def m_ta_idx (m)
"engonverts meters to quantized ind' ces.
return roundin/ / [ield_reaolution .astype top. int1)) 15 nos risinstance (m, List) isinscence (m, tuplel) else [zound(1 feedd_resolution) for in
Height_fiell ap. zeros in goals ze.zeros (8, 21)
idto_idxwwidth)))
Set up foatform and ramp dinenatons Ne make the platform height neaz 0 at minima difficulty so the quadzuped leara to climb up
plaiIoos_length
difficulty
platforn_length - m_to_ (pto__idb
platfesm_widt:: ra.random, fniform(....
Jecrease platform widch
platform_wiath m_tt__ddeeeeooo aopllttorm__idde platfora_heiggt_mmb,, platform_height_nm
D.C + 0.2 ditriculty, 0.05 + 0.25
difficulty
remp_height_sin, ramp herg_height_aax 0.0 0.5 difficulty, 0.05 + 0.55 difficulty difficulty Decrease gap length
gep_length
gap_length idxp__i
mid_y m_to_idx(wiidtii //
def add sdd_piatform(ssrttt.. end_x, mig_y):
waltwwithh wistformwiithh // x1, x2 start_x, ead
17

---

yi, y2 mid_y half_width, mid_y half_width cenfformhhiighe np. ra.dommm hommplattoromhhighttttt.... teeg y1:y2] platform_heightt
der add ranpraattttart and.x, rid_y, direction);
hell_width platfozm_width // xi, xi start_x, end_x yt, y2 mid_y half_width, mic_y half_width ramp_ homp_height np.random.in (ramp, (ramp_helght__mmg ramp_height heighttthttaaa max) ip..inepace(0, ramp_height, nummy2-y11 [srdicection] slant = alant [None, teight_fteld(M]=kk, yl:y2] slant
Add dimension for breadcasting to
di-min, dx -0.1, 0.1 dx_min, dy dy_min, dy_max = 0.0, 0.4 & Polasity of dy will alternate instead of being randon dy_min, dy_max a_te (dy_nint, i_to_idx
m_to_idx min_ m_to_idx (dx_max)
set spawn area to flat ground spagr_lennth (Lto_idx(2) height_ (irddffeed[[::paam first spassn :] =
spawn
goals|3] [epawn_ m_to_idk (0.55), mid_y]
Set remaining area Ne do this to force the robot junp from platform to platform Otherwise, the robet just jump down and climb back up height_ffelld (pw.fccddddonnnnnnnntt. !] -1.0
pit
Add first platform
cur_x spawn length
dx np.rendom. randint (dx_min, dx_max)
dy ap-.endom. randint (dy_min, dy_nax) add_platform
platform_length dx, mid_y dy)
goals[1] = [our_x (plattfomm_aane cur_x leantoom_lennga
аx) / 2, mid dy]
gap_length
sange!l, 6):
up ramps
dx = rg.raanom. (dx_min, 6x_max)
dy = mp. iap.random. (dy_ming 6y_max)
diraction (-1) $ Alternate left and right ramps
dy = dy direction Alternate positive and regative offaets
ramp (cur_x, cur_x platform dx, mid_y dy, direction)
# put goal in the center of the ramp goals(it1] = [eur_x + (platfom__ennth dx) / 2, mid dy)
Add gap
cuc_x platforn_length
gap_length.
samp, goals|-1] = [cur_x m_to_idxid.s) 10.3), mid_y] height_fiode fiogd___ffllddcr_x. :] 0
goel behind
remaining gap
return height_Field, guals
Listing 4: Evolution Example. The initial example is exactly the initial example above, and we provide an example of evolving it; in our example, replace some flat platforms with slanted ramps.
Herre's allernstive Lerrain generation function that Iocuses creating sezies of ramps for the quadruped robot to navigate through. This course will test the robot's ability to maintain balance.
handle inclined
"pythor.
import numpy
inport random
def set set_terrain width, field_cesolution, difficulty): *"*Ranos varying angles the clinb descend, a**
m_ta_idx (m):
"*"Converts meters to quant sed indicess **a
seturn roundun field_resolutinn aastype (np. int16) 15 noz fisinstanceds (m, List) or isinssence tuplel) else [round(i / resouuist)) for in m]
height_ field zeros in=so_idx (length), muto !
18

---

np.zeros 21)
Initialise parametera
ramp_ = 1.0 difficulty. 2.0 # Make rempa langer and steeper with higher
sifficulty
ranp_length m_to_idx lengg)))
remp_height 0.1 difficulty 3.4 # Remp height varies with difficulty ramp_height_cheng ramp_height ramp_length seight change per quantized step ramp_width 1.0 & set ramp width to meter
ramp_width (it__idx(ramp_vitth
mid_y m_too
def 2dd remp X, endxx, midyy, upward=irue):
"*"add remp either upиaгd cowmard starting from start_x.""" for in range (start_,,__, ene_x):
height harp_hetght_chaap
start_x) :amp hampp_hiightchhang
upward (end_x
height-[ielddi, midLy ramp_width 2: midyy ramp_width // 2] height if not upward:
mid_y ramp_width // 2: mid_y ramp_widch // 2
rarp_height Adjust for downward ramp
min, dx_min, 6x_max m_te_idk (dx_nin), (_krind m_to_idx dy dy_mia, dy_max 0.4, 0.4 dy_min, dy_max m_to_idx (dy_mminn iido_idx
-0.1, 0.1
ground
spawn_length n_to_idx(22 (iggght_fiilddddaaaa_niaaa leagth,
gools = [spawn_ (0. mid_y] # sirst goal apaиn
Initial height is cuY_x spawn_ length
cur_upward True
with upward ramp
sange dx = random. randint (dx_min, 6x_max) dy = ap. rap.random (dy_min, dy_max) ramp_length dx
up ramps
end_x
add (cur_x, end_x, mid_y dy, upwardrous_uppacce
place goal goal [1++)] [[our_x end_x] mid_y dy)
cencer somp
Switch rictticnn our ous_upward = not our_upward cus_x
down)
goal goals|-1] = [cur_x m_to_Idxxi 10.5), mid_y] height_ figgt___ffddccu_xi :] =
romp
return height_Field, guals
Listing 5: Example LLM Output. This output generates multiplc ramps in configuration that's distinctly different from any previously-seen terrains.
A.3 Generated Terrain Examples
In Figure 6, we visualize some of the terrains generated by Eurekaverse. We see that the complexity and difficulty generally increases each iteration, and the quadruped learns to pass through progressively more difficult obstacle courses.
A.4 Environment Check and Automatic Fixing
For every ILM-generated terrain function, first check that it is executable and feasible before using it for training. We check the former by running the training script and the latter with simple heuristies:
19

---

heration 1
lteration 2
abion
Iteration
Iteratiin 5
Figure 6: Visualizing subset of generated terrains, collected across iterations within run of Eurekaverse.
whether the maximum height below meters and whether the maximum height difference between goals is below 0.8 meters (double the Unitree Gol's 's standing height).
In our experiments, over 50% of terrains generated by GPT-4o pass the environment check. Common errors include mis-scaling the height values to incorrect units, out-of-bounds index access, and referencing missing helper functions or variables.
To save token usage and query costs, we also automatically fix invalid terrains: moving out-of-bounds goals, setting the quadruped spawn area to flat ground, and expanding obstacles that are too skinny. Note that these fixes can be easily replaced with assertions and additional LLM queries.
A.5 Simulation Setup
Environment Specification. We adopt the simulation framework from Cheng et al. [21]. which constructs an obstacle course terrain by quantizing the ground plane into a 2D grid and specifying the height at each index in the grid. Along with the terrain heights, the course includes goal positions indicating both progression and heading the policy should follow. This specification is completely defined via a Python function, which takes difficulty parameter as input and outputs a 2D array for terrain heights and list of 8 (z, y) coordinates for the goal positions. In Eurekaverse, the LLM is instructed to output precisely this function, which defines terrains fg modulated by the difficulty argument.
Policy Training, We train the policy on multiple terrains parallel: specifically, one round of training consists of 10 terrain classes, each with 10 difficulties. Policies are initialized at low difficulties, and if they reach Opromote goals, they are promoted to the next difficulty level; conversely, if they fail to reach Gdtemote goals, they are demoted to the previous difficulty. Otherwise, they stay the current difficulty with probability Pstay or go to random previous difficulty with 1 PPsta-
Following Cheng et al. [21], we first train a teacher policy using privileged scandot sensing. After fully training a privileged teacher, we then distill a student policy that takes in depth frames from front-facing camera. Apart from the scandot or depth, the policy also receives proprioception and heading and speed commands. We train the policy with PPO [57] with heading and velocity tracking rewards, regularizations like action rate and torque, and penalties for foot placements near terrain edges. We use this same reward across all methods and ablations.
Sim-To-Real. To facilitate sim-to-real transfer, as Cheng et al. [21], we use regularized online adaptation (ROA) [64], which trains an adaptation module that estimates environment properties from observation history. We also domain randomize over physical properties like friction, mass, and motor strength. During distillation, we introduce action delay and depth sensing delay of I simulation step (0.02s). and we update the depth frame only once every 5 steps (10Hz). Finally, we introduce noise in the depth input, adding Gaussian noise to its true value and randomly setting some pixels to O.
Simulation Benchmark. In Figure 7, we render each of the 20 obstacles in our parkour benchmark. Each of these obstacles is instantiated with 10 difficulties during evaluation, and these renders capture the obstacles at medium difficulty.
20

---

SBEEEt
Figure 7: Visualizing our simulated parkour benchmark, roughly organized by skill: climbing boxes, walking on slopes, jumping, traversing stepping stones, climbing stairs, navigating narrow hallways, moving zig-zag through agility poles, and balancing.
A.6 Environment Resampling
Previously, Eurekaverse also included resampling mechanism: each evolution iteration, we selected high quality terrains from prior iterations for training. Specifically, terrains were chosen with probability proportional to learning progress [65-67], the difference in the policy's ' performance on that terrain before and after training on our previous results, this increased the number of evaluation goals reached.
Additional experiments revealed that this mechanism is not necessary if we tune the inner difficulty progression (described in 5.1). Specifically, we adjusted it to diversify the parkour courses experienced across parallel agents during training, setting Gpromote = 8 * 0.8, Idemote = 8 * 0.4, Pstay = 0.75. In doing so, the policy is much less likely to fall into the local minima of an individual parkour course, increasing its robustness malformed difficult terrain. After increasing diversity, we find that this change negates the benefit derived from environment resampling, whose effect was avoid local minima by improving the quality of terrains used for training.
Since we observe that a careful configuration of the inner difficulty progression offsets the benefit of environment resampling, we opt to remove the latter from our method. This change streamlines our method while achieving higher evaluation results.
A.7 Deployment Details
We deploy on the Unitree Gol, quadrupedal robot with 12 degrees of freedom. When standing, the robot is 64.5 cm long, 28 em wide, and 40 em tall. We use the 3D camera mount introduced by Zhuang et al. [20] to attach an Intel RealSense D435 camera onto the Gol's 's head. Our vision policy runs onboard the Nvidia Jetson Xavier NX, with the depth encoder running asynchronously at 10 Hz and the policy at 50 Hz. Before depth inputs are passed through the depth encoder, we crop the left and right edges to remove dead pixels; we then apply hole-filling and temporal filters and down- wwwn-sample the resolution from 270x480 60x90.
21

---

A.8 Ablations
In Figure 8, we plot the performance of ablations from Figure (right). All ablations plateau or degrade except for Eurekaverse.
Ours
LUM Example
Initial Enus
Final Envs
Na Feedback
Diversity Only
Mean Denchmark Goals Reached
4-
2k
4k
6k
8k
10k
Number of Parallel Steps
Figure 8: Comparing sim benchmark performance across training steps for Eurekaverse and ablations.
22


Reviews:
[{"review_id": "YdNB6AVglF", "reviewer": "Reviewer_VvEd", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Curriculum learning is an effective technique for training an agent to perform increasingly difficult and complex tasks. However, in most cases, the curriculum is designed by domain experts. In the current paper, the authors propose leveraging LLMs for generating the environment curriculum, which is over different terrains. The method starts by initializing an i.i.d. set of environments and then training a population of agents on subsets of all the environments. Each agent is then evaluated on the entire set of environments, and the best performing agent is retained. This agent serves as the initialization for the population in the next iteration. The next step in the method involves querying the LLM to output a new set of environments that can be used to improve the policy’s performance further. The method then alternates between these two steps for several iterations to finally obtain the best performing policy. The authors also evaluate their method on the real robot.", "summary_of_recommendation": "While the idea of using an LLM to act as a “domain expert” in itself is not novel, the authors use it in a novel setting, backed by a compelling and thorough set of real world testing experiments, demonstrating the efficacy of their proposed method.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- Given that this is an automated method, what is the wall-clock time taken from start to finish? What computing setup is used to train the policies?\n- What is the cost in USD for obtaining the final policy that can be deployed on the real robot?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CK5g1jYS9U", "reviewer": "Reviewer_XMA4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces Eurekaverse, an unsupervised environment design algorithm that uses large language models (LLMs) to generate progressively more challenging and diverse environments for training robotic skills. The key idea is to leverage LLMs' code generation capabilities to create varied obstacle courses for quadrupedal parkour learning. The authors validate Eurekaverse on a quadruped robot traversing various obstacle courses, demonstrating effective sim-to-real transfer. The automatically generated curriculum enables gradual learning of complex parkour skills in simulation and successful transfer to real-world scenarios, outperforming manually designed training courses.", "summary_of_recommendation": "I recommend a weak accept for this paper. The proposed Eurekaverse algorithm presents a novel and potentially impactful approach to unsupervised environment design using LLMs. The validation on quadrupedal parkour and demonstration of sim-to-real transfer are significant strengths. However, the paper would benefit from addressing the limitations mentioned, particularly regarding the capabilities of LLMs in generating complex environments, the comparison to simpler methods, and the explanation of emergent capabilities. With these improvements, the paper could make a valuable contribution to the field of robotic learning and environment design.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. How do you ensure that the environments generated by the LLM are physically valid and increasingly complex? What are the limitations of current LLMs in this context?\n\n2. Given the effectiveness of random sampling, how does Eurekaverse compare to simpler methods of generating diverse environments and ordering them by difficulty?\n\n3. Can you provide more insight into the emergent capabilities observed in real-world trials? What aspects of the training process or environment design might contribute to this robustness?\n\n=== post rebuttal \nThe authors have made a good effort to address my comments. I am happy with 2. and 3. I am more positive now. However, there is no \"accept\" rating. I do not think the paper crossed the threshold to be a strong accept.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GwLas1ggUS", "reviewer": "Reviewer_LPUg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a co-evolution algorithm that utilizes LLMs to create an automatic curriculum of environments for an RL agent to train on.", "summary_of_recommendation": "The clarity of presentation, quality of experiments, and the impressive results (besides my questions above), assure me that this is an excellent paper that is of high relevance to CoRL.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Besides addressing the weakness above I have the following questions:\n1. I would like to understand how much of the contribution of this work is algorithmic novelty (which I suspect it mostly is!) and how much is prompt engineering? what effort did you make to make sure the baseline had the best prompt engineering available to it as well (and not one that is tailored to your algorithm)? it would be helpful to describe this process. In particular, I would like to understand the gap between the value of using a non-iterative solution compared to using your algorithm. If you think the iteration-1 and human-design reflect that, I guess I just need for you to describe why they had the best prompts they could utilize. \n\n2. I am missing a baseline that is like iteration-1, but where the tasks are not IID during generation. I.e., every tasks is conditioned on being different from all previous tasks. This would help show that diversity is not enough, but a full curriculum where the curriculum takes into the account how the agent progresses is required. In my opinion, *initial envs* is not enough because there is no dependence between envs, and *final env* doesn't capture all the diversity the RL agent has seen (since it was initialized using past environments that might be gone by this time).\n\n3. Compute: please discuss how much more compute you require compared to other baselines. If you require orders of magnitude more compute, discuss why running the baselines for longer will not improve their results (and of course if it will improve, show it!).\n\n4. *Real world results:* how does the sim2real gap manifest here? If I understood correctly $\\theta$, only takes care of obstacle sizes, not things like weight, friction, engines effective torque etc'. How come it is taken care of here? Is it part of the prompting or something similar? is the simulator very accurate?\n\n\nMinor technical issues: \n1. Line 95, the image of T should probably be a simplex over states to account for stochastic environments.\n2. Line 100, mention that $\\Pi$ is the space of policies. \n3. Line 187: \"we first pre-train a simple walking policy with 1000 parallel training steps.\"  What do you mean by parallel here? There is only one task -- walking.\n4. Line 233: \"Eurekaverse nearly matches the oracle\" where can I see that. If I understand correctly figure 5 shows that there is yet a big gap in the harder tasks between oracle and iteration 5.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
1. The paper introduces a novel approach using  LLMs for generating environment curricula, leveraging their code generation capabilities for creating progressively more challenging environments.
2. Demonstrates effective sim-to-real transfer and validates the method on real-world tasks, specifically quadrupedal parkour.
3. The paper is well-written with a good flow, providing detailed implementation, pseudocode, examples, and a progress reweighing function, enhancing clarity and reproducibility.
4. Addresses a fundamental problem in robotics and RL tasks, making the solution potentially applicable to various scenarios requiring multiple skills.
5. The experiments are thorough, showing the necessity of different components in the pipeline and highlighting the efficacy of the proposed method.

Weaknesses:
1. The population-based evolutionary algorithm can be computationally expensive, and the paper lacks a discussion on the wall-clock time and computing setup required.
2.  The surprising effectiveness of random sampling raises questions about the necessity of the proposed method compared to simpler solutions that generate diverse environments ordered by difficulty.
3. The paper does not clearly explain the emergent capabilities observed in real-world trials, such as robustness to unseen obstacles.
4. This work is missing comparisons with baselines that could use simpler non-iterative solutions or diverse environment generation conditioned on being different from previous tasks.
Decision: Accept