Environment Curriculum Generation via Large Language Models

Authors: william  liang
sam  wang
hung-ju  wang
osbert  bastani
dinesh  jayaraman
jason  yecheng
  ma
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 08:40:48

Abstract: Recent work has demonstrated that a promising strategy for teaching robots a wide range of complex skills is by training them on a curriculum of progressively more challenging environments. However, developing an effective curriculum of environment distributions currently requires significant expertise, which must be repeated for every new domain. Our key insight is that environments are often naturally represented as code. Thus, we probe whether effective environment curriculum design can be achieved and automated via code generation by large language models (LLM). In this paper, we introduce Eurekaverse, an unsupervised environment design algorithm that uses LLMs to sample progressively more challenging, diverse, and learnable environments for skill training. We validate Eurekaverse's effectiveness in the domain of quadrupedal parkour learning, in which a quadruped robot must traverse through a variety of obstacle courses. The automatic curriculum designed by Eurekaverse enables gradual learning of complex parkour skills in simulation and can successfully transfer to the real-world, outperforming manual training courses designed by humans.

Main paper: 
 introduction: 
 It is often hypothesized that the evolution of intelligent motor control in biology is driven by the need to habituate in and adapt to varied environment conditions [1, 2] . Inspired by this observation, training robot policies over a curriculum of environments has been shown to be an effective strategy for the acquisition of complex robot skills [3] [4] [5] [6] [7] . Designing a useful curriculum of environments is, however, time-consuming and requires domain-specific knowledge to work well [8] . In limited cases, when the environment variations can be succinctly represented by scalar parameters [9, 10] , unsupervised environment design methods have shown capability in automatically generating progressively more challenging tasks in low-dimensional simulation domains [9, 11, 12] . However, whether these methods can scale up to challenging real-world robotic tasks is an open question.Our key insight is that, in many practical robotics scenarios, environment variations can be represented by diverse programs. This allows us to solve environment design using large language models (LLMs) since they have been shown to excel at code generation in diverse domains [13] [14] [15] [16] . We introduce Eurekaverse, a language model powered environment curriculum design algorithm that can generate and evolve environments in code space. At a technical level, Eurekaverse instructs an LLM to generate novel environments to teach the target task, trains policies for each environment, and evaluates the best-performing policies to inform the LLM when it generates the next round of environments. Given that large language models (LLMs) have exhibited strong code generation and self-improving capabilities on other domains [17] [18] [19] , we hypothesize that using LLMs for environment curriculum design could also enable learning complex robot skills. In particular, we validate Eurekaverse's efficacy on the task of quadrupedal parkour [20, 21] , in which a quadruped robot is tasked with traversing diverse and unknown challenging courses with varied terrains and obstacles such as gaps, hurdles, boxes, and ramps. In order to train a policy deployable in the real world, prior parkour methods carefully designed training courses in simulation that enable real-world policy transfer [20, 21] . Because parkour courses are represented as programs that specify detailed course terrain and geometry (see Fig. 3 ), we posit that parkour is an ideal testbed for testing Eurekaverse's capability in handling the complexity of evolving environment programs. Parkour involves a rich variety of challenging locomotion skills so our results could also be of independent interest for scaling up locomotion learning.We demonstrate that the automatically generated curriculum of environments from Eurekaverse leads to policy training that continuously improves over time without plateauing; in contrast, baselines that only train over a fixed set of terrains, designed by either humans or an LLM, tend to overfit on these training terrains, resulting in worse generalization to test simulation and real-world courses. Specifically, on a set of 20 held-out simulation test courses that are carefully designed to holistically evaluate robot parkour skills, Eurekaverse-trained policies outperforms various baselines and ablations. Furthermore, on several real-world courses, Eurekaverse-trained policies successfully transfer and exhibit more robust and adaptive behavior than policies trained using limited or manually designed training courses.In summary, our contributions are:1. Eurekaverse, an LLM-based unsupervised environment design algorithm that can automatically generate curriculums of environment programs for robot skill learning.2. Extensive simulation and real-world validation of Eurekaverse on quadrupedal robot parkour. 
related work: 
 Large Language Models for Robotics. Recent work has demonstrated that LLMs can be used in robotics domains in various ways. They have demonstrated capabilities as high-level semantic planners [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] and control policies [35] [36] [37] . Leveraging the code generation capability of LLMs, recent work has also explored using LLMs to guide policy learning via reward function design [38] [39] [40] [41] and environment design [42] [43] [44] . In this latter category, prior work shows that LLMs can generate varied environments for top-down pick-and-place tasks that enable semantic generalization [42] or diverse tasks across different embodiments without clear intra-task dependencies [43] . In contrast, our work is the first to demonstrate how LLMs can be used in an evolutionary approach to generate full environment curricula that can guide the learning of complex physical skills such as parkour. Finally, Ma et al. [44] proposes using LLMs to automatically sample physics parameter randomization for sim-to-real policy learning [45] [46] [47] ; physics randomization is a restricted class of environment design, involving selecting a value range for each physics parameter in an enumerated list. Our domain specification here is much larger, more complex, structured: it is a full-fledged program.In our specific case study of robot parkour, this program specifies one out of a very large space of environment geometries.Environment Design and Curriculum Learning. Beyond using LLMs for environment program generation, environment design and curriculum learning have been extensively studied in prior literature. Prior works have considered framing environment design as a multi-agent game between the environment generator and the policy [9] [10] [11] [12] [48] [49] [50] [51] ; however, these methods require manually designed environment generators and can handle only a small space of environment variations (e.g., layouts in a 2D maze), limiting their scalability to challenging problems such as parkour that has a large and complex design space. Curriculum learning has been used in robotics to learn complex skills [5, [52] [53] [54] [55] [56] , but prior works use manually designed curricula or confined environment variation space that require extensive domain expertise and tuning. Our work demonstrates how to scale up adaptive environment curriculum for challenging physical skills by using LLMs to automate environment generation and modification.should be relative to the quadruped's size, and the entire obstacle should fit within the terrain bounds. You should be creative when designing your obstacles, and You may draw inspiration from features seen in dog parks, playgrounds, and urban environments. 3. Write a function that generates the obstacle course according to your plan. 
problem setting: 
 In this section, we formalize the unsupervised environment design setting as introduced in [9] . First, we model a fully specified environment as a Partially Observable Markov Decision Process (POMDP).Here, an environment is a tuple M = (A, S, R, O, T , I, γ), where A is the space of actions, S is the set of states, O is the set of observations, R :S × A → R is the reward function, T : S × A → ∆(S)is the transition dynamics function, I : S → O is the emission function, and γ is the discount factor. Given M , the goal of reinforcement learning is to learn a policy π : S → ∆(A) that maximizes the cumulative discounted sum of rewards:V M (π) = E π,T [ T t=0 γ t R(s t , a t )].Then, a POMDP template is a tuple M = (A, S, R, O, Θ, T , I, γ), where Θ represents the space of environment variations that can be incorporated into the transition dynamics T : S × A × Θ → ∆(S). While environment configuration can include parameter variations such as simulation physics values [44] , in this work, we focus on semantic and geometric variations (e.g., different parkour terrains) that are represented as programs. Given this, an environment curriculum can be parameterized by a sequence of programs ⃗ θ = (θ 1 , ..., θ T ) ∈ Θ T . Then, in unsupervised environment design (UED), we are interested in an environment generator Λ : Π → Θ that can generate a curriculum of environments ⃗ θ for the policy π ∈ Π to continue learning. The goal of UED is to learn a policy that can zero-shot generalize to unseen environments, maximizing i V Mi (π) for a set of unseen environments {M test,i } parameterized by Θ test = {θ test,i }. Therefore, Λ must be able to generate both learnable and useful environments for policy learning in order to maximize generalization capability.In this work, we investigate whether LLMs can be an effective choice of Λ for designing robot parkour courses for quadrupedal robots, where Θ can be thought of as the space of programs that captures different terrain configurations. 
methods: 
 At a high level, Eurekaverse proceeds as follows. First, it uses the LLM to generate an initial set of environments to train on. Then, it uses a process we call agent-environment co-evolution, where it iteratively uses reinforcement learning (RL) to train agents on the current set of environments, followed by updating the set of environments so they continue to challenge the best current policy without being too difficult for learning. Intuitively, as the best current policy learns to act more effectively, the environments are evolved so they can more effectively improve this policy. This strategy resembles a traditional curriculum learning pipeline, with the key difference being that the curriculum itself is generated automatically by the LLM.Throughout this section, we ground the exposition in the context of quadruped parkour to concretize the method discussion; we note that the general algorithmic principle is broadly applicable to other environment design problems. Implementation details and pseudocode are in the Appendix.In order to design an effective environment curriculum, we require a mechanism for generating an initial set of diverse environments. We first query the LLM, given a description of the task and a single in-context example environment program; then, the LLM responds with an environment program θ. To obtain a diverse set of training environments, we can sample the LLM multiple times with nonzero temperature. In particular, given in-context example θ incontext , we sample environmentsθ j ∼ Λ init LLM (θ incontext ), j = 1, ..., N J,where N is the number of RL agents and J is the environments per agent (see Sec. 4.2 for details).In quadruped parkour, this configuration θ j is an obstacle course terrain, specified as a height field-a 2D grid H that defines the height of coordinates along the ground plane-and a list of goal coordinates C = (x 1 , y 1 ), . . . (x 8 , y 8 ). Both H and C are implemented as Python code and defined via for loops and Numpy operations. Due to this programmatic format, our generation procedure also includes a simple validity check for maximum height and maximum height difference thresholds to filter out obviously impossible or incorrect terrains; details are in Appendix. See Figure 3 for the generation prompt, in-context example, LLM output, and environment rendering.The initial environments can already be used for training RL agents. However, a key issue is that the LLM has zero knowledge about the RL agent's capabilities and limitations, so they may not be very useful for training. Furthermore, they are sampled i.i.d., so they are unlikely to form a progressively more challenging curriculum. Indeed, as we show in our experiments, a single policy trained on these environments is sub-optimal, as much of the training budget is likely spent on environments that are too difficult or too easy for the agent's current capabilities.To address this issue, we propose a co-evolution approach of environments and agents. Specifically, this process alternates between training RL agents to improve on the current set of environments, and using the LLM to evolve the set of environments to further improve the agent. This alternating strategy naturally leads the environments to form a curriculum for the corresponding RL agents. We describe both of these steps in more detail below.Policy evolution. At the beginning of each iteration, we use RL to train a population of N agents {π i } N i=1 , each on its own training library of J environments {θ i j } J j=1 ; this training happens independently and in parallel. Then, we evaluate each agent on the union of all environments Θ proxy = ∪ i {θ i j } J j=1 across all iterations, and select the best-performing policy π best . In the first iteration, the RL agents are randomly initialized; in the subsequent iterations, they are initialized to π best from the previous iteration. Thus, the policies progressively become more effective across iterations. Intuitively, maintaining a population of agents increases the likelihood that some agents are trained on useful environments and not affected by bad environments; then, evaluating policies on the union of all environments selects the one that generalizes best to unseen environments.Environment evolution. In the first iteration, the initial environments generated as described in Sec. 4.1 are randomly split among the N RL agents. In subsequent iterations, we create new environments by evolving those that were used to train π best , {θ best j } J j=1 . Since π best is effective, these environments are known to be effective for training. We then use the LLM to evolve them: for each θ j in this set, we provide the LLM θ j and ask it to produce a variation on that environment:EQUATIONHere, θ j serves as a previous LLM response, and our LLM prompt contains additional information including environment statistics (e.g. maximum terrain height difference) and policy training statistics (e.g. reward terms and success rate) as well as an in-context evolution example. See Appendix for prompt and details. We perform this procedure independently N times, which produces a new set of training libraries {θ i j } J j=1 , i = 1, . . . , N for the N policy training runs in the following iteration. 
experimental setup: 
 Simulation Training. We adopt the simulation framework from Cheng et al. [21] , which trains parkour policies for the Unitree Go1 robot on obstacle course terrains defined by a height field and goal coordinates. During training, terrains are instantiated at multiple difficulties affecting obstacle dimensions and spacing. Policies are initialized at low difficulties, and they are promoted or demoted to higher or lower difficulties depending on the number of goals reached. This procedure gradually introduces the policy to harder obstacles and ensures that the thousands of parallel training robots cover a wide, diverse set of courses to avoid overfitting behaviors at local minima.Using PPO [57] , we train a teacher policy that takes in privileged scandot sensing (i.e., terrain heights at specific positions around the robot). To deploy, we distill the teacher into a student that receives depth frames from a front-facing camera. All methods and ablations use the same reward and training setup. Additional training details are in the Appendix.Algorithm Details. Eurekaverse uses GPT-4o as the LLM, and we run 5 iterations of generation, each with 8 parallel policy training runs of 2000 steps. To teach our policy a stable walking gait (which requires reward terms not relevant to parkour, e.g., torso orientation), we first pre-train a simple walking policy with 1000 policy update steps. One full run of our method takes around 24 hours on 8 A6000 GPUs, each with 48 GB VRAM, and incurs an OpenAI API cost of around $15.Methods. We compare Eurekaverse with the Human-Designed environments from Cheng et al. [21] , which were also designed for a generalist quadruped parkour agent. We emphasize that this comparison does not take into account the time and training steps required to first design Human-Designed optimally, whereas Eurekaverse must perform both environment design and policy training within the allocated steps. Additionally, we compare to a Random baseline that places randomly positioned ramps and boxes; these obstacles are sized randomly between half and double the bounding box of the quadruped, making them feasible for a well-trained policy. Finally, we consider an Oracle policy trained directly on the testing environment. Random serves as a lower bound-confirming whether environment design is actually necessary-and Oracle serves as an upper bound-if we have access to the true testing environment and can directly train on it.Ablations. We also compare against various ablations to probe the importance of our algorithm components. First, we run a variant that always asks the LLM to make the environments more challenging, without providing policy performance feedback (No Feedback). Next, we consider training a policy for the same number of steps on only the initial set of LLM-generated environments (Initial Envs), the set of environments in the final iteration (Final Envs), and a set of environments where each terrain is generated sequentially, conditioned on previously-generated terrains, to maximize diversity (Diversity Only). Finally, we train a policy on the in-context example provided to the LLM (LLM Example).To compare the zero-shot generalization performance of different policies in simulation, we design a parkour benchmark suite consisting of 20 diverse parkour tasks as our testing environments. These tasks are variations of real world obstacles seen in prior robot parkour learning works, such as box climbing, forward ramps, sideways ramps, A-frames, box jumps, stepping stones, staircases, narrow passages, agility poles, and balance beam [21, [58] [59] [60] [61] [62] [63] . Note that these benchmark environments are independent from the Human-Designed ones and not revealed to the LLM for environment generation, holistically testing all methods' generalization capability. For each of the 20 tasks, we instantiate 10 versions scaled by difficulty. The benchmark is visualized in the Appendix.In Figure 4 (left), we report the benchmark performance of Eurekaverse, Human-Designed, and Random across training steps as well as Oracle's best performance. In Figure 4 (right), we compare our method's final performance with that of various ablations. Additionally, in Figure 5 , we compare Eurekaverse's iterations with Human-Designed on each benchmark obstacle.Environment design is necessary for parkour. First, we see that the Random baseline does not increase performance on the benchmark, indicating it fails to learn in randomly-generated environments; thus, careful environment design is needed to learn complex quadruped parkour skills.Eurekaverse outperforms human-designed environments. On average, our method's final policy achieves nearly 2 additional goals over the policy trained on human-designed environments. The latter learns quickly but plateaus; intuitively, though well-designed, the relatively small number of distinct terrains limits the capacity for generalization. On the other hand, while our policy learns slower in the first few iterations due to imperfect initial environments, it exhibits a stronger upward trend since the environments are iteratively adapted to the current best policy, allowing the policy to continuously improve. Similarly, we see in Figure 4 (right) that Eurekaverse surpasses LLM Example, the human-created in-context example for the LLM queries; while the well-designed example teaches the policy to succeed in a specific subset of benchmark tasks, the LLM can use it to creatively construct a more diverse array of environments, thereby teaching a more generalist and performant policy.Eurekaverse nearly matches the oracle. After 5 iterations of design, our policy nears the performance of Oracle in Figure 4 , despite not receiving any information about the benchmark.Eurekaverse generates a curriculum for continuous learning. From Figure 4 (right), we see that Initial Envs and Final Envs achieve subpar results. Thus, neither teaches the policy a complete set of parkour skills; the former was generated without information on the policy's training progress, and the latter is tuned for a policy that has already learned skills in previous iterations, making it unsuitable for a beginner policy. Additionally, Diversity Only underperforms, showing that diverse environments alone, without an adaptive curriculum, are insufficient for optimal performance. No Feedback is also inferior; though it improves similarly to Eurekaverse in the initial iterations, policy performance degrades as the evolved terrains fail to match the policy's capabilities (see Appendix). Thus, a curriculum tailored to policy training is necessary, rather than one that simply increases difficulty. By building a curriculum that bridges the initial and final environments while increasing diversity and taking into account policy feedback, Eurekaverse trains a policy that gradually improves over each iteration; we see evidence of this in Figure 5 , where our policy generally improves at each benchmark obstacle type.Robot Deployment Details. We deploy on the Unitree Go1, a quadrupedal robot with 12 degrees of freedom. Note that the Go1 weighs 12kg and is equipped with motors that have a maximum torque of 23.70 Nm, which is significantly lower than prior work [20, 21] with the A1, whose motors reach 33.50 Nm and also weighs 12kg. Additional details are in Appendix.Real-World Tasks. Figure 1 shows Eurekaverse's and Human-Designed's performance on box climbing, gap crossing, forward ramp, and stairs tasks, four representative tasks in our simulation benchmark as well as prior works [20, 21, 59, 60] . Box climbing involves getting on top of a box of varying heights, and gap crossing involves jumping from one platform to another. In stairs, the robot must traverse as many steps of stairs as possible, and in forward ramp, the robot must step onto and go across a ramp, with difficulty determining the ramp angle.Results. We deploy distilled policies from Eurekaverse and Human-Designed and compare their success rates across the four tasks, each of which vary in difficulty corresponding to the key obstacle property. In box climbing, gap crossing, and forward ramp, we perform 10 trials for each difficulty; in stairs, we perform 10 trials total and measure the number of steps traversed. For each obstacle, we initialize the robot at 5 preset locations with varying distance to the obstacle.In all tasks, Eurekaverse generally outperforms Human-Designed across difficulties, succeeding at jumps up to 75cm (above the Go1 length), climbing up over 50cm (above the Go1 height), walking up a 30 degree ramp, and traversing the entire staircase. Our policy is also much more safe and stable than Human-Designed, which often trips our controller's motor power protection fault; in the forward ramp task, Human-Designed incurred physical damages on our robot hardware, so we refrained from completing the remaining trials (dotted line in Figure 1 ).We also observe Eurekaverse exhibiting recovery behavior, where the policy quickly reacts to slipping or misplaced feet; for example, if it misses a jump, it quickly kicks up its hind legs, allowing the front feet to grab a more stable footing and climb up. Moreover, in the staircase trials, the policy is robust to seeing chairs and railings on either side, despite never seeing similar features in simulation. We hypothesize that this behavior is the result of two mechanisms: during training, policies fail but have a chance to recover before the environment resets, and during depth distillation, random blackout and noise augmentation increases robustness against out-of-distribution depth readings. 
conclusion: 
 We have presented Eurekaverse, a novel method for automated environment curriculum design with large language models (LLMs). Eurekaverse generates effective environment programs that adapts to the policy's current training progress, enabling it to learn new skills and hone existing ones. We validate our method on quadruped parkour, where our policy outperforms prior humandesigned ones in both zero-shot simulation and real environments. We believe that Eurekaverse demonstrates the potential for LLMs to generate an infinite variation of environments, enabling continuous policy learning and serving as a path toward truly open-ended, generalist robot agents. 
 
 Appendix: Our system requires a moderate amount of LLM samples to consistently generate and select valuable environments; a important future direction necessary for scaling up is increasing sample efficiency, for example by fine-tuning the LLM. Additionally, our evolution prompt only uses textual feedback. A potential extension can additionally provide environment visualizations, leveraging recent multimodal foundation models to potentially perform stronger spatial reasoning.We provide additional implementation details for Eurekaverse below. We first describe the core algorithm loop in pseudocode. Next, we describe "soft" policy selection, a more robust way to select the best policy using evaluation on the proxy (all generated training) environments.Algorithm Soft Selection. During co-evolution, we select the best policy π best via performance on all generated environments Θ proxy , which tests the policies' generalization and serves as a proxy for the true testing environment. Empirically, we find that this proxy accurately orders the policies, where the best policy in Θ proxy also performs the best in our simulation benchmark. Nonetheless, there are rare cases where the best-performing benchmark policy is not the best one chosen by the proxy. To increase robustness against these inaccuracies, we "softly" select the best policy during each iteration of co-evolution: the selected policy is chosen randomly, where the best policy has probability p 1 , second-best has probability p 2 < p 1 , and so on. Each agent in the next iteration performs this selection independently (so that N p 1 are initialized to the best policy, N p 2 are initialized to the second-best, etc.). In practice, we select p 1 = 0.75, p 2 = 0.25, p 3 , . . . , p N = 0.You are a reinforcement learning engineer trying to write environment functions as effectively as possible for a quadruped robot parkour task. Please keep in mind that this robot will be trained in simulation and deployed in a real world obstacle course, so we want the obstacles to be realistic and challenging.To do so, you should perform the following steps: 1. Carefully read the problem statement, specifications, and tips below.You are given a quadruped robot that needs to navigate through an obstacle course in 3D space.The floor of the course is represented as a 2D numpy array, height_field, where each element represents the height of the ground at that point, in meters. To direct the quadruped, the course has 8 goals that the quadruped must reach in order. The goal locations should be stored in a 2D numpy array, goals, as (x, y) indices in height_field.To create your course, you will write a function set_terrain(length, width, field_resolution, difficulty) that creates and returns the height_field and goals arrays. The function takes in the length and width of the course, in meters, the scale of quantization for the obstacle course, in meters, and the difficulty of the obstacle course, a float between 0 (easiest) and Environment Specifications: 1. height_field quantizes the ground plane into a grid of points. field_resolution = 0.05, in meters, is the quantization of the ground's (x, y) axes, which represents the distance between adjacent points in the height_field array. 2. The ground plane size is 12 x 4 meters, so the height_field.shape = (12 / field_resolution, 4 / field_resolution) = (240, 80). Please make sure that your obstacle course spans exactly these dimensions. 3. The quadruped's standing size (length, width, height) is 0.645 x 0.28 x 0.4 meters. Keep these dimensions in mind when designing the size of your obstacles. 4. The quadruped will spawn with its center at (x, y, z) = (1, width / 2, 0) meters. You must place obstacles at indices with x >= 2 / field_resolution to avoid the quadruped spawning inside an obstacle. 5. Please make sure that your obstacles have a width of at least 1 meter. However, in rare cases, we will allow narrow obstacles with length and width of at least 0.4 meters and no smaller, as long as the quadruped is not expected to climb up or down. 6. The goal coordinates should be within the bounds of the course. Even if you have fewer than 8 obstacles, you must set all 8 goal positions in the goals array. 7. The quadruped will be instructed to travel in a straight line from one goal to the next.Thus, if you want the quadruped to turn, you must place a goal at the turning point. 8. Your course should test a particular one of the robot's skills, for example by repeating a single type of obstacle. Make the course relatively consistent throughout. There will be other courses to test other skills and different obstacles. 9. The heights in field_height can be negative. The quadruped's spawning area will always be at a height of 0 meters, so you can use negative heights to create pits or gaps in the terrain. Use this to force the quadruped to walk or jump across the tops of your obstacles without climbing up or down. Listing 1: System Prompt. In the function template, we provide a conversion function m to idx to simplify the conversion from coordinates, in meters, to the index in the array. §The following is an example of a terrain generation function. Please reference the example provided, but make your terrain different.''' import numpy as np def set_terrain(length, width, field_resolution, difficulty): """Multiple platforms traversing a pit for the robot to climb on and jump across.""" def m_to_idx(m): """Converts meters to quantized indices.""" return np.round(m / field_resolution).astype(np.int16) if not (isinstance(m, list) or isinstance(m, tuple)) else [round(i / field_resolution) for i in m] height_field = np.zeros((m_to_idx(length), m_to_idx(width))) goals = np.zeros((8, 2)) # Set up platform dimensions # We make the platform height near 0 at minimum difficulty so the quadruped can learn to climb up platform_length = 1.0 -0. Listing 2: Initial Example. This example demonstrates the main structure of a terrain function, including helper functions and a for loop over multiple obstacles. §We trained a quadruped policy to perform parkour on the obstacle course created by the generation code above (as well as others), and we tracked the values of individual reward components as well as other metrics such as the number of goals reached (out of 8), episode length, and the rate of edge violations (feet too close to edge of obstacles): <INSERT POLICY STATISTICS HERE> We have also computed statistics for the terrain height of the direct path between goals across multiple difficulties. Note that this is a heuristical shortest-path between goals that approximates the locations an optimal quadruped would traverse. The statistics do not reflect the actual path taken by the quadruped policy, nor does it include the height of gaps, pits, and other obstacles that the quadruped would not step on: <INSERT TERRAIN STATISTICS HERE> Please carefully analyze the statistics above and provide a new, improved generation function.You should pay attention to which parts of the course the quadruped successfully learned and which parts it struggled with. The goal of your course is to balance difficulty and feasibility for the quadruped robot, allowing it to learn and perform better. Thus, if the robot is getting stuck on a certain obstacle or goal, you should consider changing or removing it. You should also follow the guidelines below: -If the number of reached goals is over 80%, please create a harder course while ensuring that it is feasible and safe for a real robot. You should consider adding more obstacles, increasing the complexity of the course, and increasing the difficulty of existing obstacles. For example, you can make climbing obstacles taller or jumping gaps wider.- whether the maximum height is below 3 meters and whether the maximum height difference between goals is below 0.8 meters (double the Unitree Go1's standing height).In our experiments, over 50% of terrains generated by GPT-4o pass the environment check. Common errors include mis-scaling the height values to incorrect units, out-of-bounds index access, and referencing missing helper functions or variables.To save token usage and query costs, we also automatically fix invalid terrains: moving out-of-bounds goals, setting the quadruped spawn area to flat ground, and expanding obstacles that are too skinny. Note that these fixes can be easily replaced with assertions and additional LLM queries.Environment Specification. We adopt the simulation framework from Cheng et al. [21] , which constructs an obstacle course terrain by quantizing the ground plane into a 2D grid and specifying the height at each index in the grid. Along with the terrain heights, the course includes 8 goal positions indicating both progression and heading the policy should follow. This specification is completely defined via a Python function, which takes a difficulty parameter as input and outputs a 2D array for terrain heights and a list of 8 (x, y) coordinates for the goal positions. In Eurekaverse, the LLM is instructed to output precisely this function, which defines terrains θ j modulated by the difficulty argument.Policy Training. We train the policy on multiple terrains in parallel; specifically, one round of training consists of 10 terrain classes, each with 10 difficulties. Policies are initialized at low difficulties, and if they reach g promote goals, they are promoted to the next difficulty level; conversely, if they fail to reach g demote goals, they are demoted to the previous difficulty. Otherwise, they stay at the current difficulty with probability p stay or go to a random difficulty with 1 -p stay . Following Cheng et al. [21] , we first train a teacher policy using privileged scandot sensing. After fully training a privileged teacher, we then distill a student policy that takes in depth frames from a front-facing camera. Apart from the scandot or depth, the policy also receives proprioception and heading and speed commands. We train the policy with PPO [57] with heading and velocity tracking rewards, regularizations like action rate and torque, and penalties for foot placements near terrain edges. We use this same reward across all methods and ablations.Sim-To-Real. To facilitate sim-to-real transfer, as in Cheng et al. [21] , we use regularized online adaptation (ROA) [64] , which trains an adaptation module that estimates environment properties from observation history. We also domain randomize over physical properties like friction, mass, and motor strength. During distillation, we introduce an action delay and depth sensing delay of 1 simulation step (0.02s), and we update the depth frame only once every 5 steps (10Hz). Finally, we introduce noise in the depth input, adding Gaussian noise to its true value and randomly setting some pixels to 0. Simulation Benchmark. In Figure 7 , we render each of the 20 obstacles in our parkour benchmark. Each of these obstacles is instantiated with 10 difficulties during evaluation, and these renders capture the obstacles at medium difficulty. Previously, Eurekaverse also included a resampling mechanism: each evolution iteration, we selected high quality terrains from prior iterations for training. Specifically, terrains were chosen with probability proportional to learning progress [65] [66] [67] , the difference in the policy's performance on that terrain before and after training on it. In our previous results, this increased the number of evaluation goals reached.Additional experiments revealed that this mechanism is not necessary if we tune the inner difficulty progression (described in 5.1). Specifically, we adjusted it to diversify the parkour courses experienced across parallel agents during training, setting g promote = 8 * 0.8, g demote = 8 * 0.4, p stay = 0.75. In doing so, the policy is much less likely to fall into the local minima of an individual parkour course, increasing its robustness to malformed or difficult terrain. After increasing diversity, we find that this change negates the benefit derived from environment resampling, whose effect was to avoid local minima by improving the quality of terrains used for training.Since we observe that a careful configuration of the inner difficulty progression offsets the benefit of environment resampling, we opt to remove the latter from our method. This change streamlines our method while achieving higher evaluation results.We deploy on the Unitree Go1, a quadrupedal robot with 12 degrees of freedom. When standing, the robot is 64.5 cm long, 28 cm wide, and 40 cm tall. We use the 3D camera mount introduced by Zhuang et al. [20] to attach an Intel RealSense D435 camera onto the Go1's head. Our vision policy runs onboard the Nvidia Jetson Xavier NX, with the depth encoder running asynchronously at 10 Hz and the policy at 50 Hz. Before depth inputs are passed through the depth encoder, we crop the left and right edges to remove dead pixels; we then apply hole-filling and temporal filters and down-sample the resolution from 270x480 to 60x90. 


Decision: Strengths:
1. The paper introduces a novel approach using  LLMs for generating environment curricula, leveraging their code generation capabilities for creating progressively more challenging environments.
2. Demonstrates effective sim-to-real transfer and validates the method on real-world tasks, specifically quadrupedal parkour.
3. The paper is well-written with a good flow, providing detailed implementation, pseudocode, examples, and a progress reweighing function, enhancing clarity and reproducibility.
4. Addresses a fundamental problem in robotics and RL tasks, making the solution potentially applicable to various scenarios requiring multiple skills.
5. The experiments are thorough, showing the necessity of different components in the pipeline and highlighting the efficacy of the proposed method.

Weaknesses:
1. The population-based evolutionary algorithm can be computationally expensive, and the paper lacks a discussion on the wall-clock time and computing setup required.
2.  The surprising effectiveness of random sampling raises questions about the necessity of the proposed method compared to simpler solutions that generate diverse environments ordered by difficulty.
3. The paper does not clearly explain the emergent capabilities observed in real-world trials, such as robustness to unseen obstacles.
4. This work is missing comparisons with baselines that could use simpler non-iterative solutions or diverse environment generation conditioned on being different from previous tasks.

Reviews: review: title: title: Official Review of Paper519 by Reviewer VvEd
paper_summary: paper_summary: Curriculum learning is an effective technique for training an agent to perform increasingly difficult and complex tasks. However, in most cases, the curriculum is designed by domain experts. In the current paper, the authors propose leveraging LLMs for generating the environment curriculum, which is over different terrains. The method starts by initializing an i.i.d. set of environments and then training a population of agents on subsets of all the environments. Each agent is then evaluated on the entire set of environments, and the best performing agent is retained. This agent serves as the initialization for the population in the next iteration. The next step in the method involves querying the LLM to output a new set of environments that can be used to improve the policy’s performance further. The method then alternates between these two steps for several iterations to finally obtain the best performing policy. The authors also evaluate their method on the real robot.
main_review: main_review: The paper is well-written and has a good flow. The authors motivate the problem well and cover the relevant related work that has been performed in this domain in the recent past. The authors combine ideas such as co-evolution (used in other problem domains like robot design), and using the baked-in knowledge and reasoning capabilities of LLMs and apply it to a previously unexplored setting of environment/terrain curriculum generation. The ablations make sense and serve to highlight the importance of the different components used in the pipeline.

Strengths:
- The authors have done a great job with extensive real-world evaluation of their method.
- The implementation details, including the algorithm pseudocode, prompts used, examples, and progress reweighing function serve to increase the clarity and reproducibility of the proposed method.

Weaknesses: 
- Given that the proposed method uses an evolutionary algorithm style population-based method, it can prove to be quite expensive, depending on the value of N and J chosen.
questions: questions: - Given that this is an automated method, what is the wall-clock time taken from start to finish? What computing setup is used to train the policies?
- What is the cost in USD for obtaining the final policy that can be deployed on the real robot?
limitations: limitations: 2
review_summary: review_summary: While the idea of using an LLM to act as a “domain expert” in itself is not novel, the authors use it in a novel setting, backed by a compelling and thorough set of real world testing experiments, demonstrating the efficacy of their proposed method.
title: title: very interesting work on designing environment curriculum, with limitations
paper_summary: paper_summary: This paper introduces Eurekaverse, an unsupervised environment design algorithm that uses large language models (LLMs) to generate progressively more challenging and diverse environments for training robotic skills. The key idea is to leverage LLMs' code generation capabilities to create varied obstacle courses for quadrupedal parkour learning. The authors validate Eurekaverse on a quadruped robot traversing various obstacle courses, demonstrating effective sim-to-real transfer. The automatically generated curriculum enables gradual learning of complex parkour skills in simulation and successful transfer to real-world scenarios, outperforming manually designed training courses.
main_review: main_review: Strengths:
1. Novel approach using LLMs for unsupervised environment design in robotics, addressing the challenge of creating effective training curricula.

2. Validation on a challenging real-world task (quadrupedal parkour), with demonstration of sim-to-real transfer. Outperforms manually designed curricula, showing potential for automating complex environment design.

3. Iterative co-evolution of environments and policies enables continuous learning improvement.

Weaknesses:
1. Limited discussion on the capabilities and limitations of LLMs in generating complex, physically valid environments. It's unclear if state-of-the-art coding LLMs can effectively evolve such environments.

2. Surprising effectiveness of random sampling raises questions about the necessity of the proposed approach. A simpler solution of generating diverse environments and ordering them by difficulty might be sufficient.

3. Lack of clear explanation for emergent capabilities observed in real-world trials, such as robustness to unseen obstacles. For instance, in the staircase trials, the policy is robust to seeing chairs and railings on either side, despite never seeing similar features in simulation. What enabled this?
questions: questions: 1. How do you ensure that the environments generated by the LLM are physically valid and increasingly complex? What are the limitations of current LLMs in this context?

2. Given the effectiveness of random sampling, how does Eurekaverse compare to simpler methods of generating diverse environments and ordering them by difficulty?

3. Can you provide more insight into the emergent capabilities observed in real-world trials? What aspects of the training process or environment design might contribute to this robustness?

=== post rebuttal 
The authors have made a good effort to address my comments. I am happy with 2. and 3. I am more positive now. However, there is no "accept" rating. I do not think the paper crossed the threshold to be a strong accept.
limitations: limitations: 1
review_summary: review_summary: I recommend a weak accept for this paper. The proposed Eurekaverse algorithm presents a novel and potentially impactful approach to unsupervised environment design using LLMs. The validation on quadrupedal parkour and demonstration of sim-to-real transfer are significant strengths. However, the paper would benefit from addressing the limitations mentioned, particularly regarding the capabilities of LLMs in generating complex environments, the comparison to simpler methods, and the explanation of emergent capabilities. With these improvements, the paper could make a valuable contribution to the field of robotic learning and environment design.
title: title: Review for "Environment Curriculum Generation via Large Language Models"
paper_summary: paper_summary: The paper proposes a co-evolution algorithm that utilizes LLMs to create an automatic curriculum of environments for an RL agent to train on.
main_review: main_review: *Quality:* think it is an excellent paper that is relevant to a major bottleneck in robotics -- creating an effective curriculum for robotic applications. Besides the questions in other sections, I'm very sure that the quality of this paper is at least worth publishing if not more. 

*Clarity:* the paper is very well-written. Besides a few technical issues I have not found a major fault in it.

Strengths:
1. The problem being solved here is fundamental in many RL + robotic tasks today where the agent is supposed to handle many "skills" at the same time. The solution presented here, seems applicable to many such cases.
2. The algorithm is described well, and the experiments convinced me that all proposed features were required to make it work.
3. This works well on a real system!

Weaknesses:
1. One major contribution of this paper is to alleviate "manual" environment coding. However, I would like the authors to describe all the places where human prior knowledge (such as prompt engineering) was required to reach these impressive results. I think it is required in order to understand how much this algorithm is applicable out-of-the-box for other tasks.
questions: questions: Besides addressing the weakness above I have the following questions:
1. I would like to understand how much of the contribution of this work is algorithmic novelty (which I suspect it mostly is!) and how much is prompt engineering? what effort did you make to make sure the baseline had the best prompt engineering available to it as well (and not one that is tailored to your algorithm)? it would be helpful to describe this process. In particular, I would like to understand the gap between the value of using a non-iterative solution compared to using your algorithm. If you think the iteration-1 and human-design reflect that, I guess I just need for you to describe why they had the best prompts they could utilize. 

2. I am missing a baseline that is like iteration-1, but where the tasks are not IID during generation. I.e., every tasks is conditioned on being different from all previous tasks. This would help show that diversity is not enough, but a full curriculum where the curriculum takes into the account how the agent progresses is required. In my opinion, *initial envs* is not enough because there is no dependence between envs, and *final env* doesn't capture all the diversity the RL agent has seen (since it was initialized using past environments that might be gone by this time).

3. Compute: please discuss how much more compute you require compared to other baselines. If you require orders of magnitude more compute, discuss why running the baselines for longer will not improve their results (and of course if it will improve, show it!).

4. *Real world results:* how does the sim2real gap manifest here? If I understood correctly $\theta$, only takes care of obstacle sizes, not things like weight, friction, engines effective torque etc'. How come it is taken care of here? Is it part of the prompting or something similar? is the simulator very accurate?


Minor technical issues: 
1. Line 95, the image of T should probably be a simplex over states to account for stochastic environments.
2. Line 100, mention that $\Pi$ is the space of policies. 
3. Line 187: "we first pre-train a simple walking policy with 1000 parallel training steps."  What do you mean by parallel here? There is only one task -- walking.
4. Line 233: "Eurekaverse nearly matches the oracle" where can I see that. If I understand correctly figure 5 shows that there is yet a big gap in the harder tasks between oracle and iteration 5.
limitations: limitations: 3
review_summary: review_summary: The clarity of presentation, quality of experiments, and the impressive results (besides my questions above), assure me that this is an excellent paper that is of high relevance to CoRL.