EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents

Authors: abhay  zala
jaemin  cho
han  lin
jaehong  yoon
mohit  bansal
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 21:59:20

Abstract: Recent state-of-the-art approaches for embodied learning via interaction directly employ large language models (LLMs) as agents to determine the next steps in an environment. Due to their world knowledge and reasoning capabilities, LLM agents achieve stronger performance than previous smaller agents based on reinforcement learning (RL); however, frequently calling LLMs is slow and expensive. This begs an interesting question: Instead of directly employing LLMs as embodied agents, can we use LLMs’ reasoning capabilities to adaptively create training environments to help smaller embodied RL agents learn useful skills that they are weak at? In this work, we propose EnvGen, a novel framework to address this question. First, we prompt an LLM to generate training environments that allow agents to quickly learn different tasks in parallel. Concretely, the LLM is given the task description and environment simulator objectives that the agents should learn and is then asked to generate a set of environment configurations (e.g., different terrains, items initially given to agents, chances of finding certain objects, etc.). Next, we train a small RL agent in a mixture of the original and LLM-generated environments. Then, we enable the LLM to continuously adapt the generated environments to progressively improve the skills that the agent is weak at, by providing feedback to the LLM in the form of the agent’s performance. We demonstrate the usefulness of EnvGen with comprehensive experiments in Crafter and Heist game environments. We find that a small RL agent trained with EnvGen can outperform SOTA methods, including a GPT-4 agent, and learns long-horizon tasks significantly faster. We also show that using an LLM to adapt environments dynamically outperforms curriculum learning approaches and how the LLM adapts training environments to help improve RL agents’ weaker skills over time. Additionally, EnvGen is substantially more efficient as it only uses a small number of LLM calls (e.g., 4 in total), whereas LLM agents require one or more LLM calls per step (resulting in thousands of LLM calls per episode). We also present detailed analyses of EnvGen’s design choices.

Main paper: 
 introduction: 
 There has been growing interest in embodied AI, where agents learn through interactions with environments instead of static datasets (Ahn et al., 2022; Duan et al., 2022; Wang et al., 2023a; Yao et al., 2023; Driess et al., 2023) . Open-world games such as Minecraft (Mojang Studios, 2009) and Crafter (Hafner, 2022) have been widely used as research environments for embodied agents, where the agents visually perceive their surroundings, traverse large terrains, and learn to unlock various achievements (e.g., collecting resources, building tools, defeating monsters, etc.). Some achievements can be easily unlocked within a few steps, whereas others are more challenging as they only become accessible after the agent completes a series of prerequisite achievements, requiring hundreds of steps (i.e., long-horizon tasks). As illustrated in Fig. 1 (a), traditional embodied agents are based on reinforcement In (c) EnvGen, we train a small RL agent with diverse LLM-generated environments that train different skills in parallel and can be adapted via feedback to help the agents progressively improve skills that they are weaker at. Our method benefits from using the world knowledge from LLMs while maintaining efficient training through a lightweight RL agent. learning (RL) (Hafner et al., 2020; 2021; 2023; Schulman et al., 2017; Burda et al., 2018; Hessel et al., 2018; Sekar et al., 2020; Moon et al., 2023) . However, these RL agents usually struggle when learning such long-horizon tasks since the reward is sparsely given only after the correct execution of successive actions, and it is very expensive to automatically find many action sequences which lead to the reward (Aytar et al., 2018; Li et al., 2022a; Yuan et al., 2023) , even after long pretraining with curiosity-driven intrinsic reward (Walker et al., 2023) .As large language models (LLMs) have shown remarkable progress in various tasks that require complex reasoning (Brown et al., 2020; OpenAI, 2023a; Touvron et al., 2023a; b; Chowdhery et al., 2023; Anil et al., 2023) , recent works study implementing embodied agents based on LLMs. As illustrated in Fig. 1 (b), these methods leverage LLMs' world knowledge with chain-of-thought reasoning (Nye et al., 2021; Kojima et al., 2022; Wei et al., 2022) by creating action plans, giving feedback, and obtaining rewards throughout the episode (Yuan et al., 2023; Wang et al., 2023c; Wu et al., 2023; Wang et al., 2023a; d; Zhao et al., 2023; Du et al., 2023) . While these LLM-based agents that verbalize their knowledge in reasoning steps have seen success in achieving better performance over previous approaches, iteratively calling LLMs throughout the episode is prohibitively slow and expensive (e.g., SPRING (Wu et al., 2023) calls GPT-4 (OpenAI, 2023a) 9 times to take any action step, which results in $270 USD to complete an episode). Du et al. (2023) use LLMs to create rewards to train smaller agents, but the training is still costly, as it requires many interactions between the LLMs and student agents. This begs the question: Instead of directly employing LLMs as embodied agents, can we use LLMs' reasoning capability to adaptively create training environments to help smaller embodied RL agents learn useful skills that they are weak at?To address this question, we propose EnvGen, a novel framework where an LLM adaptively generates training environments to teach smaller embodied RL agents. We aim to generate environments that can create various conditions (e.g., have different terrains or some subgoals are already achieved) so that agents can learn different skills in parallel and obtain more frequent rewards for challenging long-horizon tasks than in the original environment. As shown in Fig. 1 (c ), EnvGen iterates over multiple training cycles, each consisting of the following four steps:• Step 1: We generate configurations for custom training environments (i.e., specifically created to train an RL agent on certain skills) by providing an LLM with a prompt including task description, controllable simulator settings, and simulator constraints (see Fig. 2 and Sec. 2 for details). Then we use the generated configurations to create different custom environments (e.g., different terrains, items initially given to agents, and chance of finding certain objects) that can teach multiple skills in parallel. • Step 2: We first train the RL agent in multiple LLM-generated environments (i.e., LLM environments), so that it can learn different useful skills in parallel. • Step 3: We then train the RL agent in the original environment to mitigate overfitting to the LLM environments. Then, we measure the current RL agent's performance in different tasks in the original environment to check which skills the agent is weak at.• Step 4: We provide the RL agent's successes/failures in different tasks (from step 3) as feedback to the LLM, so that the LLM can adapt the custom training environments to focus on progressively improving the skills that the agent is weak at.Note that EnvGen only requires a few LLM calls (e.g., 4) for environment generation/updating during the entire RL agent training process, whereas other works based on LLM agents query an LLM once or multiple times every step (resulting in thousands of LLM calls for a single episode).We study the usefulness of EnvGen in different game environments: Crafter (Hafner, 2022) and Heist (Cobbe et al., 2020) . In the Crafter environment, a simple PPO-based (Schulman et al., 2017) lightweight (< 5M parameters) RL agent trained with our LLM-generated environments outperforms strong baselines including a GPT-4 based agent that queries an LLM multiple times at every step, and RL agents that use extensive pretraining (e.g., 150M steps vs. less than 1M steps for us). When compared to just training longer in the original Crafter environment and curriculum learning approaches such as easy-tohard and adversarial environments, an RL agent trained with EnvGen achieves significant improvements on the overall score and long-horizon tasks. In Heist, we also show that our LLM-generated environments can improve overall agent performance and training stability.We also show a qualitative study on how the LLM adapts training environments to help improve RL agents' weaker skills over time. Finally, we provide comprehensive analysis and ablation studies of the design choices of EnvGen, including dynamically updating LLM environments (i.e., using adaptive environments) vs. curriculum learning methods, frequency of environment updates, EnvGen vs. longer training in the original environment, different LLMs for generating environments, the number of LLM-generated environments, and the mixture ratio between the original and LLM environment during training. 
envgen: generating and adapting environments via llms for training embodied agents: 
 We propose EnvGen, a novel framework where an LLM adaptively generates training environments to train smaller embodied RL agents, enabling them to accomplish various tasks within an environment, particularly long-horizon tasks. During the training process, the LLM is given feedback (in the form of the agent's performance) and can adaptively update the training environments to progressively focus on improving the tasks that the agent is weak at. In the following, we first explain why it is challenging to explore longhorizon tasks in open-world games (Sec. 2.1). Then we explain our method details, including how we generate environments and how agents are trained in EnvGen (Sec. 2.2).In the RL framework, agents explore various states along a trajectory and amplify policies based on the rewards received from those trajectories. However, exploration for longhorizon tasks is slow and computationally expensive, as rewards for such tasks are sparsely given only after a sequence of successful actions that often involve achieving multiple subgoals. For example, the goal in Crafter (Hafner, 2022) is to unlock 22 achievements, where some achievements can be unlocked quickly through several simple actions and others require long chains of prerequisites (e.g., collect iron requires make stone pickaxe, which must be preceded by collect stone, ... etc.); see Sec. 3.1 for details. As shown in Hafner (2022) , existing agents in Crafter spend most exploration steps learning low-level achievements but fail to unlock high-order achievements with many prerequisites.We introduce EnvGen, where we train an embodied RL agent in multiple LLM-generated environments (we refer to these as 'LLM environments' in the paper) that progressively adapt to improve agent performance in multiple skills. The generated environments can provide various conditions (e.g., different terrains, or some subgoals are already achieved) ```json { "environment_settings": { "target_biome": "grassland", "tree_rarity": "common", "coal_rarity": "common" }, "inventory_settings": { "wood": 3, "stone": 0, "wood_pickaxe": 1 } } ``P urpose: The agent can learn to collect wood and craft items like a wood sword or a table ...You are an environment designer agent for a game called "Crafter". Your job is to design a few environments which can be used to teach an agent how to play...Here is a list of things an agent would need to learn how to do: • collect_coal, collect_diamond ...Here is a list of parameters you can control when making an environment: "environment_settings": { "target_biome": "mountain", "coal_rarity": "common", "iron_rarity": "common", "diamond_rarity": "rare" }, "inventory_settings": { "wood": 5, "stone": 3, "wood_pickaxe": 1, "stone_pickaxe": 1, "wood_sword": 1 } } ``P urpose: ...```json { "environment_settings": { "target_biome": "natural", "tree_rarity": "common", "coal_rarity": "common", "iron_rarity": "common", "diamond_rarity": "rare" }, "inventory_settings": { "sapling": 2, "wood": 0, "stone": 0, "coal": 0, "iron": 0, "diamond": 0 } } ``P urpose: ...Those environments resulted in the agent improving up to these scores:• collect_coal: 38% +/-6% • defeat_skeleton: 10% +/-4% • make_stone_pickaxe: 31% +/-3% • … Could you generate new environments based on these scores?Step 1: Generate training environmentsStep 2: Train small agent in generated environments so that agents can learn different skills in parallel and obtain more frequent rewards for longhorizon tasks. As shown in Fig. 2 , EnvGen iterates N Cycle training cycles, each consisting of the following four steps:Step 1: Generate training environments with an LLM. As illustrated in step 1 of Fig. 2 , we use an LLM (e.g., GPT-4 (OpenAI, 2023a) ) to first generate N LLM-Env custom training environment configurations1 that can cover various objectives and skills that are required in the original environment. The following describes the LLM input prompt components used to create environment configurations.1. Task description: We provide a brief description of the environment and what the LLM should do (e.g., "generate a set of training environments..."). 2. Game/simulator details: We provide a list of objectives that need to be achieved in the environment (e.g., "collect coal, collect iron, etc." for Crafter); a list of which simulator settings can be controlled (e.g., terrain, agent inventory); and a list of constraints/rules that the simulator has (e.g., "skeletons only spawn in mountains; ..." for Crafter). 
output environment configuration template:: 
 We provide a blank output configuration template (i.e., a JSON object where the environment settings are empty) to the LLM, and request it to fill in the values, creating N LLM-Env environment configurations. Along with filling the templates, we also ask the LLM to verbally explain the purpose for each environment (e.g., what the environment would teach the agent); this would help users easily understand the environment generation process. 4. Adaptation feedback based on the RL agent's performance: We provide the LLM with the performance of the RL agent from the original environment (measured in step 3 and summarized in step 4), as feedback for adapting LLM environments to focus on skills that the RL agent is weak at. The feedback is given at the end of each cycle, so it is only provided to LLM from the second cycle onwards.The obtained environment configurations are then rendered in the game's simulator. Fig. 2 presents the summary of input prompt and output environments from the GPT-4 model. We provide more prompt details in Appendix F. Step 2: Train a small RL agent in the LLM-generated environments. As shown in step 2 of Fig. 2 , we train the small RL agent in the LLM-generated environments. Concretely, we train the agent in the N LLM-Env LLM environments for T LLM-Env total steps in parallel.Step 3: Train and measure the RL agent's performance in the original environment. It is important to note that the goal of EnvGen is to improve the RL agent's performance in the original environment, instead of the performance only in the LLM environments.To help the RL agent effectively adapt to the original environment and provide the LLM with the current agent's performance as feedback, we train the agent and measure its performance in the original environment, as shown in step 3 of Fig. 2 . First, to mitigate the overfitting to LLM environments, we train the agent in the original environment for T Orig-Env steps.2 Next, to find the skills that the RL agent needs to improve at, we test the agent in the original environment, without any parameter updates. Concretely, we measure individual success rates for each environment task (e.g., Crafter achievements). The agent performance is summarized (in step 4) and is provided to LLM as feedback (in step 1) to adapt training environments in the next cycle. Moreover, importantly, to obtain a more calibrated estimation of agent performance, we calculate the average and variance of the task-specific scores by testing agents with multiple random seeds (i.e., 12).Step 4: Send feedback to LLM to adapt environments (to focus on weak skills). We provide the LLM with the agent's performance from the original environment (measured in step 3), as feedback for updating LLM environments. Concretely, we list the agent's average task-specific success rate in percentages along with one standard deviation (e.g., ". . . collect coal: 38% ± 6%, defeat skeleton: 10% ± 4% . . . "), as shown in step 4 of Fig. 2 . In step 1 of the next cycle, the LLM can adaptively generate new environments (by using the agent's performance as feedback) to better help the RL agent learn the skills it is weak at (e.g., defeat skeleton). EnvGen iterates this four-step training cycle N Cycle times.In the following subsections, we present the benchmarks in which we evaluate EnvGen framework on (Sec. 3.1) and the agent architectures that we use for experiments (Sec. 3.2).Crafter. Crafter (Hafner, 2022 ) is an open-world 2D survival game focused on evaluating a broad range of agent capabilities (see Fig. 3 ). Crafter features 22 achievements that an agent can unlock during an episode of play. Some achievements can be unlocked in a few steps (e.g., collect wood, collect sapling, etc.), but other achievements, such as make iron pickaxe or collect diamond, require many training/exploration steps and several prerequisite achievements to be unlocked (see Fig. 3 b ). For example, to make an iron pickaxe, an agent must first collect enough wood to make a table and a wooden pickaxe, then go collect stone and return to the table (or collect more wood to make a new one) and then construct a stone pickaxe. Then the agent still needs to make a furnace, collect coal, and collect iron before the option to make the iron pickaxe is possible.For EnvGen setup, we use N Cycle = 4 training cycles during agent training (see Table 3 for ablation of having a different number of cycles). Each cycle uses 0.12M LLM-generated environment steps (i.e., Crafter EnvGen steps, see step 2 in Fig. 2 ) and 0.12M Crafter steps (step 3 in Fig. 2 ) and then we train for 1M steps in Crafter. In total, we train for 1.96M steps ((0.12M + 0.12M) × 4 + 1M). Note that in order to maintain a fair score comparison to baselines, we do not count any achievement during our training cycle for score calculation since the training scores derived from LLM environments and the original environment are not directly comparable. Instead, we only take into account the achievements from the last 1M training steps in Crafter for the score calculation. See Appendix B for scoring details.We also experiment with giving the baseline model additional original environment steps to match the number of EnvGen steps (i.e., an additional 0.96M steps) to ensure that EnvGen is not better simply because of more steps. We report the average performance with 30 runs (= 3 different initial LLM-generated Crafter EnvGen environments × 10 different random seeds).Heist. Heist is part of the OpenAI Procgen (Cobbe et al., 2020) benchmark. In this environment, agents must successfully 'steal' the gem after navigating a maze and opening all locks. See more details in Appendix C.2.Our base RL agent. For both Crafter and Heist, we test the EnvGen framework with a simple (CNN + linear layer) and lightweight (<5M) agent used in Moon et al. (2023) , which is slightly modified from the agent architecture used in IMPALA (Espeholt et al., 2018) . Following Moon et al. (2023) , we train the agent with a PPO (Schulman et al., 2017) objective. At every step, the agent takes an RGB image (surroundings for Crafter, entire maze for Heist) as input and outputs the value estimates and policy (action probability). See Fig. 3 (a) for an agent visual input example. We provide additional model details in Appendix E.Baseline methods. For Crafter, we compare our method to two groups of recent baselines -(1) methods that use frequent (i.e., more than thousands of) LLM calls during training or inference: SPRING (Wu et al., 2023 ) (based on GPT-4) and ELLM (Du et al., 2023) (based on Codex (Chen et al., 2021) ) and (2) methods that do not use an LLM: DreamerV3 (Hafner et al., 2023) , MuZero + SPR (Walker et al., 2023) , LSTM-SPCNN (Stanić et al., 2023) , PPO (Schulman et al., 2017) , and Achievement Distillation (AD) (Moon et al., 2023) . For Heist, we compare against the PPO agent. For the PPO and AD agents, we follow the implementation of Moon et al. (2023) . See Appendix E for the PPO/AD agent details. 
results and analysis: 
 We demonstrate the usefulness of the EnvGen method with comprehensive experiments and analysis. Crafter EnvGen (i.e., Crafter environments generated with EnvGen) to state-of-the-art baseline methods. As shown in Table 1 , we find that a small (4M parameters) PPO agent with EnvGen achieves an average score of 32.2% and significantly outperforms the baselines (and also in terms of the average reward). Note that some baseline agents have many more parameters or pretraining steps such as SPRING (GPT-4 agent; 27.3%), and MuZero + SPR (150M pretraining steps; 16.4%). Our method also only uses orders of magnitude fewer LLM calls (only 4) than works like SPRING (2.7K on average) and ELLM (5M), making it much cheaper/more efficient. EnvGen can also work with other RL agents such as Achievement Distillation (AD) (Moon et al., 2023) to achieve an even higher score (35.3%).Next, we analyze where EnvGen improves the overall score by checking individual achievement success rates in detail. For this, we compare the same PPO agent architecture trained with different setups: (1) an agent trained on Crafter for 1.96M steps and (2) an agent trained on Crafter EnvGen for 0.96M steps (0.24M steps × 4 training cycles, see Sec. 2.2) and then trained on Crafter for 1M steps. We measure the success rate (Fig. 4 ) of each achievement and unlocking speed (Fig. 5 ) of iron tools in the last 1M training steps.EnvGen helps RL agents to tackle challenging long-horizon achievements. Fig. 4 Cycle 2Here are new environments that can help the agent improve the low scores: ... Environment 1: Introduction to Mining and Crafting ```json { "target_biome": "mountain", "wood_pickaxe": 1, ... } ``P urpose: This environment is designed to teach the agent how to mine coal ... ...Collect coal is 2% Make stone pickaxe is 1% …Here are new environments that can help the agent improve the low scores: ... Environment 2: Introduce combat and advanced tool crafting ```json { "wood": 3, "stone": 2, ... } ``P urpose: With stone in the inventory, the agent is nudged to craft a stone pickaxe, a task it hasn't learned yet ... ... At the end of each cycle, the RL agent's performance is given to the LLM as feedback (e.g., 'Collect coal is 2%'). The LLM uses the feedback to adaptively generate new environments that can help the agent progressively tackle skills it was previously weak at.prerequisites; see Fig. 3 ) such as stone and iron tools. Fig. 5 shows that after unlocking the stone pickaxe, the RL agent trained in Crafter EnvGen is significantly faster in unlocking iron tools. In Appendix C.1, we also compare two AD agents, and show that Crafter EnvGen improves the success rate of the most challenging achievement -'collect diamond'.Fig. 6 shows how the LLM adaptively generates new training environments based on the intermediate performance of our PPO-based RL agent. In the intermediate performance plots, we compare the baseline agent trained only in Crafter and our RL agent trained in Crafter EnvGen . In the cycle 2, given the feedback that the current RL agent is not good at collecting coal, the LLM generates an environment to help the agent focus on it, improving the agent's performance for the skill. Likewise, in the cycle 3, given the feedback that the agent is weak at making stone pickaxes, the LLM generates an environment to help the agent more easily craft the stone pickaxe, helping the agent improve it's score for the skill.Powered by the adaptive LLM environment generation of EnvGen, our agent learns to unlock these two achievements significantly faster than the baseline agent.In the following, we provide analysis and ablation studies of EnvGen: dynamically updating LLM environments (i.e., using adaptive environments) vs. curriculum learning methods, and different frequencies of environment updates. In Appendix C, we show comprehensive analysis and ablation studies of EnvGen method: EnvGen vs.  
related works: 
 LLMs as open-world game agents. Recent works study using LLMs to create action plans (i.e., a list of subgoals or skills to target) for embodied agents in open-world games like Minecraft and Crafter (Hafner, 2022) . Most of these methods require calling LLMs frequently (e.g., at every step) for planning the next steps (Yuan et al., 2023; Wang et al., 2023c; Wu et al., 2023; Wang et al., 2023a; d; Zhao et al., 2023) . Other methods, such as Li et al. (2024) ; Kwon et al. (2023); Ma et al. (2023) ; Du et al. (2023) , have used LLMs to create/adjust rewards to train agents. Although these works show initial promising results leveraging the world knowledge of LLMs to tackle long-horizon tasks, iteratively calling LLMs throughout episodes is prohibitively slow and expensive (e.g., running a single episode in the Crafter environment with SPRING (Wu et al., 2023) costs around $270 USD as they have 2.7K LLM calls on average). EnvGen only calls LLMs a few times (e.g., 4 in total) to create training environments that focus on helping the RL agent progressively improve its weaker skills.Deep learning-based game/simulator content generation. Procedural content generation (PCG) for games is about the automatic generation of levels, landscapes, items, rules, quests, or other types of game contents (Shaker et al., 2016) . While traditional PCG methods are based on search/solver/rule/grammar-based methods, recent works apply deep learning methods such as GAN (Goodfellow et al., 2014) for PCG (Liu et al., 2021; Kumaran et al., 2020; Schubert et al., 2022) . Several works have explored using LLMs to generate game content such as difficulty levels (Sudhakaran et al., 2023; Todd et al., 2023) and scenes/environments (Kumaran et al., 2023; Wang et al., 2023b; Afshar & Li, 2024) . While these works aim to help developers create new game content, we aim to improve RL agent performance in the original environment. A line of work proposes unsupervised environment design (UED) that manipulates the difficulty level of environments to be more challenging to RL agents (Dennis et al., 2020; Jiang et al., 2021; Parker-Holder et al., 2022) . While these works use a learned environment manipulator or evolutionary algorithms to maximize the 'regret' (the difference between the expected return of the current and optimal policies) in simple games such as MiniGrid (Chevalier-Boisvert et al., 2023) , we use the world knowledge of LLMs to generate and adapt training environments that can improve weaker skills based on comprehensive skill-specific feedback from RL agents in open-world games with many challenging long-horizon tasks. To help agents generalize to unseen tasks in a text-based dialogue game, Ammanabrolu et al. ( 2022) augment new tasks with LMs and use a manually designed, fixed curriculum. Unlike this work, we adaptively generate training environments using LLMs' world knowledge and automatically learning a dynamic curriculum based on the RL agent's feedback, so as to improve the agent's weaker skills in open-world games with visual inputs. Beyond game content generation, several works visually augment vision-and-language navigation (VLN) simulators (e.g., rendering environments with different styles) using image generation models (Li et al., 2022b; Wang et al., 2023e; Li & Bansal, 2023) . Such works could complement our LLM environments (e.g., augmenting our environments with diverse colors and textures). 
conclusion: 
 We propose EnvGen, a novel framework to improve embodied RL agent performance by utilizing the world knowledge of LLMs to adaptively generate training environments. In EnvGen, we give an LLM a prompt describing a game/simulator and ask the LLM to generate the configurations to create new environments that can teach different skills. Next, we train an agent in the LLM-generated environments, give feedback to the LLM by testing the agent in the original environments, and then ask the LLM to update the environments to teach agents skills they are weaker at. In two challenging games, Crafter and Heist, we show that training in LLM-generated environments is significantly more effective than training longer in the original environments. We also show that using an LLM to adapt environments dynamically outperforms curriculum learning approaches and how the LLM adapts training environments to help improve RL agents' weaker skills over time. Moreover, a lightweight model (< 5M parameters) trained with LLM-generated environments even outperforms an LLM agent with significantly fewer LLM calls. We hope our work can guide future works in leveraging LLMs for embodied agents.target task. Another recent line of work studies using LLMs to adjust reward functions to help RL agents progressively learn certain tasks (Li et al., 2024; Kwon et al., 2023; Ma et al., 2023; Du et al., 2023) . Instead of designing new rewards, in EnvGen, an LLM adaptively generates training environments that can help the RL agent learn multiple skills it is weak at with fewer training steps than in the original environment; reward design could be complementary to our method. 
 
 Appendix: Crafter. The score for Crafter is computed as the geometric mean of individual success rates of each achievement for each episode it is completed within 1M training steps: S = exp( 1 22 ∑ 22 i=1 ln(1 + s i )) -1, where s i is the average success rate of the ith achievement across all episodes that occurred during training.Heist Environment. Heist is part of the OpenAI Procgen (Cobbe et al., 2020) benchmark. In this environment, agents must successfully 'steal' the gem after navigating a maze and opening all locks (see Fig. 7 ). The gem is behind three layers of color-coded locks, each requiring that the previous lock be unlocked first (e.g., to unlock the green lock, the blue lock must first be unlocked). Following Moon et al. (2023) , the final score is calculated as the average success of the agent in stealing the gem in 100 test episodes in 10 different seeds (i.e., 1,000 runs in total). For agent training, we use a total of 5M steps in the LLM-generated environments (i.e., 5M Heist EnvGen steps) and a total of 20M in the actual Heist environment. As the game only provides scores on the final objective ('steal gem') and the game is simple enough for the LLM-generated environments to cover all scenarios with one generation, we only use N Cycle = 1 training cycle. C.1 Design Choices, Ablations, and Other Agent ArchitecturesIn the following, we show comprehensive design choice and ablation studies of EnvGen method: EnvGen vs. longer training in the original environment, different LLMs for generating environments, the number of LLM environments, and the ratio of training steps in the LLM environments to the original environment. Unless otherwise noted, we use the PPO-based agent (Moon et al., 2023) (described in Sec. 3.2) on the Crafter (Hafner, 2022) benchmark (described in Sec. 3.1) with 0.96M steps in Crafter EnvGen and average results for 30 runs (10 different seeds, 3 different initial environments).EnvGen vs. longer training in the original environment. Environment parameter naming: obscure vs. original. To determine whether or not the LLM in EnvGen is leveraging world knowledge when generating environments we conduct an analysis experiment. We replace environment parameter names from the original ones with obscure names (see Fig. 9 ), in order to remove the use of prior knowledge/expectation of how each parameter is useful to which skills. We find that the performance decreases, from 32.2 ± 0.6 → to 28.5 ± 0.6, indicating that the world knowledge/prior knowledge of the LLM is beneficial in helping the LLM generate more suitable environments.Achievement Distillation + EnvGen. As mentioned in the Sec. 4.1, we also experiment using EnvGen with the Achievement Distillation (AD) (Moon et al., 2023) agent. As shown in Fig. 8 , similar to our results on the PPO-based agent, we find that by applying EnvGen, there is performance gain in long-horizon tasks like making iron tools and collecting diamonds. EnvGen can generalize to Heist. We also evaluate the effectiveness of EnvGen framework with Heist. We compare the PPO-based agent trained with and without EnvGen (i.e., Heist EnvGen environments). Table 8 shows that training an agent with Heist EnvGen environments is effective in improving performance by increasing average scores (25.9% → 37.7%) and rewards (4.1% → 5.5%), while also stabilizing training by reducing the score variance (i.e., standard deviation goes down 13.2% → 7.5%).In the following, we describe the two baseline implementation details described in Table 2 : easy-to-hard and adversarial curricula.Easy-to-hard curriculum. Similar to Ammanabrolu et al. (2022) , we create an easy-to-hard curriculum. An easy-to-hard curriculum has a pre-defined order of training environments. The agent first trains in "easy" environments and then by the end of the training process will be training in the "hard" environments. To do this, we first ask the LLM to generate a set of 16 random environments and train a validation agent (an agent only for the purpose of testing an environment difficulty; not used during final agent training) on each environment. Then the performance of the validation agent is measured and the environments are sorted from easiest to hardest (i.e., from environments that resulted in higher agent scores to lower agent scores). Then we train an agent on the top four easier environments first and for every 0.24M steps we replace the training environments with the next four environments in the sorted set (i.e., easy-to-hard curriculum).Adversarial curriculum. Similar to Parker-Holder et al. (2022) , we create an adversarial curriculum. The adversarial curriculum approach involves updating the agent's training environments to ones where it has struggled. To do this, we first generate a set of 16 random environments and train a validation agent on each environment. Then, we measure and sort the environments by difficulty (i.e., sorted from lowest to highest based on the validation agent's score). We take the top four hardest environments and train the final agent on these. Then, generate a new set of environments and test the current agent on this set, again sorting by difficulty. Then again, we take the top four hardest environments and resume training on these. This process then repeats four times (every 0.24M steps) creating an adversarial curriculum.PPO agent. We use the PPO-based (Schulman et al., 2017) agent used in (Moon et al., 2023) , which modifies the default ResNet (He et al., 2016) architecture in IMPALA (Espeholt et al., 2018) by increasing channel size and hidden size and adding a layer normalization Ba et al.(2016) before each linear/convolutional layer. We slightly modify this architecture further to place the layer norm after the final linear layer instead of before. Hyperparameters for this model are shown in Table 9 .Achievement distillation (AD) agent. Moon et al. (2023) We find that N=4 works well; see Table6for details.We find that T LLM-Env = T Orig-Env works well; see Table7for details. 


Decision: The paper proposes to use LLM to generate environments for training embodied agents.  Based on the agent's performance, new environments are iteratively generated to focus on training agents on the skills where they are weak.  Experiments on Crafter and Heist show that the proposed method can lead to agents that performs better than baselines.

Reviewers are in general positive on the submission, noting that the paper proposes a relatively novel use of LLMs for training agents.  Reviewers also noted the paper is well written (oNhz,y9sf) with experiments that support the usefulness of the proposed idea (oNhz).  However, reviewers had questions about the applicability of the method to more complex environments (Nrar), for what environments and why are LLMs helpful (oNhz,y9sf), and lack of discussion of relation to curriculum learning (oNhz).  Despite the weaknesses noted by the reviewers, reviewers remain positive on the work.  The AC agrees the work presents a novel and interesting perspective on the use of LLMs for training agents and thus recommends acceptances.

The AC encourages the authors to incorporate feedback from the reviewers and clarifications from the rebuttal for the camera ready, including:
- Additional experiments comparing against curriculum learning (oNhz, y9sf)
- Improve discussion of relation to curriculum learning and limitations of using LLMs to generate environments (what knowledge is leveraged, handling of complex environments, does generated environments need to match 'real' environments, etc)
- Address other points of clarifications noted by reviewers

Reviews: review: paper_summary: paper_summary: The paper EnvGen, which uses LLM to construct custom training environment configurations for RL agent training in a iterative process. Experiments on Crafter and Heist game environments shows this method allows training RL agent more efficiently and achieve higher performance, while also outperforming baselines which directly promps a LM.
strength_weakness: strength_weakness: - It is unclear if such methods can be applied in broader, more complex scenarios. As noted by the author, “EnvGen also requires that the environment simulators can [skipped] accept configurations in standard formats (e.g., JSON, CSV, YAML, TOML, etc.),". This is in fact a very strong assumption. Games like Minecraft, modern video games like GTA, or environments in other domains like robot training, might have too many degrees of freedom where such high-level configurations do not exist or require extensive human engineering to make.
questions: questions: - Minor: Figure 5 is a bit confusing to me. While the blue curve is supposed to show how the PPO agent trained on Crafter  improves over its 1.96M steps, the x-axis only has 1M steps. It would be good if you can elaborate that in the caption
paper_summary: paper_summary: This paper proposes a new framework, EnvGen, that uses large language models (LLMs) to generate training environments for embodied agents. Instead of (1) directly employing LLMs as embodied agents, or (2) directly training RL agents in the environment, the authors propose to use LLM reasoning capabilities to adaptively create training environments to help smaller embodied RL agents learn useful skills that they are weak at.

The framework consists of 4 main steps:
1. Ues LLM to generate training environments that allow agents to quickly learn different tasks in parallel.
2. Train small RL agents in the generated environments and potentially learn different skills.
3. Train and measure agent performance in the original environment
4. Use the agent's performance to provide feedback to the LLM, and ask LLm to adapt the generated environments to improve the agent's performance.

The experiment results show that with this EnvGen framework, a small RL agent can outperform large LM and RL agents and learn long-horizon tasks significantly faster.

**My general review on the paper:**

I think this paper is presenting a new perspective on **utlizing** LLMs in embodied learning. Instead of training the LLM or calling LLMs at each step, the authors propose to use LLMs as part of a curriculum learning framework (even though the paper does not explicitly mention this). The LLM is essentially used to generate training environments that help the RL agent learn faster and design curriculum that is tailored to the agent's weaknesses. 

The experiments demosntrate the effectiveness of the proposed framework, and I think they are comprehensive as it compares with non-LLM and LLM-based methods. Analsysis also show insights on how the method can help tha agent learn more efficiently.

There are a few downsides of the paper too. First or all, I think the paper is very closely related to **curriculum learning**, but the authors do not mention this in the paper, nor do they compare with existing curriculum learning methods. Second, the paper does not demosntrate the **upper bound of performance** of this method (e.g., using a larger RL agent or train with much more steps). Currently, the paper only emphasizes the efficiency of the method, but it is not clear how well the method can scale to more complex tasks and larger agents.
strength_weakness: strength_weakness: 1. A lack of comparison with existing curriculum learning methods (both fixed curriculum and adaptive curriculum).
2. Not enough comparison with other content generation methods. A lot of work are menthioned in the related work section "Deep learning-based game/simulator content generation", but it seems none of them are compared with the proposed method.
questions: questions: 1. Why can the LLM generate training environments that help the agent learn faster? What is the intuition behind this? Is it because the LLM has prior knowledge about the environment or is the LLM only inferrencing from the task description?

2. In page 2, you have "Step 2: We train the RL agent ...", but then "Step 3: We first train the RL agent". The word "first" seems a little confusing since you first trained in the LLm-generated environment in step 2.

3. In Figure 4, you compare (1) Baseline: trained in Crafter for **1.96M** steps, and (2) Ours: trained in **0.96M** steps in Crafter-EnvGen and **1M** in Crafter. Why is this comparison useful? Is training in Crafter-EnvGen more efficient than training in the original Crafter environment?

4. In Section 5, "To help agents generalize to unseen tasks in a text-based dialogue game, Ammanabrolu et al. (2022) augment new tasks with LMs and use a manually designed, fixed curriculum." Do you think the proposed method can be applied to this setting? If so, how would your method compare with the existing method?
paper_summary: paper_summary: The paper proposed EnvGen, and LLM based small RL agent training environment dataset generation framework. The method prompts an LLM with game environment task descriptions and objectives, and obtains a more diverse gaming environment configurations for small RL model training. The method also evaluates the performance of the models and provides feedback to prompt LLM to generate game configurations that are more tailored to the RL models in order to improve their performance on the weaker tasks.

The paper conducted experiments on two gaming environment: Crafter and Heist, and compared to both LLM based zero shot agents and trained RL based agent. The paper demonstrated that the proposed method only requires a few LLM API calls and can improve RL agents' performance.
strength_weakness: strength_weakness: 1. To use LLMs to generate training data, it is under the assumption that these generated data is appropriate and close to the real environment. There wasn't any evaluation on the quality of the generated data  
2. To demonstrate the benefits of the feedback loop and curriculum learning, it would be more convincing to have some more comprehensive quantitative results other than one example (w/"collect coal", "make stone pickaxe" ) in Figure 6.

Comments: title: Reviewer Response to Rebuttal
comment: Thank you for your response! I am keeping my scores.
Thanks for your clarifications.