RL-GPT: Integrating Reinforcement Learning and Code-as-policy

Authors: shaoteng  liu
haoqi  yuan
minda  hu
yanwei  li
yukang  chen
shu  liu
zongqing  lu
jiaya  jia
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 11:13:52

Abstract: Large Language Models (LLMs) have demonstrated proficiency in utilizing various tools by coding, yet they face limitations in handling intricate logic and precise control. In embodied tasks, high-level planning is amenable to direct coding, while low-level actions often necessitate task-specific refinement, such as Reinforcement Learning (RL). To seamlessly integrate both modalities, we introduce a two-level hierarchical framework, RL-GPT, comprising a slow agent and a fast agent. The slow agent analyzes actions suitable for coding, while the fast agent executes coding tasks. This decomposition effectively focuses each agent on specific tasks, proving highly efficient within our pipeline. Our approach outperforms traditional RL methods and existing GPT agents, demonstrating superior efficiency. In the Minecraft game, it rapidly obtains diamonds within a single day on an RTX3090. Additionally, it achieves SOTA performance across all designated MineDojo tasks.

Main paper: 
 introduction: 
 Building agents to master tasks in open-world environments has been a long-standing goal in AI research [1] [2] [3] . The emergence of Large Language Models (LLMs) has revitalized this pursuit, leveraging their expansive world knowledge and adept compositional reasoning capabilities [4] [5] [6] . LLMs agents showcase proficiency in utilizing computer tools [7, 8] , navigating search engines [9, 10] , and even operating systems or applications [11, 12] . However, their performance remains constrained in open-world embodied environments [1, 7] , such as Minedojo [13] . Despite possessing "world knowledge" akin to a human professor, LLMs fall short when pitted against a child in a video game. The inherent limitation lies in LLMs' adeptness at absorbing information but their inability to practice skills within an environment. Proficiency in activities such as playing a video game demands extensive practice, a facet not easily addressed by in-context learning, which exhibits a relatively low upper bound [7, 4, 6] . Consequently, existing LLMs necessitate human intervention to define low-level skills or tools.Reinforcement Learning (RL), proven as an effective method for learning from interaction, holds promise in facilitating LLMs to "practise". One line of works grounds LLMs for open-world control through RL fine-tuning [14] [15] [16] [17] [18] [19] . Nevertheless, this approach necessitates a substantial volume of domain-specific data, expert demonstrations, and access to LLMs' parameters, rendering it slow and resource-intensive in most scenarios. Given the modest learning efficiency, the majority of methods continue to operate within the realm of "word games" such as tone adjustment rather than tackling intricate embodied tasks. In particular, by reasoning about the agent behavior to solve the subtask, the LLM generates code to provide higher-level actions in addition to the original environment actions, improving the sample efficiency for RL.Addressing this challenge, we propose to integrate LLMs and RL in a novel approach: Empower LLMs agents to use an RL training pipeline as a tool. To this end, we introduce RL-GPT, a framework designed to enhance LLMs with trainable modules for learning interaction tasks within an environment. As shown in Fig. 3 , RL-GPT comprises an agent pipeline featuring multiple LLMs, wherein the neural network is conceptualized as a tool designed for training the RL pipeline. Illustrated in Fig. 1 , unlike conventional approaches where LLMs agents and RL optimize coded actions and networks separately, RL-GPT unifies this optimization process. The line chart in Fig. 1 illustrates that RL-GPT outperforms alternative approaches by seamlessly integrating both knowledge iteration and skill practice.We further point out that the pivotal issue in using RL is to decide: Which actions should be learned with RL? To tackle this, RL-GPT is meticulously designed to assign different actions to RL and Code-as-policy [20] , respectively. Our agent pipeline entails two fundamental steps. Firstly, LLMs should determine "which actions" to code, involving task decomposition into distinct sub-actions and deciding which actions can be effectively coded. Actions falling outside this realm will be learned through RL. Secondly, LLMs are tasked with writing accurate codes for the "coded actions" and test them in the environment.We employ a two-level hierarchical framework to realize the two steps, as depicted in Fig. 3 . Allocating these steps to two independent agents proves highly effective, as it narrows down the scope of each LLM's task. Coded actions with explicit starting conditions are executed sequentially, while other coded actions are integrated into the RL action space. This strategic insertion into the action space empowers LLMs to make pivotal decisions during the learning process. Illustrated in Fig. 2 , this integration enhances the efficiency of learning tasks, exemplified by our ability to more effectively learn how to break a tree.For intricate tasks such as the ObtainDiamond task in the Minecraft game, devising a strategy with a single neural network proves challenging due to limited computing resources. In response, we incorporate a task planner to facilitate task decomposition. Our RL-GPT framework demonstrates remarkable efficiency in tackling complex embodied tasks. Specifically, within the MineDojo environment, it attains good performance on the majority of selected tasks and adeptly locates diamonds within a single day, utilizing only an RTX3090 GPU. Our contributions are summarized as follows:• Introduction of an LLMs agent utilizing an RL training pipeline as a tool.• Development of a two-level hierarchical framework capable of determining which actions in a task should be learned.• Pioneering work as the first to incorporate high-level GPT-coded actions into the RL action space.2 Related Works 
agents in minecraft: 
 Minecraft, a widely popular open-world sandbox game, stands as a formidable benchmark for constructing efficient and generalized agents. Previous endeavors resort to hierarchical reinforcement learning, often relying on human demonstrations to facilitate the training of low-level policies [21] [22] [23] .Efforts such as MineAgent [13] , Steve-1 [24] , and VPT [25] leverage large-scale pre-training via YouTube videos to enhance policy training efficiency. However, MineAgent and Steve-1 are limited to completing only a few short-term tasks, and VPT requires billions of RL steps for long-horizon tasks. DreamerV3 [26] utilizes a world model to expedite exploration but still demands a substantial number of interactions to acquire diamonds. These existing works either necessitate extensive expert datasets for training or exhibit low sample efficiency when addressing long-horizon tasks.An alternative research direction employs Large Language Models (LLMs) for task decomposition and high-level planning to offload RL's training burden using LLMs' prior knowledge. Certain works [27] leverage few-shot prompting with Codex [28] to generate executable policies. DEPS [29] and GITM [30] investigate the use of LLMs as high-level planners in the Minecraft context. VOY-AGER [1] and Jarvis-1 [31] explore LLMs for high-level planning, code generation, and lifelong exploration. Other studies [32, 33] delve into grounding smaller language models for control with domain-specific finetuning. Nevertheless, these methods often rely on manually designed controllers or code interfaces, sidestepping the challenge of learning low-level policies.Plan4MC [34] integrates LLM-based planning and RL-based policy learning but requires defining and pre-training all the policies with manually specified environments. Our RL-GPT extends LLMs' ability in low-level control by equipping it with RL, achieving automatic and efficient task learning.Several works leverage LLMs to generate subgoals for robot planning [35, 36] . Inner Monologue [37] incorporates environmental feedback into robot planning with LLMs. Code-as-Policies [20] and ProgPrompt [38] directly utilize LLMs to formulate executable robot policies. VIMA [39] and PaLM-E [2] involve fine-tuning pre-trained LLMs to support multimodal prompts. Chameleon [4] effectively executes sub-task decomposition and generates sequential programs. ReAct [6] utilizes chain-of-thought prompting to generate task-specific actions. AutoGPT [7] automates NLP tasks by integrating reasoning and acting loops. DERA [40] introduces dialogues between GPT-4 [41] agents. Generative Agents [42] simulate human behaviors by memorizing experiences. Creative Agent [43] achieves creative building generation in Minecraft. Our paper equips LLMs with RL to explore environments.Since LLMs and RL possess complementary abilities in providing prior knowledge and exploring unknown information, it is promising to integrate them for efficient task learning. Prior works include using decision trees, finite state machines, DSL programs, and symbolic programs as policies [44] [45] [46] [47] [48] [49] [50] .Most work studies improve RL with the domain knowledge in LLMs. SayCan [35] and Plan4MC [34] decompose and plan subtasks with LLMs, thereby RL can learn easier subtasks to solve the whole task. Recent works [51] [52] [53] [54] studies generating reward functions with LLMs to improve the sample efficiency for RL. Some works [55] [56] [57] [58] [59] used LLMs to train RL agents. Other works [14, 15, 60, 61, 16, 17, 62] finetune LLMs with RL to acquire the lacked ability of LLMs in low-level control. However, these approaches usually require a lot of samples and can harm the LLMs' abilities in other tasks. Our study is the first to overcome the inabilities of LLMs in low-level control by equipping them with RL as a tool. The acquired knowledge is stored in context, thereby continually improving the LLMs skills and maintaining its capability. 
methods: 
 Our framework employs a decision-making process to determine whether an action should be executed using code or RL. The RL-GPT incorporates three distinct components, each contributing to its innovative design: (1) a slow agent tasked with decomposing a given task into several sub-actions and determining which actions can be directly coded, (2) a fast agent responsible for writing code and instantiating RL configuration, and (3) an iteration mechanism that facilitates an iterative process refining both the slow agent and the fast agent. This iterative process enhances the overall efficacy of the RL-GPT across successive iterations. For complex long-horizon tasks requiring multiple neural networks, we employ a GPT-4 as a planner to initially decompose the task.As discussed in concurrent works [63, 64] , segregating high-level planning and low-level actions into distinct agents has proven to be beneficial. The dual-agent system effectively narrows down the specific task of each agent, enabling optimization for specific targets. Moreover, Liang et al. highlighted the Degeneration-of-Thought (DoT) problem, where an LLM becomes overly confident in its responses and lacks the ability for self-correction through self-reflection. Empirical evidence indicates that agents with different roles and perspectives can foster divergent thinking, mitigating the DoT problem. External feedback from other agents guides the LLM, making it less susceptible to DoT and promoting accurate reasoning.As previously mentioned, we view the RL training pipeline as a tool accessible to LLMs agents, akin to other tools with callable interfaces. Summarizing the interfaces of an RL training pipeline, we identify the following components: 1) Learning task; 2) Environment reset; 3) Observation space; 4) Action space; 5) Reward function. Specifically, our focus lies on studying interfaces 1) and 4) to demonstrate the potential for decomposing RL and Code-as-policy.In the case of the action space interface, we enable LLMs to design high-level actions and integrate them into the action space. A dedicated token is allocated for this purpose, allowing the neural network to learn when to utilize this action based on observations.We consider a task T that can feasibly be learned using our current computing resources (e.g., lablevel GPUs). We employ a GPT-4 [41] as a slow agent A S . A S is tasked with decomposing T into sub-actions α i , where i ∈ {0, ..., n}, determining if each α i in T can be directly addressed through code implementation. This approach optimally allocates computational resources to address more challenging sub-tasks using Reinforcement Learning techniques. Importantly, A S is not required to perform any low-level coding tasks; it solely provides high-level textual instructions including the detailed description and context for sub-actions α i . These instructions are then transmitted to the fast agent A F for further processing. The iterative process of the slow agent involves systematically probing the limits of coding capabilities.For instance, in Fig. 3 , consider the specific action of crafting a wooden pickaxe. Although A S is aware that players need to harvest a log, writing code for this task with a high success rate can be challenging. The limitation arises from the insufficient information available through APIs for A S to accurately locate and navigate to a tree. To overcome this hurdle, an RL implementation becomes necessary. RL aids A S in completing tasks by processing complex visual information and interacting with the environment through trial and error. In contrast, straightforward actions like crafting something with a table can be directly coded and executed.It is crucial to instruct A S to identify sub-actions that are too challenging for rule-based code implementation. As shown in Table 6 , the prompt for A S incorporates role description {role_description}, the given task T , reference documents, environment knowledge {minecraft_knowledge}, planning heuristics {planning_tips}, and programming examples {programs}. To align A S with our goals, we include the heuristic in the {planning_tips}. This heuristic encourages A S to further break down an action when coding proves challenging. This incremental segmentation aids A S in discerning what aspects can be coded. Further details are available in Appendix A.The fast agent A F is also implemented using GPT-4. The primary task is to translate the instructions from the slow agent A S into Python codes for the sub-actions α i . A F undergoes a debug iteration where it runs the generated sub-action code and endeavors to self-correct through feedback from the environment. Sub-actions that can be addressed completely with code implementation are directly executed, as depicted in the blue segments of Fig. 3 . For challenging sub-actions lacking clear starting conditions, the code is integrated into the RL implementation using the temporal abstraction technique [65, 66] , as illustrated in Fig. 2 . This involves inserting the high-level action into the RL action space, akin to the orange segments in Fig. 3 . A F iteratively corrects itself based on the feedback received from the environment.In Fig. 4 , we have devised a two-loop iteration to optimize the proposed two agents, namely the fast agent A F and the slow agent A S . To facilitate it, a critic agent C is introduced, which could be implemented using GPT-3.5 or GPT-4.The optimization for the fast agent, as shown in Fig. 4 , aligns with established methods for code-aspolicy agents. Here, the fast agent receives a sub-action, environment documents D env (observation and action space), and examples E code as input, generating Python code. It then iteratively refines the code based on environmental feedback. The objective is to produce error-free Python-coded sub-actions that align with the targets set by the slow agent. Feedback, which includes execution errors and critiques from C, plays a crucial role in this process. C evaluates the coded action's success by considering observations before and after the action's execution. Within Fig. 4 , the iteration of the slow agent A S encompasses the aforementioned fast agent A F iteration as a step. In each step of A S , A F must complete an iteration loop. Given a task T , D env , and E code , A S decomposes T into sub-actions α i and refines itself based on C's outputs. The critic's output includes: (1) whether the action is successful, (2) why the action is successful or not, (3) how to improve the action. Specifically, it receives a sequence of outputs Critic i from C about each α i to assess the effectiveness of action planning. If certain actions cannot be coded by the fast agent, the slow agent adjusts the action planning accordingly.Our primary pipeline is tailored for tasks that can be learned using a neural network within limited computational resources. However, for intricate tasks such as ObtainDiamond, where it is more effective to train multiple neural networks like DEPS [29] and Plan4MC [34] , we introduce a task planner reminiscent of DEPS, implemented using GPT-4. This task planner iteratively reasons what needs to be learned and organizes sub-tasks for our RL-GPT to accomplish. 
environment: 
 MineDojo MineDojo [13] stands out as a pioneering framework developed within the renowned Minecraft game, tailored specifically for research involving embodied agents. This innovative framework comprises a simulation suite featuring thousands of tasks, blending both open-ended challenges and those prompted by language. To validate the effectiveness of our approach, we selected certain long-horizon tasks from MineDojo, mirroring the strategy employed in Plan4MC [34] . These tasks include harvesting and crafting activities. For instance, Crafting one wooden pickaxe requires the agent to harvest a log, craft planks, craft sticks, craft tables, and craft the pickaxe with the table. Similarly, tasks like milking a cow involve the construction of a bucket, approaching the cow, and using the bucket to obtain milk.ObtainDiamond Challenge It represents a classic challenge for RL methods. The task of obtaining a diamond demands the agent to complete the comprehensive process of harvesting a diamond from the beginning. This constitutes a long-horizon task, involving actions such as harvesting logs, harvesting stones, crafting items, digging to find iron, smelting iron, locating a diamond, and so on.LLM Prompt We choose GPT-4 as our LLMs API. For the slow agents and fast agents, we design special templates, responding formats, and examples. We design some special prompts such as "assume you are an experienced RL researcher that is designing the RL training job for Minecraft". Details can be found in the Appendix A. In addition, we encourage the slow agent to explore more strategies because the RL task requires more exploring. We encourage the slow agent to further decompose the action into sub-actions which may be easier to code.The training and evaluation are the same as Mineagent or other RL pipelines as discussed in Appendix C. The difference is that our RL action space contains high-level coded actions generated by LLMs. Our method doesn't depend on any video pretraining. It can work with only environment interaction. Similar to MineAgent [13] , we employ Proximal Policy Optimization (PPO) [67] as the RL baseline. This approach alternates between sampling data through interactions with the environment and optimizing a "surrogate" objective function using stochastic gradient ascent. PPO is constrained to a limited set of skills. When applying PPO with sparse rewards, specific tasks such as "milk a cow" and "shear a sheep" present challenges due to the small size of the target object relative to the scene, and the low probability of random encounters. To address this, we introduce basic dense rewards to enhance learning efficacy in these tasks. It includes the CLIP [68] Reward, which encourages the agent to exhibit behaviors that align with the prompt [13] . Additionally, we incorporate a Distance Reward that provides dense reward signals to reach the target items [34] . It costs 3K steps for harvesting a log referring to Table. 16 . For the diamond task, the evaluation ends when the diamond is found or the agent is dead. It will cost around 20K steps. The frame rate for the game is 30 fps. Further details can be found in the appendix C. , our RL-GPT can reasonably decompose the task and correct how to learn each sub-action through the slow iteration process. RL-GPT decomposes the task into "find a tree" and "cut a log", solving the former with code generation and the latter with RL. After a few iterations, it learns to provide RL with a necessary high-level action (attack 20 times) and completes the task with a high success rate. Best viewed by zooming in.MineDojo Benchmark Table 1 presents a comparative analysis between our RL-GPT and several baselines on selected MineDojo tasks. Notably, RL-GPT achieves the highest success rate among all MineAgent, as proposed in [13] , combines PPO with CLIP Reward. However, naive PPO encounters difficulties in learning long-horizon tasks, such as crafting a bucket and obtaining milk from a cow, resulting in an almost 0% success rate for MineAgent across all tasks. Another baseline, MineAgent with autocraft, as suggested in Plan4MC [34] , incorporates crafting actions manually coded by humans. This alternative baseline achieves a 46% success rate on the milking task, demonstrating the importance of code-as-policy. Our approach demonstrates superiority in coding actions beyond crafting, enabling us to achieve higher overall performance compared to these baselines.Plan4MC [34] breaks down the problem into two essential components: acquiring fundamental skills and planning based on these skills. While some skills are acquired through Reinforcement Learning (RL), Plan4MC outperforms MineAgent due to its reliance on an oracle task decomposition from the GPT planner. However, it cannot modify the action space of an RL training pipeline or flexibly decompose sub-actions. It is restricted to only three types of human-designed coded actions. Consequently, our method holds a distinct advantage in this context.In tasks involving and , the agent is tasked with crafting a stick from scratch, necessitating the harvesting of a log. Our RL-GPT adeptly codes three actions for this: 1) Navigate to find a tree; 2) Attack 20 times; 3) Craft items. Notably, Action 2) can be seamlessly inserted into the action space.In contrast, Plan4MC is limited to coding craft actions only. This key distinction contributes to our method achieving higher scores in these tasks.To arrive at the optimal code planning solution, RL-GPT undergoes a minimum of three iterations.As illustrated in Fig. 5 , in the initial iteration, RL-GPT attempts to code every action involved in harvesting a log, yielding a 0% success rate. After the first iteration, it decides to code navigation, aiming at the tree, and attacking 20 times. However, aiming at the tree proves too challenging for LLMs. As mentioned before, the agent will be instructed to further decompose the actions and give up difficult actions. By the third iteration, the agent correctly converges to the optimal solution-coding navigation and attacking, while leaving the rest to RL, resulting in higher performance.In tasks involving crafting a wooden pickaxe and crafting a bed , in addition to the previously mentioned actions, the agent needs to utilize the crafting table. While Plan4MC must learn this process, our method can directly code actions to place the crafting table on the ground, use it, and recycle it. Code-as-policy contributes to our method achieving a higher success rate in these tasks.In tasks involving crafting a furnace and a stone pickaxe , in addition to the previously mentioned actions, the agent is further required to harvest stones. Plan4MC needs to learn an RL network to acquire the skill of attacking stones. RL-GPT proposes two potential solutions for coding additional actions. First, it can code to continuously attack a stone and insert this action into the action space. Second, since LLMs understand that stones are underground, the agent might choose to dig deep for several levels to obtain stones instead of navigating on the ground to find stones.In crafting a milk bucket and crafting wool , the primary challenge is crafting a bucket or shears. Since both RL-GPT and Plan4MC can code actions to craft without a crafting table, their performance is comparable. Similarly, obtaining beef and obtaining mutton needs navigating.ObtainDiamond Challenge As shown in Tab. 2, we compare our method with existing competitive methods on the challenging ObtainDiamond task.DreamerV3 [26] leverages a world model to accelerate exploration but still requires a significant number of interactions. Despite the considerable expense of over 100 million samples for learning, it only achieves a 2% success rate on the Diamond task from scratch.VPT [25] employs large-scale pre-training using YouTube videos to improve policy training efficiency. This strong baseline is trained on 80 GPUs for 6 days, achieving a 20% success rate in obtaining a diamond and a 2.5% success rate in crafting a diamond pickaxe.DEPS [29] suggests generating training data using a combination of GPT and human handcrafted code for planning and imitation learning. It attains a 0.6% success rate on this task. Moreover, an oracle version, which directly executes human-written codes, achieves a 60% success rate.Plan4MC [34] primarily focuses on crafting the stone pickaxe. Even with the inclusion of all human-designed actions from DEPS, it requires more than 7 million samples for training.Our RL-GPT attains an over 8% success rate in the ObtainDiamond challenge by generating Python code and training a PPO RL neural network. Despite requiring some human-written code examples, our approach uses considerably fewer than DEPS. The final coded actions involve navigating on the ground, crafting items, digging to a specific level, and exploring the underground horizontally.Framework Structure In Tab. 4, we analyze the impact of the framework structure in RL-GPT, specifically examining different task assignments for various agents. Assigning all tasks to a single agent results in confusion due to the multitude of requirements, leading to a mere 34% success rate in crafting a table. Additionally, comparing the 3rd and 4th rows emphasizes the crucial role of a critic agent in our pipeline. Properly assigning tasks to the fast, slow, and critic agents can improve the performance to 65%. Slow agent faces difficulty in independently judging the suitability of actions based solely on environmental feedback and observation. Incorporating a critic agent facilitates more informed decision-making, especially when dealing with complex, context-dependent information.Two-loop Iteration In Tab. 3, we ablate the importance of our two-loop iteration. Our iteration is to balance RL and code-as-policy to explore the bound of GPT's coding ability. We can see that pure RL and pure code-as-policy only achieve a low success rate on these chosen tasks. Our method can improve the results although there is no iteration (zero-shot). In these three iterations, it shows that the successful rate increases. It proves that the two-loop iteration is a reasonable optimization choice. Qualitative results can be found in Fig. 5 . Besides, we also compare the results with and without special prompts (SP) to encourage the LLMs to further decompose actions when facing coding difficulty. It shows that suitable prompts are also essential for optimization.Recent works [51, 52] explore the use of LLMs for RL reward design, presenting an alternative approach to combining RL and code-as-policy. With slight modifications, our fast agent can also generate code to design the reward function. However, as previously analyzed, reconstructing the action space proves more efficient than designing the reward function, assuming LLMs understand the necessary actions. Tab. 5 compares our method with the reward design approach. Our method achieves a higher average success rate and lower dead loop ratio on our selected MineDojo tasks. 
conclusion: 
 In conclusion, we propose RL-GPT, a novel approach that integrates Large Language Models (LLMs) and Reinforcement Learning (RL) to empower LLMs agents in practicing tasks within complex, embodied environments. Our two-level hierarchical framework divides the task into high-level coding and low-level RL-based actions, leveraging the strengths of both approaches. RL-GPT exhibits superior efficiency compared to traditional RL methods and existing GPT agents, achieving remarkable performance in the challenging Minecraft environment. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide computational cost in Section 4 of the main paper.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] .Justification: We discussed the broader impacts of this work in Section G of the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We use the openai API to do research.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We follow the license for each asset and cite all the original papers that contribute to our paper. We will also give credits to the used code and follow the license, copyright information, and terms of use in our released code.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Prompt details of fast and slow agent including {role_description}, {planning_tips},{act_info}, and {obs_info} are listed in Table 8 ,9,10. We provide several human-written code blocks following DEPS [29] . We list some codes in Table 11 . These human-provided codes are for crafting tasks since it is difficult for GPT-4 to write correct crafting actions. Our agent will not directly use these crafting codes as API. It will writing code by itself based on these contexts. We also provide the critic agent's prompt details in Table 13 . We have provided a comprehensive overview of helpful and unhelpful low-level skills written by LLMs in Table 15 and Table 14 . These examples showcase the range of skills generated by LLMs, including those beneficial for task completion and others that are less effective. The useful low-level skills identified will be integrated into the action space, as illustrated in Fig. 2 of our paper. It's important to note that unhelpful low-level skills typically fall into two categories: (1) Wrong in details: Examples include actions like attacking a tree a limited number of times or attempting to "use" an item ineffectively for a crafting task. (2) Too difficult actions: While actions such as adjusting sight to the tree before an attack may be reasonable, they can be overly complex to code and execute efficiently.It is difficult to code all actions in this game. We only want to code as many sub-actions as possible. The task of you is to tell me which sub-actions can be coded by you with Python.At each round of conversation, I will give you Task: T Context: ... Critique: The results of the generated codes in the last round Here are some actions coded by humans: {programs} You should then respond to me with Explain (if applicable): Why these actions can be coded by python? Are there any actions difficult to code? Actions can be coded: List all actions that can be coded by you.You should only respond in the format as described below: Explain: ... Actions can be coded: 1) Action1: ... 2) Action2: ... 3) ... Here are the attributes of the obs that can be used: {obs_info}Here are the guidelines of the act variable: {act_info} At each round of conversation, I will give you Task: ... Context: ... Code from the last round: ... Execution error: ... Critique: ... You should then respond to me with Explain (if applicable): Can the code complete the given action? What does the chat log and execution error imply? You should only respond in the format as described below: {code_format} Table 7 : Fast Agent's prompt: Write Python codes.RGB frames provide an egocentric view of the running Minecraft client that is the same as human players see. Data type: numpy.uint8 Shape: (3, H, W), height and width are specified by argument image_size obs["inventory"]["name"]: Names of inventory items in natural language, such as "obsidian" and "cooked beef". Data type: str Shape: (36,) We also provide voxels observation (3x3x3 surrounding blocks around the agent). This type of observation is similar to how human players perceive their surrounding blocks. It includes names and properties of blocks.Names of surrounding blocks in natural language, such as "dirt", "air", and "water". Properties of traced blocks include blocks' names and distances from the agent.Names of traced blocks in natural language in the fan-shaped area ahead of the agent, such as "dirt", "air", and "water". Data type: str Shape: (num_rays,) obs["rays"]["block_distance"]: Distances to traced blocks in the fan-shaped area ahead of the agent. Data type: numpy.float32 Shape: (num_rays,) decompose tasks and learn subtasks with different solutions autonomously. Subtasks requiring simple locomotion skills, such as navigation and reaching, might be easily acquired with code-as-policy via motion planning. Nevertheless, there are other complex skills, such as object manipulation, which necessitate using RL tools to address. In Fig. 6 , we present a qualitative demonstration in the Furniture environment [69] . The motion planning action effectively aids in hole-finding tasks during table assembly, highlighting the practical utility of our approach in complex assembly scenarios. Fig. 7 illustrates the RL training process in the Kitchen environment [70] . The vertical axis represents the success rate, and the horizontal axis represents the number of training steps. Inserting coded motion planning into the action space accelerates learning. Our method learns faster compared to the baseline. In Fig. 8 , we present a qualitative demonstration of the Furniture environment [69] . The motion planning action effectively aids in hole-finding tasks during table and chair assembly. The baseline struggles to find the correct location at the same training step. Modifications are needed for different domains, such as adjusting the task descriptions in the prompts. The powerful zero-shot capability of GPT should ensure generalization ability.We count the average tokens on different tasks for the first 3 iterations in Tab. 17. Works like Voyager only consider high-level planning, using human-coded skill libraries to bypass the need for low-level control. Our method considers both high-level planning and low-level actions, directly facing the Minedojo action space. To acquire low-level policies autonomously, RL is necessary. We visualize these in Tab. 18 . We show the comparison on the "harvest a log" task in Tab. 19. The performance of Claude is similar to GPT-4. Vicuna-13b has lower performance due to its poor coding ability. VLMs can function as more effective critic agents in our framework. While LLMs can only indicate whether the agent succeeded with its coded actions, VLMs can explain why it failed in the environment. As shown in Fig. 9 , GPT-4V provides more detailed feedback across different environments. For example, in Minecraft, it can identify that the agent keeps attacking the ground instead of finding the cow. In the driving simulation environment, it can note that the vehicle is gradually drifting off the road. This feedback can be used by both our fast and slow agents for self-improvement. This framework relies on the LLMs being able to initialize the environment and code reasonable actions. As a result, the upper bound of our method is limited by LLMs' ability. LLMs may exhibit limited out-of-sample generalization capabilities, meaning they may struggle to generalize on unseen data. This implies that in real-world applications, models may exhibit unpredictable or unstable behavior under unknown circumstances, posing challenges to system security. This is particularly critical in domains requiring high reliability and stability, such as autonomous driving or medical devices. Besides, due to the expense of GPT, experiments are completed with fewer random seeds.The integration of RL-GPT in real-world interactions presents exciting opportunities but also raises security concerns. While RL-GPT facilitates seamless collaboration between agents and humans, enabling efficient task execution and adaptive responses, its reliance on large-scale language models introduces vulnerabilities. These vulnerabilities include susceptibility to adversarial attacks, where malicious inputs can manipulate the model's behavior, potentially leading to unintended actions or compromised system integrity. Privacy and security protection are crucial issues when using LLMs for natural language processing. LLMs may handle vast amounts of personal data and sensitive information during training and deployment, necessitating stringent privacy protection measures to prevent data breaches or misuse, thereby safeguarding user privacy and security. • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper discusses how one can design an agent empowered by LLMs that can break down a task into subtasks and dynamically decide whether to solve such a subtask by code generated by the LLM or by a "traditional" RL agent. The method is validated in the game Minecraft. This is an interesting paper; all reviewers unanimously recommended its acceptance, and I'm doing the same.

However, an important point about the number of independent runs was made, and the concerns raised are real. I'm ok accepting this paper because it is a _proof of possibility. It shows how such an idea can be instantiated, and it shows that such an instantiation can work. This is how it should position itself instead of making claims that "it attains State-of-the-Art (SOTA) performance" (line 60). It is hard to imagine someone justifiably claiming SOTA performance without reporting some notion of variability or robustness. Thus, I strong suggest the authors to change their wording accordingly.

Additionally, some of the comparisons to RL methods are unfair, mainly when talking about their sample complexity. In line 75, for example, the paper states that the problem with other techniques is that they "requires billions of steps for fine-tuning long-horizon tasks". It is crucial to understand that LLMs are generated by going over many, many, many samples (orders of magnitude more than billions). Thus, the comparison above is unfair. LLMs allow one to amortize the cost of such samples because LLMs are a general-purpose model that can bypass the lack of initial knowledge of RL agents. I expect the final version of this paper to also better contrast the different alternatives as per this discussion.

Reviews: review: paper_summary: paper_summary: The paper introduces RL-GPT, a novel framework that integrates Large Language Models (LLMs) with Reinforcement Learning (RL) to enhance the performance of LLM-based agents in complex, embodied environments. The primary goal is to address the limitations of LLMs in executing intricate logic and precise control, especially in open-world tasks like those found in the Minecraft game.

The RL-GPT framework employs a two-level hierarchical approach, consisting of a slow agent and a fast agent. The slow agent is responsible for decomposing tasks and determining which actions can be coded directly, while the fast agent generates the corresponding code and integrates RL configurations. This division of labor allows for efficient handling of both high-level planning and low-level execution.

Key contributions of the paper include:
1. Two-level hierarchical framework to determine which actions should be learned by RL and which ones can be coded (e.g. in a python).
2. The introduction of a mechanism where high-level actions coded by LLMs are appended to the RL agent’s action space, instead of only relying on low level actions.
3. Empirical validation showing that RL-GPT outperforms traditional RL methods and existing LLM agents in the Minecraft environment, particularly excelling in the ObtainDiamond task and other MineDojo tasks.

In experiments, RL-GPT demonstrated superior performance, achieving state-of-the-art (SOTA) results in several tasks, including rapidly obtaining diamonds and excelling in long-horizon tasks with limited computational resources.
strength_weakness: strength_weakness: 1. The work focus on Minecraft, while useful as a testbed, may not fully represent the range of challenges present in other open-world environments. Expanding the scope of testing to other domains would enhance the significance of the work

2. While the integration approach is novel, the individual components (LLMs, RL, task decomposition) are well-known techniques. The paper could benefit from a more explicit discussion of how these components are synergistically combined to create a unique solution.

3. Some sections, particularly those describing the two-loop iteration process and the role of the critic agent, could benefit from more detailed explanations or examples to improve understanding. I am aware that the full prompts are included in the appendix and they do not fit in the main paper, however.
questions: questions: 1. The slow agent can all a sub-action that the RL agent executes. But can the fast agent which uses “code as policies” also call the RL agent inside its loop?
2. What is the architecture and inputs of the PPO RL agent? In particular, it was said that “PPO is limited to a set of skills”: how is the sub-action (e.g. “Harvest log” sub-action 2 in Figure 3) from the slow agent represented to the policy network? Is it just a one-hot vector (“PPO is constrained to a limited set of skills”), or are you encoding the sub-action text instruction as an embedding vector, etc., or are you learning a different set of policy weights for each sub-action, or something else?
3. How would the proposed framework adapt to other open-ended environments beyond Minecraft? Are there specific modifications required for different domains?
4. (RL Interface Ablation) Do you have any hypotheses on why the action space reconstruction is more effective than designing the reward function? Is the action space shortening the horizon of the task and makes the reward less sparse? 
5. Table 4: is this showing the %? In Section 4.4, it was reported as 0.34% and 0.65% success rate. Was this a typo and it’s supposed to be 34% and 65% success percentage for crafting a table?

Post rebuttal: I have increased the score after the rebuttal.
limitations: limitations: The authors acknowledge some limitations of their work, notably the reliance on the capabilities of LLMs and the potential challenges in generalizing to unseen data. However, there are additional limitations and assumptions that could be more explicitly addressed:
1. The approach assumes an environment with a structured observation space that can support coding as policies. This might not be feasible in more complex or less structured environments. Environments need to be compatible with the two-level hierarchical framework and provide sufficient information for task decomposition and action coding. It’ll be helpful to provide guidelines for adapting the framework to different types of environments, including those with less structured observation spaces. 
2. The framework's reliance on multiple interacting agents and iterative refinement could complicate implementation and debugging. The need for specific prompts and hand written examples add to the complexity, making it challenging for broader adoption.
paper_summary: paper_summary: This work is a variation of the code as policies, which utilizes LLMs to write code robotic policies in code snippets.  This work examines minecraft, proposing that certain tasks can be composed into two sets: those solvable using LLM generated code and those best left to be solved using a standard RL agent.  They utilize 2 LLM prompting styles.  The first is for an LLM agent which decomposes tasks and determines which ones can be learned as code or using RL.  The 2nd actually implements the code and inserts it into the action space for use by the RL agent.  A critic LLM determines if the action was successful and how to improve it.  This is used for iterative improvement of the code generating LLM.
strength_weakness: strength_weakness: Alot of manual design is needed for each specific environment. 

A large amount of calls to an LLM API are used and this method becomes very expensive quickly.  

I would like to see how VLMs could be leveraged in other environments.

The work does not explore envs outside of Minecraft.  I would like to see how applications of this work could extend to other common RL envs.
questions: questions: Can the authors discuss the broader impact of this method in envs outside of Minecraft and how it can apply to other types of environments in robotics?
limitations: limitations: Limitations are clearly the expense of this work, could the authors discuss a bit more about this?
paper_summary: paper_summary: This paper introduces RL-GPT, a hierarchical framework that uses LLMs to first break down complex embodied tasks into sub-actions that are suitable for coding or learnable through RL, and then write codes or RL configurations to execute the actions. The authors evaluate the framework on MineDojo benchmark and the challenging ObtainDiamond task, showcasing better performance and efficiency compared to existing baselines.
strength_weakness: strength_weakness: The proposed framework is only evaluated on the MineCraft games, which the GPTs have extensive knowledge about due to the massive relevant contents on the internet. It's unclear if the framework could be easily extended to novel domains like simple MuJoCo simulated environments, new games, or more real-world tasks, e.g. navigation or household tasks with real robots.
questions: questions: 1. Have you tried using any smaller / open-source LLMs instead of GPT-4? How does the performance change?
2. When the fast agent fails to code up a sub-action, how does the slow agent decide if it should further break down the action into steps, or use an RL training pipeline?
3. Does the framework keep an archive of solutions (high level plans + codes / RL agents)?
limitations: limitations: Yes.
paper_summary: paper_summary: The authors present a framework to give LLMs the ability to code and train RL agents as a tool for completing tasks. They perform experiments in MineDojo.
strength_weakness: strength_weakness: **Minor Issues:**

- some relevant related works using LLMs to train RL agents: [https://rlvlmf2024.github.io/,](https://rlvlmf2024.github.io/) https://mihdalal.github.io/planseqlearn/, https://clvrai.github.io/boss/, https://clvrai.github.io/sprint/, https://instructionaugmentation.github.io/, https://gimme1dollar.github.io/lift/,

**Clarity:**

- The term “Code-as-Policy” is introduced without definition or citations in the introduction. Is this a common term? Is this referring to the [Code as Policies](https://code-as-policies.github.io/) paper? This will confuse readers who haven’t seen this term before.
- The related works section doesn’t do a great job differentiating this work against prior work. For example section 2.2 doesn’t mention the current work in relation to prior work at all, same with the first paragraph of 2.3.
- The method section isn’t really clear, it’s missing details, references to appendix sections, or references to figures clarifying things. Examples:
    - L133: If the focus is not on the reward function then how is the reward obtained?
    - L136: “A dedicated token is allocated for [integrating high-level actions]…” how is this trained or used?
    - L181: What are the inputs and outputs of $C$? image observatoins or other details from the environment state?
    - Overall, there’s not enough examples or details here to understand the method and how things are learned/trained without having to thoroughly dig through the appendix. For space reasons obviously not all details can be here, but I think this methods section can be rewritten to be much better.
- Experiments:
    - L230 “It costs 3k steps for harvesting a log” what does this mean? This is the minimum number of timesteps required? or is this the empirical number the RL agent that the authors trained requires?
    - L232: please link to a specific section in the appendix.

**Experiments:**

- There are no standard deviations on any numbers, or information about # seeds in the main paper. How many seeds did you run?
- Task-directed Voyager seems like a very direct comparison that uses LLMs for everything instead of RL. Why is this not compared?
questions: questions: See weaknesses
limitations: limitations: Yes, but i think it's typical to include this in the main paper instead of the appendix.

Comments: comment: Thank you for your insightful comments and questions, which have greatly improved the quality of our paper. We deeply appreciate your recognition and the score increase! We will carefully integrate the results discussed with you into the revision.
Thank you for the detailed response addressing my questions and concerns regarding generalization to other environments, how the components are combined and more detailed explanations. I have increased my score to 7.
Thank you for recognizing our work and raising the score. We sincerely appreciate your valuable suggestions and will ensure they are incorporated in the revision.
Thanks for the response. I have increased my score given the additional results.
Thank you for your thoughtful feedback and for raising your score! We appreciate your insights and will carefully revise the writing issues you highlighted. We also take your point about using multiple seeds seriously and will incorporate experiments with multiple seeds in the revision.
Thanks for the response!. Overall, due to NeurIPS' rebuttal format, I cannot determine how the writing issues will be addressed (unless authors provide direct quotes for each part).
However, I have re-read the paper again, with the proposed changes in mind, and I also saw some areas where I simply missed the author's text in answering some of my questions.

Regarding experiments, I fundamentally disagree that experiments in simulation should only be run with 1 seed. VPT does use just one seed, but just because one paper uses one seed doesn't mean that should be the standard everywhere. 

I'm raising my score but keeping it at a borderline simply due to this standard deviation issue as I believe the paper itself has merit.