RL-GPT: Integrating Reinforcement Learning and Code-as-policy
Authors: Shaoteng Liu
Haoqi Yuan
Minda Hu
Yanwei Li
Yukang Chen
Shu Liu
Zongqing Lu
Jiaya Jia
Venue: NeurIPS 2024 oral (2024)
Keywords: Agent
Large Language Models (LLMs)
Reinforcement Learning (RL)
Primary area: reinforcement_learning

Abstract: Large Language Models (LLMs) have demonstrated proficiency in utilizing various tools by coding, yet they face limitations in handling intricate logic and precise control. In embodied tasks, high-level planning is amenable to direct coding, while low-level actions often necessitate task-specific refinement, such as Reinforcement Learning (RL). To seamlessly integrate both modalities, we introduce a two-level hierarchical framework, RL-GPT, comprising a slow agent and a fast agent. The slow agent analyzes actions suitable for coding, while the fast agent executes coding tasks. This decomposition effectively focuses each agent on specific tasks, proving highly efficient within our pipeline. Our approach outperforms traditional RL methods and existing GPT agents, demonstrating superior efficiency. In the Minecraft game, it rapidly obtains diamonds within a single day on an RTX3090. Additionally, it achieves SOTA performance across all designated MineDojo tasks.

RL-GPT: Integrating Reinforcement Learning and Code-as-policy
Shaoteng Liu', Haoqi Yuan', Minda Hu', Yanwei Li!, Yukang Chen," Shu Liu,² Zongqing Lu,4. Jiaya Jia
The Chinese University of Hong Kong SmartMore The Hong Kong University of Science and Technology
School of Computer Science, Peking University Beijing Academy of Artificial Intelligence attps://sites http://.ssss..glleeeeee/e wwt..coo/vvaaeeeeee com/view/rl-gpt.
Abstract
Large Language Models (LLMs) have demonstrated proficiency in utilizing various tools by coding, yet they face limitations in handling intricate logic and precise control. In embodied tasks, high-level planning is amenable to direct coding, while low-level actions often necessitate task-specific refinement, such as Reinforcement Learning (RL). To seamlessly integrate both modalities, we introduce two-level hierarchical framework, RL-GPT, comprising a slow agent and a fast agent. The slow agent analyzes actions suitable for coding, while the fast agent executes coding tasks. This decomposition effectively focuses each agent on specific tasks, proving highly efficient within our pipeline. Our approach outperforms traditional RL methods and existing GPT agents, demonstrating superior efficiency. In the Minecraft game, it possibly obtains diamonds within single day RTX3090. Additionally, achieves good performance on designated MineDojo tasks.
Introduction
Building agents to master tasks in open-world environments has been a long-standing goal AI research [1-3]. The emergence of Large Language Models (LLMs) has revitalized this pursuit, leveraging their expansive world knowledge and adept compositional reasoning capabilities [4-6]. LLMs agents showcase proficiency in utilizing computer tools [7, 8], navigating search engines [9, 10], and even operating systems or applications [11, 12]. However, their performance remains constrained in open-world embodied environments [1, 7]. such Minedojo [13]. Despite possessing "world knowledge" akin to a human professor, L.MS fall short when pitted against a child in a video game. The inherent limitation lies in LLMs adeptness at absorbing information but their inability to practice skills within an environment. Proficiency in activities such as playing video game demands extensive practice, facet easily addressed by in-context learning, which exhibits relatively low upper bound [7, 4, 6]. Consequently, existing LLMs necessitate human intervention to define low-level skills or tools.
Reinforcement Learning (RL), proven as an effective method for learning from interaction, holds promise in facilitating L.Mss "practise". One line of works grounds L.MMs for open-world control through RL fine-tuning [14-19]. Nevertheless, this approach necessitates substantial volume of dopeii- data, expert demonstrations, and access to LLMs' parameters, rendering it slow and resource-intensive in most scenarios. Given the modest learning efficiency, the majority of methods continue operate within the realm of "word games" such intricate embodied tasks.
adjustment rather than tackling
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

cnv-actioonssee _once
Optimized Coded Actions
Optimized Neural Natwork
Optimized Actions + Neural Network
Action Space Design
Action
nuattacklets, 20) for Jf Coaetsines7 ylald net
GPT
RL
RL-GPT (Ours)
/-+code
Succossta
0.4
codee
Policy Network
0.2
Observation
0
200
400 600
800
1000
Epachs
Figure 2: To learn subtask, the LLM can gen- crate environment configurations (task, observa- tion, reward, and action space) to instantiate RL. In particular, by reasoning about the agent be- havior to solve the subtask, the LLM generates code to provide higher-level actions in addition to the original environment actions, improving the sample efficiency for RL.
Figure An overview of RL-GPT. After the optimization in an environment, LLMs agents obtain optimized coded actions, RL achieves an optimized neural network, and our RL-GPT gets both optimized coded actions and neural networks. Our framework integrates the coding parts and the learning parts.
Addressing this challenge, we propose to integrate LLMs and RL in a novel approach: Empower LLMs agents to use an RL training pipeline as a tool. To this end, we introduce RL-GPT, framework designed to enhance LLMs with trainable modules for learning interaction tasks within an environment. As shown in Fig. 3, RL-GPT comprises
agent pipeline featuring multiple LLMs, wherein the neural network is conceptualized as a tool designed for training the RL pipeline. Illustrated in Fig. 1, unlike conventional approaches where LLMs agents and RL optimize coded actions and networks separately, RL-GPT unifies this optimization process. The line chart in Fig. 1 illustrates that RL-GPT outperforms alternative approaches by seamlessly integrating both knowledge iteration and skill practice.
We further point out that the pivotal issue in using RL is to decide: Which actions should be learned with RL? To tackle this, RL-GPT is meticulously designed to assign different actions to RL and Code-as-policy [20], respectively. Our agent pipeline entails two fundamental steps. Firstly, LLMs should determine "which actions" to code, involving task decomposition into distinct sub-actions and deciding which actions can be effectively coded. Actions falling outside this realm will be learned through RI. Secondly, LLMs are tasked with writing accurate codes for the "coded actions" and test them in the environment.
We employ a two-level hierarchical framework to realize the two steps, as depicted in Fig. 3. Allocating these steps to two independent agents proves highly effective, as it narrows down the scope of each LLM's task. Coded actions with explicit starting conditions are executed sequentially, while other coded actions are integrated into the RL action space. This strategic insertion into the action space empowers LLMs to make pivotal decisions during the learning process. Illustrated in Fig. 2, this integration enhances the efficiency of learning tasks, exemplified by our ability to more effectively learn how to break a tree.
For intricate tasks such as the Obtain task in the Minecraft game, devising a strategy with single neural network proves challenging due limited computing resources. In response, we incorporate a task planner to facilitate task decomposition. Our RL-GPT framework demonstrates remarkable efficiency in tackling complex embodied tasks. Specifically, within the MineDojo environment, it attains good performance on the majority of selected tasks and adeptly locates diamonds within a single day, utilizing only an RTX3090 GPU. Our contributions are summarized as follows:
Introduction of an LLMs agent utilizing an RL training pipeline as a tool.
Development of a two-level hierarchical framework capable of determining which actions in a task should be learned.
Pioneering work as the first to incorporate high-level GPT-coded actions into the RL. action space.
2

---

2 Related Works
2.1 Agents in Minecraft
Minecraft, widely popular open-world sandbox game, stands as formidable benchmark for constructing efficient and generalized agents. Previous endeavors resort to hierarchical reinforcement learning, often relying on human demonstrations to facilitate the training of low-level policies [21-23]. Efforts such as MineAgent [13], Steve-1 [24], and VPT [25] leverage large-scale pre-training via YouTube videos to enhance policy training efficiency. However, MineAgent and Steve-1 are limited to completing only a few short-term tasks, and VPT requires billions of RL steps for long-horizon tasks. DreamerV3 [26] utilizes world model to expedite exploration but still demands substantial number of interactions to acquire diamonds. These existing works either necessitate extensive expert datasets for training exhibit low sample efficiency when addressing long- horizon tasks.
An alternative research direction employs Large Language Models (LLMs) for task decomposition and high-level planning to offload RL's training burden using LLMs' prior knowledge. Certain works [27] leverage few-shot prompting with Codex [28] to generate executable policies. DEPS [29] and GITM [30] investigate the use of LLMs as high-level planners in the Minecraft context. VOY- AGER [1] and Jarvis-1 [31] explore LLMs for high-level planning, code generation, and lifelong exploration. Other studies [32, 33] delve into grounding smaller language models for control with domain- specific finetuning. Nevertheless, these methods often rely on manually designed controllers or code interfaces, sidestepping the challenge of learning low-level policies.
PC [34] integrates LLM-based planning and RL-based policy learning but requires defining and pre-training all the policies with manually specified environments. Our RL-GPT extends LLMs' ability in low-level control by equipping it with RL., achieving automatic and efficient task learning.
2.2 LLMs Agents
Several works leverage L.Mss to generate subgoals for robot planning [35, 36). Inner Monologue [37]
incorporates environmental feedback into robot planning with LLMs. Code-as-Policies [20] and
ProgPrompt [38] directly utilize LLMS to formulate executable robot policies. VIMA [39] and
PaLM-E [2] involve fine-tuning pre-trained LLMs to support multimodal prompts. Chameleon [4] effectively executes sub-task decomposition and generates sequential programs. ReAct [6] utilizes chain-of-thought prompting to generate task-specific actions. AutoGPT [7] automates NLP tasks by integrating reasoning and acting loops. DERA [40] introduces dialogues between GPT-4 [41] agents. Generative Agents [42] simulate human behaviors by memorizing experiences. Creative Agent [43] achieves creative building generation in Minecraft. Our paper equips LLMs with RL to explore environments.
2.3 Integrating LLMs and RL
Since LLMs and RL possess complementary abilities in providing prior knowledge and exploring unknown information, it is promising to integrate them for efficient task learning. Prior works include using decision trees, finite state machines, DSL programs, and symbolic programs policies [44-50].
Most work studies improve RL with the domain knowledge in L.Mss. SayCan [35] and Plan4MC [34] decompose and plan subtasks with LLMs, thereby RL can learn easier subtasks to solve the whole task. Recent works [51-54] studies generating reward functions with LLMs to improve the sample efficiency for RL. Some works [55-59] used LLMs to train RL agents. Other works [14, 15, 60, 61, 16, 17, 62] finetune LLMs with RL to acquire the lacked ability of LLMs in low-level control. However, these approaches usually require lot of samples and can harm the LLMs' abilities in other tasks. Our study is the first to overcome the inabilities LLMs in low-level control by equipping them with RL tool. The acquired knowledge is stored in context, thereby continually improving the LLMSs skills and maintaining its capability.
3 Methods
Our framework employs a decision decision-making process to determine whether an action should be executed using code or RL. The RL-GPT incorporates three distinct components, each contributing to its
3

---

Action Space Design
for iouu daaeng act:)=2
Craft
Objective
Navigate to find
Action TV
Artion
Tran
Poliy Netwak
Slow
Plan Subvetion onn
Fast Agent
Feedback
Agent
Sale Autionn 22
7
Enwrcoment
Sa Sul-Action @
Implementation
Whetherts implement Sabastion
code
Ensironment Feedback
Code Implementation
High lovel Action Feedback
Subv-Action:: d
act chann)
Sub-Auction 25--
Agent
Interact
Envrrnmenn
Dirrect Code Implementation
Figure 3: Overview of RL-GPT. The overall framework consists of a slow agent (orange) and fast agent (green). The slow agent decomposes the task and determines "which actions" to learn. The slow agent will improve the decision based on the high-level action feedbacks. The fast agent writes code and RL configurations. The fast agent debugs the written code based the environment feedback ("Direct Code Implementation"). Correct codes will be inserted into the action space as high-level actions ("RL Implementation")
Action Code
Sub-Act
Slow Agent
Sub-Actt (2:...
Fast Agent
Environment
Sub-Act (3)
Observation before after action
Critic (
Critic @
Critic (
Critic Agent
Critic
For Sub-Act Z
Figure 4: The two-loop iteration. We design method to optimize both slow agent and fast agent with a critic agent.
innovative design: (1) slow agent tasked with decomposing given task into several sub-actions and determining which actions can directly coded, (2) fast agent responsible for writing code and instantiating RL configuration, and (3) iteration mechanism that facilitates iterative process refining both the slow agent and the fast agent. This iterative process enhances the overall efficacy of the RL-GPT across successive iterations. For complex long-horizon tasks requiring multiple neural networks, we employ GPT-4 as planner to initially decompose the task.
As discussed in concurrent works [63, 64], segregating high-level planning and low-level actions into distinct agents has proven to be beneficial. The dual-agent system effectively narrows down the specific task of each agent, enabling optimization for specific targets. Moreover, Liang al. highlighted the Degeneration-ol ofeeruution----huuuhtt (DoT) problem, where an LLM becomes overly confident in its responses and lacks the ability for self-correction through self-reflection. Empirical evidence indicates that agents with different roles and perspectives can foster divergent thinking, mitigating the Do'T problem. External feedback from other agents guides the I.IM, making it less susceptible to DoT and promoting accurate reasoning.
3.1 RL Interface
As previously mentioned, we view the RI. training pipeline tool accessible LLMSs agents, akin to other tools with callable interfaces. Summarizing the interfaces of an RL training pipeline,

---

identify the following components: 1) Learning task; 2) Environment reset; 3) Observation space; 4) Action space; 5) Reward function. Specifically, our focus lies on studying interfaces 1) and 4) to demonstrate the potential for decomposing RI. and Code-as-poliey.
In the case of the action space interface, we enable L.MSs design high-level actions and integrate them into the action space. A dedicated token is allocated for this purpose, allowing the neural network to learn when utilize this action based on observations.
3.2 Slow Agent: Action Planning
We consider a task T that can feasibly be learned using our current computing resources (e.g., lab- level GPUs). We employ GPT-4 [41] as a slow agent As. As is tasked with decomposing 7 into sub-actions a;,, where i E {0, ..., ni), determining if each a; in ' can be directly addressed through code implementation. This approach optimally allocates computational resources to address more challenging sub-tasks using Reinforcement Learning techniques. Importantly, As is not required to perform any low-level coding tasks; it solely provides high-level textual instructions including the detailed description and context for sub-actions a; These instructions are then transmitted to the fast agent Ap for further processing. The iterative process of the slow agent involves systematically probing the limits of coding capabilities.
For instance, in Fig. 3, consider the specific action of crafting a wooden pickaxe. Although As is aware that players need to harvest a log, writing code for this task with a high success rate can be challenging. The limitation arises from the insufficient information available through APIs for As to accurately locate and navigate to a tree. To overcome this hurdle, an RL implementation becomes necessary. RL aids As in completing tasks by processing complex visual information and interacting with the environment through trial and error. In contrast, straightforward actions like crafting something with a table can be directly coded and executed.
It is crucial to instruct As to identify sub-actions that are too challenging for rule-based code imple- mentation. As shown in Table 6, the prompt for As incorporates role description (role description). the given task T, reference documents, environment knowledge (minecraft knowledge,,, planning heuristics {planning tips). and programming examples fprograms). To align As with our goals, we include the heuristic in the {planning tips]. This heuristic encourages Ag to further break down an action when coding proves challenging. This incremental segmentation aids As in discerning what aspects can be coded. Further details are available in Appendix A.
3.3 Fast Agent: Code-as-Policy and RL
The fast agent Ar is also implemented using GPT-4. The primary task is to translate the instructions from the slow agent Ag into Python codes for the sub-actions aq. AF undergoes debug iteration where it runs the generated sub-action code and endeavors to self-correct through feedback from the environment. Sub-actions that can be addressed completely with code implementation are directly executed, as depicted in the blue segments of Fig. 3. For challenging sub-actions lacking clear starting conditions, the code is integrated into the RL implementation using the temporal abstraction technique [65, 66]. as illustrated in Fig. 2. This involves inserting the high-level action into the RL action space, akin to the orange segments in Fig. AP iteratively corrects itself based on the feedback received from the environment.
3.4 Two-loop Iteration
In Fig. 4. we have devised a two-loop iteration to optimize the proposed two agents, namely the fast agent Ap and the slow agent As. To facilitate it, critic agent C' is introduced, which could be implemented using GPT-3.5 or GPT-4.
The optimization for the fast agent, as shown in Fig. 4, aligns with established methods for code-as- policy agents. Here, the fast agent receives a sub-action, environment documents Denv (observation and action space), and examples Ecode as input, generating Python code. It then iteratively refines the code based on environmental feedback. The objective to produce error-free Python-coded sub-actions that align with the targets set by the slow agent. Feedback, which includes execution errors and critiques from C, plays a crucial role in this process. C evaluates the coded action's success by considering observations before and after the action's 's exccution.
5

---

Within Fig. 4, the iteration of the slow agent As encompasses the aforementioned fast agent Ap iteration step. In each step of As. Ap must complete an iteration loop. Given task T, D orve and Faode, As decomposes 7 into sub-actions o; and refines itself based on C's outputs. The critic's output includes: (1) whether the action is successful, (2) why the action successful or not, (3) how to improve the action. Specifically, it receives sequence of outputs Critic, from C' about each og to assess the effectiveness of action planning. If certain actions cannot be coded by the fast agent, the slow agent adjusts the action planning accordingly.
3.5
Task Planner
Our primary pipeline is tailored for tasks that can be learned using neural network within limited computational resources. However, for intricate tasks such as ObtainDiamond, where is more effective to train multiple neural networks like DEPS [29] and PIan4MC [34], we introduce task planner reminiscent of DEPS, implemented using GPT-4. This task planner iteratively reasons what needs to be learned and organizes sub-tasks for our RL-GPT to accomplish.
4 Experiments
4.1 Environment
MineDojo MineDojo [13] stands out as a pioneering framework developed within the renowned Minecraft game, tailored specifically for research involving embodied agents. This innovative framework comprises a simulation suite featuring thousands of tasks, blending both open-ended challenges and those prompted by language. To validate the effectiveness of our approach, we selected certain long-horizon tasks from MineDojoo, mirroring the strategy employed in Plan4MC [34]. These tasks include harvesting and crafting activities. For instance, Crafting one wooden pickaxe requires the agent to harvest a log, craft planks, craft sticks, craft tables, and craft the pickaxe with the table. Similarly, tasks like milking a cow involve the construction of a bucket, approaching the cow, and using the bucket obtain milk.
Ohtain Diamond Challenge It represents classic challenge for RL methods. The task of obtaining a diamond demands the agent to complete the comprehensive process of harvesting a diamond from the beginning. This constitutes a long-horizon task, involving actions such as harvesting logs, harvesting stones, crafting items, digging to find iron, smelting iron, locating a diamond, and so on.
4.2 Implementation Details
LLM Prompt We choose GPT-4 as our LLMs API. For the slow agents and fast agents, we design special templates, responding formats, and examples. We design some special prompts such as "assume you are an experienced RL researcher that is designing the RL training job for Minecraft". Details can be found in the Appendix A. In addition, we encourage the slow agent to explore more strategies because the RI. task requires more exploring. We encourage the slow agent to further decompose the action into sub-actions which may be easier to code.
PPO Details The training and evaluation are the same as Mineagent or other RL pipelines as discussed in Appendix C. The difference is that our RL action space contains high-level coded actions generated by LLMs. Our method doesn't 't depend on any video pretraining. It can work with only environment interaction. Similar to MineAgent [13], we employ Proximal Policy Optimization (PPO) [67] as the RI baseline. This approach alternates between sampling data through interactions with the environment and optimizing "surrogate" objective function using stochastic gradient ascent. PPO is constrained to limited set of skills. When applying PPO with sparse rewards, specific tasks such as "milk cow" and "shear a sheep" present challenges due the small size of the target object relative to the scene, and the low probability of random encounters. To address this, we introduce basic dense rewards to enhance learning efficacy in these tasks. includes the CLIP [68] Reward, which encourages the agent to exhibit behaviors that align with the prompt [13]. Additionally, we incorporate a Distance Reward that provides dense reward signals to reach the target items [34].
costs 3K steps for harvesting log referring to Table. 16. For the diamond task, the evaluation ends when the diamond found or the agent dead. It will cost around 20K steps. The frame rate for the game is 30 fps. Further details can be found in the appendix C.

---

Table Comparison of several tasks selected from the Minedojo benchmark. Our RL-GPT achieves the highest success rate on all tasks. All values in our tables refer to the actual successful rate.
TASK
MINEAGENT
0.00
0.00
0.00
0.00
0.00
0.00
MINEAGENT
0.00
0.03
0.00
0.00
0.00
0.46
0.50
0.33
0.35
0.00
(AUTOCRAFT)
PLAN4MC
0.30
0.30
0.53
0.37
0.17
0.83
0.53
0.43
0.33
0.17
RL-GPT
0.65
0.65
0.67
0.67
0.64
0.85
0.56
0.46
0.38
0.32
Table 2: Main results in the challenging ObtainDiamond task in Minecraft. Existing strong baselines in ObtainDiamond either require expert data (VPT, DEPS), hand-crafted policies (DEPS- Oracle) for subtasks, or take huge number of environment steps to train (DreamerV3, VPT). Our method can automatically decompose and learn subtasks with only little human prior, achieving ObtainDiamond with great sample efficiency.
METHOD
TYPE
SAMPLES
SUCCESS
DREAMERV3
100M 16.8B
VPT DEPS-BB BC
IL+RL IL+LLM
20%
0.6%
DEPS- ORACLE
LLM
60%
PLAN4MC
RL+LLM
7M
0%
RL-
RL+LLM
3M
MinsAgent
Success Rate 10%
(FL
RL
RL-GPP (iter O)
Success Rate 0%
Code: Harvest
RL-GPT (iter 1)
Success Rate 18%
Code: Navigate to find
RL
Cut
Code: Aim the Attack limes
RL-GPT (tler 3)
Success Rate 58%
Navigate find
Code:
Figure 5: Demonstrations of how different agents learn to harvest log. While both RL agent and LLM agent learn a single type of solution (RL or code-as aol-ass-oolicy, our RL-GPT can reasonably decompose the task and correct how to learn each sub-action through the slow iteration process. RI. GPT decomposes the task into "find a tree" and "eut log", solving the former with code generation and the latter with RL. After a few iterations, it learns to provide RL with a necessary high-level action (attack 20 times) and completes the task with a high success rate. Best viewed by zooming in.
4.3 Main Results
MineDojo Benchmark Table presents comparative analysis between our RL-GGT and several baselines selected MineDojo tasks. Notably, RL-GPT achieves the highest success rate among all

---

Table 3: Ablation the RL and Code-as-policy com- ponents and the iteration mechanism. RL-GPT outper- forms its Pure RL and Pure Code counterparts. Given more iterations, RL-GPT gets better results.
Table Ablation on the agent structure.
Structure
One Agent
0.34
0,42
Method
Slow Fast
0.52
0.56
Slow + Fast + Critic
0.65
0.67
Pure RL
0.00
0.00
0.00
0.00
Pure Code
0.13
0.02
0.00
0.00
Table 5: Ablation on the RL interfaces.
Ours (Zero-shot)
0.26
0.53
0.79
0.32
Interface
Success f
Dead Loop &
Ours (Iter-2 w/o SP)
0.26
0.53
0.79
0.30
Ours (Iter-2)
0.56
0.67
0.88
0.30
Reward
0.418 0.585
=00.6
Ours (Iter-3)
0.65
0.67
0.93
0.32
Action
s=0.3
baselines. All baselines underwent training with 10 million samples, and the checkpoint with the highest success rate was chosen for testing.
MineAgent, as proposed in [13], combines PPO with CLIP Reward. However, naive PPO encounters difficulties in learning long-horizon tasks, such as crafting a bucket and obtaining milk from a cow, resulting in almost 0% success rate for MineAgent across all tasks. Another baseline. MineAgent with autocraft, as suggested in PIan4MC [34]. incorporates crafting actions manually coded by humans. This alternative baseline achieves a 46% success rate the milking task, demonstrating the importance of rode-as-policy. Our approach demonstrates superiority in coding actions beyond crafting, enabling to achieve higher overall performance compared to these baselines.
PMan4MC [34] breaks down the problem into two essential components: acquiring fundamental skills and planning based these skills. While some skills are acquired through Reinforcement Learning (RL), Plan4MC outperforms MineAgent due to its reliance on an oracle task decomposition from the GPT planner. However, it cannot modify the action space of an RL training pipeline or flexibly decompose sub-actions. is restricted to only three types of human-designed coded actions. Consequently, our method holds distinct advantage in this context.
In tasks involving / and 1 the agent is tasked with crafting stick from scratch, necessitating the harvesting of a log, Our RL-GPT adeptly codes three actions for this: /) Navigate to find tree; 2) Attack 20 times; 3) Craft items. Notably, Action can be seamlessly inserted into the action space. In contrast, Plan4MC is limited to coding craft actions only. This key distinction contributes to our method achieving higher scores in these tasks.
To arrive at the optimal code planning solution, RL-GPT undergoes a minimum of three iterations. As illustrated in Fig. 5, in the initial iteration, RL-GPT attempts code every action involved in harvesting log, yielding a 0% success rate. After the first iteration, it decides to code navigation, aiming the tree, and attacking 20 times. However, aiming at the tree proves too challenging for LLMs. As mentioned before, the agent will be instructed to further decompose the actions and give up difficult actions. By the third iteration, the agent correctly converges to the optimal solution coding navigation and attacking, while leaving the rest to RL, resulting in higher performance.
In tasks involving crafting wooden pickaxe > and crafting bed 4 in addition to the previously mentioned actions, the agent needs to utilize the crafting table. While Plan4MC must learn this process, our method can directly code actions place the crafting table on the ground, use it, and recycle it. Code-as-policy contributes to our method achieving a higher success rate in these tasks.
In tasks involving crafting furnace and stone pickaxe 7t, in addition to the previously mentioned actions, the agent is further required to harvest stones. Plan4MC needs to learn an RL network to acquire the skill of attacking stones. RL-GPT proposes two potential solutions for coding additional actions. First, it can code to continuously attack a stone and insert this action into the action space. Second, since LLMs understand that stones are underground, the agent might choose to dig deep for several levels to obtain stones instead of navigating on the ground to find stones.
In crafting milk bucket @ and crafting wool the primary challenge is crafting a bucket or shears. Since both RL-GPT and PIan4MC can code actions to craft without crafting table, their performance is comparable. Similarly, obtaining beef and obtaining mutton & needs navigating.
8

---

ObtainDiamond Challenge As shown in Tab, 2, we compare our method with existing competitive methods on the challenging ObtainDiamond task.
DreamerV3 [26] leverages a world model to accelerate exploration but still requires a significant number of interactions. Despite the considerable expense of over 100 million samples for learning, it only achieves 2% success rate on the Diamond task from scratch.
VPT [25] employs large-scale pre-training using YouTube videos to improve policy training efficiency. This strong baseline trained on 80 GPUs for days, achieving a 20% success rate in obtaining a diamond and 2.5% success rate in crafting diamond pickaxe.
DEPS [29] suggests generating training data using a combination of GPT and human handcrafted code for planning and imitation learning. It attains 0.6% success rate on this task. Moreover, an oracle version, which directly executes numan-written codes, achieves 60% success rate.
Plan4MC [34] primarily focuses on crafting the stone pickaxe. Even with the inclusion of all human-designed actions from DEPS, it requires more than million samples for training.
Our RL-GPT attains an over 8% success rate in the ObtainDiamond challenge by generating Python code and training PPO RL neural network. Despite requiring some Homan--ritteen code examples, our approach uses considerably fewer than DEPS. The final coded actions involve navigating on the ground, crafting items, digging specific level, and exploring the underground horizontally.
4.4 Ablation Study
Framework Structure In Tab, 4, we analyze the impact of the framework structure in RL-GPT, specifically examining different task assignments for various agents. Assigning all tasks to single agent results in confusion due to the multitude of requirements, leading to a mere 34% success rate in crafting a table. Additionally, comparing the 3rd and 4th rows emphasizes the crucial role of critic agent in our pipeline. Properly assigning tasks to the fast, slow, and critic agents can improve the performance to 65%. Slow agent faces difficulty in independently judging the suitability of actions based solely on environmental feedback and observation. Incorporating a critic agent facilitates more informed decision-making, especially when dealing with complex, context-dependent information.
Two-loop Iteration In Tab. 3, we ablate the importance of our two-loop iteration. Our iteration is to balance RL and code-as asllic explore the bound of GPT's coding ability. We pure RL and pure code-as as- ass-ooicc only achieve a low success rate on these chosen tasks. Our method improve the results although there is iteration (zero-shot). In these three iterations, it shows that the successful rate increases. It proves that the two-loop iteration is a reasonable optimization also compare the results with and
that
choice. Qualitative results can be found in Fig. 5. Besides, without special prompts (SP) to encourage the LLMs to further decompose actions when facing coding difficulty. It shows that suitable prompts are also essential for optimization.
RL Interface Recent works [51, 52) explore the use of LLMs for RL reward design, presenting an alternative approach combining RL and code-as-policy. With slight modifications, our fast agent can also generate code to design the reward function. However, as previously analyzed, reconstructing the action space proves more efficient than designing the reward function, assuming LLMs understand the necessary actions. Tab. 5 compares our method with the reward design approach. Our method achieves a higher average success rate and lower dead loop ratio on our selected MineDojo tasks.
Conclusion
In conclusion, we propose RL-GPT, novel approach that integrates Large Language Models (LLMs) and Reinforcement Learning (RL) to empower LLMs agents in practicing tasks within complex, embodied environments. Our two-level hierarchical framework divides the task into high-level coding and low-level RL-based actions, leveraging the strengths of both approaches. RL-GPT exhibits superior efficiency compared traditional RL methods and existing GPT agents, achieving remarkable performance in the challenging Minecraft environment.

---

Acknowlegdement This work was supported in part by the Research Grants Council under the Areas of Excellence scheme grant A0E/E-601/22-R
References
[1] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlckar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. 3,3
[2] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian lchter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, ct al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.0337 2023. 3
[3] Lei Wang. Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang. Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308. 11432, 2023. 1
[4] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304, 09842, 2023. 3,
[5] Hongru Wang. Huimin Wang, Lingzhi Wang, Minda Hu, Rui Wang, Boyang Xue, Hongyuan Lu, Fei Mi, and Kam-Fai Wong. Tpe: Towards better compositional reasoning over conceptual tools with multi-persona collaboration. arXiv preprint arXiv:2309. 16090, 2023.
[6] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du. Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03622 2022. 1,3
[7] S Gravitas. Auto-gpt: An experimental open-source attempt to make gpt-4 fully autonomous. github. retrieved april 17, 2023, 2023. 3,3
[8] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, al. Metagpt: Meta programming for multi- agent collaborative framework. arXiv preprint arXiv:2308.0m 00352, 2023.
[9] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang. Zihan Wang, Yuxiao Dong, Ming Ding, al. Cogagent: A visual language model for gui agents. arXiv preprint vrXiv:2312.08914 2023.
[10] Izzeddin Gur, Hiroki Furuta, Austin Huang. Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust, A real-world webagent with planning, Iong context understanding, and program synthesis. arXiv preprint 2307 /2856 2023.
[11] Zhao Yang. Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents smartphone users. arXiv preprint 13Xiv:2312.13771, 2023.
[12] Yingqiang Ge, Yujie Ren, Wenyue Hua, Shuyuan Xu, Juntao Tan, and Yongfeng Zhang. Lim (Imao)) agents as apps: Envisioning aios, agents and the aios-agent ecosystem. arXiv preprint arXiv:3312.03815. 2023.
[13] Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, pages I 18343-18362, 2022. 1, 6, 21
[14] Wanpeng Zhang and Zongqing Lu. Rladapter: Bridging large language models to reinforcement learning in open worlds. arXiv preprint arXiv:2309.1711 2023. 1,3
[15] Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li. Ziyue Wang, Chencheng Jiang, Haoran Tan, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vasion-language programmer from environmental feedback. arXiv preprint arXiv: 2rXiv:2310008888 10,08888, 2023. 3
[16] Chengrun Yang. Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le. Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXivv2309.0344 2023. 3
[17] Hao Sun. Offline prompt evaluation and optimization with inverse reinforcement learning, arXiv preprint arXiv:2309.0o 06553,, 2023. 3
10

---

[18] Thomas Carta, Clément Romac. Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Y Oudeyer. Grounding large language models in interactive environments with online reinforcement learning arXiv preprint arXiv:2302.00 0262220222222222 2023.
[19] Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannanch Hajishirzi, Sameer Singh, and Roy Fox. Do emhodied agents dream of pixelated sheep?: Embodied decision making using language guided world modelling. arviv preprint arXiv:2301. 12050, 2023.
[20] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA). pages 1493-9500, 2023. 2,3
[21] William Guss, Brandon Houghton, Nicholay Topin, Phillip Wang. Cayden Codel, Manucla Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv: 1xki::997..13444 13440, 2019.
[22] Anssi Kanervisto. Stephanie Milani, Karolis Ramanauskas, Nicholay Topin, Zichuan Lin, Junyou Li. Jianing Shi, Deheng Ye, Qiang Fu, Wei Yang, et al. Minerl diamond 2021 competition: Overview, results, and lessons learned. NeurlPS 2021 Competitions and Demonstrations Track. pages 13-28, 2022.
[23] Haoqi Yuan, Zhancun Mu, Feiyang Xie, and Zongging Lu, Pre-training goal-based models for sample- efficient reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. 3
[24] Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila Mcllraith, Steve-1: A generative model for text-to- -ee-vviob in minecraft. arXiv preprint arXiv:2306. 00937, 2023, 3
[25] Bowen Baker, lige Akkaya, Peter Zhokov, Joost Huizinga, Jic Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Vidco pretraining (vpt): Learning to act by watching unlabeled online vidcos. Advances in Neural Information Processing Systems, pages
2022, 3, 9
[26] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint urXiv:2301 04104, 2023. 3,
[27] Ryan Volum, Sudha Rao, Michael Xu, Gabriel DesGarennes, Chris Brockett, Benjamin Van Durme, Olivia Deng, Akanksha Malhotra, and William B Dolan. Craft an iron sword: Dynamically generating interactive game characters by prompting large language models tuned on code. In Proceedings of the 3rd Wondplay: When Language Meets Games Workshop (Wordplay 2022), pages 25-43, 2022. 3
[28] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprins arXiv:2107.03374 2021.
[29] Zihao Wang. Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Deseribe, explain, plan and seleet: Interactive planning with large language models enables open-world multi-task agents. arxiv preprint arXiv:2302.08 5600, 2023. 3, 6, 9, 14
[30] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijic Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. arXiv preprint 2rXiv:230S.77144
2023. 3
[31] Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei I Lin, Zhaofeng Hc, Zilong Zheng, Yaodong Yang, et al. Jarvis-l: Open-world multi-task agents with memory- multimodal language models. arXiv preprint arXiv:231 1.05997, 2023.
[32] Yicheng Feng, Yuxuan Wang, Jiazheng Liu, Sipeng Zheng, and Zongqing Lu. Llama rider: Spurring large language models to explore the open world. arXiv preprint arXiv:2310.0892 2023. 3
[33] Sipeng Zheng, Jiazheng Liu, Yicheng Feng, and Zongqing Lu. Steve-eye: Equipping l1m- based embodicd agents with visual perception in open worlds. arXiv preprint arXiv:2310. 13255, 2023. 3
[34] Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongging La. Plan4me: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303.16562 2023. 3, 6, 8,
[35] Michael Ahn, Authony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan. Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169 2022. 3
11

---

[36] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 18-9147, 2022. 3
[37]
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.00 05608, 2022. 3
[38] Ishika Singh. Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523- 11530, 2023. 3
[39] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv, 2022.
[40] Varun Nair, Elliot Schumacher, Geoffrey Tso, and Anitha Kannan. Dera: enhancing large language model completions with dialog-enabled resolving agents. arXiv preprint vrXiv:2303.10077 2023.
[41] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2xxi:22333007772 2023. 5,5
[42] Joon Sung Park, Joseph C O' O'Brinn, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.00 2304.0344, 2023. 3
[43] Chi Zhang, Penglin Cai, Yuhui Fu, Haoqi Yuan, and Zongqing Lu. Creative agents: Empowering agents with imagination for creative tasks. arXiv preprint arXiv:2312.02519 2023. 3
[44] Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmect Kohli, and Swarat Chaudhuri. Pro- grammatically interpretable reinforcement learning. In International Conference on Machine Learning. pages 5045-5054, 2018. 3
[45] Abhinav Verma, Hoang Le, Yisong Yue, and Swarat Chaudhuri. Imitation-projectee programmatic reinforcement learning. Advances in Neural Information Processing Systems, 2019.
[46] Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 2018
[47] Jeevana Priya Inala, Osbert Bastani, Zenna Tavares, and Armando Solar-Lezama. Synthesizing program- matic policies that inductively generalize. In 8th International Conference on Learning Representations, 2020.
[48] Dweep Trivedi, Jesse Zhang, Shao- Sun, and Joseph J Lim. Learning synthesize programs as interpretable and generalizable policics. Advances in neural information processing systems, pages 25146-25163, 2021.
[49] Wenjie Qiu and He Zhu, Programmatic reinforcementt learning without oracles. In International Conference on Learning Representctions, 2021.
[50] Guan- Liu, En-Pei Hu, Pu-Jen Cheng. Hung- Yi Lee, and Shao-Hua Sun. Hierarchical programmatic reinforcement learning via learning compose programs. In International Conference on Machine Learning, pages 116722216999 2023. 3
[51] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 12.11111,, 2023. 3,
[52] Hao Li, Xue Yang, Zhaokai Wang. Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, and Jifeng Dai. Auto me-reward: Automated dense reward design with large language models for minecraft. arXiv preprint arXiv:2312.092333 2023. 9
[53] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2rearr: Automated dense reward function generation for reinforcement learning. arXiv preprint arXiv:230911188, 2023.
12

---

[54] Wenhao Yu, Nimrod Gileadi, Chuyuan Fu. Sean Kirmani, Kuang-Huet Lee, Montse Gonzalez Arenas. Hao-Tien Lewis Chiang. Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language rewards for robotic skill synthesis. arXiv preprint urXiv:2306. 08647, 2023. 3
[55] Yufei Wang. Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. RI-vIm-f: Reinforcement learning from vision language foundation model feedback. arXiv preprint arXiv:2402.03682 2024. 3
[56] Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, and Ruslan Salakhutdinov. Plan-seq leq--eem-eamm Language model guided rl for solving long horizon robotics tasks. arXiv preprint arXiv:2405.013 01534, 2024.
[57] Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. arXiv preprint Xriiv23100. 10021, 2023.
[58] Jesse Zhang, Karl Pertsch, Jiahui Zhang. Taewook Nam, Sung Ju Hwang, Xiang Ren, and Joseph J Lim. Sprint: Scalable semantic policy pre-training via language instruction relabeling. 2022.
[59] Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint wxXiv2221..11555 2022. 3
[60] Danyang Zhang, Lu Chen, Situo Zhang. Hongshen Xu, Zihan Zhao, and Kai Yu. Large language model is semi parameeramcttic reinforcement learning agent. arXiv preprint arXiv:2306.0799 2023. 3
[61] Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. Query-dependent prompt evaluation and optimization with offline inverse rl arXiv e-prints, 2023. 3
[62] Ryan Shea and Zhou Yu. Building persona consistent dialogue agents with offline reinforcement learning. arXiv preprint arXiv:2310.1077
2023. 3
[63] Tian Liang. Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang. Rui Wang, Yujuu Yang. Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305. 19118, 2023. 4
[64] XAgent Team. Xagent: An autonomous agent for complex task solving, 2023.
[65] Richard S Sutton. Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for
temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2)111-211, 1999. 5
[66] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pages 188 204. PMLR, 2021.
[67] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
[68] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages $748-8763, 2021.
[69] Youngwaon Lee, Edward S Hu, and Joseph Lim. Ikea furniture assembly environment for long-horizon complex manipulation tasks. 2021 ieee international conference on robotics and automation (icra), pages 3343-6349, 2021, 22
[70] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning, arXiv preprint 2019, 22
13

---

Appendix
A Agent Prompt Details
Prompt details of fast and slow agent including (role_ description). [planning [apt) _ino), and {obs_ine _nfoo are listed in Table 8,9,10. We provide several human-written code blocks following DEPS [29]. We list some codes in Table 11. These human-providee provided codes are for crafting tasks since is difficult for GPT-4 write correct crafting actions. Our agent will not directly use these crafting codes as API. It will writing code by itself based on these contexts. We also provide the critic agent's s prompt details in Table 13. We have provided comprehensive overview of helpful and unhelpful low-level skills written by LLMSs in Table 15 and Table 14. These examples showcase the range of skills generated by L.Mss, including those beneficial for task completion and others that are less effective. The useful low-level skills identified will be integrated into the action space, illustrated in Fig. 2 of our paper. It's important to note that unhelpful low-level skills typically fall into two categories: (1) Wrong in details: Examples include actions like attacking tree limited number of times or attempting to "use" an item ineffectively for crafting task. (2) Too difficult actions: While actions such as adjusting sight to the tree before attack may be reasonable, they can be overly complex to code and execute efficiently.
It is difficult to code all actions in this game. We only want to code as many sub-actions as possible. The task of you is to tell me which sub-actions can be coded by you with Python.
Here are some basic actions coded by humans: (programs p
Please inherit the class CodeAgent. You are only required to overwrite the function main function.
At each round of conversation, will give you
Task: T
Context:
Here are some reference examples written by me:
Critique: The results of the generated codes in the last round
Here are some actions coded by humans: (programs)
Here are the attributes the obs that can used: jobs_info)
You should then respond to me with Explain (if applicable): Why these actions can be coded by python? Are there any ac- tions difficult to code? Actions can be coded: List actions that can be coded by you.
Here are the guidelines of the act variable: (act_inf()
At each round of conversation, will give you Task:
Context
Important Tips: (planning tips)
Code from the last round:
Execution emror:
Critique:
You should only respond in the format as described below:
You should then respond me with Explain (if applicable): Can the code complete the given action? What does the chat log and execution error imply?
Explain:
Actions can be coded:
1) Actionl:
2) Action2:...
You should only respond in the format as de- scribed below:
3)... ...
Table 6: Slow Agent's prompt: Decom-
Table 7: Fast Agent's prompt: Write Python codes.
pose a task into sub-actions.
14

---

(role description):
You are playing the game Minecraft. Assume are Python programmer.. You want to write python code complete parts of this game.
{planning tips):
1) If is unsuccessful to code one action in the last round, it means the action is too difficult for coding.
2) If one action in the last round is too difficult to code, try to further subdivide the action. For example, if "attacking the 20 times" is difficult, try "simply attacking 20 times".
3) Please refer the additional knowledge about Minecraft. is very useful.
Table Slow Agent's 's prompt details
(role description):
We want to write python code to complete some actions Minecraft, You are helpful assistant that helps to write the code for the given action tasks.
(act info]:
We design a compound action space. each step the agent chooses one movement action (forward, backward, camera actions, etc.) and one optional functional action (attack, use, craft, etc.). Some functional actions such as craft take one argument, while others like attack does not take any argument. This compound action space be modelled in autoregressive manner.
Technically, our action space is multi-discrete space containing eight dimensions: > >>> emaaction, space MultiDiscrete((3, 3, 4, 25, 25, 8, 244, 36])
Index Forward and backward; noop, 1: forward, back
Index 1; Move left and right: 0: noop, 1: move left, 2: move right
Index 2; Jump, sneak, and sprint; 0: noop, 1: jump. 2: sneak, 3:sprint
Index 3; Camera delta pitch; 0: -180 degree, 24: 180 degree
Index 4; Camera delta yaw; 0: -180 degree, 24: 180 degree
Index Functional actions: O: noop, use, 2: drop. 3: attack, craft, equip. place, 7: destroy
Index 6; Argument for "craft"; All possible items to be crafted
Index 7; Argument for "equip", "place". and "destroy"; Inventory slot indice
Table 9: Fast Agent's prompt details
15

---

obs("rgb"|:
RGB frames provide egocentric view of the running Minecraft client that is the same human players see.
Data rype: numpy.uint8
Shape: (3, H, W), height and width are specified by argument image_size
obs[ "inventovv"""" nammee""
Names of inventory items natural language, such "obsidian" "cooked beef""
Data rype:
Shape: (36.)
We also provide voxels observation (3x3x3 surrounding blocks around the agent). This type of observa- tion is similar human players perceive their surrounding blocks. It includes names and properties of blocks.
obs( voxe"vvxe"""""l" bloodss""""""__oaaaa" _|
Names of surrounding blocks in natural language, such "dirt", "air", and "water".
Data rype:
Shape: (3, 3, 3)
obs[ "locatona stats" "Iats""!"pos"; The xyz position the agent.
Data type: numpy.float32
Shape: (3.)
obs(" dos[[location__ stats" "aw"" and "ooratton_stat"[["paa Yaw and pitch the agent.
Data type: numpy.float32
Shape: (1.)
obs[ "locationn stats" "iits" id"]: Biome ID of the terrain the agent currently occupies.
Data type: numpy.int64
Shape: (1.)
Lidar observations are grouped under obs["rays" includes three parts: information about traced entities, properties of traced blocks, and directions of lidar rays themselves.
abs["rays"T]] " name"]:
Names of traced entities.
Data rype: str Shape: (num_rays.)
obs""rays"I"eenttt distance"]:
Distances to traced entities.
Data type: numpy.float32
Shape: @oum_rays..
Properties of traced blocks include blocks' names and distances from the agent.
obs[ Aos["rays"ll "I"yss"""lookk [[""loockname""i
Names of traced blocks in natural language in the fan-shaped area ahead of the agent, such as "dirt", "air", and "water".
Data type: Shape: (num_rays.)
obs[ obs["rays"]""boce distance"]: Distances to traced blocks in the fan-shaped area ahead of the agent.
Data rype: numpy.float22
Shape: (num (num_rays.)
Table 10: Observation information (obs_ jins) of Fast Agent
16

---

# look specific direction def look to(self, deg =0==0); 0):
#accquire info obs, reward, done, info = self.accquire info()
obs self.env.obs_
while obs abs[""loatioon pittc"" < deg:
act secff..rvaactoonsspcc......
act[]]= = =
yield act
obs, reward, done, info self.accquire selfaccquire_info _nffo0
while obs["location_saae "esecttio_stat"""" "|atonsssaast"[[""" "["as""[["ichh"" > deg:
self.env.action_space cenvvaction_ssacceno__op
act[3]= = act[5] 3
yield act obs, reward, done, info = seffaacuurruiireinnnoo i_nfo(
place the item the hands def place(self, goal):
slot self.indxx slot(goal)
if slot =1 -1
return False
act = wew..enw.action_.eeeee spprorsspccece_ppp
act[2] =
act[5]
act[7] =
yield act
# place the table in the hands and use it
def place dofn(self, goal):
self index, sefffiinxxsssogodaddd slot(goal) ==
return None
for act in chain(
self. self.look_nee saffllookttocccg_3a to(dcg=83), self.attack(2), self place(
yield act
# recycle the table after using it def recycle(self, goal, times 20):
for i in range(times):
self.en ee.f.env.action__. cittiiiosspacnno_oppl act[5]=
obs, reward, done, info www...ew.....ppeeeeeeee .en.sttcp(actt
if any([iemm name'] = goal for item in iiffooiiventtoy"" I]): break
yield self.env.action awwaaacctttcccnnnnnne space.no_opp vo_op() for act in chain( self.look_to(O), yield act
Table 11: Human written code examples for crafting actions. (Continued in Table 12)
17

---

# directly craft something without crufting table
def seff.cnv.actionsssacc....
goal):
act
act[5]= =
act[6] = self.craft_ sssesssmeetttt items.index(goa) print(goal, self.craft_smelt settiismmttttttssiiite index((a)d)
yield act
craft something def craft w table(self, goal);
crafting table
#print 'here,, self. pandx_ssorf crrftting_aable if selffindex_slee s_ssss(oo' grafting_table' == -1:
return None
for act in chain(
self. place pllace_down((((e crafting table"),
serratt_wo_tttee woo_able(goal),
selfcrecycle( cecccccccccraattingge baatting_tbbees 200),
print((" goal: act") yield act
smelt something with a furnace
def smelt w furnace(self,, goal);
#print Here', sedfi slotf furnace') ))
if self selffindex_stee _sootssoo furnace') = -1:
return None
for act in chain( placee pww_c_ddowwweee""e down('furnace furnace') )
self.craft wo table(goal). recycle" furnace', (fcrcccl""
200),
yield act
# directly smelt something without a furnace
def smelt wo furnace(self, goal): for act in self.craft_ww seraaf_ww_taahl too_tablc(goal): yield act
Table 12: written code examples for crafting actions.
18

---

We want to write python code complete some actions in Minecraft. You assistant that assesses whether the coded actions effective. Can the code complete the target action? You need to analyze why the code is successful or not. Please give detailed reasoning and critique.
will give you the following information:
code action: The action we want code. Context for the action: The context the coded action.
Code: The code written by to complete the action.
Observations before running coded action:
Inventory Name: Names of inventory items natural languages, such as "obsidian" and "cooked beef". Inventory Quantity
Blocks in lidar rays: Names of traced blocks.
Entities in lidar rays: Names of traced entities.
Around blocks: Names surrounding blocks natural language, such "dirt", "air", and "water".
Observations after running the coded action: Inventory Name Inventory Quantity Blocks lidar rays Entities in lidar rays
Around blocks
You should only respond JSON format as described below:
"reasoning": "reasoning",
"success": boolean,
"critique"; "critique",
Table 13: Critic Agent's prompt details
# unuseful skills # continuously attacking 10 times (not enough break a log) for act in chain(
self.attack(1O),
):
yield act
"use" instead of "craft"
act[5] 3 act[6] = ceelf.cratt_smeeee litgl_ssseeslssiiiiiieeee
items wttiiittemeeettee index(goal)
# look at the (difficult succeed) while not telf.target sight(obs, 'wood', max dis=5):
act [3 = 13
act 5] 3
yield act
look at tree attack (the free may not in the front) for act in chain( self.look to o_front,
selfattack.
yield act ):
Table 14: Unuseful low-level skills designed by the LLMs
19

---

# useful skills
# continuously attacking 20 times def attack(self,, times 20): for range(times):
act[5= = yield act
for act in chain( self.attack(20),
):
yield act
# craft an item
act]5] =4
act[6] self.craft sweettstmaatttttteeeeeeeeeeeeeee items wwwps/ttttmsiieeeeeeeeeeeeeepeep
look to front
while ohs( [on["location,_ stats pitchh"" < 50:
act[3] = 13
act[5] 3
yield act
while obs[ "llocation stats"][" pitccch"" > 50:
act[3] = 11
act[5] 3
yield act
# move forward for in range(10):
act[0] = yield act
# navigate to find cow
if random,randint((, 20) == 0:
act[4]
if rrandom.anddit( 20) =
act[0]= =1
for act in chain(
self.ookk to front, folf.forward,
self.attack,
)
yield act
place the crafting table
slot self.index slot( crafting table")
act[2] = 1
act[5] 6
act| slot
yield act
# mine deep to depth if self.env.obs] "location "tats"]["pos"" depth:
self.env.obs (wwycato"oo stats"]["piin "["ptchh" < 80:
act[3]= = 13
yield act
else:
act[5] =
yield act
Table 15: Usefuu low-level skills designed by the LLMs.
20

---

B Algorithms
Algorithm RL-GPT's Two-loop Iteration
Input: Task T. Slow agent As, Fast agent Ar, Critic agent C, Prompt for slow agent Ps. Prompt for fast agent Pp. Critic, None repeat
a0,.,,, As(T, Ps) for = to n do
repeat
Code = A (a,, Pr. Critic;)
act_space config(Code)
Obs; = rl_ rrt__rrininnaatt_spaaae
Critic; CF (rl_config, code, Obs;)
until no bug
end for Ps Ps + Critico +... + Critic, until T is complete
Actions deemed unsuitable for coding are not included in ao,,., On. ensuring they are learned naturally through RL neural networks during training.
C Details in PPO
CLIP reward. The reward incentivizes the agent to generate behaviors aligned with the task prompt. 31 task prompts are selected from the entire set of MineDojo programmatic tasks as negative samples. Utilizing the pre-trained MineCLIP model [13], we calculate the similarities between the features extracted from the past 16 frames and the prompts. The probability is then computed, indicating the likelihood that the frames exhibit the highest similarity to the given task prompt: p [softmax (S (fe: fi), {S (fer f--))->-l lo- where fr, is the task prompt, and 1 are negative prompts. The CLIP reward is:
video features and prompt features, [
TCLIP = max {
{
32 0
Distance reward. The distance reward offers dense reward signals for reaching target items. In combat tasks, the agent receives a distance reward when the current distance is closer than the minimum distance observed in history:
do
T Tdistonce max
min de
P<t
For mining tasks, where the agent needs to remain close to the block for several time steps, adapt the distance reward to promote maintaining a small distance:
dd-1 de. 1.5 S de S to0
Ydistonce
2, d, < 1.5
-2 dj +00,
where de is the distance between the agent and the target item time step 4, detected through lidar rays in the simulator.
D Other Environments
The powerful zero-shot capability of GPT serves as a guarantee of its generalization ability. also represents an advantage of RL-GPT over pure RL methods. Our method mainly works in complex environments with both high-level planning and low-level controlling, like the real world. In robotic planning and motor execution, RL-GPT is promising framework to
tasks requiring
21

---

Table 16: Settings for MijeDojo tasks our paper.
Task Icon
Target Name
Initial Tools
Biome
Max Steps
stick
plains
3000
crafting _rafting_table____
table_
plains
3000
nearby
>
wooden_pickaxe
forest
3000
furnace furnace_nearby
10
hills
5000
7
stone_pickaxe
forest_hilly
10000
milk_bucker
33
plains
3000
wool
#2
plains
3000
beef
plains
3000
mutton
plains
3000
bed
plains
10000
Table 17: OpenAl tokens it takes to "speed up" RL.
ITER-1
ITER-2
ITER-3
TOKENS 10K
15K
16K
decompose tasks and learn subtasks with different solutions autonomously. Subtasks requiring simple locomotion skills, such navigation and reaching, might be easily acquired with code-as code-as-policy via motion planning. Nevertheless, there are other complex skills, such as object manipulation, which necessitate using RI. tools to address. In Fig. 6, present qualitative demonstration in the Furniture environment [69]. The motion planning action effectively aids hole-finding tasks during table assembly, highlighting the practical utility of our approach in complex assembly scenarios. Fig. 7 illustrates the RL training process in the Kitchen environment [70]. The vertical axis represents the success rate, and the horizontal axis represents the number of training steps. Inserting coded motion planning into the action space accelerates learning. Our method learns faster compared to the baseline. Fig. 8, we present qualitative demonstration of the Furniture environment [69]. The motion planning action effectively aids in hole- mool-ffnding tasks during table and chair assembly. The baseline struggles to find the correct location at the same training step. Modifications are needed for different domains, such adjusting the task descriptions in the prompts. The powerful zero-shot capability of GPT should ensure generalization ability.
E
Other Details
We count the average tokens different tasks for the first 3 iterations in Tab. 17. Works like Voyager only consider high-level planning, using human-coded skill libraries to bypass the need for low-level control. Our method considers both high-level planning and low-level actions, directly facing the Minedojo action space. To acquire low-level policies autonomously, RL is necessary. We visualize these in Tab. 18. We show the comparison on the "harvest log" task in Tab. 19. The performance of Claude is similar to GPT-4. Vicuna-13b has lower performance due to its poor coding ability. VLMs can function as more effective critic agents in our framework. While LLMs can only indicate whether the agent succeeded with its coded actions, VLMs can explain why it failed in the environment. As shown Fig. 9, GPT-4V provides more detailed feedback across different environments. For example, in Minecraft, it can identify that the agent keeps attacking the ground instead of finding the cow. In the driving simulation environment, it can note that the vehicle gradually drifting off the road. This feedback can be used by both our fast and slow agents for improvempeeeetent.
22

---

Table 18: High-level comparison among different methods.
METHOD
LONG-HORIZON TASK
LOW-LEVEL CONTROL SAMPLE-EFFICIENCE SELF-IMPROVEMENT
MINEAGENT
X
VPT
X
DEPS
XX
VOYAGER
RL--GTT
Table Comparison among different LLMs.
LLMs
SUCCESS RATE
DEAD LOOP
VICUNA-138
0.36
0.8
CLAUDE
0.64
0.3
GPT-4
0.65
0.3
Baseline: Insert the wrong position
Ours: Motion planning (as an action) to the hole
Figure 6: Qualitative results on Furniture. Using motion planning as an action can help the robot arm find the hole more efficiently.
F Limitations
This framework relies on the LLMs being able initialize the environment and code reasonable actions. As result, the upper bound of our method is limited by LLMs' ability. LLMs may exhibit limited out-of-sample generalization capabilities, meaning they may struggle generalize data. This implies that in real-world applications, models may exhibit unpredictable or unstable behavior under unknown circumstances, posing challenges system security. This is particularly critical in domains requiring high reliability and stability, such as autonomous driving or medical devices. Besides, due the expense of GPT, experiments completed with fewer random seeds.
unseen
G Broader Impact
The integration of RL-GPT in real-world interactions presents exciting opportunities but also raises security concerns. While RL-GPT facilitates seamless collaboration between agents and humans, enabling efficient task execution and adaptive responses, its reliance on large-scale language models introduces vulnerabilities. These vulnerabilities include susceptibility to adversarial attacks, where malicious inputs can manipulate the model's behavior, potentially leading unintended actions or compromised system integrity. Privacy and security protection are crucial issues when using LLMs for natural language processing. LLMs may handle vast amounts of personal data and sensitive information during training and deployment, necessitating stringent privacy protection measures to prevent data breaches misuse, thereby safeguarding user privacy and security.
23

---

Kettle
Hinge Cabinet
A0
Success
0.6
0.6
0.4
100K
100K
1.50K
200K
Number timesteps
Number timvesteeps
Slide Cabinet
Tbp Burner
3.0
1.0
0.0
0.8
0.6-
0.6-
0.4-
32K
100k
50K
100K
150k
Number of timestops
Number timestopp
Figure 8: Qualitative demonstrations in the Fur- niture environment show that our motion plan- ning action effectively aids in furniture assembly, whereas baselines struggle to find the correct Io- cation.
GPT
Figure 7: The training curves in the Kitchen environment. Integrating coded motion planning and RL accelerates learning. RL-GPT learns faster compared to the RL baseline.
The agent keeps attacking the ground instead finding cow
The agent needs adjust the view to horizantal, Rewrtte code as follows:
Harvest fillk
The agent is gradually aporoaching
good.
Fast Agent
The car needs to micdle
vehicle gradually leoving the road.
position. Rewritt code as follows.
Drive
vehicle maintains its pesition in the middle of road.
Code is good.
Figure 9: VLMs can offer more precise critiques based vision feedback.
Badii
Oun
Figure 10: MuloCo example. GPT-4 can code action to reverse the and then move it forward.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we clearly state the claims made, including the contributions made the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Section F of the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
25

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: We provide sufficient details to reproduce the prompt engineering in Section and model training results in Section 4.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [No]
Justification:
We will release all the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wttpccccpppcccaeeccccccccccccclt /tneeeeeueeeeeeeeemiiiiii ssssssooiccyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide experimental details in Section of the main paper and Section A of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computational cost Section of the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: :
wwtppccccpcccceeeeeceeeeeeeeeece http:/hhiiccieeeeeeeeeeeeeee ines?
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification:
We discussed the broader impacts of this work in Section G of the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We use the openai API to do research.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We follow the license for each asset and cite all the original papers that contribute our paper. We will also give credits the used code and follow the license, copyright information, and terms of use in our released code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwoee com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "NcOTA4VHRG", "reviewer": "Reviewer_LWBp", "summary": "The paper introduces RL-GPT, a novel framework that integrates Large Language Models (LLMs) with Reinforcement Learning (RL) to enhance the performance of LLM-based agents in complex, embodied environments. The primary goal is to address the limitations of LLMs in executing intricate logic and precise control, especially in open-world tasks like those found in the Minecraft game.\n\nThe RL-GPT framework employs a two-level hierarchical approach, consisting of a slow agent and a fast agent. The slow agent is responsible for decomposing tasks and determining which actions can be coded directly, while the fast agent generates the corresponding code and integrates RL configurations. This division of labor allows for efficient handling of both high-level planning and low-level execution.\n\nKey contributions of the paper include:\n1. Two-level hierarchical framework to determine which actions should be learned by RL and which ones can be coded (e.g. in a python).\n2. The introduction of a mechanism where high-level actions coded by LLMs are appended to the RL agent’s action space, instead of only relying on low level actions.\n3. Empirical validation showing that RL-GPT outperforms traditional RL methods and existing LLM agents in the Minecraft environment, particularly excelling in the ObtainDiamond task and other MineDojo tasks.\n\nIn experiments, RL-GPT demonstrated superior performance, achieving state-of-the-art (SOTA) results in several tasks, including rapidly obtaining diamonds and excelling in long-horizon tasks with limited computational resources.", "questions": "1. The slow agent can all a sub-action that the RL agent executes. But can the fast agent which uses “code as policies” also call the RL agent inside its loop?\n2. What is the architecture and inputs of the PPO RL agent? In particular, it was said that “PPO is limited to a set of skills”: how is the sub-action (e.g. “Harvest log” sub-action 2 in Figure 3) from the slow agent represented to the policy network? Is it just a one-hot vector (“PPO is constrained to a limited set of skills”), or are you encoding the sub-action text instruction as an embedding vector, etc., or are you learning a different set of policy weights for each sub-action, or something else?\n3. How would the proposed framework adapt to other open-ended environments beyond Minecraft? Are there specific modifications required for different domains?\n4. (RL Interface Ablation) Do you have any hypotheses on why the action space reconstruction is more effective than designing the reward function? Is the action space shortening the horizon of the task and makes the reward less sparse? \n5. Table 4: is this showing the %? In Section 4.4, it was reported as 0.34% and 0.65% success rate. Was this a typo and it’s supposed to be 34% and 65% success percentage for crafting a table?\n\nPost rebuttal: I have increased the score after the rebuttal.", "limitations": "The authors acknowledge some limitations of their work, notably the reliance on the capabilities of LLMs and the potential challenges in generalizing to unseen data. However, there are additional limitations and assumptions that could be more explicitly addressed:\n1. The approach assumes an environment with a structured observation space that can support coding as policies. This might not be feasible in more complex or less structured environments. Environments need to be compatible with the two-level hierarchical framework and provide sufficient information for task decomposition and action coding. It’ll be helpful to provide guidelines for adapting the framework to different types of environments, including those with less structured observation spaces. \n2. The framework's reliance on multiple interacting agents and iterative refinement could complicate implementation and debugging. The need for specific prompts and hand written examples add to the complexity, making it challenging for broader adoption.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "**Originality**: \n- The paper proposes a novel integration of RL and LLMs, leveraging the strengths of both to improve task efficiency in open-world environments.\n- The fast agent's method of combining code-as-policy with RL by inserting high-level coded actions into the RL action space is a creative and original solution that sets RL-GPT apart from other frameworks.\n\n**Quality**: \n- Claims are well supported by extensive experimental results, comparing with previous SOTA methods in MineDojo\n\n\n**Clarity**: \n- The paper is generally well-written and structured, with detailed explanations of the framework components and their interactions.\n- Figures and tables effectively illustrate the performance improvements and workflow of RL-GPT.\n\n\n**Significance**: \n- There is often a question about the what is the right abstraction level for LLM agents to operate on: high level APIs/code-as-policies, or low level mouse/keyboard, etc. RL-GPT addresses this by trying to leverage the both, with the higher level actions appended to the RL agent’s action space. \n- The state-of-the-art performance in MineDojo tasks highlights its potential impact on the field, though the focus on Minecraft limits the generalizability of the results.", "weaknesses": "1. The work focus on Minecraft, while useful as a testbed, may not fully represent the range of challenges present in other open-world environments. Expanding the scope of testing to other domains would enhance the significance of the work\n\n2. While the integration approach is novel, the individual components (LLMs, RL, task decomposition) are well-known techniques. The paper could benefit from a more explicit discussion of how these components are synergistically combined to create a unique solution.\n\n3. Some sections, particularly those describing the two-loop iteration process and the role of the critic agent, could benefit from more detailed explanations or examples to improve understanding. I am aware that the full prompts are included in the appendix and they do not fit in the main paper, however.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer LWBp,\n\nThank you for appreciating our work with valuable suggestions. We address your questions below.\n\n**Q1. Generalization to other environments.**\n\n**A1.** Thanks for this suggestion! We acknowledged this concern and addressed it to some extent in Appendix Section D. It is difficult to find real-world environments like Minecraft, which require **both high-level planning and low-level control**. We applied our methods to some robotic tasks that demand both long-horizon planning and precise motor execution. Some results are shown in **Fig. 7 and Fig. 8 in the attached pdf**.\n\n- Kitchen Environment Training: **Fig.7** illustrates the RL training process in the Kitchen environment[1]. The vertical axis represents the success rate, and the horizontal axis represents the number of training steps. Inserting coded motion planning into the action space accelerates learning. Our method learns faster compared to the baseline.\n- Furniture Environment Demonstration: In **Fig.8**, we present a qualitative demonstration of the Furniture environment[2]. The motion planning action effectively aids in hole-finding tasks during table and chair assembly. The baseline struggles to find the correct location at the same training step.\n\n[1] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning, 2019\n\n[2] Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks, 2021\n\nYes, modifications are needed for different domains, such as **adjusting the task descriptions in the prompts**. The powerful zero-shot capability of GPT should ensure generalization ability.\n\n**Q2. How are these components synergistically combined?**\n\n**A2.** Thanks for this suggestion! The key technical novelty is illustrated in **Fig.2** of the paper. Incorporating high-level actions generated by LLMs into the RL action space captures the core ideas presented. This core design allows the agent to choose between RL and code-as-policy.\n\n**Q3. More detailed explanations or examples.**\n\n**A3.** Thanks for this suggestion!\n- **Slow Agent:** This agent will decide which parts can be coded and which parts should be learned. The first iteration loop is designed to correct decision-making errors.\n- **Fast Agent:** This agent will code the coding tasks from the slow agent. The second iteration loop is designed to correct coding errors.\n- **Critic Agent:** The critic agent provides feedback from the environment for each code execution. The output from the critic agent after one step serves as feedback for the fast agent, while outputs from a sequence of steps serve as feedback for the slow agent.\n\n**Q4. Can the fast agent also call the RL agent inside its loop?**\n\n**A4.** Sorry for the confusion. The fast agent cannot perform RL training. Its role is to generate code based on the requirements provided by the slow agent.\n\n**Q5. The architecture and inputs of the PPO RL agent.**\n\n**A5.** Sorry for the confusion. Yes, we will learn the weights for each RL sub-action (marked orange in **Fig. 3**).\n1. The \"Harvest log\" is sent from the slow agent to the fast agent in text format.\n2. The fast agent generates high-level codes for the action.\n3. Coded actions are inserted into the RL action space.\n4. RL training is performed after this insertion.\n\n**Q6. Action space vs reward function.**\n\n**A6.** Thanks for this good question! We have conducted ablation studies, as shown in **Tab. 5**. Using an action space with higher-level actions can **shorten the task horizon and reduce reward sparsity**. This design is more efficient as it leverages the coding ability and common sense of LLMs. For instance, LLMs understand that it takes 10 attacks to break a tree. When designing a reward function, even though \"10 attacks\" yields a high reward, randomly sampling those 10 individual attacks is a time-consuming process. In contrast, with action space design, an action like \"attack 10 times\" can be directly generated, **resulting in an immediate high reward**.\n\n**Q7. Typos.**\n\n**A7.** Thank you for pointing that out. The correct values are 34% and 65%. We will correct this in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sDxFVO7f23", "reviewer": "Reviewer_m13r", "summary": "This work is a variation of the code as policies, which utilizes LLMs to write code robotic policies in code snippets.  This work examines minecraft, proposing that certain tasks can be composed into two sets: those solvable using LLM generated code and those best left to be solved using a standard RL agent.  They utilize 2 LLM prompting styles.  The first is for an LLM agent which decomposes tasks and determines which ones can be learned as code or using RL.  The 2nd actually implements the code and inserts it into the action space for use by the RL agent.  A critic LLM determines if the action was successful and how to improve it.  This is used for iterative improvement of the code generating LLM.", "questions": "Can the authors discuss the broader impact of this method in envs outside of Minecraft and how it can apply to other types of environments in robotics?", "limitations": "Limitations are clearly the expense of this work, could the authors discuss a bit more about this?", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This deals with one of the most important problems in utilizing LLMs and code as policies, the fact that some actions are just not well suited for code and should be learned using standard RL.\n\nAblations including removing the critic are shown.\n\nThe improvements in Minecraft appear to be substantial.\n\nOverall, the paper presents a good improvement in an area of interest to many RL researchers. I generally support acceptance of this work if the limitations/broader impact are discussed.", "weaknesses": "Alot of manual design is needed for each specific environment. \n\nA large amount of calls to an LLM API are used and this method becomes very expensive quickly.  \n\nI would like to see how VLMs could be leveraged in other environments.\n\nThe work does not explore envs outside of Minecraft.  I would like to see how applications of this work could extend to other common RL envs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer m13r,\n\nThank you for appreciating our work with valuable suggestions. We address your questions below.\n\n**Q1. Manual design is needed for each specific environment.**\n\n**A1.** Yes, we acknowledge that some manual design is necessary. However, compared to existing agents like Voyager, which require humans to write low-level action code, our **RL training reduces manual design by 90%**. As LLMs' zero-shot capabilities continue to advance, the need for manual design will further decrease.\n\n**Q2. Expensive API call.**\n\n**A2.** Thanks for this good question! We agree that calling API has a cost.\nHere are the statistics. We count the average tokens on different tasks for the first 3 iterations.\n\n|      | iter-1 | iter-2 | iter-3 |\n|  ----  | ----  | ----  | ----  |\n| tokens  | 10K | 15K  | 16K  |\n\nGPT-4-32k will cost $0.12 per 1,000 tokens. The newly released **GPT-4o Mini** will be more cost-effective.\n\nHere is the comparison on the ''harvest a log'' task. Vicuna-13b has lower performance due to its poor coding ability. Mixtral-8x7B works much better than Vicuna-13b. Open-sourced methods are continuously improving, making them promising for future agent development.\n\n|   LLMs   | Success Rate | Dead loop |\n|  ----  | ----  | ----  |\n| Vicuna-13b | 0.36 | 0.8 |\n| Mixtral-8x7B | 0.55 | 0.5 |\n| Claude  | 0.64 | 0.3 |\n| GPT-4  | 0.65 | 0.3 |\n\n**Q3. How VLMs could be leveraged in other environments?**\n\n**A3.** Thanks for this good question! VLMs can function as **more effective critic agents** in our framework. While LLMs can only indicate whether the agent succeeded with its coded actions, VLMs can explain why it failed in the environment. As shown in **Fig. 9 of the attached pdf**, GPT-4V provides **more detailed feedback** across different environments. For example, in Minecraft, it can identify that the agent keeps attacking the ground instead of finding the cow. In the driving simulation environment, it can note that the vehicle is gradually drifting off the road. This feedback can be used by both our fast and slow agents for self-improvement.\n\n**Q4. Generalization to other environments.**\n\n**A4.** We acknowledged this concern and addressed it to some extent in Appendix Section D. It is difficult to find real-world environments like Minecraft, which require **both high-level planning and low-level control**. We applied our methods to some robotic tasks that demand both long-horizon planning and precise motor execution. Some results are shown in **Fig. 7 and Fig. 8 in the attached pdf**.\n\n- Kitchen Environment Training: **Fig.7** illustrates the RL training process in the Kitchen environment[1]. The vertical axis represents the success rate, and the horizontal axis represents the number of training steps. Inserting coded motion planning into the action space accelerates learning. Our method learns faster compared to the baseline.\n- Furniture Environment Demonstration: In **Fig.8**, we present a qualitative demonstration of the Furniture environment[2]. The motion planning action effectively aids in hole-finding tasks during table and chair assembly. The baseline struggles to find the correct location at the same training step.\n\n[1] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning, 2019\n\n[2] Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks, 2021\n\nModifications are needed for different domains, such as adjusting the task descriptions in the prompts. The powerful zero-shot capability of GPT should ensure generalization ability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DXSPlBUflG", "reviewer": "Reviewer_efny", "summary": "This paper introduces RL-GPT, a hierarchical framework that uses LLMs to first break down complex embodied tasks into sub-actions that are suitable for coding or learnable through RL, and then write codes or RL configurations to execute the actions. The authors evaluate the framework on MineDojo benchmark and the challenging ObtainDiamond task, showcasing better performance and efficiency compared to existing baselines.", "questions": "1. Have you tried using any smaller / open-source LLMs instead of GPT-4? How does the performance change?\n2. When the fast agent fails to code up a sub-action, how does the slow agent decide if it should further break down the action into steps, or use an RL training pipeline?\n3. Does the framework keep an archive of solutions (high level plans + codes / RL agents)?", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The integration of RL training pipeline is novel and generally applicable to other domains.\n2. Strong empirical results on Minecraft tasks. Detailed ablation studies on key design choices.\n3. The paper is clearly written and well presented.", "weaknesses": "The proposed framework is only evaluated on the MineCraft games, which the GPTs have extensive knowledge about due to the massive relevant contents on the internet. It's unclear if the framework could be easily extended to novel domains like simple MuJoCo simulated environments, new games, or more real-world tasks, e.g. navigation or household tasks with real robots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer efny,\n\nThank you for appreciating our work with valuable suggestions. We address your questions below.\n\n**Q1. Generalization to other environments.**\n\n**A1.** We acknowledged this concern and addressed it to some extent in Appendix Section D. It is difficult to find real-world environments like Minecraft, which require **both high-level planning and low-level control**. We applied our methods to some robotic tasks that demand both long-horizon planning and precise motor execution. Some results are shown in **Fig. 7 and Fig. 8 in the attached pdf**.\n\n- Kitchen Environment Training: **Fig.7** illustrates the RL training process in the Kitchen environment[1]. The vertical axis represents the success rate, and the horizontal axis represents the number of training steps. Inserting coded motion planning into the action space accelerates learning. Our method learns faster compared to the baseline.\n- Furniture Environment Demonstration: In **Fig.8**, we present a qualitative demonstration of the Furniture environment[2]. The motion planning action effectively aids in hole-finding tasks during table and chair assembly. The baseline struggles to find the correct location at the same training step.\n\n[1] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning, 2019\n\n[2] Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks, 2021\n\nAdditionally, VLMs can be integrated into our critic agent, enabling future expansion to vision-based environments like driving, as shown in **Fig. 9** of the attached pdf. Simple MuJoCo simulated environments may not be ideal for our method due to their **focus on low-level controls** and lack of high-level planning or decision-making. However, our method performs well in some cases, as illustrated in **Fig. 10**. For instance, GPT-4 can code an action to reverse the car and then move it forward based on the topography.\nModifications are needed for different domains, such as adjusting the task descriptions in the prompts. The powerful zero-shot capability of GPT should ensure generalization ability.\n\n**Q2. Smaller / open-source LLMs.**\n\n**A2.** Thanks for this good question! Here is the comparison on the ''harvest a log'' task. The performance of Claude is similar to GPT-4. Vicuna-13b has lower performance due to its poor coding ability. Mixtral-8x7B works much better than Vicuna-13b. Open-sourced methods are continuously improving, making them promising for future agent development.\n\n|   LLMs   | Success Rate | Dead loop |\n|  ----  | ----  | ----  |\n| Vicuna-13b | 0.36 | 0.8 |\n| Mixtral-8x7B | 0.55 | 0.5 |\n| Claude  | 0.64 | 0.3 |\n| GPT-4  | 0.65 | 0.3 |\n\n**Q3. When the fast agent fails to code up a sub-action.**\n\n**A3.** \nThanks for this good question! When the fast agent is unable to code an action, it implies that at least part of the action requires RL training. The slow agent will decompose this action, which is a process of gradually analyzing which specific sub-action needs RL. For the steps that can be coded, the code is written. If decomposition fails to resolve the issue after a certain number of iterations, the task will be handed over to RL.\n\n**Q4. Keeping an archive of solutions.**\n\n**A4.** Yes, both successful coded actions and well-trained RL networks will be preserved during agent optimization. They will be executed as skills with specific names, similar to Voyager's vector database.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XstzUOdhYD", "reviewer": "Reviewer_ep8Y", "summary": "The authors present a framework to give LLMs the ability to code and train RL agents as a tool for completing tasks. They perform experiments in MineDojo.", "questions": "See weaknesses", "limitations": "Yes, but i think it's typical to include this in the main paper instead of the appendix.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**Experiments:** Barring the standard deviation issue mentioned in teh weaknesses, the results are a significant improvement over Plan4MC.\n\n**Idea:** the method of using RL as a tool for crafting subfunctions to complete tasks via LLMs is interesting, intuitive, and simple.", "weaknesses": "**Minor Issues:**\n\n- some relevant related works using LLMs to train RL agents: [https://rlvlmf2024.github.io/,](https://rlvlmf2024.github.io/) https://mihdalal.github.io/planseqlearn/, https://clvrai.github.io/boss/, https://clvrai.github.io/sprint/, https://instructionaugmentation.github.io/, https://gimme1dollar.github.io/lift/,\n\n**Clarity:**\n\n- The term “Code-as-Policy” is introduced without definition or citations in the introduction. Is this a common term? Is this referring to the [Code as Policies](https://code-as-policies.github.io/) paper? This will confuse readers who haven’t seen this term before.\n- The related works section doesn’t do a great job differentiating this work against prior work. For example section 2.2 doesn’t mention the current work in relation to prior work at all, same with the first paragraph of 2.3.\n- The method section isn’t really clear, it’s missing details, references to appendix sections, or references to figures clarifying things. Examples:\n    - L133: If the focus is not on the reward function then how is the reward obtained?\n    - L136: “A dedicated token is allocated for [integrating high-level actions]…” how is this trained or used?\n    - L181: What are the inputs and outputs of $C$? image observatoins or other details from the environment state?\n    - Overall, there’s not enough examples or details here to understand the method and how things are learned/trained without having to thoroughly dig through the appendix. For space reasons obviously not all details can be here, but I think this methods section can be rewritten to be much better.\n- Experiments:\n    - L230 “It costs 3k steps for harvesting a log” what does this mean? This is the minimum number of timesteps required? or is this the empirical number the RL agent that the authors trained requires?\n    - L232: please link to a specific section in the appendix.\n\n**Experiments:**\n\n- There are no standard deviations on any numbers, or information about # seeds in the main paper. How many seeds did you run?\n- Task-directed Voyager seems like a very direct comparison that uses LLMs for everything instead of RL. Why is this not compared?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ep8Y,\n\nDespite the negative score, we really appreciate your detailed review. We address your questions below.\n\n**Q1. Related works.**\n\n**A1.** Thanks for mentioning these papers. We will add some of them to the related work part in the revision.\n- RL-VLM-F introduces visual feedback to the reward function design.\n- Plan-Seq-Learn combines motion planning and RL skills to solve long-horizon tasks, using LLM as the high-level planner. \n- BOSS chains short-term skills to construct long-term skills with LLM supervision.\n- SPRINT uses LLMs to relabel instructions.\n- DIAL uses foundation models for dataset augmentation.\n- LiFT uses foundation models for task instruction proposal and reward design.\n\nThe motivation, technical novelty, and experimental results of our work distinctively set it apart from these works:\n- Our **motivation** is to equip LLMs with RL capabilities to explore environments and autonomously decide which actions to learn using RL.\n- Our **technical novelty** involves inserting high-level actions coded by LLMs into the RL action space.\n- Our **main results** demonstrate that our agent can successfully determine which actions to learn and complete long-horizon targets in Minecraft.\n\nWe will modify Sections 2.2 and 2.3 following your suggestions.\n\n**Q2. Code-as-Policy.**\n\n**A2.** Sorry for the confusion. Yes, the definition of \"Code-as-Policy\" in this paper refers to LLMs writing code that is then executed in the environment to control the agent. We will reference this paper to clarify this concept.\n\n**Q3. Writing issues.**\n\n**A3.** Thanks for these valuable suggestions!\n- L133: It means our agent will not focus on reward design. Instead, default reward functions, such as sparse rewards and distance rewards, are used for agent training in the environment.\n- L136: It means that one of the output dimensions of the RL network will represent the coded high-level action. This high-level action will be executed when the network selects that option.\n- L181: You can find the outputs in lines **185 to 187**. The inputs are simply observations from the environment, as shown in **Tab. 10**.\n\nWe will clarify our method based on your suggestions in future versions.\n\n**Q4. Experiment issues.**\n\n**A4.** Thanks for these valuable suggestions!\n- L230: For these tasks, the maximum exploration steps are capped at 3K. We will compare the success rate for these tasks. More details can be found in **Tab. 16**.\n- L232: It is linked to **Section C**.\n- Seeds: For quantitative ablation studies on harvest tasks, we used **3 seeds**. Due to the large number of tasks, the rest were tested on 1 seed. This approach aligns with practices in the field. Given the low simulation speed of the Minecraft game, other significant works, such as VPT [1], also report outcomes based on a single RL training seed.\n\n[1] Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos, 2022\n\nWorks like Voyager focus solely on **high-level** planning, using human-coded skill libraries to bypass the need for **low-level control**. Our method integrates both high-level planning and learning low-level skills, directly addressing the challenge of playing the Minecraft game using keyboard and mouse actions. RL is necessary to acquire low-level policies autonomously.\n\n- We are the first agent that utilizes an RL training pipeline as a tool and decides which actions to be learned by RL.\n- We are the first to incorporate GPT-coded actions into the RL action space, enhancing the sample efficiency for RL.\n- The critic agent is similar to Voyager, but Voyager doesn't have a two-level hierarchical framework since it only contains high-level planning.\n\nHere is a system-level comparison table:\n\n|   Method   | long-horizon task | low-level control | sample-efficiency | self-improvement |\n|  ----  | ----  | ----  | ----  | ----  |\n| MineAgent  | &#10007; |\t&#10004; | &#10007; | &#10007; |\n| VPT  | &#10004; | &#10004;  | &#10007; | &#10007; |\n| DEPS  | &#10004; | &#10007; | &#10004; | &#10007; |\n| Voyager  | &#10004; | &#10007; | &#10004; | &#10004; |\n| RL-GPT  | &#10004; | &#10004; | &#10004; | &#10004; |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear all reviewers,

We sincerely thank your effort in the review with valuable comments and suggestions. **We appreciate reviewers _LWBp_, _m13r_, and _efny_ for recognizing our work**. Additional figures are attached in the **_6379_rebuttal_figs.pdf_**, which we will reference in the following specific responses.

Meta-review: This paper discusses how one can design an agent empowered by LLMs that can break down a task into subtasks and dynamically decide whether to solve such a subtask by code generated by the LLM or by a "traditional" RL agent. The method is validated in the game Minecraft. This is an interesting paper; all reviewers unanimously recommended its acceptance, and I'm doing the same.

However, an important point about the number of independent runs was made, and the concerns raised are real. I'm ok accepting this paper because it is a _proof of possibility. It shows how such an idea can be instantiated, and it shows that such an instantiation can work. This is how it should position itself instead of making claims that "it attains State-of-the-Art (SOTA) performance" (line 60). It is hard to imagine someone justifiably claiming SOTA performance without reporting some notion of variability or robustness. Thus, I strong suggest the authors to change their wording accordingly.

Additionally, some of the comparisons to RL methods are unfair, mainly when talking about their sample complexity. In line 75, for example, the paper states that the problem with other techniques is that they "requires billions of steps for fine-tuning long-horizon tasks". It is crucial to understand that LLMs are generated by going over many, many, many samples (orders of magnitude more than billions). Thus, the comparison above is unfair. LLMs allow one to amortize the cost of such samples because LLMs are a general-purpose model that can bypass the lack of initial knowledge of RL agents. I expect the final version of this paper to also better contrast the different alternatives as per this discussion.
Decision: Accept (oral)