Voyager: An Open-Ended Embodied Agent with Large Language Models
Authors: Guanzhi Wang
Yuqi Xie
Yunfan Jiang
Ajay Mandlekar
Chaowei Xiao
Yuke Zhu
Linxi Fan
Anima Anandkumar
Venue: Accepted by TMLR ()

Abstract: We introduce Voyager, the first LLM-powered embodied lifelong learning agent in an open-ended world that continuously explores, acquires diverse skills, and makes novel discoveries without human intervention in Minecraft. Voyager consists of three key components: 1) an automatic curriculum that maximizes exploration, 2) an ever-growing skill library of executable code for storing and retrieving complex behaviors, and 3) a new iterative prompting mechanism that incorporates environment feedback, execution errors, and self-verification for program improvement. Voyager interacts with GPT-4 via blackbox queries, which bypasses the need for model parameter fine-tuning. The skills developed by Voyager are temporally extended, interpretable, and compositional, which compounds the agent’s capability rapidly and alleviates catastrophic forgetting. Empirically, Voyager demonstrates strong in-context lifelong learning capabilities. It outperforms prior SOTA by obtaining 3.1x more unique items, unlocking tech tree milestones up to 15.3x faster, and traveling 2.3x longer distances. Voyager is able to utilize the learned skill library in a new Minecraft world to solve novel tasks from scratch, while other techniques struggle to generalize.

Published in Transactions on Machine Learning Research (03/2024)
Voyager: An Open-Ended Embodied Agent with Large Language Models
Guanzhi Wang 2 Yuqi Xie² Yunfan Jiang Ajay Mandlekar" Chaowei Xi 15, Yuke Zhu Linxi "Jim" Fan 1 E Anima Anandkumar
'NVIDIA, CCaltech, *UT Austin, "Stanford, UUW Madison
"Equal contribution 'Equal advising g Corresponding authors
https://voyager... http://yyyger.miieddj..oo.ooo minedojo.org
Reviewed OpenReview:
Abstract
We introduce VOYAGER, the first L.M--ooered embodied lifelong learning agent in Minecraft that continuously explores the world, acquires diverse skills, and makes novel discoveries without human intervention. VOYAGER consists three key components: 1) an automatic curriculum that maximizes exploration, 2) an ever-growing skill library of executable code for storing and retrieving complex bohaviors, and 3) a new itcrative prompting mechanism that incorporates environment feedback, execution errors, and self-verification for program improvement. VOYAGER interacts with GPT-4 via blackbox queries, which bypasses the need for model parameter fine-tuning. The skills developed by VOYAGER are temporally extended, interpretable, and compositional, which compounds the agent's 's abilities rapidly and alleviates catastrophic forgetting. Empirically, VOYAGER shows strong in-context lifelong learning capability and exhibits exceptional proficiency in playing Minecraft. It obtains 3.3x more unique items, travels 2.3x longer distances, and unlocks key tech tree milestones up to 15.3x faster than prior SOTA. VOYAGER able ut ililize the learned skill library in new Minecraft world to solve novel tasks from scratch, while other techniques struggle to generalize.
Minecraft Tech Tree
Diamond
60
Wooden
Diamond Tool
Number of Distinct Items
50
Tool
Tool
Tool
40
J
J
30
Iron
Tool
Tool
>>
Stone Tool
Stone Tool
Wooden Tool
20
Wooden Tool
10
75
100
150
Voyager (Qurs) Voyager w/o Skill Library
ReAct
Reflexion
AutoGPT
Figure 1: VOYAGER discovers Minecraft items and skills continually by self-driven exploration, significantly outperforming the baselines. X-axis denotes the number of prompting iterations. The shaded region represents the standard deviation of the items acquired by each agent over three experimental runs. Voyager is the only method that can obtain a diamond tool in one of the three runs.

---

Published in Transactions on Machine Learning Research (03/2024)
Automatic Curriculum
Iterative Prompting Mechanism
Skill Library
a5yne Tunction conbatzonole
// Equio a aoapon
Mine Wood Log
const sward bot, inventery. findImentorry
actltt, toensflytionel sttnne,, sword") 1d1d; (sword)
Make Crafting Table
Make Crafting Table
zwat bot.equipt svard, "hand");)
Craft Stone Sword
New Task
maat sraftStconSwwesttbtt
Make Furnace
Combat Zombie
1/ and equto sntetd
Retrioval
awalt crafrshitllddoott
Craft Shield
Cook Steak
Log
Combat Zombio
Env Feadback Execution Errors
Code as Actions
Refine Program
x
MINECBRET
?
Mine Diamond
Update Exploration Progress
Add New Skill
Environment
Self-Verification
Figure 2: VOYACER consists of three key components: an automatic curriculum for open-ended exploration, skill library for increasingly complex behaviors, and iterative prompting mechanism that uses code action space.
1 Introduction
Building generally capable embodied agents that continuously explore, plan, and develop new skills in
open-ended worlds is a grand challenge for the Al community (Kolve et al. 2017 Savva et al. 2019
Zhu et al.
2020
Xia et al. 2019 Shen et al. 2020). Classical approaches employ reinforcement learning (RL)
(Kober
ell al.
2013
Arulumaran et al. 2017) and imitation learning (Baker et al.
2022 Team et al.
2021
Vinyals
let al.
2019
that operate on primitive actions, which could be challenging for systematic exploration
Ecoffet
et al.
2019
Huizinga & Clune 2022 Wang et al. 2020 Kanitscheider et al. 2021 Dennis et al.,
2020
inteppretabllity (Liang et al.] 2022 Sun et al. 2020 Zhao et al. 2021), and generalization (Jiang al.
2022
Shridhar et al.
2021 Fan et al. (001).. Recent advances in large language model (T.MM based agents harness
the world knowlodge encapsulated in pre-trained LLMs to generate consistent action plans exccutable
policies {Liang et al. 2022 Singh et al. 2022 Jiang el al.
They are applied to embodied tasks like games and robotics (Fan et al. 2022] Zeng et al. 2022 Ahn et al. 2022 Huang et al. 2022b a), as well as
2022
Shinn et al. 2023). However, these agents are not lifelong learners that can progressively acquire, update, accumulate, and transfer knowledge over extended time spans (Parisi et al. 2019 Wang et al. (023a)..
NLP tasks without embodiment (Richards) 2023 Yao et al. 2022
Let us consider Minccraft an example. Unlike most other games studied in AI MMih et al.) 2013 OpenAl et al. 2019 Vinyals et al.) 2019), Minecraft does not impose predefined end goal or fixed storyline but rather provides a unique playground with endless possibilities (Fan et al. 2022). Minecraft. requires players to explore vast, procedurally generated 3D terrains and unlock a tech tree using gathered resources. Human players typically start by learning the basics, such as mining wood and cooking food, before advancing to more complex tasks like combating monsters and crafting diamond tools. We argue that an effective lifelong learning agent should have similar capabilitics as human players: (1) proposc suitable tasks basod on its current skill level and world state, e.g., learn to harvest sand and cactus before iron if it finds itself in a desert rather than forest; (2) refine skills based environmental feedback and commit mastered skills to memory for future reuse in similar situations (e.g. fighting zombies is similar to fighting spiders): (3) continually explore the world and seek out new tasks in a self-driven manner.
Towards these goals, we introduce VOYAGER, the first LLM- embodied lifelong learning agent to drive exploration, master a wide range of skills, and make new discoverics continually without human intervention in Minecraft. VOYAGER is made possible through three key modules (Fig. 2 1) an automatic curriculum that maximizes exploration; 2) skill library for storing and retrieving complex behaviors; and 3) new
2

---

Published in Transactions on Machine Learning Research (03/2024)
iterative prompting mechanism that generates executable code for embodied control. We opt to use code as the action space instead of low-level motor commands because programs can naturally represent. temporally extended and compositional actions {Liang et al. 2022 Singh et al.
2022), which are essential for
many long-horizon tasks in Minccraft. VOYAGER interacts with a blackbox LLM (GPT-4 (OpenAI 2023
through prompting and in-context learning (Wei et al.] 2022a Brown al.
2020 Raffel et al. 2020). Our
approach bypasses the need for model parameter access and explicit gradient- based training or finetuning.
More specifically, VOYAGER attempts to solve progressively harder tasks proposed by the automatic curriculum, which takes into account the exploration progress and the agent's state. The curriculum is generated by GPT-4 based on the overarching goal of "discovering as many diverse things as possible". This approach can be perceived as an in-context form of novelly search (Eysenbach et al. 2019 Conti et al.) 2018). VOYAGER incrementally builds a skill library by storing the action programs that help solve a task successfully. Each program is indexed by the embedding of its description, which can be retrieved in similar situations in the future. Complex skills can be synthesizod by composing simpler programs, which compounds VOYAGER's capabilities rapidly over time and alleviates catastrophic lorgetting in other continual learning methods Parisi et al. 2019 Wang et al. 2023a)..
However, LLMs struggle to produce the correct action code consistently in one shot (Chen et al. 2021a). To address this challenge, we propose an iterative prompting mechanism that: (1) executes the generated program to obtain observations from the Minecraft simulation (such as inventory listing and nearby creatures) and error trace from the code interpreter (if any); (2) incorporates the feedback into GPT-4's prompt for another round of code refinement; and (3) repeats the process until verffectrtnnntion module confirms the task completion, at which point we commit the program to the skill library (e.g., StaneSSveeho () and munbat//ombieithhhwwoo
and query the automatic curriculum for the next milestone (Fig, 2
Empirically, VOYAGER demonstrates strong in-context lifelong learning capabilities. It can construct an ever-growing skill library of action programs that are reusable, interpretable, and generalizable to novel tasks. We evaluate VOYAGER systematically against other LLM-based agent techniques (e.g., ReAct (Yoo eet al.] 2022). Reflexion (Shinn et al. 2023). AutoGPT (Richards) 2023)) in MineDojo (Fan et al. 2022): an open-source Minecraft Al framework. VOYAGER outperforms prior SOTA by obtaining 3.3x more unique items, unlocking key tech tree milestones up to 15.3x faster, and traversing 2.3x longer distances. We further demonstrate that. VOYAGER is able to utilize the learned skill library in a new Minecraft world to solve novel tasks from scratch, while other methods struggle to generalize.
2 Method
VOYAGER consists of three novel components: (1) an automatic curriculum (Sec. 2.1 that suggests objectives
for open-ended exploration, (2) skill library (Sec. 2.2 for developing increasingly complex behaviors, and (3) an iterative prompting mechanism (Sec. 2.3 that generates executable code for embodied control. The pseudocode of VOYAGER algorithm is shown in Pseudocode Full prompls are presented in Appendix, Sec.
2.1 Automatic Curriculum
Embodied agents encounter a variety of objectives with different complexity levels in open-ended environments. An automatic curriculum offers numerous benefits for open-ended exploration, ensuring a challenging but manageable learning process, fostering curiosity-driven intrinsic motivation for agents to learn and explore, and encouraging the development of general and flexible mooving strategies (Wang et al. 2019 Portelas et al.] 2020 Forestier et al., 2022). Our automatic curriculum capitalizes on the internet-scale knowledge contained within GPT-4 by prompting it to provide a steady stream of new tasks or challenges. The curriculum unfolds in a bottom-up fashion, allowing for considerable adaptability and responsiveness to the exploration progress and the agent's 's current state (Fig, 3h. As VOYAGER progresses to harder self-driven goals, it naturally learns a varicty of skills, such as "mining diamond".
The input prompt to GPT-4 consists of several components:
3

---

Published in Transactions on Machine Learning Research (03/2024)
inventory (oak_panks" 3, 'stick:
Reasoning: Since you have wooden pickaxe some stones, would upgrade your
4, d'cratting_sablee 1, 'sstoee:
GPT-4
efficiency.
wwooden_pickxee":
Task: Craft pickaxe.
Biome: river
Since have fishing rod inventory and neer river blome, would be good opportunity to catch some fish for food and experiance. Tesk: Catch 1 fish.
Reasoning:
Inventory (4/36): ('sugar_cane': 'paper:
GPT-4
3, Tishing rod't prooden.pikaaee
Nearby entities: pig, cat, villager
Reasoning: Your hunger is at 0, which means you need to find food to replenish your hunger. Since there ard pigs nearby, you can one to abtain raw porkchops. Task: Kill pig.
Health: 12/20
GPT-4
Hunger: 0/20
Inventory (8/36): (llumace: 1,
You have raw Iron and coel, and you have fumace: It's time to smelt Task: Smelt ingots,
Reasoning:
'stone_pickaace planks'
"crattiin_teblle: 4, 'coall:
right
zomble nearby, good opportunity killing the zombie now that you stone sword and shield equipped. Task: Kill zomble.
Reasoning:
night
Nearby entities: zombie
GPT-4
Equipment: 'stone_sword,, 'shield'
Figure 3: Tasks proposed by the automatic curriculum. We only display the partial prompt for brevity. See Appendix, Sec. A.2 for the full prompt structure.
(1) Directives encouraging diverse behaviors and imposing constraints, such "My ultimate goal is to discover as many diverse things as possible ... The next task should not be too hard since I may not have the necessary resources or have learned enough skills to complete it yet.";
(2) The agent's current state, including inventory, equipment, nearby blocks and entities, biome, time, health and hunger bars, and position;
(3) Previously completed and failed tasks, reflecting the agent's current exploration progress and capabilities frontier;
(4) Additional context: We also leverage GPT-3.5 to self-ask questions based on the agent's current state and exploration progress and self-answer questions. We opt to use GPT-3.5 instead of GPT-4 for standard NL. tasks due to budgetary considerations.
2.2 Skill Library
With the automatic curriculum consistently proposing increasingly complex tasks, it is essential to have a skill library that serves as a basis for learning and evolution. Inspired by the generality, interpretability, and universality of programs (Ellis et al. 2020), we represent each skill with executable code that scaffolds temporally extended actions for completing a specific task proposed by the automatic curriculum.
The input prompt to GPT-4 consists of the following components:
(1) Guidelines for code generation, such as "Your function will be reused for building more complex functions. Therefore, you should make it generic and reusable.";
(2) Control primitive APIs, and relevant skills retrieved from the skill library, which are crucial for in-context learning (Wei et al. 2022a Brown al. 2020 Raffel et al.) (0200) to work well;
(3) The generated code from the last round, environment feedback, execution errors, and critique, based on which GPT-4 can self-improve (Sec. 2.3
(4) The agent's current state, including inventory, equipment, nearby blocks and entities, biome, time, health and hunger bars, and position;
Chain-of-thought prompting (Wii al. 2022b to do reasoning before code generation.
4

---

Published in Transactions on Machine Learning Research (03/2024)
Program Generated GPT-4
Program Description
Skill Library
Sacio
The furctton equtgpinn stote sword to conear zacbue, stose svord La oot found,
Mine Wood Log
const werd
inventrrv.yy Indizwentery2tatt
crart ase. protection. A Aftenwrtss, it poaceeds to cosw sticks to arder rastore barger. does barger replonisted, actively
Make Crafting Table
Craft Wooden Pickaae
1.id
"xtone, svend
GPT-3.6
shteld
Embedding
Key
Craft Stone Sword
Make Furnace
3 eloe
sentt tot-ecutol evend, "hand"l
Add
keilt craftSttowwbeendttttt
cegages combut }
11 Craft and eaxtp seteld'
Value
Craft Iron Axe
Combat Zomble
11 Lack for and conoue zocole
Task: Craft Iron Pickaxe
Top-5 5 Relevant Skills
To craft iran pickaxe, need to iron ingots and sticks, have gathered the materials....
Smelt fron Ingot
How craft iron pickazee
GPT-3.5
Embedding
Query
Library
Retrieve
Stick
Minecraft?
Make Crafting Table
Make Fumace
Environment Feodback
skill. Each time GPT-4 generates and verifies new skill, we add it to the skill library, represented by a vector database. The key is the embedding vector of the program description (generated by GPT-3.5), while the value is the program itself. Bottom: Skill retrieval. When faced with a new task proposed by the automatic curriculum, we first leverage GPT-3.5 to generate a general suggestion for solving the task, which combined with environment feedback as the query context. Subsequently, we perform querying to identify the top-5 relevant skills.
Figure 4: Skill library. Top: Adding
We iteratively refine the program through a novel iterative prompting mechanism (Sec. 2.3 incorporate it into the skill library new skill, and index it by the embedding of its description (Fig. top). For skill retrieval, we query the skill library with the embedding of self- ceff-generated task plans and environment feedback (Fig. 4 bottom). By continuously expanding and refining the skill library, VOYAGER can learn, adapt, and excel in a wide spectrum of tasks, consistently pushing the boundaries of its capabilities in the open world.
2.3 Iterative Prompting Mechanism
We introduce an iterative prompting mechanism for self- reff-improvement through three types of feedback:
(1) Environment feedback, which illustrates the intermediate progress of program execution (Fig- left). For example, "I cannot make iron chestplate because need: more iron ingots" highlights the cause of failure in crafting an iron chestplate. We use bot. chat () inside control primitive APIs to generate environment feedback and prompt GPT-4 to use this function as well during code generation;
(2) Execution errors from the program interpreter that reveal any invalid operations synlax errors in programs, which are valuable for bug fixing (Fig. 5 right);
(3) Self Self-verification for checking task success. Instead of mamually coding success checkers for each new task proposed by the automatic curriculum, we instantiate another GPT-4 agent for self-verifieation. By providing VOYAGER'S 's current state and the task to GPT-4, we ask it to act as a critic (Mnih et al., 2016 Schulman et al.] 2017 Lillicrap et al. 2016) and inform us whether the program achieves the task. In addition, if the task fails, it provides critique by suggesting how to complete the task (Fig. 6p. Hence, our oelffcerifiationn is more comprehensive than self- reflection (Shinn et al. 2023 by both checking success and reflecting mistakes.
5

---

Published in Transactions on Machine Learning Research (03/2024)
Environment Feedback
Execution Error
throw Error( No item named S(name) );;
I cannot make stick because I need: 2 more planks
No item named acacia_axe at line 18:await craftitem(bot, "acacia_axe",
cannot soon_shhoell because need: more stick
GPT-4
GPT-4
async function craftstoneehoveeWWtttbbblddd { // If not enough cobblestone, mine cobblestone
async function craftAcaclaaxe((ba { +async function craftWoodenAxe(bal { // Craft anacia planks and sticks
// If not enough sticks, check there are // enaugh planks in the inventory if (sticksCount 2)
// Place the crafting table near the bot
const planksCount bot. inventory count( mcData. iyNme acacia_planks.idl
if (planksCount 2) {
Craft acacia using acacta planks and 2 sticks
// lelt craft planks await mineBlock(bot, "acacia_log", 1); await craftItem(bot, bot, "acacia_planks" 1);
await craftItem( bot, "acacia_axee 1); bot.chat( "Acacia axe crafted.");
// Craft wooden axe using 3 acacta planks // and 2 sticks
// Craft sticks using planks await craftItem(bor bot, "stick", 1);} await craftItem(bor bot, "stone_shovel", 1);
await craftItem( bot, "wooden axe", 1); bot.chat( "Wooden axe crafted.");
Figure Left: Environment feedback. GPT-4 realizes needs 2 more planks before crafting sticks. Right: Execution error. GPT-4 realizes it should craft wooden axe instead of an acacia axe since there is acacia in Minecraft. We only display the partial prompt for brevity. The full prompt structure for code generation is in Appendix, Sec. A.3
During each round of code generation, we execute the generated program to obtain environment feedback and exccution errors from the code interpreter, which are incorporatod into GPT-4's prompt for the next round of code refinement. This iterative process repeats until self-verifiattoo validates the task's completion, which point add this new skill to the skill library and ask the automatic curriculum for new objective (Fig. 2). If the agent gets stuck after 4 rounds of code generation, then we query the curriculum for another task. This iterative prompting approach significantly improves program synthesis for embodied control, enabling VOYAGER to continuously acquire diverse skills without human intervention.
3 Experiments
3.1 Experimental Setup
We leverage OpenAT's gpt-4-0314 OOeenA 2023 and 3. 5--.-3.-tuurb-0311 (chatgpt APIs for text completion, along with text-embedding--da- temperatures to 0 except for the automatic curriculum, which uses temperature task diversity. Our simulation environment is built top of MineDojo (Fan et al.
(embedding) API for text embedding. We set all
0.1 encourage
2022 and leverages
Mineflayer (PrismarineJS 2013 JavaSeript APIs for motor controls. See Appendix, Sec.
B. for more details.
3.2 Baselines
Because there is no LLM-based agents that work of the box for Minecraft, make our best effort to select number of representative algorithms baselines. These methods originally designed only for NLP tasks without embodiment, therefore we have to re-interpret them to be executable in MineDojo and compatible with our experimental setting:

---

Published in Transactions on Machine Learning Research (03/2024)
Inventory (8/36): foa.__lankks 5, "cabblestene":2
Reasoning:
Mining Minecraft get
your
'porkchop": 2, wwooden_sword": "coall: 5, pickaxe,
GPT-4
inventory.
''oa_logg:3 3, 'dint:: 9)
Success: True
Task: Mine coal pres
Inventory (10/36): coopert: 'eopper.inget=-.
Reasoning: craft spygless, you need copper ingots and amethyst shard. You have 3 cooper ingots, but you don't have any amethyst shards.
aacaci__planks: 'raw_iron:: 'sttick: 1, 'roo_sword":
'iro__ pickaxek iron_ingot: 3, ttrafting_tabble' 'furnace' 1)
GPT-4
False
Task: Craft spyglass
Critique:
amethyst underground.
Inventory (7/36): ('bak] Log'llog: 5, ''oak,pankk".. wooden_pickaxe" 1, wwoodon_sword"; 1, |parkchop!: 2,
Reasoning: white_wool mutton your inventory, you sheep.
GPT-4
sheep.
White wool: 2, mmuttonn 8)
Task: sheep
Critique:
Find and shoop complete the task.
Inventory (9336)((strrnng 'coal:: 1, hotten_fleshh:
Reasoning: You have rotten_flesh your inventory, which moans have killed leest 1 zombie.
sworssw 1, "umacce:1 'eirtt stone_shovel:: 1,
GPT-4
prooden.pickaxe". 'granttet:5
Task Kill zombie
Success: True
Figure 6: Self- velfficeiiination examples. We only display the partial prompt for brevity, See Appendix, Sec. for the full prompt structure.
A.4
Pseudocode 1: VOYAGER algorithm.
def voyager
environnent, environment that code action space curriculum_agen agent,, curriculum agent for proposing the task action_agent, action agent for code generation critic_agent, critic agent for self-verification skill_manager, skill manager adding
retrieval
agent_state onvironment.
while True:
curriculum_
curriculum_aee agent gett cwwpleted_ tasks (),
jurriculum_agent get_failed, taaks (),
task = curriculun propose_next_tace agent_state, exploratiion.rrgreass
code None ceatronnnnt_feddaaee None
execution_errors None critique None success
try range(4); skills skill maaile_mnnggr.... retrieve_ekilled
moving on next
task, environzent_
code action agent. ceaettggeerrtteedeeeo
taak, code.
sevtronment_feedbaee
execution_errors
critique,
skills.
agent_atate, aavironment_feddeee
execution_error environment. step (code)

---

Published in Transactions on Machine Learning Research (03/2024)
succeas, critique critic_ageni check taskk taccesucessa task, stane_
if success:
break
success:
MAAl__maaager.add_22 (code)
Strrcullu_agent add cowpletedppeccccccoococcooe tass (task)
elae:
curriculum_agee agentt add_ failed_ (alaaaleedttttteeeeeeeaa (task)
ReAct Yao et al. 2022 uses chain-of-thoughtt prompting (Wei et al. 2022b) by generating both reasoning traces and action plans with L.MSs. We provide it with our environment feedback and the agent states obscrvations.
Reflexion SShinn et al. 2023 is built on top of ReAct Yao et al., 2022) with self-reflection to infer more intuitive future actions. We provide with execution errors and our verificaiiinntoon module.
AutoGPT (Richards 2023 a popular software tool that automates NLP tasks by decomposing a high-level goal into multiple subgoals and executing them in ReAct-style loop. We re-implement AutoGPT by using GPT-4 to do task decomposition and provide it with the agent states, environment feedback, and exccution errors as observations for subgoal exccution. Compared with VOYAGER, AutoGPT lacks the skill library for accumulating ledge, sell-verification for assessing task success, and automatic curriculum for open-ended exploration.
Note that we do not directly compare with prior methods that take Minecraft screen pixels as input and output low-level controls (Nottingham et al., 2023 Cai et al.) 2023 Wang al. 2023b It. would not be apple-to-applc comparison, because control the agent. Our work's focus is on pushing the limits of GPT-4 for lifelong embodied agent learning, rather than solving the 3D perception or sensorimotor control problems. VOYAGER is orthogonal and can be combined with gradient-based approaches like VPT (Baker et al. 2022) as long as the controller provides a code API. We make system-level comparison between VOYAGER and prior Minecraft agents in Table, A.2
rely the high-level Mineflayer (PrismarineJS 2013 API to
3.3 Evaluation Results
We systematically evaluate VOYAGER and basclines their exploration performance, tech tree mastery, map coverage, and zero-shot generalization capability to novel tasks in new world.
Significantly better exploration. Results of exploration performance are shown in Fig. ! VOYAGER'S superiority is evident in its ability to consistently make new strides, discovering 63 unique items within 160 prompting iterations, 3.3x many novel items compared to its counterparts. On the other hand, AutoGPT lags considerably in discovering new items, while ReAct and Reflexion struggle to make significant progress, given the abstract nature of the open-ended exploration goal that is challenging to execute without appropriate curriculum.
Consistent tech tree mastery. The Minecraft tech tree tests the agent's ability to craft and use a hierarchy of tools. Progressing through this tree (wooden tool + stone tool - iron tool + diamond tool) requires the agent to master systematic and compositional skills. Compared with baselines, VOYAGER unlocks the wooden level 15.3x faster (in terms of the prompting iterations), the stone level 8.5x faster, the iron level 6.4x faster, and VOYAGER is the only one to unlock the diamond level of the tech tree (Fig. and Table. D. This underscores the effectiveness of the automatic curriculum, which consistently presents challenges of suitable complexity to facilitate the agent's progress.
Extensive map traversal. VOYAGER is able to navigate distances 2.3x longer compared to baselines by traversing a variety of terrains, while the baseline agents often find themselves confined to local areas, which significantly hampers their capacity to discover new knowledge (Fig.
Efficient zero-shot generalization unseen tasks. To evaluate zero-shot generalization, we clear the agent's 's inventory, reset it to newly instantiated world, and test it with unseen tasks. For both VOYAGER
8

---

Published in Transactions on Machine Learning Research (03/2024)
Table Tech tree mastery, Fractions indicate the number of successful trials out three total runs. 0/3 means the method fails to unlock level of the tech within the maximal prompting iterations (160). Numbers are prompting iterations averaged over three trials. The fewer the iterations, the more efficient the method.
Method
Wooden Tool
Stone Tool
Iron Tool
Diamond Tool
ReAct Yao et al. 2022
N/A (%/3
N/A (%/)
N/A (%/)
N/A (%/)
Reflexion Shinn et al. 2023
N/A (%/3
N/A (%/)
N/A (%/)
N/A (%/3
AutoGPT Richards 2023
92 72 (3/a)
94 72 (3/a)
135 103 ($/a)
N/A (%/)
VOYAGER w w/o Skill Library
7 2 (%/a
9 + (3/a)
29 + 11 (3/a)
N/A (%/a
VOYAGER (Ours)
6 2 (3/3)
11 = 2 (3/3)
21 + 7 (8/3)
102 (1/a)
Voyager (Ours)
ReAct
Reflexion
AutoGPT
Starting Point
Figure 7: Map coverage: Two bird's eye views of Minecraft maps. VOYAGER able traverse 2.3x longer distances compared to baselines while crossing diverse terrains. Trajectories are plotted based on the positions where each agent interacts with GPT-4.
Table 2: Zero-shot generalization unseen tasks. Fractions indicate the number of successful trials out of throe total attempts. 0/3 means the method fails to solve the task within the maximal prompting iterations (50). Numbers are prompling iterations averaged over three trials. The fewer the iterations, the more elficient the method.
Method
Diamond Pickaxe
Golden Sword
Lava Bucket
Compass
ReAct Yao et al.
2022
N/A (%/a
N/A (%/a)
N/A (%/3
N/A (%/a
Reflexion (Shmn al. 2023
N/A (%/a
N/A (%/a
N/A (%/)
N/A (%/3
AutoGPT Richards
2023
N/A (%/a
N/A (%/)
N/A (9/3
N/A (%/3
AutoGPT Richards
2023
w/ Our Skill Library
39 (1/s)
(4/s)
N/A (%/s
(2/s)
VOYAGER w/o Skill Library
36 (2/s)
30 + 9 (3/a)
27 9 (3/a)
26 + (3/a)
VOYAGER (Ours)
19 + (2/a)
18 1 (3/3)
21 4 (8/a)
18 + (3/a)
and AutoGPT, we utilize GPT-4 to break down the task into a series of subgoals. Table. and Fig. show VOYAGER can consistently solve all the lasks, while baselines cannol solve any task within 50 prompting iterations. What's interesting to note is that our skill library constructed from lifelong learning not only enhances VOYAGER'S performance but also gives boost to AutoGPT. This demonstrates that the skill library serves as versatile tool that can be readily employed by other methods, effectively acting a plug-and-play asset to enhance performance.

---

Published in Transactions on Machine Learning Research (03/2024)
Craft a Golden Sword
Collect a Lava Bucket
golden
bucket lava source
gold Ingot
raw gold
2
bucket
Iron pickaxe
7
iron ingot
iron ingot
iron
furnaee
furnace
stone pickaxe
>
stone pickaxe
wooden pickaxe
wooden pickaxe
crafting table
crafting table
wood log
wood log
30
40
10
20
50
Prompting Iterations in Code Generation
Prompting Iterations Code Generation
Voyager (Ours) Voyager w/o Skill Library
AutoGPT
AutoGPT w/ Our Skill Library
Figure 8: Zero-shot generalization to two tasks. See Appendix, Sec. B.4.3 for the other two tasks. We do not plot ReAct and Reflexion since they do not make any mcaningful progross.
lasks. We visualize the intermediate progress of each method
3.4 Ablation Studies
We ablate design choices (automatic curriculum, skill library, environment feedback, execution errors, self-verification, and GPT-4 for code generation) in VOYAGER and study their impact on exploration performance (see Appendix, Sec. B.3 for details of each ablated variant). Results shown in Fig. 9 We highlight the key findings below:
Automatic curriculum is crucial for the agent's consistent progress. The discovered item count drops by 93% if the curriculum is replaced with random one, because certain tasks may be too challenging if attempted out of order. On the other hand, a manually designed curriculum requires significant Minecraft-speciff expertise, and does not take into account the agent's 's live situation. It falls short in the experimental results compared to our automatic curriculum.
Voyager w/o skill library exhibits tendency to plateau in the later stages. This underscores the pivotal role that the skill library plays in VOYAGER. Tt helps create more complex actions and steadily pushes the agent's 's boundaries by encouraging new skills to be built upon older ones.
Self-verificate is the most important among all the feedback types. Removing the module leads to significant drop (-73%) in the discovered item count. Self-verification serves as critical mechanism to decide when to move on to new task reattempt previously unsuccessful task.
GPT-4 significantly outperforms GPT-3.5 in code generation and obtains 5.7x more unique items, as GPT-4 exhibits a quantum leap in coding abilities. This finding corroborates recent studies the literature (Bubeck et al. 2023 Liu et 2023b).
4 Limitations and Future Work
Multimodal Feedback. VOYAGER does not currently support visual perception, because the available version of GPT-4 API text-only at the time of this writing. However, VOYAGER has the potential to augmented by multimodal perception models (Liu al., 2023a Driess al.) 2023) to achieve more impressive tasks. We demonstrate that given human feedback, VOYAGER is able to construct complex 3D structures in Minecraft, such as Nether Portal and house (Fig. 10 There two ways to integrate human feedback:
10

---

Published in Transactions on Machine Learning Research (03/2024)
60
Number of Distinct Items
50
50
Number Distinct
40
40
30
30
20
20
10
10
0
0
25
50
75
100
125
150
25
50
75
100
125
150
Prompting iterations in Code Generation
Prompting Iterations Code Generation
Skill Library
GPT-3.5
w/o Environment Feedback
Manual Curriculum
Random Curriculum
w/o Execution Error
w/o Sett-verificaton
Figure 9: Left: Ablation studies for the automatic curriculum, skill library, and GPT-4. GPT-3.55 means replacing GPT-4 with GPT-3.5 for code generation. VOYAGER outperforms all the alternatives, demonstrating the critical role of each component. Right: Ablation studies for the iterative prompting mechanism. VOYAGER surpasses all the other options, thereby highlighting the essential significance of each type of feedback in the iterative prompting mechanism.
Build Nether Portal
Build House
Figure VOYAGER builds 3D structures with human leedback. The progress of building designs that integrate human input demonstrated from left right.
(1) Human as a critic (equivalent to VOYAGER's 's velfieeeri module): humans provide visual critique to VOYAGER, allowing it to modify the code from the previous round. This feedback is essential for correcting certain errors in the spatial details of a 3D structure that VOYAGER cannot perceive directly.
(2) Human as a curriculum (equivalent to VOYAGER's automatic curriculum module): humans break down a complex building task into smaller steps, guiding VOYAGER to complete them incrementally. This approach improves VOYAGER'S ability to handle more sophisticated 3D construction tasks.
Cost. The GPT-4 API incurs significant costs. It 15x more expensive than GPT-3.5. Nevertheless, VOYAGER requires the quantum leap code generation quality from GPT-4 (Fig. 9b. which GPT-3.5 and open-source LLMs cannot provide (Touvron al. 2023).
Inaccuracies. Despite the iterative prompting mechanism, there are still cases where the agent gets stuck and fails to generate the correct skill. The automatic curriculum has the flexibility to reattempt this task at a later time. Occasionally, self-verification module may also fail, such as not recognizing spider string as a success signal of beating a spider.
Hallucinations. The automatic curriculum occasionally proposes unachicvable tasks. For example, it may ask the agent to craft a "copper sword" or "copper chestplate', which are items that do not exist within the game. Hallucinations also occur during the code generation process. For instance, GPT-4 tends to
11

---

Published in Transactions on Machine Learning Research (03/2024)
use cobblestone a fuel imput, despite being an invalid fuel source in the game. Additionally, it may call functions absent in the provided control primitive APIs, leading to code execution errors.
We are confident that improvements in the GPT API models as well as novel techniques for finetuning open-source LLMs will overcome these limitations in the future.
5 Related work
Decision-making Agents in Minecraft. Minecraft is an open-ended 3D world with incredibly flexible game mechanics supporting broad spectrum of activities. Built upon notable Minecraft benchmarks (Fan)
et al. 2022 Guss et al. 2019b 2021 Kanervisto et al. 2022 Johnson et al.) 2016), Minecraft learning
algorithms can be divided into two categories: 1) Low-level controller: Many prior efforts leverage hicrarchical reinforcement learning to learn from human demonstrations (Lin et al. 2021
Mao et al.) 2021 Skrynnik le al. 2021). Kanitscheider et al. (2021) design a curriculum based on success rates, but its objectives are limited to curated items. MineDojo (Fan et al. (022) and VPT (Baker et al.
2022 utilize YouTube vidcos
for largo-scale pro-training. DreamerV3 (Hafner et al. 2023), on the other hand,
learns a world model to (2022) leverage few-shot.
explore the environment and collect, diamonds.. 2) High-level planner: Volum et. al.
prompting with Codex (Chen et al.) 2021a) to generate executable policies, but they require additional human interaction. Recent works leverage LLMs as a high-level planner in Minecraft by dccomposing a high-level (ask into several subgoals following Minecraft recipes (Wang et al. 2023b Nottingham et al. 2023
Yuan et al. 2023), thus lacking full exploration flexibility. Like these latter works, VOYAGER also uses LLMs as a high-level planner by prompting GPT-4 and utilizes Mineflayer @PrismarineJS 2013) as a low-level controller following Volum et al. (022)). Unlike prior works, VOYAGER employs an automatic curriculum that unfolds in bottom-up manner, driven by curiosity, and therefore enables open-ended exploration.
Large Language Models for Agent Planning. Inspired by the st rong emergent capabilities of LLMss, such
as zero-shot prompting and complex reasoning (Bommasaní et al., 2021 Brown et al., 2020 Raffel et al. 2020,
Wei et al.] 2022a Chowdhery et al. 2022 Chung et al. 2022). embodied agent research (Duan et al. 2022 Batra
et al. 2020 Ravichandar et al. 2020 Collins et al. 2021 has witnessed significant increase in the utilization of
LLMS for planning purposes. Recent efforts can be roughly classified into two groups. 1) Largge language models
for robot learning: Many prior works apply LLMs to generate subgoals for robot planning (Huang et al. 2022a a
Ahn et al. 2022 Min et al., 2021 Blukis et al. 2021). Inner Monologue (Huang el al. 2022b) incorporates envi-
ronment feedback for robot planning with LLMs. Code as Policies (Liang et al. 2022 and ProgPrompt (Singh
ct al. 2022) directly leverage LLMs to generate exccutable robot policies. VIMA (Jinng ct al. 2022 and
PaLM-E (Driess et al., 2023) fine-tune pre-trained LLMs to support multimodal prompts. 2) Large language
models for text agents: ReAct (Yao et al. 2022 leverages chain-of-thought prompting (Wei et al. 2022b and
generates both reasoning traces and fask-specife actions with LLMs. Reflexion (Shinn et al. 2023 is built
upon ReAct (Yao et al. 2022) with self-reftection to enhance reasoning. AutoGPT (Richards] 2023 is a popular
tool that automates NLP tasks by crafting a curriculum of multiple subgoals for completing a high-level goal
while incorporating ReAct (Yao et al. 2022 's reasoning and acting loops. DERA (Nair et al.
2023 frames a
task as a dialogue between two GPT-4
OpenAA 2023) agents. Generative Agents (Park et al.
2023 leverages
ChatGPT (chatgpt) to simulate human behaviors by storing agents' experiences as memories and retrieving
those for planning, but its agent actions are not executable. SPRING (Wu et al., 2023) is a concurrent work that uses GPT-4 to extract game mechanics from game manuals, based on which it answers questions arranged in directed acyclic graph and predicts the next action. All these works lack a skill library for developing more complex behaviors, which are crucial components for the success of VOYAGER in lifelong learning.
Code Generation with Execution. Code generation has been a longstanding challenge in NLP (Chn ct al. 2021a Nijkamp et al. 2022. Le et al. 2022 Chowdhery et al. 2022 Brown et al. 2020). with various works leveraging execution results to improve program hests. Execution-guides approaches leverage inter- mediate execution outcomes to guide program search (Chen et al., 2019 2021b Ellis et al. 2019). Another line of research utilizes majority voting to choose candidates based on their exccution performance (Li et al. 2022 Cobbe et al. 2021). Additionally, LEVER (Ni et al.) 2023) trains a verifier to distinguish and reject incorrect programs based on execution results. CLAIRIFY (Skreta et al. 2023), on the other hand, generates code for
12

---

Published in Transactions on Machine Learning Research (03/2024)
planning chemistry experiments and makes use of rule-based verifier to iteratively provide error feedback to LLMs. VOYAGER distinguishes itself from these works by integrating environment feedback, execution errors, and self-verification (to assess task success) into an iterative prompting mechanism for embodied control.
6 Conclusion
In this work, we introduce VOYAGER, the first LLM-powered embodied lifelong learning agent, which leverages GPT-4 to explore the world contimuously, develop increasingly sophisticated skills, and make new discoveries consistently without human intervention. VOYAGER exhibits superior performance discovering novel items, unlocking the Minecraft tech tree, traversing diverse terrains, and applying its learned skill library to unseen tasks in a newly instantiated world. VOYAGER serves starting point to develop powerful generalist agents without tuning the model paramcters.
7 Broader Impacts
Our research is conducted within Minecraft, a safe and harmless 3D video game environment. While VOYAGER is designed to be generally applicable to other domains, such as robotics, its application to physical robots would require additional attention and the implementation of safety constraints by humans ensure responsible and secure deployment.
8 Acknowledgements
We are extremely grateful to Ziming Zhu, Kaiyu Yang, Rafal Kocielnik, Colin White, Or Sharir, Sahin Lale, De-An Huang, Jean Kossaifi, Yuncong Yang, Charles Zhang, Minchao Huang, and many other colleagues and friends for their helpful feedback and insightful discussions. This work is done during Guanzhi Wang's ' interuship at NVIDIA. Guanzhi Wang is supported by the Kortschak fellowship Computing and Mathematical Sciences at Caltech.
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Tchter, Alex Trpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jelfrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang- Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarck Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as can, not say: Grounding language robotic affordances. arXiv preprint Xiv: Arziv- 20loo2224.006999 2022.
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Authony Bharath. Deep reinforcement. learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26 38, 2017.
Bowen Baker, 1lge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jell Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arxiv: Arziv-2206. 11795, 2022.
Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su. Rearrangement: A challenge for embodied ai. arXio preprint arXiv: 20mi-20111.0999 2020.
Valts Blukis, Chris Paxton, Dicter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial semantic representation for high-level natural language instruction execution. In 5th Annual Conference on Robot Learning, 2021. URL https::// http:://oeeeeeeeeeeeeeeee/t//aaa
13

---

Published in Transactions on Machine Learning Research (03/2024)
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bolg,, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portclance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frioda Rong. Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Sant hanahanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. arXiv preprint ar Xiv: Arziv-2 10..000000 100.07258 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Cerbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jelfrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances Neural Information Processing Systems 39: Annual Conference on Neu- vol Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https; //proceedings.o http:/ngs..niiisccccccooee
2ww.a22e64---bbrrraeeaeea Abstract..rrre......... html
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. orXie preprint Xrxi Arziv- sacii-2208.127/1 2023.
Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. arxiv preprint or Xix: 20tv-2301.1003.. 2023.
chatgpt. Introducing chatgpt, 2022. CRL
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alee Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv: Arrio-2107.03374 2021a.
Xinyun Chen, Chang Liu, and Dawn Song. Exocution-guided neural program synthesis. In 7th Interna- tional Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. Reviewvi,,,ne,, 2019.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Xinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis. arv preprint arvi Arzio-2107.00101 2021b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Scbastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvasshchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret, Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Morcira, Rewon Child, Oleksandr Polozov, Katherine Loe, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Calasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprinl arviv: Arziv-2204 220--02144,,,1,, 02311, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahmna, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-finetuuee language models. arXiv preprint ar arxiv: Araio-2210.111 11116, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv: Arxiv-2110. 1418.,,68,,, 2021.
Jack Collins, Shelvin Chand, Anthony Vanderkop, and David Howard, A review of physics simulators for robolic applications. IEEE Access, 9:51416 51431, 2021.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via population of novelty-seeking agents. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi and Roman Garnett (cds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Syslems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 5032 5043, 2018 URL
.eugrips..rp.... cc/paape:/018///eeeeeeee hhahh/
00001144efffaccc5511100/////2222 wbssdeeeeessAAsaaatttttao..a html
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (cds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 12, 2020, virtual, 2020. URL /htp::tt/////ttt htttps://proceedings http//aahh///ae 15109//9111111000010000012222229 0tw.:000000000001199999444455559 oaraaaae011000000--aooooaooeaaa
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. arXiv preprint arXiu: Arriv-2303. 2303.03978 2023.
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied Al: from simulators to rescarch tasks. IEEE Trans. Emerg. Top. Comput. Intell, 6(2):230-244, 2022. doi: 00.110//ETTC112222.11111
1109/TETCIII... 201.0///ETI..222...... URL https:://doiii http////o..r//......./////.....
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiu preprint ar Xiv: Arzin- 19tt.1901.111.0 10995, 2019.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Kevin Ellis, Maxwell I. Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama. Write, execute, assess: Program synt hesis with a REPL. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché- Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouser, BC, Canada, pp. 9165 9174, 2019. URL httpp:://proceeenng...tt newraps....pp.. cc// paper/ http///000...0022227770600060990
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lozama, and Joshua B. Tenenbaum. Dreamcoder: Growing generalizable, interpretable kuowledge with wake-sleep bayesian program learning. arXiv preprint arxiv: Arriv-2006.0838.1 2020.
embedding. New and improved embedding model, 2022.
URL https:://ppena..comm
new awwaadd-impp aadd-mpoooedeeeeddinne model
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In 7th International Conference on Learning Representations, TCT.R 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRevvew.net, 2019. URL httpp:://ppnrevieee.eeet// nettoorrmm
Rd
Linxi Fan, Guanzhi Wang. De-An Huang, Zhiding Yu, Li Fci-Fei, Yuke Zhu, and Animashroe Anandkumar. SECANT: self-expert cloning for zero-shot generalization of visual policies. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 3088-3099. PMLR,
2021. URL http:://procsecingss.. mlr httppp/ccccoog.....pprrss////a/a html
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXie preprint arXiv: Armiv-2206.088 08855, 2022.
Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre- Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. The Journal of Machine Learning Research, 23 (D::688--6858, 2022.
William H. Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanic Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, Manuela Veloso, and Phillip Wang. The minerl 2019 competition on sample efficient reinforcement learning using human priors. arXiv preprint Xiv: Arzio-1904. 1904. 2019a.
William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. In Sarit Kraus (ed.), Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 16, 2019, pp. 2442 24488 ijeai.org, 2019b. doi: (0.24963//jeal) /t1///aaa..///333/ URL https://doi.org 10. /ttp///0229633/caa.2200000 2019//33
William H. Guss, Mario Ynocente Castro, Sam Devlin, Brandon Houghton, Noboru Sean Kuno, Crissman Loomis, Stephanie Milani, Sharada Mohanty, Keisuke Nakata, Ruslan Salakhutdinov, John Schulman, Shinya Shiroshita, Nicholay Topin, Avinash Ummadisingu, and Oriol Vinyals. The minerl 2020 competition on sample efficient reinforcement learning using human priors. arXiv preprint arxiv: Arzio-2101.11011, 2021.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arxiv: (rmzw-2301.041000 2023.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9118 9147. PMLR, 2022a. URL httpp::/pproceeeiggs mlr wwlr.press/vie//neee haw.//22hhaggaaaaamm html
16

---

Published in Transactions on Machine Learning Research (03/2024)
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Tau, Sergey Levine, Karol Hausman, and Brian lchter. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arxi:: Arzio-22012 1rzio-2207.05600 2022b.
Joost Huizinga and Jeff Clune. Evolving multimodal robot behavior via many stepping stones with the combinatorial multiobjective evolutionary algorithm. Evolutionary compulation, 30(2):131-164, 2022.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yonggiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi (Jim) Fan. Vima: General robot manipulation with multimodal prompts. ARXIV.ORG, 2022. doi: 10.48550/ 10.48550/arXiv (at55///aXxv.221...0.. 22w.i.22220009......
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In Subbarao Kambhampati (ed.), Proceedings of the Twenty-Fiit International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pp. 4246-4222 4247. IJCAT/AAAT Press, 2016. URL http::/www.::::t iijaiiccaaia... org/ Abstract/16/644
Anssi Kanervisto, Stephanie Milani, Karolis Ramanauskas, Nicholay Topin, Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, Wei Yang, Weijun Hong, Zhongyue Huang. Haicheng Chen, Guangjun Zeng, Yue Lin, Vincent Micheli, Eloi Alonso, François Fleuret, Alexander Nikulin, Yury Belousov, Oleg Svidchenko, and Aleksei Shpilman. Minerl diamond 2021 competition: Overview, results, and lessons learned. arXiv preprint Xie: 2222 10583, 2022.
Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton, Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, Oleg Klimov, and Jeff Clune. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint arxiv: Arriv-2106. 14876, 2021.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robolics Research, 32(11):1238 1274, 2013.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXio preprint arxiv; Araio-1712.0544 2017.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. arXiv preprint orXiv: Arpio-2207.01720 7807780 2022.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cypricn de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. arXiv preprint arXiv: Arziv-2205 2208.0781. 2022.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pele Florence, and Andy Zeng. Code policies: Language model programs for embodied control. arXie preprint ar Xiv: Aro0v-2209.00 07759, 2022.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:// wwwp:/oggg/aa/aaeee 02971
Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. Juewu-me: Playing minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint arxiv: Arziv-21 12. 04904 2021.
17

---

Published in Transactions on Machine Learning Research (03/2024)
Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Prismer: A vision-language model with an ensemble of experts. arXiv preprint or Xin: Arrio-2308 02306, 2023a.
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao Hc, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, and Bao Ge. Summary of chatgpt/ research and perspective towards the future of large language models. arXiv preprint arXiv: Arziv-2304. 01852, 2023b.
Hangyu Mao, Chao Wang, Xiaotian Hao, Yihuan Mao, Yiming Lu, Chengjie Wu, Jianye Hao, Dong Li, and Pingzhong Tang. Seihai: A sample-efficient hierarchical ai for the minerl competition. arXiv preprint arXiv: Arzio-2111.08833 2021.
So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and R. Salakhutdinov. Film: Following instructions in language with modular methods. International Conference on Learning Represen- tations, 2021. URL
07342v3
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, arXiv preprinl arxiv: Arzio-1312.5600 2013.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria- Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1928-1933 JMLR.org, 2016. URL http://proceeddngs. mlr press/ w48/mniha16 h
Varun Nair, Elliot Schumacher, Geoffrey Tso, and Anitha Kannan, Dera: Enhancing large language model completions with dialog-enabled resolving agents. arXiv preprint arXio: Arzio-23033 17011, 2023.
Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen tan Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with exccution. aix preprint arXi:: Arzio-2302.08400 08466, 2023.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. arXio preprint arxi:: 1atii-2209.144777 2022.
Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hanna Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep?: Embodied decision making using language guided world modelling. ARXIV. ORG, 2023. doi: 10.48550 aavv 2301.12000
Ap Gpt-4 technical report. arxie preprint Xrxiv: (itii-2303.08777 2023.
OpenAL, :, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. ar preprint arXiv: Arziv-1912.0me 2019.
German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019. doi: 2019.01.012. URL httpp:://aoi.org/11.. 10116 www.et. 201...1.000000
Joon Sung Park, Joseph O'Brien, Carric J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv: Arziv- 2304.03444 2023.
18

---

Published in Transactions on Machine Learning Research (03/2024)
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yve Oudeyer. Automatic curriculum learning for deep RI: A short survey. In Christian Bessiere (ed.), Proceedings of the Twenly-Ninlh International Joint Conference on Artificial Intelligence, LJCAI 2020, pp. 4819 4825. ijeai.org, 2020. doi: 10.24963 /ttp::aaaa2222///// URL https://doi..rrg
24963/ijcai.
1 2020/671
Prismarino.S. Prismarincjs/minntann api., 2013. URL
Create minocraft bots with powerful, stable, and high level javascript http:/twwww......aaaaaaaaaaaaaae
Colin Raffel, Noam Shazoer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. 2020. URL http://jmlr http://ww..a.rpppppp///////////0
J. Mach. Learn. Res., 21:140:1
Harish Ravichandar, Athanasios $ Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3:297-330, 2020.
Toran Bruce Richards. Signifeant-graviisssaattto--s An experimental open-source attempt attomp to make gpt-4 fully autonomous., 2023. URL
Manolis Savva, Jitendra Malik, Devi Parikh, Dhruv Batra, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, and Vladlen Koltun. Habitat: A platform for embodied AT research. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- November 2, 2019, pp. 9338 9346. IEEE 2019. doi: 10.1109/1CCV 2019.00943, URL https://doi org/ 10. 1ww1//199ICCC// 2019 00943
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprinl or Xiv:: Arciv-1707.060
2017.
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin-Mann Martin, Linxi Fan, Guanzhi Wang, Claudia Pérez- D' D'Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent Vainio, Josiah Wong, Li Fei-Fei, and Silvio Savarese. igibson 1.0: simulation environment for interactive tasks in large 2020.
realistic scenes. arXiv preprinl Xrv:
Noah Shimn, Bock Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reftection. arXiv preprint arXiv: Arziv-2308. 21..22888.......6 11366, 2023.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. arXiv preprint arXiv:
2021.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv: Arziv-2209. 11302, 2022.
Marta Skreta, Naruki Yoshikawa, Sebastian Arullhno-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXie preprint arXiv: Arziv-2303. 14 100, 2023.
Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, Kirill Aksenov, Vasilii Davydov, and Aleksandr Panov. Hicrarchical deep q-network from imperfect demonstrations in minecraft. Cogn. Syst. Res., 65:74-78, 2021. doi: 10.1016/j.aa www.:jooooos..ooooo.
URL httpt:://doi.rrg////// wtw.//..oogys.......... wwg.sy..00..000......
Shao-Hua Sun, To-Lin Wu, and Joseph J. Lim. Program guided agent. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview net, 2020.
URL http:p/://pppaaeeeeeeeeee///o///
DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Mansi Gupta, Tim Harley, Felix Hill, Peter C Humphreys, Alden Hung Jessica Landon, Timothy Lillicrap, Hamza Merzic, Alistair Muldal, Adam Santoro, Guy Scully, Tamara von Glehn, Greg Wayne, Nathaniel Wong, Chen Yan, and Rui Zhu. Creating multimodal
19

---

Published in Transactions on Machine Learning Research (03/2024)
interactive agents with imitation and self-supervised learning. arXiv preprint orxe 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv: Arziv-2802, 2897,1991, 2023.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojeiech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al. Alphastar: Mastering the real-time strategy game starcraft ii DeepMind blog, 2, 2019.
Ryan Volum, Sudha Rao, Michael Xu, Gabriel DesGarennes, Chris Brockett, Benjamin Van Durme, Olivia Deng, Akanksha Malhotra, and Bill Dolan. Craft iron sword: Dynamically generating interactive game characters by prompting large language models tuned code. In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop ( (Wordplay 2022), pp. 25-43, Seattle, United States, 2022. Association for Computational Linguistics. doi: 10.18653 (118653/v1//8 11/122222222///// wordplay- URL attpps:/accaatthol..g horgt/ooooooogg/o/////ooo org/ 2022. wodd.ooddlay
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXic preprint aivi Arzio-2302.004 00487, 2023a.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer (poct): Endlessly generating increasingly complex and diverse learning environments and their solutions. arv preprinl arXiv: 19ziv-1901.00 01 0173,, 2019.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jef[rey Clune, and Kenne(h O. Stanley. Enhanced POET: open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Mochine Learning Research, pp. 9940-9951. PMLR, 2020. URL. http:://proceeedingg.ooo mlr
http://pooeannng....rersss////// ppwps//v19//aan200111 html.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXie preprint arXiv: Arziv-2302 230.00300,, 2023b.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. arXiv preprint arXiv: 1rniv22206.76688 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv: Arrio-2201.111 11903, 2022b.
Yue Wu, Shrimai Prabhumoye, So Ycon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, and Yuanzhi Li. Spring: Gpt-4 out-performs rl algorithms by studying papers and reasoning. arXiv preprint arXiv: 2305.15486, 2023. URL https://arxiv wtggabs/2305.... 15tp//2605.11466222
Fei Xia, William B. Shen, Chengshu Li, Priya Kasimbeg, Micael Tchapmi, Alexander Toshev, Li Fei-Fei, Roberto Martin-Martin, and Silvio Savarese. Interactive gibson benchmark (igibson 0.5): A benchmark for interactive navigation in cluttered environments. arXiv preprint arXiv: Arzie-19100 14442, 2019.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arxiv: Arziv-2210. 221.003600.,9 2022.
Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xic, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4me: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint orXiv: 2303.16563, 2023. URT. https: http://ax.vvvo/// hbse/baa/2200 2atbb///333..30 16563y1
20

---

Published in Transactions on Machine Learning Research (03/2024)
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Compasing zero-shot multimodal reasoning with language. arXiv preprint arXiv: Araio-2204.00598 2022.
Zelin Zhao, Karan Samel, Binghong Chen, and Le Song. Proto: Program-guided transformer for program-guided tasks. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Aduances in Newral Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 17021-17036, 2021. URL
html
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin. robosuite: A modular simulation framework and benchmark for robot learning. arXic preprint arxiv: Arziv-2009. 12293, 2020.
21

---

Published in Transactions on Machine Learning Research (03/2024)
A Method
A.1 Prompting
GPT-4 and GPT-3.5 offer users the ability to designate the role of each prompt message among three options:
System: A high-level instruction that guides the model behavior throughout the conversation. It sets the overall tone and objective for the interaction.
User: A detailed instruction that guides the assistant for the next immediate response.
Assistant:
A response message generated the model.
See https://platformm openai wttpppp.co//ccc///////o.cctc httppp/aa..///o/////////a///aaaa
for more details.
To save token usage, instead of engaging multi-round conversations, we concatenate a system prompt and user prompt to obtain each assistant's response.
A.2 Automatic Curriculum
A.2.1
Components in the Prompt
The input prompt to GPT-4 consists of several components:
(1) Directives encouraging diverse behaviors and imposing constraints (so that the proposed task is achicvable and verifiable): Soe Sec. A.2.4 for the full prompt:
(2) The agent's is current slate:
Inventory: dictionary of items with counts, for example, ('cobblestone': 4, 'furnace': 1, 'stone pickaxe': 'oak planks': 7, 'dirt': 6, 'wooden boadden_pickaxe" 1, 'crafting table': 1, 4, 'coal': 1};
Equipment:
Armors or weapons equipped by the agents;
Nearby blocks: A set of block names within a 32-block distance to the agent, for example, 'dirt,, 'water', 'spruce 'spruce_planks? "grass_ block', 'dirt path', 'sugar__cnne" 'fern;;
Other blocks that are recently seen: Blocks that are nearby or in the inventory;
Nearby entities: A set of entity names within 32- block distance to the agent, for example, 'pig', 'cat', 'villager', 'zombic';
A list of chests that are seen by the agent: Chests are external containers where the agent can deposit items. If chest is not opened before, its content is "Unknown", Otherwise, the items inside each chest are shown the agent.
Biome: For example, 'plains', ffower_forest!; 'meadow', 'river,, 'beach', 'forest', Ssnowy_slopce
'rooenn peaks', 'old growth birch forest', 'ocean', 'sunflower plains", 'stony shore';
Time: One of 'sunrise', 'day', 'noon', 'sunset', 'night', 'midnight';
Health and hunger bars: The max value is 20;
Position: 3D coordinate (x, 2) of the agent's position in the Minccraft world;
(3) Previously completed and failed tasks;
(4) Additional context: See Sec. A.2.2
(5) Chain-of-thought prompting Wei al. 2022b) response: We request GPT-4 first reason about the current. progress and then suggest the next task.
22

---

Published in Transactions on Machine Learning Research (03/2024)
A.2.2 Additional Context
We leverage GPT-3.5 to self-ask questions to provide additional context. Each question is paired with a concept. that is used for relrieving the most relevant document. from the wiki knowledge base (Fan el al. 2022). We feed the document content to GPT-3.5 for self-answering questions. In practice, using wiki knowledge base is optional since GPT-3.5 already possesses good understanding of Minecraft game mechanics. However, the external knowledge base becomes advantageous if GPT-3.5 is not pre-trained in that specific domain. See Sec. A.2.4 for the full prompt.
A.2.3 uprm-up Schedule
In practice, adopt a warm-up schedule to gradually incorporate the agent's state and the additional context into the prompt based on how many tasks the agent has completed. This ensures that the prompt is exposed to increasing amounts of information over the exploration progress and therefore begins with basic skills and progressively advances towards more intricate and diverse ones. The warm-uu setting that we use across all the experiments shown in Table. A.1
Table A.1: Warm-up schedule for automatic curriculum.
Information in the prompt
After how many tasks completed
core inventory (only including log, planks, stick, craft- ing table, furnace, dirt, coal, pickaxe, sword, and axe)
equipment
0
nearby blocks
0
position
0
nearby entities
5
full inventory
7
other blocks that are recently seen
10
biome
10
health bar
15
hunger bar
15
time
15
additional context
15
A.2.4 Full Prompt
Prompt 1: Full system prompt for automatic curriculum. The list of question-answer pairs represents the additional context.
You are a helpful assistant that tells me the next immediate task to Minecraft. My ultimate goal is to discover as many diverse things as possible, accomplish as many diverse tasks as possible and becone the best Minecraft player in the world.
will give you the following information:
Question 1:
Answer:
Question 2:
Ansuer:
Question 3:
Ansuer:
Bione:
Time:
Nearby blocks:
Other blocks recently seen:
Hearby entities (nearest farthest):
23

---

reasoning about what the Published in Transactions on Machine Learning Research (03/2024)
Health: Higher than
I'm healthy.
Hunger: Higher
I'm hungry.
Position: ...
Equipment:
better armor my inventory, you should equip it.
Inventory (xx/36):
Cheats: You can ask me to depoeit or take items from these chesta. There also might be some unknown chest, you should ask me to open and check items inside the unknown chest.
Completed tasks so far: Failed tasks that are too hard:
You must follow the following criteria:
1) You should act as mentor and guide me to the next task based on current learning progress.
2) Please be
specific about what resources need to collect, what I need to craft, what mobs need to kill.
3) The next task should follow concise format, such "Mine [quantity] [block]*, "Craft [ quantity] [item]", "Smelt [quantity] [iten]", "Kill [quantity] [mab]", "Coak [quantity] should be single phrase. Do propose multiple taska anything else.
[food]", "Equip [item]" Do
4) The task should learned enough skills complete yet.
since may have necessary resources
The next task should be novel my equipment and tools using better materials, and discover new things. should not be doing the same thing over
interesting. should look for rare resources, upgrade
over again.
6) may sonetimes need to repeat some tasks if need to collect more resources to complete nore difficult tasks. Only repeat tasks if necessary.
7) Do not ask me to build or dig shelter even if it's at night. I vant to explore the world and discover new thinga. I don't want to stay in one place.
8) Tasks that require information beyond the player's status to verify should be avoided. For instance, "Placing 4 torches" and "Dig 2xix2 hole" are not ideal since they require visual confirmation from the screen. All the placing, building, planting, and trading tasks should be avoided. Do not propose task starting with these keywords.
You should only respond in the format described below: RESPONSE FORMAT:
Ressoning: Based the information liated should be.
Task: The task.
Here 's an example response:
Reasoning: The inventory empty now, chop down tree to get sone wood.
Task: Obtain a wood log.
Prompt 2: Full system prompt for asking questions. We provide both good and bad examples as few-shot. exemplars.
You helpful assistant that asks questions help decide the immediate task Minecraft. My ultimate goal discover as many things possible, accomplish Minecraft player in world.
many tasks possible become
will give you the follosing information:
Bione:
Time:
learby blocks:
Other blocks that are recently seen:
Nearby eatities (nearest to farthest):
Health:
Hunger:
Position:
Equipment:
Inventory (xx/36):
Chests:
Completed tasks far:
Failed tasks that hard:
follov following criteria:
24

---

Published in Transactions on Machine Learning Research (03/2024)
1) You should least 5 questions (but imnediate task do. Each question should followed by the concept that question
10 questions) help decide
Your question should specific
concept in Minecraft.
Bad example (the question too general):
Question: What the beat way to play Minecraft?
Concept: unknown
Bad example (axe is still general, you should specify the type of axe such as wooden axe):
What are the benefits of using an axe to gather resources?
Concept: axe
Good example:
Question: Now to make wooden pickaxe?
Concept: wooden pickaxe
3) Your questions should be self-contained and not require any context.
Bad example (the question requires the context of ny current biome): find in my current biome?
the blocks that Concept: unknown
Question: What
Bad example (the queation requires the context of my
inventory):
Question: What
the resources you need the nost currently?
Concept: unknown
example (the question requires
context my current inventory):
Question: you have any gold or emerald resources?
Concept: gold
Bad example (the question requires the context of my nearby entities):
Question: Can you see any aninals nearby that you can kill for food?
Concept: food
Bad example (the question requires the context of ny nearby blocks);
Question: Is there any water source nearby?
Concept: water
Good example:
Question: What the blocks that I can find in the sparse jungle?
Concept: sparse jungle
4) Do not ask questions about building tasks (such building shelter) since they hard for
do.
Let say your
bione is sparse jungle. that find the sparse jungle?
questions like:
Question:
Concept: sparse jungle
Question: What are the mobs that can find the sparse jungle?
Concept: sparse jungle
Let's say you see a creeper nearby, and you have not defeated creeper before. You can ask a question like:
Question: How to defeat the creeper?
Concept: creeper
Let's say last completed task is "Craft wooden pickaxe". You can ask question like: Question: What are the suggested tasks that I can do after crafting wooden pickaxe? Concept: vooden pickaxe
Here
question and concept examples:
Question: What the ores that I can find in the sparse jungle?
Concept: sparse jungle
"ore" because sparse jungle) Question: How can you obtain food in the sparse jungle?
(the above concept should
look up the page of "sparse jungle'
find
find
Concept: sparse jungle
(the above concept should not "food" because need to look up the page "sparse jungle " to find out what food can obtain in the sparse jungle)
Question: How can you use the furnace to upgrade your equipment and make useful itens? Concept: furnace
Question: How to obtain a diamond ore?
Concept: diamond ore
Question: What are the benefits of using a stone pickaxe over wooden pickaxe?
Concept: stone pickaxe
Question: What are the tools that you can craft using good planks and sticks?
Concept: vood planks
25

---

Published in Transactions on Machine Learning Research (03/2024)
You should only reapond in the format deacribed below:
RESPONSE FORMAT:
Ressoning:
Question
Concept
Question 2:
Concept 2:
Question 3:
Concept 3:
Question 4:
Concept 4:
Question 5:
Concept 6:
Prompt 3: Full system prompt for answering questions. Context represents the optional content from wiki knowledge base.
You are helpful assistant that anaver my question about Minecraft.
I will give you the following information: Question:
You will knozledge of Minecraft.
the question based the context (only if available and helpful) and your
1)
Start your
with "Answer:
2) Ansver "Answer: Unknown" " you don't 't know the
A.3 Skill Library
A.3.1 Components in the Prompt
The input prompt to GPT-4 consists of the following components:
(1) Guidelines for code generation: See Sec A.3.2 for the full prompt:
(2) Control primitive APIs implemented by us: These APIs serve a dual purpose: they demonstrate the usage of Mineflayer APIs, and they can be directly called by GPT-4.
exploreUntil (bot, direction, maxTime = 60, callback): Allow the agent to explore in a fixed direction for maxTime. The callback is the stopping condition implemented by the agent to determine when to stop exploring;
mineBlock (bot,, name, count 1): Mine and collect the specified number of blocks within
32-block distance;
craftItem (bot, name, count 1): Craft the item with a crafting table nearby;
placeItem (bot, name, position): Place the block at the specified position;
smeltItem (bot, itemName, fuelName, count 1): Smelt the item with the specified fuel.
There must be a furnace nearby;
AillMob(bot, mobName, timeout 300): Attack the mob and collect its dropped item; petItemFromChest. (bot, chestPosition, itemsToGet): Move to the chest at the specified
position and get items from the chest;
depositItemIntoChees (bot, chestPosition, itemsToDeposit): Move the chest the specified position and deposit items into the chest;
Control primitive APIs provided by Mineflayer:
await bot bot.pathf inder. wan.e..gtooeeeeeeooooooooooooooo (goal): Go a specific position. See below for how the goal;
26

---

Published in Transactions on Machine Learning Research (03/2024)
new GoalNear (x, y, 2, range): Move the bot to block within the specified range of the specified block;
new GxalXZ z): For long-range goals that don't have specific Y level;
new GoalGetToBlock(x,
2): Not get into the block, but get directly adjacent to it. Useful for fishing, farming, filling a bucket, and using a bed.;
new GoalFollow(entityy range): Follow the specified entity within the specified range;
new GoalPPaaceBloc22(pstttio bot. world, +O) Position the bot in order to place a block;
new GoaLLootAtBlocccpppsttttoo bot. world, O)) Path towards a position where a face of the block at position visible;
bot. LsABeeeeeeebeBlcctt Return true if bedBlock bod;
bot. www.booocttttssstoooo (position); Return the block at position;
await bot. bot.equip (item, destination): Equip the item in the specified destination. destination must be one of "hand", "head", "torso", "legs", "feet", "off-hand";
await bot. consume (); Consume the item in the bot's hand. You must equip the item to consume first. Useful for eating food, drinking potions, etc.;
await fishfish(): Let bot fish. Before calling this function, you must first get to a water block and then equip a fishing rod. The bot will automatically stop fishing when it catches a fish;
await bot. slee..leppccc sweeppeedBlee.eoeo (bedBlock): Sleep until sunrise. You must get to a bed block first;
await bot. www.vattellcc (block): This is the same as right-clicking a block in the game. Useful for buttons, doors, etc. You must get the block first;
await bot.looka, (position)) Look the specified position. You must go near the position before you look at it. To fill bucket with water, you must look at it first:
await bot activateItem() (): This is the same right-clicking to use the item in the bot's hand. Useful for using bucket, etc. You must equip the item to activate first;
await bot useun (entitt): This is the same as right-clicking an entity in the game. Useful for shearing a sheep. You must get to the entity first;
(4) Retrieved skills from the skill library;
(5) Generated code from the last round;
(6) Environment feodback: The chat log in the prompt;
(7) Execution errors;
(8) Critique from the self-verification module;
(9) The agent's current state: See Sec. A.2.1 for each element of the agent's 's state;
(10) Task proposed by the automatic curriculum;
(11) Task context: We prompt GPT-3.5 to ask for general suggestions about how solve the task. In practice, this part is handled by the automatic curriculum since it has systematic mechanism for question-answering (Sec. A.2.2
(12) Chain-of-thought prompting (Wei et al. 2022b) response: We ask GPT-4 first explain the reason why the code from the last round fails, then give step-by-step plans to finish the task, and finally generate code. See Sec. A.3.2 for the full prompt.
27

---

Published in Transactions on Machine Learning Research (03/2024)
A.3.2 Full Prompt
Prompt 4: Full system prompt for code generation.
You are a helpful assistant that writes Mineflayer javascript code to complete any Kinecraft task specified by me.
Here are some useful programs written with Mineflayer APIS.
/+
Explore until find iron_ore, (0, -1, 0) because iron ores usually underground await exploreUntil (bot, V0, 0), 60, () =>
iron_ore bot fin
matching: a.Data.blocksByllee. iron_oree ore "] id,
maxDistance:
});
return iron_ore;
});
Explore until find a pig, use (1,3 0, 1) because pige are usually on the surface let pig avait exploreUntil new Ve, 0, 1), 60, O -> const pig bot meearestEatity(ennt... return
=>
entity. name -n- "pig" &2 entity position. distanceTo (bot. enttet position) 32
);
}); return pig:
});
asyne function exploreUntil (bot, (bot, direction, maxTine 60, callback)
/*
Implementation
function amitted.
direction: Vec3, can only contain value -1,
maxTime: number, the max time for exploration
callback: function, early stop condition, will be called each second, exploration will stop if return value is not null
Return: mull if explore timeout, othervise return the return value of callback +/
// Mine 3 cobblestone: mineBlock (bot, "stone", 3) aayne function mineBlock (bot, nane. count 1) { blocks bot. (i matching: (block) => return block
name;
maxDistance: 32,
count: count,
});
const targete []:
for (let 1 0; Math Math. (blocks .length, count); i++) { targets. push (bot. blockAt (blocks [i]));
awalt bot. collectBlock collect (target,,,,,,lleeeetttttttt, { igmorelloPath: true });
11 Craft 8 oak_planks from 2 oak_log (do the recipe 2 times): craftItem (bot, "oak_planke", 2)
// You must place crafting table before calling this function asyne function craftIten (bot, name, count 1) {
item mcData. itensByNane [name];
craftingTable bot findBlock((
matching: meData. blockeByHame. crafting. table.
maxDistance:
28

---

Published in Transactions on Machine Learning Research (03/2024)
));
await bot. pat.paatfi goto ceall position, bot.world)
const recipe bot recipeeooo (iten. id, null, 1, craftingTable) [o];
await bot, cractpr count, craftingTable);
// Place crafting.table near the player, V1,3 0, 0) is just an example, you shouldn 't always use that: placeIteu(bot, ccraftingtablle. bot bot.ente entity Peatity.position position offaet (1, o, 0)); asyne function placeIten (bot,, name, position) { const item bot.
(mcData. itemsByllane [nane]. maaeelide id);
// find reference block
const faceWectors -
Ve, 1, 0),
Vec3 -1, 0),
(1c3 0, 0).
(-13 0),
Vec3 0, 1).
Ve,3 0, -1),
]:
let referenceBlock null; let faceVector mull;
for (const vector of faceVectors) {
const block bot blocktt (pt.biocktt minus (vector)),
if (block?. name === "air")
referenceBlock = block;
faceVector vector; break;
}
// You must first await bot pathfinderr gota (new GoalPlaceBlock (position, borld, (1));
the block position you want place
// You nust equip the item right before calling placeBlock awalt bot. equip "hand"); await bot. oet.pllaeelocc
faceVector);
Smelt rav_iron into iron_ingot using oak_planke as fuel: smeltIten (bot, "raw_ iron".
"oak_planke")
// You must place a furnace before calling this function
async function smeltIten (bot, itemllane, fuelllae, count = 1) {
const item mcData m itensByllame [itemllame]; const fuel mcData. Tanbata.itensBylllaa. [fuellamme]; const furnaceBlock
hot.ffinBlace
matching: mcData. Bboata.boockBBylaat furnace. maxDistance: 32, });
await bot pathfinifi ( GoalLookAtBlock (furnaceBlocck.ossiee
vorld)
furnace await openFurnsce (furnacellock))
(let await furnace, putFuel (fuel. id, null, 1);
count; i++)
awalt furnace. putInput
mull, 1);
// Wait 12 seconds for the furnace to smelt the item awalt bot waitforTicke (12 20):
avait furnace. takeoutput(). ();
await furnace. c)ose();
// Kill pig and collect the dropped item: killMob (bot, "pig", 300); sayne function killNob (bot, mobilane, timeout 300) { entity bot nesrestEntity
29

---

Published in Transactions on Machine Learning Research (03/2024)
(entity)
entity mobName
entity preition.ddistanee (bot entity position)
await pop. attack(entttt)))
await bot pathfinder goto( new GoalBlock (entiit position.x, position.y, entity .asttyppooee position.2 );
11 Get torch from chest at (30, 65, 100): getItenPronChest (bot, new Vec3 (30, 66, 100), {" torch": ;
// This function will work matter how far the bot is from the chest. sayne function setitemFromChestthoot chestPosition, itemsToGet) { cheatPosition);
await moveToChest (bot, (bot,
const chestBlock bot wot.bblcccAttchheePootttttt chest await bot. www..popoonnaaieeecceo... (chestBlock)) for (const const AtemByName const item chest. findContainerItem (itemBylemee.
itemsToGet)
itemsByName [name];
awalt chest. withdraw (item. (itteettttt...........et..mee. null, itemsToGet [
await closeChest (bot, chestBlock);
// Deposit a torch into chest at (30, 65, 100): depositIteeIntoChhes (bot, new Vec3 (30, 65, 100), ("torch": 1));
// This fuaction will work no natter how far the bot is from the chest. asyne function depositItemIntoChhettt (bot, chestPosition, itemsToDeposit) { (bot, (bot, chestPosition);
await moveToChest
const chestBlock = bot wot.oloocccchhhccootttii const chest await bot..ppenConts.nee (chestBlock)) for (const const StemByName mcData www.seyeameeeeeeeeeeeeeeeeeeae [name];
in itemsToDeposit)
const item bot inventory -inndnnenntrryTeer (itemByName... id))
awalt dlest.deposie (1tem. type. mull, itensToDeposit [name]);
await closeChest (bot, chestBlock);
Check the items inside the chest at (30, 65, 100): checkItenInstdeChher (bot, new Vec3 (30, 65, 100));
You only need to call this function once vithout any action to finish task checking items inside the chest.
asynс function checkIteeiiniideChstttt (bot, chestPosition)
await moveToChest (bot, (bot, chestPosition);
const chestBlock bot bicchllc await bot openCoonnnnn (chestBlock); // You nust close the chest after opening it if are asked to open chest await closeChest (bot, chestBlock);
avait pathiinder goto (goal); very useful function. This function may change your main-hand equipment.
Follooing
Goals you use:
new GoalNear(x, y. 2, range); // Move the bot to a block within the specified range of the specified block, 'x', 'y', 'z', and 'range' are 'number' new (xalX2 z); // Useful for long-range goals that don't t have specific level. and '2' are 'number!
new GoalGetToBlock (x, 8, z); // Not get into the block, but get directly adjacent to it. Useful for fishing, farming, filling bucket, and beds. 'x', 'y', and 'z' are 'number" new GoalFollos (entity, range); // Follow the apecified entity vithin the apecified range. entity is 'Entity', 'range' is 'number'
new GoalPlaceBlock (position, bot borld, 0); // Position the bot in order to place block. 'position' is 'Vec3'
new CoalLookAABBoockpppstto
bot. world, (); // Path into position where blockface of the block at position is visible. 'position' is 'Vec3'
30

---

Published in Transactions on Machine Learning Research (03/2024)
These are Mineflayer functions you use: bot. iet (bedBlook)))) Return bot. blockkt (position); // Return block 'position'. 'position" 'Vec3*
'bedBlock' r bed
These are other Mineflayer ааyпс functions you can use: await bot. equ.equup(1aee (item. destination); // Equip the item in the specified destination. 'item"
is 'Item', 'destination" can only be "hand", "head", "torso", "lege", "feet", "off -aand" await bot. consume // Consume the iten in the bot's hand. You must equip the item to consume first. Useful for eating food, drinking potions, etc.
await fith (); // Let bot fish. Before calling this function, you must first get to water block and then equip fishing rod. The bot will automatically stop fishing when it catches fish await bot. slee (bedBlock); // Sleep until sunrise. You must to bed block first await bot activateBlok (block); // This is the same right-clicking Useful for buttons, doors, using hoes, etc. You must to the block first suait bot JockAt
block in the game.
// Look the specified position. You must go the position before you look at it. To fill bucket with water, you lookkt first. 'position is 'Vec3"
await activateltem // hand. Useful using buckets, avait (entity); // This for shearing sheep, equipping harnesses, etc. You must get to the entity first
the You equip item activate first right-clicking an entity the game. Useful
right-clicking
the
bot's
(retrieved_ek1lle)
Åt each round of conversation, will give you Code from the last round: Execution error:
Chat log:
Bione:
Time:
Nearby blocks:
Nearby entities (nearest farthest):
Health:
Hunger:
Position:
Equipment:
Inventory (xz/36):
Cheats:
Task:
Context:
Critique:
You should then respond to me with Explain (if applicable): Are there steps missing in your plan? Why does the code not complete the task? What does the chat log and execution error imply?
Plan: How to complete the task step step. You should pay attention to Inventory since it tells what you have. The task completeness check is also based your final inventory.
Code:
1) Write asyne function taking the bot the only argument.
2) Reuse the above useful programs much ponsible. Use (bot,,lock nane, count) collect blocks. Do
'bot.dig directly
Use feraftIten (bot, name, count)' craft 'smeltIten name count) to smelt items. Do not use 'bot. openFurrneee directly.
Do
'bot.craft' directly.
Use 'placeIten name, position) to place blocks. not use 'bot 'bt. ploceplacllock'
directly.
Use 'killNob (bot, name, timsout) to kill mobs. Do not use 'bot. directly. 3) Your function will be reused for building more complex functions. Therefore, you should make it generic and reusable. You should not nake strong assumption about the inventory (as it may be changed at later time), and therefore you should alvays check whether you have the required items before using then. If not, you should first collect the required itens and reuse the above useful programs.
4) Functions in the "Code from the last round" section will not be saved or executed. Do not reuse functions listed there.
31

---

Published in Transactions on Machine Learning Research (03/2024)
5) Anything defined outside function will ignored, define all your variables inside your functions.
6) Call 'bot.chat''
internediate progress.
7) Use 'exploreUntil (bot, direction, maxDistance, callback)*
you
find
something. You should frequently call this before mining blocks or killing mobs. You should select direction at randon every time instead of constantly using (1, 0, 1). 8) "maxDietance" should always be 32 for 'bot. findlllkkss and 'bot. findBokkk cheat.
not
9) Do not write infinite loops or recursive functions.
10) Do not use 'bot.on' or 'bot.once" to register event listeners. You definitely do not need then.
11) Nane your function in neaningful way (can infer the task from the name).
You should only respond in the format described below: RESPONSE FORNAT:
Explain:
Plan:
1)
2)
Code:
''Javascript
helper functions (only if needed, try to avoid then)
// main function after the helper functions aаyис function yourMainnunnctionllmm (bot) {
//
Prompt 5: Full system prompt for generating function descriptions. This is used when adding a new skill to the skill library. We give one-shot example in the prompt.
You are a helpful assistent that writes description of the given function written in Mineflayer javascript code.
1) Do not mention the function nane.
2) Do not mention anything about 'bot.chat" or helper functions.
3) There night be some helper functions before the main function, but you only need to describe the main function.
4) Try to sumzarize the function in nore than 6 sentences.
5) Your reaponse should be a single line of text.
For example, if the function is:
asyne function mineCobblestone (bot)
// Check the wooden pickaxe the inventory, if craft woodenPickaxe inventory htw.:/aatttoy..illleeerrrrrooooe findInventoryItenn (mcData. itensByMame ["wooden_pickaxe id);
((voodenPickaxe)
bot chat ("Crafting vooden pickexe."); await sattWoodenPiccaxe(bbb))..
voodenPickaxe bot. inventory. www.ntor..findnvveteeee findInventorrIten (ncData. itemsByNamee "'xooden_pickaxe "]. id)
}
// Equip the wooden pickaxe if it exists if (xoodenPickaxe) {
await bot. equip (woodenPickaxe, "hand");
// Explore until we find stone block
await exploreUntil (bot, Vec3 (1, -1, 1), 60, O {
bot. findBllikl
matching: b.Data.blocksBylii (*stone ") id, maxDistance: 32 });
32

---

Published in Transactions on Machine Learning Research (03/2024)
(stone) true;
});
// Mine cobblestone blocks using the wooden pickexe bot. chat ("Found stone block. Mining cobblestone blocke. await mineBlock(bot, "stone", 8); bot, chat (" Successfully mined 8 cobblestone blocks.");
// Save the event of mining 8 cobblestone
bot save ("woobblettooneeineeee
} else
bot. chat ("Failed to craft wooden pickaze. Cannot mine cobblestone.") ");
}
The function is 'mineCobblestone
Then you would write:
The function is about mining cobblestones using a wooden pickaxe. First check if a wooden pickaxe is in the inventory. If not, craft one. the wooden pickaze is available, equip the wooden pickaxe in the hand. Next, explore the environment until finding stone block. Once stone block is found, mine a total of cobblestone blocks using the wooden pickaxe.
A.3.3 Examples
Skill library example 1: craftWoodenPanks..
sayne function eraftNoodenPlanke (bot) {
const logNames ["oak_log", bbrchh_log", "spruce_log", "jungle_log", "acacia_log"" dark_oak_log" "mangrove_log");
plankNames I'oak_planka", "birch_planke", "apruce_planxe", "jungle_planke",
acacia_planke", idark_oak_plent "zangrove.planke")
loginInventory logNames. find (logNane inventory count (mcData. itemsByName
logNamel 0);
(#loginInventory)
bot chat wooden log inventory. Mining wooden log...");
await mineWoodLog (bot);
const logIndex logNames index0f (logInInventory))
const plankName = plankNames [logIndex]; bot. chatchha a $(plankWame)...
await craftItem (bot, plankName, 1);
chat $(planklano} crafted. ));
Skill library example 2: mineTlaCCobllcDepsseeeeddd'''''
sвyпс function aiettanccbleebeeppssteeeeloooo
// Equip the iron pickaxe
const tronPickaxe bot. inttiiventory.ffee findInventoryItenn (mcData. itensByMame ["iron_pickaaee "]. id)
await bot. (tt.equup(rrooPcckeee "hand");
// Find cobbled_ blocks below Y=0
const coblledDeepslateBllocs await exploreUntil (bot, new Vec3 -1, 1). 60, 0 => const cobbledDeepslate = bot findBlock
matching: acData. blockeByNaee cobbled_deepslatttt 1ddd,
maxDistance: 32,
position: pos => pos.y });
cobbledDeepslate;
));
33

---

Published in Transactions on Machine Learning Research (03/2024)
// Mine 10 cobbled, deepalate blocks
await mineBlock (bot, "cbbleed_deepsatee" 10); 10 cobbled_deepslate mined below y=0.");
Skill library example 3: smeltFiveRawTron2...
ааynс function InaSultablePosttion { const offsets = [new (1, 0, 0), new Vec3(-1, 0, 0), Voc3 0, 1), -1)];
Vec3 (O, o,
for (const offset of offsets) position
www.eeetttt.tiiiii...e offset (offset.x, offset.y, offset if (block. blocklockAt (position);
block bot
"air") {
position;
return null;
sвyne function sxeltFiveRaurroa(ttt)) // Check if there is furnace in the inventory const furmaceIten bot www.:attorrr.dddddna.o...
(mcData. LtemsByNams furnace...... 1d);
// If not, craft a furnace using the available cobblestone if (IfurnaceItem) await (aftFurrace(bot));
1/ Find suitable position place the furnace furnacePositioon await finduitablePostionn (bot))))ooo(oo))); (bot); if (IfurnacoPositioom) {
chat (" Could find suitable position place furnace. "); return;
// Place the furnace at the suitable position await placeIten (bot, "furnace", furnacePosition);
// Smelt 5 raw iron using the available coal as fuel await smeltIten (bot, "ras_iron", "coal", 5); bot.chat chat ras iron smelted.*);
Skill library example 4: fillBucket WithWater.
asyne function fsllBucketWithWateer (bot) Find water block nearby
const waterBlock avait exploreUntil (bot, new (ec3 0, 1). 60, O const water bot (indBlock
matching: mcData. blockeByName, id,
msxDistance: 32
));
return water;
});
if (IwaterBlock) {
bot. chat (" Could not find water.");
return;
}
// Go the sater block adjacentBlock= vaterBlock. owfiition.offat 0); asait pathffitfr g GoalGetToBlock (adjacentBlockk adjacentBlock. y, adjacentBlock.2)
34

---

Published in Transactions on Machine Learning Research (03/2024)
// Look await looklo (vaterBlock position, true);
block
Equip bucket
const bucket bot. inventory findInventoryIten (mcData. itemsByName. buckeet id) await bot. equiipbuckkett "hand");
bucket
// Activate the bucket to collect water
await bot aotiiattvaeet)) bot. ("Filled the bucket with vater.");
Skill library example 5: catchFiveFishSafol.
asyne function catchFiveFishsafely (bot) // Check if the bot has a fishing rod in its inventory let fishingRod bot. intentryy FindInventoryIten (ncertttteemnaaaaaa (mcData. wwansyyllaee..sshingg... rd); if (IfiehingRod) { await (afttisshiggoddbot)) (bot);
fishingRod bot.innottrry bot. (mcData... itemsByllane fishing.fiiiniiiniee rod. id);
// Find nearby water block
let waterBlock;
while (IwaterBlock)
waterBlock await exploreUntillbot. Vec3 1), foundWaterBlock bot. findBlock(( matching: meData. ncData.blocker blocksBylane water.id, id,
maxDistance: 32
});
return foundWaterBlock;
));
if (IuaterBlock)
bot chat (No path to the water block. Trying to find another water block..."))
}
}
// Move to a block adjacent to the water block const adjacentBlock waterBlock position offset 1, 0) await bot pathffiedd goto (oslBlock(adJaceetBBll adjacentBlock.y, adjacentBlock.22 );
// Look await bot. lookAt (vaterBlock, position);
block
Equip fishing rod. asait bot. equip(fisshingddd, "hand");
// Fish in the water times for (let 0; <<5, i++) {
try {
avait bot. fish chat Fiihh $(1 + 1) caught. 9);
catch (error) {
if (error.
-.- "Fishing cancelled") { bot. ("Fishing was cancelled. Trying again....)) "); 1--: // Betry the same iteration else {
throv
35

---

Published in Transactions on Machine Learning Research (03/2024)
A.4 Self- Sefffeeiiiiitionn
A.4.1 Components in the Prompt
The input prompt to GPT-4 consists of the following components:
(1) The agent's state: We exclude other blocks that are recently seen and nearby entitics from the agent's 's state since they are not useful for assessing the lask's completeness. See Sec. A.2.1 for each element. of the agent's state;
(2) Task proposed by the automatic curriculum;
(3) Task context: We prompt GPT-3.5 to ask for general suggeslions about how to solve the task. In practice, this part is handled by the automatic curriculum since it has a systematic mechanism for question-answering (Sec. A.2.2
(4) Chain-of- Chain-of-thought prompting (Wei et al. 2022b) in response: We request GPT-4 to initially reason about the task's success or failure, then output a boolcan variable indicating the task's outcome, and finally provide a critique to the agent if the task fails.
(5) Few-shot examples for in-context learning (ei et al. 2022a Brown et al. 2020 Raffel et al. 2020
A.4.2 Full Prompt
Prompt 6: Full system prompt for self-verification.
You assistant that guidance.
my progress playing Minecraft and provides useful
required evaluate if requirements also considered success while failing to meet requires you provide critique to help me improve.
requirements. Exceeding
will give you the following information:
Bione: The biome after the task execution.
Time: The current time.
Nearby blocks: The surrounding blocks. These blocks are not collected yet. Hovever this is useful for some placing or planting tasks.
Health: current health.
Hunger: current hunger level. For eating task, if hunger level is 20.0, then I successfully ate the food.
Position: Ny current position.
Equipment: final equipment. For crafting tasks, sometimes equip the crafted item. Inventory (Ix/36); final inventory. For mining and smelting tasks, you only need check inventory.
Chests:
requires ne to place
chest, you find chest information
here.
Task: The objective need accomplish.
Context: The context of the task.
You should only respond in JSON format as described below:
"reasoning"; "reasoning", "success"; boolean. "critique": "critique",
Ensure the reaponse can be parsed by Python 'json. loads', e.g.: trailing commas single quotes, etc.
Here examples:
INPUT:
Inventory (2/36); l'oakllo''' :2, 'spruce_log log :2
Nine wood logs
36

---

Published in Transactions on Machine Learning Research (03/2024)
RESPONSE:
"ressoning": "You need wood logs. logs spruce logs, which up to wood logs.", "success": true.
"critique";
INPUT:
Inventory (3/36): (' t'arattig_tablle'. 1, "apruce_planka : 6, 'atick': 4)
Task:
Craft wooden pickaxe
RESPONSE:
{
"reasoning": "You have enough materials craft wooden pickaxe, but you didn't 't craft it.",
"success": false,
"critique": "Craft wooden pickaxe with crafting table using spruce planke and sticks."
INPUT: Inventory (2/36): ('rav_iron', 5, "stone_pickaxe 1}
Task:
Nine 5 iron_ore
RESPONSE:
{
"reasoning"; "Kining iron_ore in Minecraft will ras_iron. You have 5 raw_iron in your inventory.", "success": true, "critique"; "*
INPUT: Bione: plains
Nearby blocke: stone, dirt, grass_block, grass, farmland, wheat
Inventory (26/36)
Task:
Plant 1 wheat seed.
RESPONSE:
{
"reasoning": "For planting tasks, inventory information is useless. In nearby blocks, there is farmland and wheat, which means you succeed to plant the wheat seed.", "success"; true,
"critique";
INPUT: Inventory (11/36): (... "'rotten_feeet
Task:
Kill zonbie
Context:
RESPONSE
"reasoning": "You have rotten flesh in your inventory, which means successfully killed one zombie.",
"success"; true,
"critique";
37

---

Published in Transactions on Machine Learning Research (03/2024)
INPUT:
Hunger: 20.0/20.0
Inventory (11/36):
Task:
Eat
Context:
RESPONSE
"For all eating task, if the player's 's hunger is 20.0, then the player successfully ate the food.",
"reasoning":
"success"; true, "critique";
INPUT: Nearby blocks: chest
Inventory (28/36) ('rail:: 1, 'coal's 2, 'oak_planks', 13, 'copper_ 'copper_block" block's 1, 'diorite': 7,
booked_bbetf ' 4, 'granite'; 22, debblled 23, 'feather', 4, 'leather's 2, cooked_chicken 3, 'shite_wool' 2, 'stick': 3, 'black_wool'' 1, 'stone_sword' 2,
stone_hoe': 1, 'stone_axe'; 2, "stone_shovell 2, "cooeed mutton's "ooked__mttonn'. 4, 'cobblestone_vel :: 18, craatting.table'' 1, 'furnace'; 1, "iron_pickaxe" 1, 'stone_pickaxe 1, rav_copper ;; 12)
Cheats:
(B1, 131, 16): ('andesite?: 2, 'dirt': 2, 'cobblestone' :: 75, boodden_pickaxe" : 1, wooden_sword : 1)
Task: Deposit useless items into the chest at (81, 131, 16)
Context:
RESPONSE
items your inventory depositing, which than 20. You need to deposit more itens from your inventory to the chest.", "success"; false.
"reasoning": "You
"critique"; "Deposit more useless items such as copper_block diorite, granite, bubbled_deepelee feather, and leather meet the requirement of having only occupied slots in your inventory."
A.5 System-level Comparison between Voyager and Prior Works
We make a system-level comparison in Table. A.2 Voyager stands out as the only method featuring a combination of automatic curriculum, iterative planning, and a skill library. Morcover, it learns play Minecraft without the need for any gradient update.
B
Experiments
B.1
Experimental Setup
Our simulation environment is built upon MineDojo Fan et al. 2022 and utilizes Mineflayer (PrismarineJS) 2013) JavaScript APIs for motor controls (Sec. A.3.2p. Additionally, we incorporate many bot. chat () into Mineflayer functions to provide abundant environment feedback and implement various condition checks along with try-catch exceptions for continuous exccution. If the bot dies, ground, and its inventory preserved for uninterrupted exploration. The bot recycles its crafting table and furnace after program execution. For detailed implementations, please refer to codebase.
resurrected near the closest
38

---

Published in Transactions on Machine Learning Research (03/2024)
Table A.2: System-level comparison between VOYAGER and prior works.
VPT
Dreamer V3
DECKARD
DEPS
Plan4MC
VOYAGER
Baker et al.
(Hffner et al.
Nottingham
Wang et al.
Yuan et al.
2022
2023
et al. 2023)
2023b
2023
Demos
Videos
None
Videos
None
None
None
Rewards
Sparse
Dense
Sparse
None
Dense
None
Observations
Pixels Only
Pixels & Meta
Pixels & Inventory
Feedback & Inventory
Pixels Meta
Feedback &
Meta &
Inventory
Actions
Keyboard & Mouse
Discrete
Keyboard & Mouse
Keyboard & Mouse
Discrete
Code
Automatic Curriculum
(in-context
GPT-4 pro- posal)
Iterative Planning
(3 types of feedback)
Skill Library
(pre-defined)
(self-
generated)
Gradient-
Free
B.2 Baselines
ReAct Yao et al. 2022 uses chain-of-thought prompting Wei et al. 2022b) by generating both reasoning traces and action plans with LLMs. We provide it with our environment feedback and the agent states as observations. ReAct undergoes one round of code generation from scratch, followed by three rounds of code refinement. This process then repeated until the maximum prompting iteration is reached.
Reflexion (Shinn et al. 2023) built on top of ReAct (Yao al. 2022) with self-reflection to infer more intuitive future actions. We provide with environment feedback, the agent states, execution errors, and our self-verification module. Similar to ReAct, Reflexion undergoes one round of code generation from scratch, followed by three rounds of code refinement. This process is then repeated until the maximum prompling iteration is reached.
AutoGPT RRichards, 2023) is a popular software tool that automates NLP tasks by decomposing a high-level goal into multiple subgoals and executing them a ReAct-style loop. We re-implement AutoGPT by using GPT-4 to do task dccomposition and provide it with the agent states, environment feedback, and execution errors as observations for subgoal execution. Compared with VOYAGER, AutoGPT lacks the skill library for accumulating knowledge, self-verification for assessing task success, and automatic curriculum for open-ended exploration. During each subgoal exccution, if no execution error occurs, we consider the subgoal completed and proceed to the next one. Otherwise, we refine the program until three rounds of code refinement (equivalent. to four rounds of code generation) are completed, and then move on to the next. subgoal. If three consecutive subgoals do not result in acquiring a new item, we replan by rerunning the task decomposition.
The task is "explore the world and get many items as possible" for all baselines.
39

---

Published in Transactions on Machine Learning Research (03/2024)
Table A.3: Comparison between VOYAGER and baselines.
ReAct
Reflexion
AutoGPT
VOYAGER
Yao
al.
(Shinn et al.
Richards
2022
2023
2023
Chain-of- Thenughh (Wei et al. 2022b
Self Verification
Environment Feedback
Execution Errors
Agent State
Skill Library
Automatic Curriculum
B.3 Ablations
We ablate design choices (automatic curriculum, skill library, environment foedback, exccution errors, self- verification, and GPT-4 for code generation) in VOYAGER and study their impact on exploration performance.
Manual Curriculum: We substitute the automatic curriculum with a manually designed curriculum: "Mine 3 wood log", "Craft 1 crafting table", "Craft 1 wooden pickaxe", "Mine 11 cobblestone", "Craft 1 stone pickaxe", "Craft furnace", "Mine 3 iron ore", "Smelt iron ore", "Craft iron pickaxe", "Craft 1 irou sword", "Kill pig", "Cook 3 porkchop", "Eat cooked porkchop", "Mine copper ore", "Smelt copper ore", "Mine amethyst", "Craft 1 spyglass", "Mine gold ore", "Smelt gold ore", "Craft gold pickaxe", "Kill 3 cow", "Cook 3 boef", "Eat 1 cooked beef", "Mine 3 diamond ore", "Craft 1 diamond pickaxe", "Kill 3 sheep", "Cook 3 mutton", "Eat 1 cooked mutton", "Mine redstone ore", "Craft 1 clock", "Craft shield", "Craft iron helmet", "Craft 1 iron chestplate", "Craft iron leggings", "Craft 1 iron boots", "Kill 3 spider", "Craft 1 fishing rod", "Fish 3 raw cod", "Fish 3 raw salmon", "Fish 3 pufferfish", "Cook raw cod", "Cook raw salmon", "Craft 1 bucket", "Collect water", "Mine obsidian". manual curriculum requires human effort design and not. scalable for open-ended exploration.
Random Curriculum:
We curate 101 items obtained by VOYAGER and create a random curriculum by randomly selecting one item as the next task.
w/o Skill Library: We remove the skill library, eliminating skill retrieval for code generation.
w/o Environment Feedback: We exclude environment feedback (chat log) from the prompt for code generation.
w/o Exccution Errors: We exclude execution errors from the prompt for code generation.
w/o Sclf- Verification: For each task, we generate code without self-verification and itcratively refine the program for rounds (equivalent to 4 rounds of code generation total).
We replace GPT-4 with GPT-3.5 for code generation. We retain GPT-4 for the automatic curriculum and the self-verification module.
GPT-3.5:
B.4
Evaluation Results
B.4.1
Significantly Better Exploration
The meaning of each icon in Fig. is shown in Fig. A.1
We three trials for each method. The items collected by VOYAGER in each trial is
40

---

Published in Transactions on Machine Learning Research (03/2024)
wood_/og
stick
raw_copper
lapis_lazuli
leather
ender_pearl
wood_plank
wooden__ickaax
copper_ingot
copper_block
cooked_mutton
pufferfish
crafting_table
wooddnnsswod
raw_iron
amethyst.block
cooked_porkcchop
salmon
chest
wooden_hoe
iron_ingot
bucket
bed
bamboo
sand
wooder_shovel
amethyst
water_bucket
compass
wood_sapling
dirt
wooden_axe
amethyst_cluster
lava_bucket
wheat_seed
cobblestone
7
stone_pickaxe
raw_gold
iron_helmet
spyglass
cactus
furnace
70
iron_pickake
gold_inget
iron_chestplate
rishing_rod
green_dye
wood_boat
goodd
redstone
iron_egggins
rail
wood_button
diamond sword
diamond
0 iron_bocts
scaffolding
Figure A.1:
Minecraft item icons with corresponding names.
'iron ingot', 'stone shovel', 'iron leggings', "fishing_ rod', 'pufferfish', 'oak 'oak_log', 'cooked mutton', 'green dye', 'flint', 'chest', 'iron sword', 'string', 'ender pearl', 'raw copper', 'crafting table', 'cactus', 'lapis lazuli', 'iron soon__ickaxee 'copper 'copper_ingon ingot', 'stone 'stone_pickaxe" pickaxe', 'wooden hoe', 'scaffolding', 'stick", 'porkchop', 'copper block', 'gravel', 'grass block', 'white bed', 'bone', 'dirt', 'mutton', 'white_ toolle_ 'oak Lak_sspling', 'coal', 'bamboo', 'wooden pickaxe', 'rot- grooden_pickaxe',
Trial 1:
ten flesh', 'cooked porkchop', 'cod', 'iron boots', 'lightning rod', 'diorite', 'water bucket', 'shears', "furnace", 'andesite", 'granite', 'bucket', 'wooden 'wooden_sword, 'sandstone', 'iron helmet', 'raw iron', 'sand', lacacia_loog 'cooked_cet cod', 'oak plankllak', 'azure 'azure_bluct', bluct', 'iron shovel', 'acacia 'acacia_plankk',, 'shicld', iron_axx', 'iron choe_t_alll, 'cobblestone';
Trial 'iron ingot', 'tuff", 'stone shovel', 'iron leggings', 'fishing rod', 'cooked mutton', "spruce_ "spruce_plankk', 'gunpowder',
'cooked_salmon',, shard', 'chest', 'string', 'cooked salmon', 'iron_sword"
'stone__iccaxxe' pickaxe', 'raw cowp__op'rr,__ 'crafting table', 'torch', 'lapis lazuli" firon pinka_pccce',, 'copper ingot', 'stone
'wooden hoe', 'stick', 'amethyst block', 'salmon', 'calcite', 'gravel', 'white bed', 'bone', 'dirt', 'mutton', 'white_ "hite__wool,, wool', 'spyglass', 'coal', wooden__pccaase 'cod', 'iron boots', 'lily dily_pad', 'cob- bled deepslate', hightning_rode rod', 'snowball', 'stone istone_axe', 'smooth basalt,, 'diorite', 'water bucket', 'furnace', 'andesite', 'bucket', 'granite', 'shicld' fron_hhelmet 'raw 'raw_iron', 'cobblestone', 'spruce_ log', 'cooked cod', 'tripwire hook', 'stone hoe', firon chestplate', 'stone sword';
'iron cron_chhstppltee 'stone pickaxe', 'leather', 'string', 'chicken', 'chest', 'diorite', 'iron leggings', bllck wool', 'cob- blestone_ paalll 'cobblestone', 'cooked chicken', 'feather', 'stone "stone_ssoord', 'raw gow_"old', 'gravel', 'birch planks', 'coal', 'cobbled deepslate', 'oak planks', 'iron pickaxe', 'granite', 'tuff', 'craft- ing_ table', 'iron helmet', 'stone hoc', 'iron ingot', "stone_axe" 'birch boat', 'stick', 'sand', "bone', 'raw iron', "beef', 'rail', 'oak sapling', 'kelp', 'gold ingot', bbirh log', 'wheat. seeds', 'cooked mutton', 'furnace', 'arrow', 'stone_ shovel', 'white _woll, 'andesite", 'jungle slab', 'mutton', 'ron sword', 'copper ingot', 'diamond', 'torch', 'oak log', 'cooked beef", 'copper block', 'flint', 'bone_ "bone_ 'raw_copper" 'wooden "wooden_pickaxe'' iron_boots"" 'wooden_sword"
Trial 3: 'spruce_ppaaiie planks', 'dirt', 'shicld', 'redstone', 'clock', ddiamond_swerr
'stone_shovels
The items collectod by ReAct Yao al. 2022) in each trial is
Trial 1: 'bamboo', 'dirt', 'sand', sceds';
Trial 2: 'dirt', 'rabbit', 'spruce_ loo',,_oo,, sppruce_sapingg
Trial 3: 'dirt', "pointed dripstone';
The items collected by Reflexion (Shinn et al. 2023 in each trial is
Trial I: 'crafting table', 'orange tulip', 'oak planks', 'oak log', 'dirt';
41

---

Published in Transactions on Machine Learning Research (03/2024)
Trial 2: "spruce_ log', 'dirt', 'clay __all, 'sand', 'gravel';
Trial 3: 'wheat seeds', 'oak log', 'dirt", 'birch log', 'sand'.
The items collected by AutoGPT Richards 2023 in each trial is
Trial 1: 'feather', "oak log', 'leather', 'stick", 'porkchop', 'chicken', 'crafting table', 'wheat seeds', plakkk, 'dirt', 'mutton";
'iron ingot', 'stone', 'coal', pppruce_planlee 'string', "ra_copper', 'crafting _talii 'diorite,, 'andesite', 'furnace', 'torch', 'spruce sspruce_sapling, 'granite', 'iron pickax'_ 'stone pickaxe', 'wooden axe', 'raw iron', 'stick", 'spruce log', 'dirt', 'cobblestone;;
Trial 2:
'wooden__pcceee
Trial 3: 'wooden shovel', 'wooden pickaxe', 'iron ingot', 'stone', 'cod', 'coal', 'oak log', Aflint, 'raw cewp_r' 'crafting _tablea 'diorite', 'furnace', 'andesite', 'torch', 'granite', sapis_lazulij 'iron pickaxe', 'stone pickaxe', 'raw iron', 'stick', 'gravel', 'oak planks', 'dirt', 'iron axe', tcobble- stone'.
B.4.2 Extensive Map Traversal
Agent trajectories for map coverage are displayed in Fig.
The terrains traversed by VOYAGER in each trial
Trial 1:
'meadow', 'desert', 'river', 'savanna'; 'forest', 'plains', 'bambboo jungle', 'dripstone caves';
Trial snowy_plainss 'frozen river', ddripstone _aaves" 'snowy taiga', 'beach;;
'flower fooesr_oorcst', 'meadow", 'old Bit__goowt_hb_ birch_forest', "snowy_sloped ppazs' "forest', 'river', 'beach', 'ocean', "sunflower plains', 'plains", 'stony shore:.
Trial 3:
The terrains traversed by ReAct Yao al. 2022 in each trial is
Trial 1: 'plains', 'desert', 'jungle';
Trial 2: 'snowy plainsi, 'snowy taiga', 'snowy slopes';
Trial 'dark 'dark_forest forest', 'dripstone_caave caves', 'grove', 'jagged_peadt peaks',
The terrains traversed by Reflexion Shinn et al. 2023 in each trial is
Trial 1: 'plains', 'flower forest_forest';
Trial 2: saioyytaiiga':
Trial 3: 'old growth birch forest', 'river', 'ocean', 'beach', 'plains',
The terrains traversed by AutoGPT Richards 2023 in each trial is
Trial 1: "plains', "dripstone_ceet " 'savanna', 'meadow;;
Trial 2: 'snowy laiga';
Trial 3: 'plains', 'stony shore', 'forest, focean'.
42

---

Published in Transactions on Machine Learning Research (03/2024)
Craft a Diamond Pickaxe
Craft a Compass
diamond pickaxe
compass redstone
diamond
iron pickaxe
iron pickaxe
iron ingot
Iron Ingot
Taw_iron
raw iron
furnace
furaace
stone pickaxe
7
stone pickaxe
wooden pickaxe
wooden pickaxe
crafting table
crafting table
wood log
wood log
10
20
50
10
20
40
50
Prompting Iterations Code Generation
Prompting Iterations Code Generation
Voyager (Ours)
Voyager w/o Skill Library
AutoGPT AutoGPT w/ Our Skill Library
Figure A.2: Zero-shot generalization to unseen tasks. We visualize the intermediate progress of each method on the other two tasks. We do not. plot ReAct and Reflexion since they do not make any meaningful progress.
B.4.3 Efficient Zero-Shot Generalization to Unseen Tasks
The results of zero-shot generalization to unseen tasks for the other two tasks are presented in Fig. A.2 Similar to Fig. S VOYAGER consistently solves all tasks, while the baselines are unable to solve any task within 50 prompting iterations. Our skill library, constructed from lifelong learning, not only enhances VOYACER's 's performance but also provides a boost to AutoGPT (Richards 2023
B.4.4 Accurate Skill Retrieval
We conduct an evaluation of our skill retrieval (309 samples in lotal) and the results are in Table. A.4 The top-5 accuracy standing at 96.5% suggests our retrieval process is reliable (note that we include the top-5 relovant skills in the prompt for syuthesizing new skill).
Table A.4: Skill retrieval accuracy.
Top-1 Acc
Top-2 Acc
Top-3 Acc
Top-4 Acc
Top-5 Acc
80.2 3.0
89.3 + 1.8
93.2 + 0.7
95.2 + 1.8
96.5 + 0.3
B.4.5 Robust to Model Variations
In the main paper, all Voyager's experiments are conducted with gpt-4-0314. We additionally run new experiments with gpt-4-0613 and find that the performance roughly the same (Fig. A.3) demonstrates that Voyager is robust to model variations.
43

---

Published in Transactions on Machine Learning Research (03/2024)
50
Number of Distinct Items
40
20
GPT-4-0314
0
GPT-4-0613
0
25
50
75
100
125
150
Prompting Iterations in Code Generation
Figure A.3: VOYAGER's performance with GPT-4-0314 and GPT-4-0613.
44


Reviews:
[{"review_id": "VmojV0RaeD", "reviewer": "Reviewer_ogTF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Very impressive overall performance. It is surprisingly and instructive to see how much can be achieved purely with black box queries to a SOTA LLM.\n* Careful and insightful ablation experiments.\n\nWeaknesses:\n* The high level action API abstracts away a large part minecraft. It remains to be seen to what extent the same performance can be obtained in other domains.\n* The paper is relatively clear but I think the overall presentation could be improved. See suggestions below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an embodied lifelong learning agent in Minecraft and explores the world, acquires and reuses skills. It consists of three components:\n* an automatic curriculum that proposes novel tasks\n* a skill library for storing and retrieving new skills\n* an iterative prompting mechanism\n\nVoyager interacts with Minecraft via code and the high-level Mineflayer API. All the different components are implemented with black box queries to GPT-4. The methods leads to quite advance and complex minecraft behaviour. It performs better than reasonable LLM-based baselines in a number of metrics. Futhermore the paper presents ablation studies in which parts of the overall system are removed. The automatic curriculum is found to be crucial and the skill library appears to help avoid plateauing. Finally GPT-4 is found to outperform GPT-3.5 in code generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main concern is that the main text lacks some details that would make it easier to read as a self-contained document. \n* Consider moving the algorithm in pseudo code to the main text.\n* Replace figure 7 with the similar figure A.2 from the appendix that shows paths instead of circles. As it stands there is not enough information to interpret fig 7. Please highlight the starting position in some way.\n* please give further details on figure 1: What do the shaded regions represent? I am assuming there are several underlying runs shown. How are the shown items related to only one of the runs? In particular this presentation may lead a reader to conclude that the diamond tool is always found. If I understand table 1 correctly, this is not the case?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QBM8gMbt1d", "reviewer": "Reviewer_HygD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The approach is an elegant composition of different ways of using LLMs that form a clearly strong game agent.\n- The experimental setup is really thorough, and the adaptation of existing approaches (ReAct, Reflexion, AutoGPT) providing access to the skill library etc. are well-tested. \n- Overall, the experiments are sufficient to show that this approach is very strong!\n\nWeaknesses:\n- My main concern is that one critical component of this approach, the automatic curriculum, is useful primarily because it's taking advantage of existing knowledge of the Minecraft tech tree and game strategies that are encoded in GPT-4. This is not a fundamental flaw, but it would be interesting to discuss more how the proposed approach would generalize to completely new games without mentions on the web that have equally complex tech trees. \n- The manual curriculum seems quite small and focused on a single goal in the tech tree (mining a diamond). Can't a larger manual curriculum be constructed automatically from the tech tree itself, e.g. by flattening it into some sequence?\n\nQuestions:\n- In Figure 5 left, it seems like the lines describing how to create a stick could be factored out into their own function, right? Since 2 sticks are being created (also, it seems like this code is only creating 1 stick even though 2 more are needed?). Where, if anywhere, in this pipeline is this kind of refactoring taking place? Or would it only be possible if there was already a function called craftStick available in the skill library?\n- In the zero-shot generalization experiment, does the evaluation of Voyager also allow for exploration and adding to the skill library, or are these two aspects fixed?\n- I'd like to see more details on what \"self-ask questions\" look like in Section 2.1.\n- The human feedback in Fig 10 is type (1) right, using humans as a critic?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a system that acts as an agent in Minecraft, which progressively acquires and represents the complex action and state space of the environment through exploration and abstraction. The main contributions of the approach are: (a) a curriculum generated from an LLM to suggest new subgoals to pursue at a particular environment state, (b) an iterative prompting mechanism where the goal is to write a piece of executable code representing this subgoal, and a (c) skill library that stores previously-generated pieces of code for later use. Experiments are performed on the Minecraft environment, and comparisons are made with existing text-based approaches that focus on self-reflection / inference-time refinement for (a) iterative refinement / critique at inference time and (b) inference-time policies that decompose the current goal into subgoals.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Discussion on how priors encoded in pretrained LMs may not be available for new games, thus the exploration phase / automatic curriculum may not transfer as well\n- Experiments using the full tech tree to create a curriculum instead of a small number of hand-picked subgoals as the manual curriculum", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cUJgefJTWI", "reviewer": "Reviewer_eudD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Results are convincing and the authors also perform some ablation to investigate components of Voyager (exploration, CL, and self-verification).\nHowever, being not very familiar with this kind of work, it is hard for me to assess its novelty and where it places itself with respect to current literature. \n\nIt seems to me that all the reasoning is done by Chat-GPT, and the Voyager framework simply provides Chat-GPT some constrains and guidance (\"learn diverse tasks\" and \"start with simple tasks\"). In the end, Voyager does not really learn, Chat-GPT does. \nThe authors often argue that they want to mimic human learning, so Voyager starts with simple tasks and then builds a library of skills over time. Yet, the learning that happens does not look like human learning to me. \nRather than being like a new human player learning Minecraft autonomously for the first time all by itself, Voyager seems a new player being instructed step-by-step by a pro player (Chat-GPT). Rather than really learning, one could argue that the player just follows instructions blindly. \nOverall, this seems more like a new way to use Chat-GPT rather than actually showing a new way of learning.\nI'd appreciate if the authors could elaborate on this.\n\nHaving more discussion on the baseline would also help understanding the impact of Voyager. I am not familiar with ReAct, Reflection, and AutoGPT, and their very brief description provided by the authors is not sufficient to understand the difference between them, and how Voyager is better. \n\nFinally, how can Voyager be applied to other domains, and impact other research? The authors briefly mention \"robotics\" at the end of the paper but more discussion on it would help. I find it hard to believe that Chat-GPT would help in real-world taks where very little data is available, and where maybe simulation is impossible or inaccurate.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents Voyager, a novel framework for autonomously training an agent to play Minecraft. Voyager uses a curriculum learning (CL) approach: it keeps exploring the world and builds a library of skills, that can be retrieved to solve new tasks. The major innovation of this framework is the aid of a \"prompting mechanism\" that eliminates the needs of human intervention. \nVoyager shows impressive results and beats all SOTA competitors. \n\nThe paper is well written, results are clear, and the proposed framework is interesting. \nTo the best of my knowledge, while the use of CL and a library of skills is not new to this type of problems, the combination with LLMs makes Voyager novel. \n\nIn particular, Voyager agent's actions are code routine: if the action is \"craft axe\" then Voyager outputs JavaScript code that produces low-level commands that try to craft an axe. If the code fails, Voyager receives some feedback from the game (e.g., \"not enough material X to craft an axe\") and refines the code. If the task is successful, the agent saves the code as skill for future reference. This way, one task at the time, it builds a library of skills of increasing difficulty. \nThe way skills are learned, in fact, are constrained to the higher goal of \"discover as many diverse things as possible\" and that \"the task should not be too hard\". This is done to reflect the behavior of human players, who start learning basic tasks and then move on to harder tasks. \n\nAll this reasoning is done by Chat-GPT. First, information about the current state of the game (e.g., inventory, biome, time, health) are passed to Chat-GPT. In turn, Chat-GPT decides what to do (find food, harvest materials, craft items, ...) and outputs code that performs actions in Minecraft.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Discussed in \"strengths and weaknesses\".", "broader_impact_concerns": "Discussed in \"strength and weaknesses\".", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers are in the agreement that the method is sound and impressive. And while it might to be clear how well it might generalize beyond MineCraft, there is value in publishing the paper.

eudD: "The paper combines known and new techniques into a novel framework that trains agents to play Minecraft better than any other existing framework. The paper is well written, resuls are convincing and well discussed, and the authors have addressed all my concerns."
ogTF: "I think this paper shows very impressive performance in Minecraft."
HygD: "While I am still unsure about whether this approach will scale with completely new games that have different dynamics than ones whose relevant documentation / discussion has been heavily included in training data for existing LLMs, I think the approach itself is noteworthy and worthy of acceptance."
Decision: Accept (poster)