STEVE-1: A Generative Model for Text-to-Behavior in Minecraft
Authors: Shalev Lifshitz
Keiran Paster
Harris Chan
Jimmy Ba
Sheila A. McIlraith
Venue: NeurIPS 2023 spotlight (2023)
Keywords: minecraft
instruction following
foundation models
sequence models
reinforcement learning
sequential decision making
goal conditioned reinforcement learning
text conditioned reinforcement learning
transformers
deep learning
TL;DR: We introduce a methodology for instruction-tuning generative models of behavior without relying on a large dataset of instruction-labeled trajectories.

Abstract: Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces a methodology, inspired by unCLIP, for instruction-tuning generative models of behavior without relying on a large dataset of instruction-labeled trajectories. Using this methodology, we create an instruction-tuned Video Pretraining (VPT) model called STEVE-1, which can follow short-horizon open-ended text and visual instructions in Minecraft. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, reducing the need for costly human text annotations, and all for only $60 of compute. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines and robustly completing 12 of 13 tasks in our early-game evaluation suite. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.

STEVE-1: A Generative Model for Text-to-Behavior in Minecraft
Shalev Lifshitz!:22 1:2+
Keiran Paster keirp0cs toronto.oo.m... edu
shalev. lifshitzzmail.......mi....... 11fehhtz/mma...uooooooooo utoronto...
Harris Chan 1,2
Jimmy Ba Bal.2
Sheila Mcllraith sheila@cs toronto.oooo... edu
hchanCcs. toronto. edu
jbaccs. toronto.... edu
Depaatment of Computer Scienee, University of Toronto, Toronto, Canada. Vector Institute for Artificial Intelligence, Toronto, Canada.
Abstract
Constructing Al models that respond to text instructions is challenging, especially for sequential decision decision-making tasks. This work introduces methodology, inspired by unCLIP, for instruction-tumat
generative models behavior without relying large dataset of instruction-laelled trajectories. Using this methodology. create an instruction-tunce Video Pretraining (VPT) model called STEVE-1. I, which follow short-horizon open-ended text and visual instructions in Minecraft STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow com- mands in MineCLIP's latent space, then training prior predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling. reducing the need for costly human annotations, and all for only $60 of compute. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE- sets new bar for open- opennended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines and robustly completing 12 of 13 tasks in our early-game evaluation suite. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-fre guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.
Introduction
The ability use text instructions to control and interact with powerful Al models has made these models accessible and customizable for the masses. Such models include ChatGPT [41], which can respond to messages written in natural language and perform a wide array of tasks, and Stable Diffusion [50], which turns natural language into image. While those models cost anywhere from hundreds of thousands to hundreds millions of dollars to train, there has been an equally exciting trend whereby powerful open-source foundation models like LLaMA [59] can be finetuned with surprisingly little compute and data to become eestruction-folloviii (e.g.. [58, 13]).
In this paper, we study whether such approach could be applicable to sequential decision- decision-making domains. Unlike in text and image domains, diverse data for sequential decision-making very expensive and often does not come with convenient "instruction" label like captions for images. We propose to instruction-tune pretrained generative models of behavior, mirroring the advancements seen in recent instruction-tumm LLMs like Alpaca [58], without relying large dataset of instruction-labell trajectories.
"quual contribution.
i Core contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Video
ercodder
Reshiet
ResNet
Reshiet
Spal
CLIP Objective
Zg
OVAE
Policy
Text
Transfomer (VPT)
Encoder
Prior
Goussian Prior
"chop
MineCLIP
(Frozen)
Mouse/Keyboard
Figure Like unCLIP [48], our approach involves two models. First, we train the policy by finetuning VPT to achieve goals given by pretrained MineCLIP [17] visual embeddings using gameplay dataset. Second, for the prior model, we train a CVAE [54] to sample MineCLIP visual embeddings given text prompt. The combination of these two models enables agent to follow text and visual instructions.
In the past year, two foundation models for the popular open-ended video game Minecraft were released: foundation model for behavior called VPT [5] and model aligning text and video clips called MineCLIP [17]. This has opened up an intriguing avenue to explore fine-tuning for instraction-following in the sequential decision-making domain of Minecraft. VPT was trained 70k hours of Minecraft gameplay. so the agent already has vast knowledge about the Minecraft environment. However, just as the massive potential of LLMs was unlocked by aligning them to follow instructions, it is likely that the VPT model has the potential for general, controllable behavior it is finetuned to follow instructions. particular, our paper demonstrates method for fine-tuning VPT to follow short- mort-horiion text instructions with only $60 of compute and around 2,000 pnrruction-labelle trajectory segments.
Our method draws inspiration from unCLIP [48], the approach used to create the popular text-to- image model DALL.E 2. We decompose the problem of creating instruction-following Minecraft agent into two models: a VPT model finetuned to achieve visual goals embedded in the MineCLIP latent space, and a prior model that translates text instructions into MineCLIP visual embeddings. We finetune VPT using behavioral cloming with self-supervised data generated with hindsight relabeling [3], avoiding the use expensive text-instruction labels favor of visual MineCLIP embeddings. We apply unCLIP with classifier free guidance [23] to create our agent called STEVE-1, which sets a new bar for open-endee instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming the baseline set by Baker et al. [5].
Our main contributions
follows:
We introduce methodology, inspired by unCLIP [48], for instruction-cuning generative models of behavior without relying on a large dataset of expensive instruction labels.
We apply this methodology to create STEVE- 1, a Minecraft agent that can follow short- horizon open-ended text and visual instructions with high degree of accuracy, all for only $60 of compute. We perform extensive evaluations of our agent, showing that it can robustly complete 12 of 13 goal-conditioned control tasks in our parly-game evaluation suite in Minecraft. For long-horizon tasks' like crafting and building, we show that a basic version of prompt chaining can dramatically improve performance.
We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-frs free guidance, data scaling, prompt-engineering, and other design choices. particular, we show that unCLIP [48] and classifier-fres free guidance [23] translate well to sequential decision making and are essential for strong performance.
We release model weights for STEVE-1 1 as well as training scripts and evaluation code to help foster more research into instructable, open-ended sequential decision-main making agents.?
'Sortt-horioo tasks require few steps: e.g., go
and chop it down, dig hole. Long-horizz tasks
take many steps: e.g.. craft complex recipes from scratch, build a house.
"Model weights, training code, videos, and an interactive demo script are hosted on our project webpage

---

2 Related Work
Minecraft as a Test-bed for AI Minecraft has gained popularity as a benchmark for Al research due to its complex and dynamic environment, making it a rich test-bed for reinforcement learning and other Al methods (e.g.. [26, 19, 17, 21, 40, 62. 38, 9]). We leverage the MineRL environment [19] to research the creation of agents that can follow open-ended instructions in complex visual environments using only low-level actions (mouse and keyboard). We build STEVE-1 on top of two recent foundation models. In order to align text and videos,
MineCIIIP [17), CLIP [47] model trained on paired web videos of Minecraft gameplay and associated captions. To train STEVE-1's 's policy,
fine-tune VPT [5], a foundation model of Minecraft behavior that is pretrained on 70k hours of web videos of Minecraft along with estimated mouse and keyboard actions. Several prior works [61, 62] have explored the use of LLMs in creating instructable Minecraft agents. These works typically use LLMs to make high-level plans that are then executed by lower-level RL [40, 62] or scripted [46] policies. Since STEVE-1 is far more flexible low-level policy, the combination of STEVE-1 1 with LLMs is a promising direction for future work. Fan et al. [17] introduced an agent trained using RL with MineCLIP as shaping reward on 12 different tasks and conditioned on text-prompts. However, this agent failed to generalize beyond the original set of tasks without further RI. finetuning using the MineCI.IP reward function. Cai et [9] proposed a Goal-Sensitive Backbone architecture for goal-conditioned control in Minecraft which is trained on fixed set of goals, while STEVE-1 learns goal-reaching behavior from large dataset in a self-supervised way without training on an explicit set of tasks.
Foundation models which are pretrained on vast amounts of data and then finetuned for specific tasks have recently shown great promise in variety of domains including language [8, 14, 59], vision [48, 10, 47], and robotics [7. 53, 25, 39, 65]. GATO [49] and RT-1 [7] have demonstrated the potential of training transformers to perform both simulated and real-world robotic tasks. With the exception of Kumar et al. [30]. which uses Q- learning, the vast majority of cases [32, 7, 49] where deep learning has been scaled to large, multitask of datasets have used supervised RL. Supervised RL (e.g.. [42, 18, 12)) works by framing the sequential
Foundation Models for Sequential Decision-Making
problem as a prediction problem, where the model is trained to predict the next action conditioned on some future outcome. While these approaches are simple and scale well with large amounts of compute and data, more work needed to understand the trade-off's between supervised RL and Q-learning policy gradient-based methods [43, 44. 6, 55]. Recent works explore the use of hindsight relabeling [3] using Lasiuanlanguage models [47, 2] to produce natural language trajectories, which is then used to select a hindsight instruction from a candidate set. Sumers et al. [56] uses Flamingo [2] zero-shot for hindsight relabeling by framing it as visual-question answering (VQA) task. In contrast, STEVE- 1 relabels goals using future trajectory segment embeddings given by the MineCLIP [17] visual embedding.
relabeling instructions. DIAL [65] finetunes CLIP [47] on human-labell
Coxt-Conditioned Generative Models There has been a recent explosion of interest in text-to-X models, including text-to-image (e.g.. [48, 51, 50]), text-to-3D (e.g.. [27, 35]), and even text-to-music (e.g.. [I]. These models are typically either autoregressive transformers modeling sequences of discrete tokens [60, 8] or diffusion models [24]. Most related to our work is unCLIP, the method used for DALL.E 2 [48]. unCLIP works by training generative diffusion model to sample images from CLIP [47] embeddings of those images. By combining this model with a prior that translates text to visual CLIP embeddings, unCLIP can produce photorealistic images for arbitrary text prompts. unCL.IP and many other diffusion- basidd approaches utilize a technique called classifier-fres guidance [23], which lets the model trade-off between mode-coverage and sample fidelity post-training. We utilize the basic procedure of unCLIP and classifier- free guidance for training STEVE-1.
3 Method
Inspired by the rapid recent progress in instruction-tuninn Large Language Models (LLMs), we choose to leverage the recently released Video Pretraining (VPT) [5] model as a starting point for our agent. Since VPT was trained on 70k hours of Minecraft gameplay, the agent already has vast knowledge about the Minecraft environment. However, just as the massive potential of L.MSs was unlocked by aligning them to follow instructions, it is likely that the VPT model has the potential
3

---

22 Zg 24 2/ 2g 27
Zg 22 Z7 21
27
21
27
Figure 2: To create conl-condit data for finetuning, randomly select timesteps from episodes and use hindsight relabeling to set the intermediate goals for the trajectory segments to those visual MineCLIP embeddingg. This self-supervised data teaches the agent which actions lead which states.
for general, controllable behavior if it is finetuned to follow instructions. In this work, we present a method for finctuning VPT to follow natural, open-ended textual and visual instructions, which opens the door for wide range uses for VPT in Minecraft.
Our approach is inspired by unCLIP, the method behind the recent text-to-image generation model, DALL 2 [48]. Our goal is create generative model of behavior in Minecraft conditioned on text instructions y. To do so, we utilize a dataset of Minecraft trajectory segments, some of which trajectory of observations and actions. We also employ a pretrained CLIP model called MineCLIP [17], which generates aligned latent variables Z7 Zy. where 17 10 is embedding of any 16 consecutive timesteps from the trajectory. MineCLIP is trained using a contrastive objective on pairs of Minecraft videos and transcripts from the web. For simplicity notation, we refer the MineCLIP embedding of the last 16 timesteps of a trajectory segment as Fgu Like unCLIP [48], we utilize a hierarchical model consisting of prior and policy:
contain instruction labels y: [(TI, vi), (T2, 1y2)...................... (Tr, 0)] where
A prior p(zr,uly)) that produces a latent variable Zrzur conditioned on text instruction y- A policy P(T (rgu)) that produces a trajectory conditioned on a latent variable 2rgeall
These two models then be combined produce generative model of behaviors conditioned on text instructions:
p(rly) p(T, Zrgua (y) p(( (T p(( ZTpa))
(3.1)
3.1 Policy
To learn our policy, we finetune VPT, a foundation model of Minecraft behaviors po(T) trained on 70k hours of Minecraft gameplay videos. Specifically, VPT consists of a ResNet [22] that processes frames of dimension 128 128 x 3, and a Transformer-XI- [15] which processes frame representations and autoregressively predicts the next action using the joint hierarchical action space described Baker et al. [5]. In order to modify the architecture to condition on goal information, we add affine transformation of z to the output of the ResNet before passing it to the transformer:
Process Frames:
ResNeta(O:) + It
[+ Conditioning on MineCLIP Embedding Goal]:
24 + Jy W9 + by
Predict Actions:
TransformerXL. .., I(+T) - a++r
In order to finetune VPT to condition goals, finetune the model using method inspired by supervised RL approaches like Decision Transformer [12], GLAMOR [42], and GCSL [18]. We use modification of hindsight relabeling which call packed hindsight relabeling (see Figure 2) to generate a new dataset of trajectories with goals pulled from future states that periodically switch. In contrast with hindsight relabeling, packed hindsight relabeling packs multiple relabeled sequences into a single sequence. Specifically, our method to generate this dataset consists of two steps:
1. Given trajectory 7 with T timesteps, randomly generate indices select goals from: i,, i 4n- These indices are chosen by starting at the first timestep and repeatedly sampling
timestep by adding random value the previous timestep. This ensures that the data reflects that some goals may take longer to achieve than others.
2. For each chosen goal at timestep i,, set the goals for timesteps iý-1 + 1,..., to be the goal at timestep ij. denoted दT.

---

Our final dataset rellabeled consists of observation sequences .e....,,r) action sequences (Q)....,ar]) and packed hindsight relabeled goals (21...., 2T). We then finetune VPT on this dataset using supervised loss to predict each action autoregressively using causal attention mask:
palicy (0) = Da I log Lo 21...)]
(3.2)
3.2 Prior
In order to condition not only embeddings of visual goals but on latent goals, need the prior, model that produces a latent variable eea conditioned on a text instruction y. Our model is simple conditional variational autoencoder (CVAE) [54, 29] with Gaussian prior and a Gaussian posterior. Rather than learn to condition directly on text, we choose to condition on frozen text representations from MineCLIP Zy- Thus, the prior learns function to translate from text embedding 2y visual embedding 2 (see Appendix C.5). Both the encoder and decoder of our CVAE are parameterized as two-layer MLPs with 512 hidden units and layer normalization [4]. We train the model our dataset, for which we have text labels Diabels using the following loss:
p (() = B (a (30) Dracck Kl.(go(2 (zy) (()))((()))))))))))))){))))))[ - (a logPo(( Pe (c, ]n)]
(3.3)
3.3
Datasets
To train our policy, we gather a gameplay dataset with 54M frames (++ month at 20FPS) of Minecraft gameplay along with associated actions from two sources: contractor gameplay and VPT-generated gameplay. To train our prior, we use a small dataset of text-video pairs gathered by humans and augmented using the OpenAl API Ept-3.5-ture model [41] and MineCLIP. See Appendix D for more detailed dataset information.
OpenAI Contractor Dataset We use 39M frames sourced from the contractor dataset which VPT [5] used to train its inverse dynamics model and finetune its policy. The dataset was gathered by hiring human contractors play Minecraft and complete tasks such as house building or obtaining diamond pickaxe. During gameplay, keypresses and mouse movements are recorded. We use the same preprocessing as VPT, including filtering null actions.
VPT-Generated Dataset We generate an additional dataset of 15M frames by generating random trajectories using the various pretrained VPT agents. The diversity of this dataset is improved by randomly switching between models during trajectories [44], randomly resetting the agent's 's memory, and randomly turning the agent to face a new direction.
Text- Video Pair Dataset To train our prior model, which learns a mapping between text embed- dings and visual embeddings. we also manually gather a small dataset of 2,000 text instructions paired with 16-frame video segments (less than second) from our gameplay dataset. This dataset corresponds less than 30 minutes of gameplay and takes just a few hours to collect. We aug- ment this dataset by using the alignment between text and video embeddings from MineCLIP. For each text instruction, we find the top & most similar gameplay segments our dataset and use the corresponding 16-frame segment as additional training data. For augmentation, we also add 8,000 text-instructions generated by the OpenAl API gpt-3. 5-tubo model [41], in addition to our 2,000 hand-labeled instructions.
3.4 Inference
At inference time, we use the prior to sample a latent goal Σrpu from the text instruction y. We then use the policy to autoregressively sample actions conditioned on the observation history o1... and the latent goal Σ Similar to the observation in Appendix I of Baker al. [5], even with conditioning the policy often fails to follow its instruction and simply acts according to its prior behavior. To mitigate this, we borrow another trick used image generation models: classifier-fres guidance. Specifically, during inference we simultaneously compute logits for the policy conditioned on the goal [Or,,... Oc+1, 2=pmn) and for the unconditional policy (o,,,..., o++1 We then compute combination of the two logits using À parameter to trade-off between the two:
5

---

logits (1 + A) Solor...... ,,,,,, 1,+1; 2r 2rgu)
...;
So(on..... 0t+11
(3.4)
conditional logits
unconditional logits
By setting higher value of X, we can encourage the policy to follow actions that are more likely when conditioned on the goal and, as demonstrated in Section 4.5, this significantly improves performance. Also, in order to train the policy to generate these unconditional logits, we occasionally dropout the goal embedding Zrgul from the policy's ss input (with probability 0.1). This lets generate both the conditional and unconditional logits using the same model with batch processing inference time.
3.5 Evaluation
Evaluating the performance of our agent is challenging task due to the wide variety of instructions that are possible and the difficulty of evaluating whether the agent has successfully achieved its task. We use a combination of programmatic evaluation metrics and automatic MineCLIP evaluation metrics to get a sense of the agent's s capability level. We collectively refer all our evaluation tasks including the 11 evaluation tasks from Figure and the two prompt chaining tasks from Section 4.3 as our early-game evaluation suite.
Programmatic Evaluation We compute programmatic evaluation metrics by monitoring the MineRL. [19] environment state throughout each evaluation episode, As done VPT [5], we compute multiple programmatic metrics including travel distance and early-game item collection. The travel distance is the maximum displacement of the agent along on the horizontal (X-Z) plane, measured from the initial spawn point. For aall-game inventory counts, we store the maximum number of log, seed, and dirt items seen in the agent's 's inventory during the episode.
MineCLIP Evaluation We explore the of tixtalisual alignment in MineCLIP latent space between trajectories and text or visual goals to evaluate our agent over a wider variety tasks where programmatic evaluation isn't practical. To determine the degree to which a task has been completed at all during an evaluation episode, we record the minimum cosine distance between the (text or visual) goal embedding and the visual MineCLIP embedding at any timestep during episode.
4 Results
In our experiments, we aim to answer the following questions:
1. How well does STEVE-1 perform at achieving both text and visual goals in Minecraft?
2. How does our method scale with more data?
3. What choices are important for the performance of our method?
4.1 Training Setup
We base our implementation off of the official VPT codebase The main STEVE-1 I is trained using Pytorch [45] distributed data parallel on four A40 GPUs for 160M frames, or just under three epochs of our gameplay dataset. Hyperparameters exception of learning rate, which we set to 4c-5. Our models are optimized using Adam [37]. See Table 1 for a full list of hyperparameters.
selected to match those in Baker et al. [5] with the
4.2 Performance on Textual and Visual Goals
Due to the hierarchical nature of our model, we can evaluate the performance of our agent at achieving either text visual goals simply by choosing whether use the prior condition text bypass the prior and condition on a MineCI.IP video embedding directly. We first tested our model on a set of 11 tasks that are achievable within the first 2.5 minutes of gameplay and which do not require multiple steps to complete (c.g., chop tree or dig hole, but not build a house). A complete list of the tasks and prompts we used for evaluation can be found in Table 3 in the appendix. To select visual goals for testing each of the evaluation tasks, we implemented a tool that searches through

---

Conditioned Prompt
undorwated
Seeda Cailasted
lnaves
wrand
steeda
flree
esplure
switt
inversory
aky
2.5
5.0
Evaluation Prompt
sky
leaves
0.10
(Ted)
0.08
Calleted
Dhe Dellettr
Jower
0.06
explore
swits
0.04
underwater
Lnvestory
(leay
(1hml)
(ee)) (Nhrad)
(a) Programmatic Evaluation
(b) MineCLIP Evaluation
Figure Left: our programmatic evaluations, STEVE-1 performed far better than the unconditional VPT agent aarly-game 2x and the text-conditions VPT agent when prompted appropriately. The asterisk * in the "VPT (Text)*" indicates that this result was taken from Appendix [5], which had twice the episode length compared our setting. On some tasks, visual outperforms text-based prompting, creating gap that can likely be bridged through better prompt engineering, Right: Across our 11 MineCLIP evaluation tasks, STEVE-1 achieves the shortest distance between the episode and the MineCLIP goal embedding when prompted appropriately except for in two cases, where it mixes up digging and dirt and swimming and going underwater. This shows the strong general performance minimum element along the and the diagonal number signifies the diagonal element's rank (0 means is the minimum row element). See Figure 14 for sample frames from each of the 11 visual goals and Figure 13 for success-rate version of this matrix.
STEVE- across a wide variety of short-horizon tasks. The dashed box marks
10% of our gameplay dataset by finding the closest 16-frame videos to a given text prompt. We then manually selected 16-frame video that clearly demonstrates the task being completed and use the corresponding MineCLIP video embedding as the goal embedding for that task. Screenshots of these visual goals can be found in Figure 14 in the appendix.
In Figure 3, we compare the performance of our text and cisual-conditioned agents with the uncon- ditional VPT agent and text-conditioned VPT agent (Appendix I in [5]) across our programmatic tasks. We find that when given the relevant text instruction, STEVE-1 collects 75x more dirt, 4.9x more wood, 22x more seeds, and travels 4.3x farther than the unconditional agent, and STEVE-1 collects 3.3x more dirt, 4.4x more wood, 8.1x more seeds, and travels 2.2 farther than the text- conditioned VPT agent. This represents significant improvement over the reported performance of coxt-conditioned VPT, which collects several times fewer resources despite having twice as long of episode to do so. We also by measuring the minimum distance of a goal embedding to any frame in the episode. As shown in Figure 3b, the distance between the goal and the episode is significantly lower when the agent is conditioned on the corresponding visual goal than otherwise. Full results for STEVE-1 with both text and visual goals can be found in Appendix F.
automatic evaluation using MineCLIP embedding distances
addition to our evaluations of STEVE-1, we also recorded several sample interactive sessions we had with the agent (controlling in real-time by giving it written text instructions specific visual goals). These sessions demonstrate STEVE- 1's ability to responsively follow instructions in real-time in variety of situations. We believe that such use-cases, where humans give agent natural instructions that it can follow to complete tasks, will become increasingly important and have practical uses the creation of instructable assistants and virtual-worke characters. These videos, as well videos of our agent performing our evaluation tasks, can found at https://sites..oontt wot.view/steeee.l......
4.3 Prompt Chaining
We also experiment with longer horizon tasks that require multiple steps, such as crafting and building. explore two different prompting methods: directly prompting with the target goal, and simple form of prompt chaining [11, 64, 16] where the task is decomposed into several subtasks and the

---

Duild Tower
Height Porision
Make Wooden Planks
Inventury
Inventoryy
in Inventory
Thned Dakuzee (23vda)
Seeds Colleced
200
1999
SXB
2000 Epcoodo Tineetop
Epssol Timertep
1.3
Wisden Enps Calleted
Daald Tooee Shoo0e
1.5
Prongt Clamin
Eirett Proseting
Wo-sbm Plans
Dir Callend
2.5
24
Caittigg
0.2
14
692
1003
2051
50.1
1000
O00S 100M
31M
5007
Nember
Frrmes
Nomber Trarzes
Figure 4: Top left: By sequentially chaining visual prompts like "get dirt" and "build tower", STEVE-I successfully gathers dirt and then uses this dirt to build a tower. The prompts switch at the dotted vertical line. Bottom left: The success rates of the chained prompts improve steadily train STEVE-1 I on more data. Right: The performance of STEVE-1 in different tasks scales in different ways when conditioning prompts (e.g., the break wood prompt is the relevant prompt for the "Wooden Logs Collected" metric, while the other prompts "irrelevant"). For instance, in the wood-collection and dirt-collection tasks, performance starts increasing after training on 60M frames of gameplay. See Figure 14 for sample frames from each visual prompt.
relevant visual prompt for the metric versus other irrelevant visual
prompts are given sequentially for a fixed number of steps. We explore prompt chaining with visual goals for two tasks: 1) building a tower and 2) making wooden planks. When using prompt chaining, we first prompt STEVE-1 to gather dirt before building a tower, and to gather wooden logs before crafting wooden planks. Figure 4 shows that directly prompting STEVE-1 with the final tasks results in near-zero success rates. However, prompt chaining allows STEVE-1 to build tower 50% of the time and craft wooden planks 70% of the time. For the tower building task, STEVE-1 immediately starts collecting dirt until the prompt switches, at which point its average height starts increasing rapidly and its dirt decreases as it builds a tower. Similarly, for the crafting wooden planks task, STEVE-1 immediately starts collecting large amount of wooden logs until the prompt switches and it rapidly converts these wooden logs into wooden planks (causing the amount of wooden logs in its inventory to immediately decrease and the number of wooden planks to increase as crafts more). Figure visualizes the average item counts and agent height for the prompt chaining episodes. See Figure 18 and Figure 19 in the appendix for visualizations of specific prompt chaining episodes.
4.4 Scaling
Recent works in language modeling have found that scaling up pretraining FLOPs, by training on more data or by training model with more parameters, can improve performance on downstream tasks [28, 57, 63]. In certain cases when measuring performance with metrics such as exact-match [52], performance improvement may appear be "emergent" [63], appearing suddenly as the model is trained with more compute. Here, we aim to gain a basic understanding of how the performance of STEVE-1 on various tasks scales by training with more data (learning rate schedule is chosen appropriately).
To assess performance gain, we first isolated the performance of the policy from the prior, measuring performance of the agent through training on programmatic tasks (travel distance, seeds, logs, dirt) with visual goals. Due to compute constraints, we chose use the 2x VPT model, which has 248M parameters. We found that both seed collection and travel distance did not improve significantly past 20M frames. From inspecting gameplay, we suspect that travel distance is relatively easy task since it is close to VPT's default behavior of running around and exploring. For seed collection, performance remains suboptimal, suggesting that further scaling may be beneficial. This hypothesis is supported by the observation that performance on log and dirt collection remained roughly level until 60M frames when it began to rapidly improve. Figure 4 shows the scaling curves for STEVE-1 on each programmatic task when conditioning on relevant vs. irrelevant visual prompts for that task. Since we do not observe regression on any tasks as we train the model with more compute, we expect the model to continue perform better as we train larger models on larger datasets.
8

---

We also evaluated the scaling properties of STEVE-1 I for our multi-step tasks with and without prompt chaining. Without prompt chaining, the tasks remain challenging for STEVE-1 throughout training. However, we note that after 60M frames, STEVE-1 learns to gather wooden logs and build a small tower when told to build a tower. This is likely because our visual prompt for tower building shows a video of tower being built out of wooden logs. With prompt chaining, the performance of STEVE-1 steadily increases with more data. We conjecture that this is because the success of a chained prompt requires the success of each element in the chain. Since different abilities emerge at different scales, one would expect chained prompts to steadily get more reliable as these subgoals become more reliably completed. In the case of building wooden planks, we note that crafting is one such task that gets significantly more reliable as the agent is trained on more data. Figure 4 shows the scaling curves for STEVE-1 on the prompt chaining tasks.
In summary, we see evidence of tasks that do not require much data for STEVE- 1 to learn, tasks that steadily get more reliable as the agent is trained longer, and tasks where capability suddenly spikes after the agent reaches some threshold. Put together, this suggests that further scaling would likely significantly improve the agent, although we leave the task of predicting exactly how much performance there is to gain to future studies.
4.5 What Matters for Downstream Performance?
Pretraining Baker et al. [5] finds that by pretraining a behavioral prior with imitation learning on internet-scale datasets for Minecraft, the learned policy can be effectively finetuned to accomplish tasks that are impossible without pretraining. In this section, we demonstrate that pretraining is also massively beneficial for instruction- tuning in Minecraft. We hypothesize that due to the strong performance of STEVE-1 1 and the relatively small amount of compute (=s 1% additional compute) used for instruction finetuning, most of the capabilities of our agent come from the pretraining rather than the finetuning. To test this hypothesis, scc-early-game-- 2x, rl-from- ferrmmffounte coandttion-22e and with
finetune several varients of STEVE- I from various
pretrained weights: foundation-25 randomly initialized weights. In this experiment, each model was finetuned on 100M frames.
Figure shows the performance of these models on our programmatic tasks with visual goals. Note that while agent trained our dataset from scratch can accomplish basic tasks like dirt collection fairly well, it is unable to find and chop down trees, in contrast to the pretrained agents. This demon- strates that the abilities present in the agent due to pretraining tuned agent. Out of all the pretrained weights we tried, we noticed that r1 i--from-ff cerrom-founnat.. condationn- performed the best, having qualitatively better performance at tasks like crafting and chopping down trees. Indeed, Figure 5 shows that this model has strong performance, likely due to the massive amount of compute it was trained with during its RL training [5].
successfully transferred to the fine-
Classifier-Free Guidance Baker et al. [5] observed that when conditioning the agent on text, it tended to ignore its instruction and instead perform the prior behavior learned during pretraining. As discussed in section 3.4, classifier-free guidance gives a knob for trading off between goal-conditioned and prior behaviors. Figure shows the effect of this parameter À on the log and dirt collection tasks. The performance of the agent reaches its maximum around X = 5.0 to X = 7.0, after which it starts to drop off. These results demonstrate the importance of classifier-frs guidance, which improves the performance of STEVE-1 1 by orders of magnitude.
Prompt Engineering Prompt engineering as discipline has rapidly emerged over the last year due the observation that the quality of the output of text-to-X models can dramatically change depending on the prompt [67]. For example, Table 5 in the appendix shows how prompt for Stable Diffusion [50] might be written. By listing out the various attributes of the image such as visual medium, style, and the phrase "trending on ArtStation", the user is able to get higher quality image [20, 36). In this section, we explore how this same style of prompt engineering can improve the performance of STEVE-1. Figure 6 shows how simple prompt of "get dirt" might be changed in order to more accurately specify the type of behavior that desired. Just like image generation models, the performance of STEVE-1 significantly improves by modifying the prompt in this fashion. By changing to more complicated prompts, STEVE-1 1 is able to collect 1.6x more wood, 2x more dirt, and 3.3x more seeds.

---

Wooden Logs Collected
Wooden Logs Collected
75
30
Det Collentl
Dirt Collected
10
X0
scratci
bc
d
Pretzained
Wrights
Conditional Seale A
Figure 5: Left: We trained STEVE-1 on 100M frames starting from four different pretrained weights: random initialization (scratch), foundation-2s (fd), bc-early-game-22 (bc), and Exrromfoonddatonn
2x agent is generally the most per- formant after fine-tuning. Using pretrained weights performs better than training from scratch, especially for more complicated tasks like collecting wood. Right: By using classifier-free guidance [23], STEVE-1 collects 7.5x more dirt and 15x more wood than when = (no guidance). See Figure 17 in the Appendix for similar results with other programmatic tasks.
(rl). The 1l-ffrom-fonndatin
Prompt
Dirt Collected
"break flower"
0.7 (-0.2, 1.6)
"collect seeds"
2.7 4.59,
"dig as far as possible"
3.9 (2.8, 5.0)
" "get dirt"
9.2 (5.7, 12.7)
"get dirt, dig hole, dig dirt, gather a ton of dirt, collect dirt"
26.7 (19.9, 33.5)
Figure 6: Similar to in image generation, switching to longer. more specific prompt dramatically improves the performance of STEVE-1, Values in parentheses are 95% confidence intervals.
5 Limitations and Conclusion
In this paper, we present a methodology for creating instruction-follooing foundation models of behavior. Specifically, by leveraging two existing pretrained foundation models: a behavioral prior (VPT [5]) and domain-specific CLIP model (MineCLIP [17]). we create a powerful Minecraft agent that follow short-horizon open-ended text and visual instructions, all for only $60 of compute. The resulting foundation model, STEVE- 1, sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines and robustly completing 12 of 13 tasks in our early- evaluation suite. We note that generalist agents such STEVE-1 can have potential negative effects society. We include thorough discussion of these issues in Appendix A.
STEVE-1 1 is significant advancement in creating generative models of text-to-bchavion but it has several limitations, as described in Appendix B. First, STEVE-1 is mostly proficient at achieving short-horizen tasks while struggling with longer-horizon tasks. While prompt chaining promising approach for improving performance on complex tasks, more can be done in future work to improve performance. Another limitation observe is that prompt engineering, with other generative models, can be unintuitive and time-consuming. Future work should investigate improving the steerability of STEVE-1 through better understanding of natural language prompts. Additionally, we note that evaluating and describing the capabilities of open-ended generalist agents is an open research problem itself since capability depends strongly preconditions, prompt engineering, and our own ability to come up with varied and challenging tasks. Finally, since our approach is not specific the Minecraft domain, we hope that the method used to create STEVE-1 can inspire future work in creating powerful generalist agents in other domains and environments.
10

---

Acknowledgements
All of the authors gratefully acknowledge funding for this research from the Natural Sciences and Engineering Research Council Canada (NSERC) and the Canada CIFAR AI Chairs Program (Vector Institute for Artificial Intelligence). SL is supported by Vector Institute internship and by an NSERC Discovery Grant. KP is supported by an NSERC PGS-D award. HC is supported by an NSERC CGS-D award. JB acknowledges funding from the Canada CIFAR AI Chairs program, Fujitsu Japan, and an Amazon Research Award. In addition to NSERC and CIFAR (Vector Institute). SM acknowledges funding from Microsoft Research. We thank Silviu Pitis, Romi Lifshitz, Forest Yang. and Yongchao Zhou for their helpful comments; Alisa Wu and Ziming Chen for their contributions to the text-video pair dataset; and Finn Paster for the logo and graphic for the website. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute for Artificial Intelligence (www. pectorinstiio tute. ai ai/partners).
References
[1] Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse H. Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matthew Sharifi, Neil Zeghidour, and Christian Havnø Frank. Musiclm: Generating music from text. CoRR, 0bs/2301.11333 2023. doi: 10.48550/rXX..2211.1333 URL https://doi. org/10 48550// arXiv 2301 11325.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:
23716-23736, 2022.
[3] Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017 Long Beach, CA, USA, pages 5048-5055 2017. URL https: //prceediigg/ neurips cc/ paper / 2017 /hw.h/ahhhaaaaaa 453
(ate:775--bbbtrrccco Abstract hhtttrrrct...............
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607.0660 2016.
[5] Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-246544 2022.
[6] David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does cenuttccondttooeed supervised learning work for offline reinforcement learning? arXiv preprint arXiv:2206.00 01079, 2022.
[7] Authony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:22 2022
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https: httprc/ee/o/cec.eeeeeeee/ec wwweeediiiinnnnnnniieeeeeeeeeeee wwturis..ccppaaaee f wwwes/ppppeeeeeeeeeeeeeeeeeeeeee /ppper/2020/fille/ Paper. pdf..
11

---

[9] Shaofei Cai, Zihao Wang, Xiaojian Ma. Anji Liu, and Yitao Liang. Open- multi-task control through goal-aware representation learning and adaptive horizon prediction. arXiv preprint arXiv:2301. 10034, 2023.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference Computer Vision (ICCV), 2021.
[11] Harrison Chase. Langchain, 2022. URL htttpp://gi.uuub.cmm//tt//ttttie
[12] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097,
2021.
[13] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023, URL
https://Imsy:: wttgblgg////o/oooooooootooogoog/ 2023 11--00333--immm-
[14] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR, 0bs/2204.00211 2022. doi: 10.48550/arXiv.2205 02311. URL 2204 02311.
https:/ //dii. org/10 10. 00..4455///r/
[15] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhut- dinov. Transformer-xi: Attentive language models beyond a fixed-length context. In Anna Korhonen, David R. Traum, and Lluís Mårquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2019, Volume 1: Long Papers, pages 2019. doi: 10.18653/v 1/p19- 1285. URL https: //woi//oooo.o/ org/10/ /00 18653/v1/p19- 1285.
Association for Computational Linguistics,
[16] David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-dickstein. Kevin Murphy, and Charles Sutton. Language model cascades, 2022.
[17] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Build- ing open-ended embodied agents with internet-scale knowledge. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neu- ral Information Processing Systems, volume 35, pages 18343-18361 Curran Associates, Inc., 2022. URL 7 11557788ccc55511FF44888483333333 aappaaacssapaaa--aaaaaaaaaaaaaaa
newrine.u..p.... cc/paper_ppppe http//ppperels____/aaaaaaaaaeeee
Datasets aat_aandddd_aeeeeeeaeeeeaaaaaa pdf.
[18) Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations, 2021. URL https: /ttpps///opnnrevier. net/forummidd<AAAAXXGG.
[19] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXhn:1907.134400 2019.
12

---

[20] Gustavosta. Stable-difftusoo--prmppss Hugging Face, 2023. URL http:://uugiiggaaae Hugging Face Datasets.
co/ (aatasocss/ussaaaootttt/taaaaal wwt.ll--Dffuuuioorrooooe
[21] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04100 2023.
[22] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 20/6 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016. doi: 10.1 109///99.22010.0009 URL https: :ttppp://doo. org/oo////// 10. www1/111.9..... 1011110/////PPP20006600
[23] Jonathan Ho and Tim Salimans. Classifier-free diffuusion guidance. CoRR, abs/2207. 2022. doi: 10. 21.48550/rxx...22...222
URL httphtsss////o...rgee org/ 400.0.885550rrraa. 2207 12598.
[24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Anmual Con- ference Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020. URL https::/ppooceee//ee
neurine httpsscccppppe///eeeeeeeeee papprr/020//aaah (cbb.feeBB88af000011111111111123 www.::11779404--bbbaabeab Abstratt........................ httml.
[25] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:22 10.03094, 2022.
[26] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In Ijcai, pages 4246 4247,, 2016.
[27] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. CoRR, abs/2305.02463, 2023. doi: (0.48550/a2Xi..2305.2222 URL.htps:://aon wtt/oo...//00..000
arXiv 2305. 2 02463.
[28] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001 108361, 2020. URL htLthtss//arxxii www.a/s..r.r///////////oo/bo hatgg//b//00000 08361.
[29] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL bctp://arxta www.aiiww./aaaa.eeeeeeeeeoooeeee htgg/abs/1311.....
[30] Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline g-learning on diverse multi-task data both scales and generalizes. In The Eleventh International Conference on Learning Representations, 2023. URL k7kUavj..
net/forum?
[31] Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12004-12019, PMLR, 17-23 Jul 2022. URL https: html.
//proceedings.c mlr. press/ https//v111111nna/e
[32] Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama. Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski. et al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35:27921-27991 2022.
[33] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: automatic evaluator of instruction-folloviin models. https: httpss:///ttttbtee wwt.://tttootttoooeeeeeeeott alpaca_eval, 2023.
13

---

[34] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga. Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksekgönül,, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang. Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang. Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. CoRR, doi: 10.48550/ARXIV.2211 1.09110...
abs/2211.099 2022.
URL https://doot org/10.
18550/arXiv
[35] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng. Xun Huang. Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[36] Vivian Liu and Lydia B. Chilton. Design guidelines for prompt engineering text-to-image to-image generative models. In Simone D. J. Barbosa, Cliff Lampe, Caroline Appert, David A. Shamma, Steven Mark Drucker, Julie R. Williamson, and Koji Yatani, editors, CHI '22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 May 2022, pages 1-8:4-3 ACM, 2022. doi: 10.1145/349100225501 825. URL https://doi... org/
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL. http:p/ss...paaaeeeeeeeeee/eeeer
[38] Federico Malato, Florian Leopold, Amogh Raut, Ville Hautamäki, and Andrew Melnik. Be- havioral cloning via search in video pretraining latent space. arXiv preprint arXiv:2212.13326, 2022.
[39] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. /2601, 2022.
[40] Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannanch Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep?: Embodied decision making using language guided world modelling. arXiv preprint 200 2023.
[41] OpenAl. Introducing ChatGPT, Nov 2022. URL.
www.blog///c//cccccccooooooooooo chatgpt. [42] Keiran Paster, Sheila A Mcllraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv preprint arXiv:2012.024199 2020.
[43] Keiran Paster, Sheila A. Mcllraith, and Jimmy Ba. You can't count on luck: Why decision transformers and rvs fail in stochastic environments. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: //tpa:///oppeeeee /pwp//veeeeeeeeeeeeee/ee/eeeeeee wwt. net/forummidd-ttbyiii
[44] Keiran Paster, Silviu Pitis, Sheila A. Mcllraith, and Jimmy Ba. Return augmentation gives supervised RL temporal compositionality. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. URL htttttt:////oeeeeeteee
net.
[45] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library.
Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Ahh- Buc, Emily B. Fox, and Roman Gamett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8034-8035, 2019. URL https: http///proceen...e//////// neur wwturpps.ccpppaee cccppape//0111 9/ashh/ bobba2288feee772222f440777020072 www.2//77777777---0------ Abstract.
14

---

[46] PrismarineJS and Others. Mineflayer. https://github. http://wwww.........aaaaa/aaaaaa
2023. URL https: /tttps://githb.. http://gttuuuu..co..//aaraaaaa comf//rtaaaaarnneB//iiieeeaaa.. GitHub repository.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning,
pages 8748-8763. PMLR, 2021.
[48] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204006/25, 2022.
[49] Scott Reed, Konrad Zoina, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint wwr.vx:220500117. 2022.
[50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[51] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Pho- torealistic texi-to-image diffusion models with deep language understanding. In NeurlPS, 2022. URL http://papers wwtpeerrrsiiiss....e.eeeeeee..es nips. cw/paccr_aaaeeeeeeeeeeeeeee files salesppane (aaper/202/hhan 7778aaaaasb00000raacccotoooaaaa Abstract- Conference html.
[52] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? CoRR, abs/2304. 15004, 2023. doi: 10.483500/nxiv..3344.5555 150044 URL
https://doi.. http:///oooooo//../// 48550/ aariv..xiivvv.... 2304. 15004.
[53] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pages 994-906, PMLR, 2022.
[54] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output rep- resentation using deep conditional generative models. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference
Neural Infor- mation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 3483-3491, 2015. URL https: http///eeeeeeee..///////// caf52296ebaa5006772277222023 Aat.oedaaaa2tAA--Aaaataae Abstract. .ttml.
www.eediinnnnuuiip.....eee neurips courips.cc/paperr. httpppppppp//////
[55] Miroslav Strupl, Francesco Faccio, Dylan R. Ashley, Jürgen Schmidhuber, and Rupesh Kumar Srivastava. Upside-down reinforcement learning can diverge in stochastic environments with episodic resets. CoRR, abs/2205.0633 2022. doi: 10.448550/arxiv.225555 06595. URL https: / //doi.org/10 40554/550//// arivv. 0006006555.
[56] Theodore Sumers, Kenneth Marino, Arun Ahuja, Rob Fergus, and Ishita Dasgupta. Distilling internet-scale vision-language models into embodied agents, arXiv preprint 2:X1v:2301.12507, 2023.
[57] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challeng- ing big-bench tasks and whether cfain-of-thought can solve them. CoRR, abs/2210.09261, 2022. doi: 10.48550//rXiv222100 09261. URL https: org/ 10. 48550/arXiii
09261.
[58] Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follooing llama model. https://ghs thub. com tatsualasaaaaa//aa staabb/staaoooo ord alpaca, 2023.
[59] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, a39/230, 2023. doi: 10.48550/arxivv2302... 23000 https: // //oi. erg/10.cr 48550/ arxiv 2w0.. 13971.
URI
15

---

[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrkee von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017. URL / http://pooceeeeeeeeeeeco neurips httpp/pppp///////p//p (0lee245544de911ffoo0occlcclaaa aateredbaaa-bbtaatttte Awstract.httm
[61] Ryan Volum, Sudha Rao, Michael Xu, Gabriel DesGarennes, Chris Brockett, Benjamin Van Durme, Olivia Deng, Akanksha Malhotra, and Bill Dolan. Craft an iron sword: Dy- namically generating interactive game characters by prompting large language models tuned on code. In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Word- play 2022), pages 25-43, Seattle, United States, July 2022, Association for Computational Linguistics. doi: 10. 00.18653/11/202.. wwt.:22wwwwdpdlyl..... URL /tt.ttts///cllattoooo wordplay-1.3.
org/2022.
[62] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01566 202001560, 2023.
[63] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. CoRR, abss2200.007682 2022. doi: 10448550krXx..2226007622 URL nttpps///oi: 4rg/00 arviv..iivvv.... 2206 07682.
[64] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL btctps://openreeyee net/ forum?id=_ forum?id=_VjllMeet VJQMMMBB_M....
[65] Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint arXiv:2211. 2022.
[66] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging Ilm-as- alm-as-a-judge with mnthbench and chatbot arena, 2023.
[67] Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2023. URL https: http///ppnre....../// httt/forumiid-22gkkk00
16

---

A
Broader Impact
With the increasing capability level of artificial intelligence comes many potential benefits and also risks. On the positive side, we anticipate that the techniques that used to create STEVE-1 could be applied to the creation of helpful agents in other sequential decision making domains, including robotics, video games, and the web. Our demonstration of such low cost approach to creating Al.
powerful, instruction foolowwwg model also has the potential to improve the democratization However, the negative side, agents pretrained on large internet datasets reflect the biases of the internet and, as suggested by our experiments, these pretraining biases can potentially remain after instruction-tuningg If not addressed carefully, this could lead devastating consequences for society. We hope that while the stakes are low, works such as ours can improve access to safety research on cntructiionfollowwiie models in sequential decision-making domains.
B
Limitations and Future Work
B.1
Goal Misgeneralization
One of the most common mistakes that STEVE-1 makes during evaluation is to overgeneralize. For instance, if we prompt STEVE-1 with video of someone punching a cow, may simply run to the nearest animal and punch that animal instead. This is related to the concept of goal misgeneralization [31]. Generalization can be helpful when the task we assign the agent is impossible to achieve from the current state and the agent instead performs closely related action, but harmful when the task is achievable. We note two things: first, we believe the powerful generalization ability of STEVE-I I probably comes from the MineCLIP embeddings and it especially improves the ability of STEVE-1 to follow visual instructions when the exact items or blocks nearby are not available in the current environment, which is an extremely common scenario. Second, we notice that the tendency of the agent to misgeneralize decreases with scale. For example, with a model trained on less data, we find that asking the agent to look up and punch a tree get wooden log often resulted in the agent looking in the air and punching nothing: training the model on more data results in the agent first walking over nearby tree and looking up to get wooden log. Future work should look to measure more closely the relationship between misgeneralization and scale.
B.2 Random Selection of Hindsight Goals
In this work, we choose to randomly select future episode timesteps as hindsight goals, rather than more sophisticated strategy. This is primarily due to the simplicity of the approach, but also to ensure a diverse and unbiased coverage of potential goals achievable within the short horizon. Future works investigate the effects of alternative approaches that filter for semantically interesting timesteps as goals.
B.3 Difference in Performance Between Text and Visual Goals
In our experiments, we observed that STEVE-1 often performed better when conditioned with visual goals compared to text goals converted through the prior. There are several potential factors that could contribute to this performance gap between text and visual conditioning. First, the prior model may not be accurately capturing the full meaning of the text prompt. Training the prior on more data or using a more powerful model architecture could potentially improve the quality of the sampled latent goals. Second, the visual goals can provide more precise demonstrations of the desired behavior, Text goals are inherently more ambiguous. Providing additional information such as the observation context to the prior and further prompt-engineering to make text prompts more detailed and less ambiguous, could help close this gap. Because text conditioning provides more ficxibility and potential for generalization, closing the performance gap between text and visual conditioning is an important direction for future work.
B.4 Challenges in Evaluating STEVE-1
See Table 4 for a non-exhaustive list of tasks that the STEVE-1 agent is able to achieve. It is worth mentioning that evaluating and describing the capabilities of open-ended generalist agents is an open
17

---

research problem itself since capability depends strongly preconditions, prompt engineering, and our own ability come up with varied and challenging tasks. For example, there are many recent works on the evaluation of LLMS (e.g., [34, 33, 66]) which highlight these challenges.
That being said, there are number of tasks which STEVE-1 is unable accomplish. As previously mentioned, long-horizon tasks such as obtaining diamond or building house are currently beyond the capability of STEVE-I. 1. Further, STEVE-1 also struggles with more complex crafting tasks like crafting an enchanting table, bookcase, or boat. Again, the virtually limitless and open-ended nature of tasks in Minecraft makes it very difficult test generalist agents this domain. We hope future works develop more sophisticated methods to evaluate the performance of generalist agents on short and long- monghooioon tasks (potentially through an extension of our MineCLIP evaluation method).
is also worth noting that is currently not possible to test STEVE-1 1 or VPT [5] in the MineDDojo [17] environment, which is meant for generalist agent evaluation, since the action spaces not compatible. We believe that bridging this gap could be greatly beneficial to the generalist agent community and we hope future works investigate this further.
B.5 Towards Improved Long-Horizon Performance
STEVE-1 I is a significant advancement in creating generative models of text-to-behavior, but has several limitations. First, STEVE-1 is mostly proficient achieving short-horizon tasks while struggling on longer-horizon tasks like obtaining diamond. Solving long-horizon tasks while taking actions using low-level mouse/keyboard controls is very challenging and exciting research direction and, while prompt chaining is a promising approach for improving performance on complex tasks, more can be done in future work improve performance.
One potential bottleneck the fact that during packed hindsight relabeling, the hindsight goals limited to at most 200 timesteps in the future (~10 seconds). Thus, tasks which require more -i-t---diitiitttionn for STEVE- 1. Although sampling
than 200 timesteps to complete are technically hindsight goals from farther into the future could theoretically enhance long-horizon performance, our experiments in Appendix C.4 indicate that the performance tends to decrease if we increase much. We suspect that while increasing this hyperparameter may be able to improve long-horing horizhoizon performance, also increases noise and comes at the cost reducing performance short-horizon goals. Investigating whether it is possible to achieve better tradeoff is an important avenue for future work. We also suspect that the long-horizon capabilities of STEVE-1 could be improved through scaling or finetuning with reinforcement learning, or leveraging LLMs or VLMs to automatically provide prompt chains to the STEVE-1 agent.
this hyperparameter
B.6 Applying the STEVE-1 Approach to Other Domains
We designed STEVE-1 1 for Minecraft due the availability of two key ingredients: (1) strong behavioral prior (VPT [5]), and (2) a powerful visual-language model which maps text and video joint embedding space (MineCLIP [17]). However, the method used create STEVE-1 is not specific to the Minecraft domain. Given the rapid development of generative models, we expect that similar models VPT and MineCLIP will become available in many other domains. As these models become available, future work could investigate the applicability of the STEVE-I approach to these other domains.
C Additional Ablations
In this section, we describe additional ablations on design choices for our method, including the use of classifier-free guidance during training, text augmentation strategies, different VAE variants, and varying chunk sizes during finetuning. We use programmatic evaluation metries compare the performance of the various ablations.
C.1 Classifier-Free Guidance During Training
We examine the importance of using classifier-free guidance during training by finetuning a model with no guidance which does not drop out the goal embedding Zzg.ut from the policy's input (i.e., Puncond = 0.0) and comparing it to the version which uses guidance (Puncond = 0.1). The chunk
18

---

Twweel Distance (Blocks)
Woodon Logs Collected
300
Soods Collected
12.5
Diet Collected
10.0
200
80
100
2.5
guidane (-
guiteme
guidemo
gulhaze (A
no gogiteeeee
gudiaze (A
no guider
gulbaze (A
goidurrs 2
Figure 7: Ablation on Guidance. the "no guidance" variant, we set Puncond = 0, meaning that we do not drop any zyyou from the policy's input during training. The "guidance" variants set Puncond = 0.1, dropping 10% of the time during training. Whereas the "no guidance" model is only compatible with À 0 at inference, the "guidance" model can À >> 0, allowing for better performance.
size set to the range 15 to and we train each policy for 100M frames. In Figure 7, we compare the performance of using visual goals (MineCLIP video embedding) the no guidance model using conditional scale =
the guidance model using conditional scales observe that while the guidance model slightly outperforms the guidance model at À across few metrics, the agent with guidance outperforms the no guidance agent by a factor of 2 to times for the inventory collection tasks when increase the conditional scale to X = 3 (which we cannot do for the no guidance model). For the travel distance metric, both the guidance versions perform similarly to the guidance version.
0 and
3. We
C.2 Text Augmentation
During finetuning, instead of using only self-supervision with future MineCLIP video embedding the goal, considered using the text embeddings from the 2,000 human labeled trajectory segments as goal embeddings, either solely or in addition to the self-supervised video embeddings. In order to more fairly compare with the CVAE prior approach, data with additional text-gameplay pairs generated as described Appendix E.2. We implement this experiment by replacing the visual embeddings used for relabeling in Algorithm I with text embeddings, when available, with 90% probability. To experiment with not using visual embeddings observe that
augment the human-labell
all, can replace the visual embeddings with zeros in the same way. In Figure using only the visual embeddings during training, in combination with the CVAE, can outperform using MineCLIP text embeddings directly in the other two baselines. In this experiment, the chunk size is set to the range 15 to 50 and we train each policy for 100M frames.
C.3 VAE Variants
We study the dataset used to train the CVAE prior model. In Figure 9, we observe that augmentation helps in some programmatic tasks, including the dirt and seed collection tasks, but slightly hurts the wooden log collection and travel distance metries. In this experiment, each CVAE variant and we tune the conditional scale À for each variant. The chunk size is set to the range 15 to 200 and train the policy for 100M frames.
the same policy with
19

---

Thwee Distance (Blodes)
Wonden Logs Collected
300
Sonds Collected
Diet Callected
10
100
coun VAT)
Lod vieaad (ror too))
vasual (fece VAL)
lect (pow toa))
vimal (now toot)
viou (fod VAL)
bext (rowr bood)
tost vieual (now toxt)
(rau
vioud (roa
tos
Figure 8: Ablation on Text Augmentation. In the "text (raw text)" ablation, we train the model using only the text labels from human labelled trajectory segments, and directly use the MineCLIP text embedding of the text label as the goal embedding during training and at inference. For the "lext + visual (raw text)" ablation, text embedding from the human labelled trajectory segments during training and use the MineCLIP text embedding during inference. Even with augmentation, the dataset only contained around 2% text embeddings. The "visual (text VAE)" version is as reported in the main method, using the CVAE to convert MineCLIP text embedding to visual embedding during inference.
both the visual embedding self-supervised manner and the
Thael Distance (Blocke)
Wanden Logss Collected
15
30
Crilletted
Dirt Collected
10
Homan Ane
Hunaan
Huna
Human Aug
Tum
Humen Ang
Dumen
Figure 9: Ablation on VAE Training Data. "Human" baseline uses only the 2,000 human-labelled trajectory segments (text-video pairs), as training example for the CVAE prior model. "Human Aug" baseline adds additional pairs of text-video examples as described Section 3.3.
C.4 Chunk Size
During finetuning, we compare different goal chunk sizes by varying the max_ gax__twn__ooas=fii 200, ,00,,00,,
while keeping the min_ men_bttn__goassst gorithm for more details. A larger max bt__bttn________ goals introduces more noise, with actions that led to achieving the further away goal being less correlated to the actions present in that goal chunk. In Figure 10, observe that the best max max_btwa wtwp_wtwwallaomoo chunk size is around 200, and increasing the chunk size beyond that causes a drop in performance. We train each policy for 160M frames and tune the conditional scale for each.
See Al-
1500
20.5
Thuel Datate (Dlrde))
330
Woode Logt Calected
Soade Colected
160
Oollcted
230
7.5
12.3
935
I0C
130
7.8
1001
2001
300
831
503
2301
200
130
30C
331
2241
210
310
250
300
00
Clanse Raz
Claunk Saz
Chark Siar
Clanak Stor
Figure 10: Ablation on Segment Chunk Size. We vary the max. bes_bttn__gaallo parameter in Algo- rithm 1. The performance is roughly the best at around 200, beginning to decline with greater values.
20

---

200
1300
ascedl
Eantal
50
52
30
20
ESItal
Figure 11: Even without training the prior on the concept of dirt or digging all, STEVE- can still be instructed to dig holes and get dirt. This demonstrates that STEVE-1 can generalize to unseen text instructions.
C.5 Generalization to Novel Text Instructions
train the prior on a dataset of human and GPT-generated instructions designed to representative of the tasks that appear in our gameplay dataset. Here, we have performed set of simple generaliza- tion experiments to measure degree to which the prior can generalize to unseen instructions.
Instruction Training Set Contamination The "Text Prompt" column in Table shows which of the prompts used in evaluation show up in training dataset. Among evaluation instructions, the bolded and italicized instructions show in the instruction instruction-trajector trajectory dataset. While some instructions do show up, most of the instructions do not show up our training (verbatim).
Training Set Decontamination To measure the effect on performance removing a concept from the training set, we ran an experiment where removed every instruction with the words "dirt" or "dig" in them and retrained the VAE model. This corresponds to around 10% of the instructions. As shown in Figure 11, found that even without training the concept of dirt digging at all, STEVE-1 can still be instructed to dig holes and get dirt. This demonstrates clearly that STEVE-1 can generalize to unseen text instructions likely because most of the exx--understanding comes from the pretrained MineCLIP model which was trained on a highly diverse dataset of YouTube videos and captions. The prior VAE only needs learn mapping between the text and visual MineCLIP embeddings. Note that there is a slight decrease in performance across all tasks likely due to the smaller VAE training set (~ 10% less).
The instruction- followwng capability of STEVE-1 shared between: the policy, which learns follow instructions in the visual MineCLIP embedding space; the MineCLIP text-encoder, which is trained align well with the visual embeddings and performs most of the eex-understanding: and our prior VAE model, which learns simple function to translate between text and visual embeddings. This modeling setup lets us fully exploit pretrained models such as MineCLIP gain impressive language understanding without relying on having our own large datasets or compute.
D
Dataset Details
D.1
Gameplay Dataset
Our gameplay dataset consists of two types of episodes: 7,854 episodes (38.94M frames) of contractor dataset made available from Baker et al. [5] and 2,267 episodes (14.96M frames) of gameplay generated by running various pretrained VPT agents.
OpenAl Contractor Dataset The majority of our data comes from the contractor data used train VPT [5]. OpenAl released five subsets of contractor data: 6.x. 7.x, 8.x. 9.x, and 10.x. We use equal
21

---

mix of 8.x, 9.x, and 10.x, which correspond to "house building from scratch", "house building from random starting materials". and "obtain diamond pickaxe". Contractors were given anywhere from 10 to 20 minutes to accomplish these goals to the best of their abilities while their screen, mouse, and keyboard were recorded.
VPT-Generated Dataset We generated additional data by generating episodes using various pre- trained VPT agents. In order to increase the diversity of data as well as to get data of the agent switching tasks randomly throughout the middle of episodes, we added random switching between the different pretrained agents during episodes. Specifically, at the beginning of an episode we ran- domly sample two VPT agents from (foundation_ng) on_nooel_2m2 2x. bc_ re__arllygaa say-gaee_-- __x, pc_hhuse__ _3x, 2x. rl_from house_ __ax and switch between them at each timestep with a probability of 1/ 1/1000. Since the RL agents all act quite similarly, we avoid sampling two RL agents at once. Additionally, with a probability of 1 1/750 each timestep, we cause the agent to spin a random number of degrees. This adds more data where the agent spontaneously changes tasks, increasing downstream steerability.
D.2 Text-Video Pair Dataset
We gathered small dataset of 2,000 human-labelled trajectory segments (text-video pairs) by manually labeling gameplay from our datasets. We used a simple web app that presented a video of 16 frames the user from a randomly sampled episode. This only corresponds to 32,000 frames of labeled data, which corresponds to labeling 0.06% of the full dataset, or 27 minutes of labeled data. However, as discussed in Appendix E.2, combining this with automatically labeled data using gpt-3. gpt-3.5-turbe. 5-turbo and MineCLIP results in strong prior model.
D.3 Prompt Design
In experiments used both short and longer prompts. The short prompts either taken from previous literature (e.g.. the inguage-connitioning experiment in the VPT appendix [5]) or they were simply the first prompt tried. The longer prompts were created by taking inspiration from the prompt engineering methods used with text-to-image models such as Stable Diffusion [50]. To design these prompts, simply strung together lot of terms related to our task in order to increase the specificity of the prompts. We were excited to discover that this style of prompt design inspired by the prompt-engineerig community works well in STEVE-1.
E
Training Details
E.1 Policy Training
STEVE-1 was trained using distributed data parallel in PyTorch [45]. During training, segments of 640 timesteps were sampled from the dataset. Due to memory constraints, these segments were further broken up into chunks of 64, which are processed sequentially. Since VPT uses Transformer- [15], this sequential processing lets the policy attend to previous batches up the limit of its context length. We optimized the weights using Adam [37] with maximum learning rate of 4c-5 and a linear warmup for the first 10M frames followed by cosine learning rate decay schedule that decays to 10% the original learning rate. See Table 1 for an exhaustive list of hyperparameters used during training.
During training, we sample data using packed hindsight relabeling (Figure 2). This involves sampling segment of an episode, randomly selecting some timesteps at which to change goals, and then filling the corresponding goal embeddings for the entire episode with the embeddings from the corresponding goal segments. See Algorithm for detailed explanantion of packed hindsight relabelling.
F.2 Prior Training
The prior model is simple CVAE [54] that conditions on MineCLIP [17] text embeddings and models the conditional distribution of visual embeddings given the corresponding text embedding. This model is trained combination of around 2,000 hand-labeled trajectory segments and augmented
22

---

Hyperparameter Name
Value
trunc_t
64
T
640
Hyperparameter Name
Value
batch_size
architecture
MLP
wor_wor
4
hidden_dim
512
veight_decay
0.039428
latent_dim
512
n_frames
160M
hidden_layers
2
learning_rate
4e-5
batch_size
256
optimizer
AdamW [37]
learning_rate
le-4
varmup_frames frames
10M
8
0.001
p_ p_uncond
0.1
n_epochs
50
btn_bttn_goals
15
n_ n_search_episodes
2000
max max_btwn, max_btwn_goals
200
k
5
vpt_architecture
2x
offset
8
Table 1: Policy Hyperparameters
Table 2: Prior Hyperparameters
Algorithm 1: Sampling Episode Segments with Packed Hindsight Relabeling
Function sample_episode. seement (T; mnin_bwwn_eee max max_btwn_ htwpt_tthohhooooooooeee goals)
segment= -asmpeeeeeeeeeteppidddeee T)
curr hurr_timestep-2 segment.start
hon__wwitchingindicee []
while curr curr_tureee curr_timestep- segment.end do
curr_times curr_timestep += uniform(minn
batprrrmmmmmi_ttt___aa__aaa max_ max_btwn_goaah
goal_switching irapiinnnniiiiesddee
relabeled goal_ embeds
for n in range(1, lentgoal_switching
switching indices)) do
L relabeled goal_egaaallee waall_mmedd__niiit 1 segment.goal_ce segmon..ggoleembbddddglsss....)
return segment.obs, segment.actions
with additional data by automatically searching for text-gameplay pairs from our gameplay dataset. This is done using the following steps:
1. Combine the 2,000 text labels with 8,000 additional labels generated by querying gpt-3.5-ture t-rbbbb....
2. For each of these 10,000 text labels, search through 1,000 episodes sampled from the gameplay dataset find the top 5 closest visual MineCLIP embeddings to the text embedding of the text label.
These 50,000 automatically-minet text-video pairs are added the original 2,000 hand-labeled examples to form the final dataset used for prior training.
We noticed when prompting STEVE-1 using visual goals that when the visual goal showed the agent hitting a block but not following through and breaking it that STEVE-1 actually avoided breaking blocks. Unfortunately, many of the automatically discovered text-gameplay clips include gameplay of this kind. In order to prevent this issue, we added an offset to the embeddings found this manner. By selecting embeddings from timestep offset steps after the originally-selctte timestep. the agent is much more likely to follow through with breaking blocks.
We trained our prior model for 50 epochs this dataset and used early-stopping with small validation set. exhaustive list of hyperparameters used for creating the prior model can found at Table 2.
23

---

E.3
Training Costs
The $60 cost we report corresponds to the cost of renting 6xA10g node using spot instances on AWS for 12 hours using our instances prices in May 2023.
F
Additional Visualizations
F1
MineCLIP Evaluation
We ran MineCLIP evaluation on both text and visual prompts. The MineCLIP evaluation results can be found in Figure 12.
F.2 Steerability with Programmatic Metrics
Similar to Figure 20 in the VPT appendix [5], we plot the programmatic metric performances (mean and 95% confidence intervals) across the different goal prompt conditioning, both using visual prompts (Figure 15) and text prompts with CVAE prior (Figure 16) conditioning, on our policy trained with hyperparameters in Table and using conditional scaling and À = 6.0 (for text prompts with CVAE prior). Each conditioning variant is run with 10 trials, each trial with a different environmental seed and with an episode length of 3000 timesteps (2.5 minutes gameplay) Across the conditioning variant, we use the same set of environmental seeds. For comparison, we also plot the metrics for an unconditional VPT (early ggamm))) agent ("VPT (wncond)") and the (ext-conditioned agent investigated in VPT appendix [5] ("VPT (text)**) when conditioned on the relevant text. When using visual goal conditioning, we the use MineCLIP video encoder to embed 16-frame clip of the agent performing the desired task taken from our training dataset. An example frame from each of the visual goals is illustrated in Figure 14. When using text VAE goal conditioning, we use the MineCLIP text encoder to encode the text prompts (Table 3) and use the CVAE prior to sample the goal embedding from the MineCLIP text embedding.
(for visual prompts)
We note several differences in our experimental setup compared to that in VPT [5]. We only run our evaluation episodes for 3000 timesteps, equivalent to 2.5 minutes of gameplay, compared to 5 minutes in the VPT paper. Due limited computational budget, we generate 10 episodes per conditioning variant, and 110 episodes for the unconditional ("VPT (uncond)"), compared to VPT's 1000 episodes. Lastly, when measuring the inventory count, we log the maximum inventory count seen throughout the episode, which is lower bound on the potential number of items collected since the agent can later throw out, place, the "VPT (text)"" legend in Figure 15 and Figure 16 with an asterisk as we use the results reported in [5] directly for comparison.
these items craft. As result of these caveats, denote
We make several observations. First, we observe that our agents is more steerable: when conditioned to collect certain items (in bold), the agent collects (relatively) many more of those items than when conditioned on other instructions unrelated to that item, as well as compared to the unconditional VPT. When conditioned tasks unrelated to the item (e.g. break flower when interested in measuring logs collected), we also observe that the agent pursues that item less than the unconditional agent. Second, we observe that for the bolded instructions which expect to stand out, we outperform VPT performance (dashed blue line) [5], even with half the amount of time the episode rollout, This suggests that agent is both more steerable relative to the unconditioned VPT agent and the text-conditioned VPT agent investigated in the VPT appendix [5].
F33 Prompt Chaining Visualization
We visualize two specific episodes from the prompt chaining experiments in Section 4.3 in Figure 18 (building a tower) and Figure 19 (crafting wooden planks).
24

---

Conditioned Prompt
Conditioned Prompt
dig
etrt
alay
prosed
perteds
flwerr
esplore
awir
anderwator
Inventory
dia
dist
maaves
Irood
flover
explure
ewin
undormtert
Iavestory
dig
0.14
0.12
0.12
Evaluation Prompt
sky
sky
leeves
leavas:
0.10
wood
wood
soads
0.08
Evaluation
0.00
flowor
0.06
0.06
explore
exploze
swim
0.04
swim
underwater
underwator
inventory
0.02
inventory
(a) MineCLIP Text Evaluation
(b) MineCLIP Visual Evaluation
Figure 12: MineCLIP Evaluation. We measure the cosine distance between the goal embedding given to the agent and the MineCLIP video embeddings throughout the episode and record the minimum across the episode. Dashed box indicates the minimum along the row, and the number in the diagonal box indicates the rank of the diagonal element in the row (0 specifies that the diagonal row). The ideal performance would where the minimum values
the minimum element of each row lie on the diagonal. That is, the agent performs specific task best when it is asked to perform that specific task. Left: We use the prior to convert the text into the goal embedding. Across our 1l text MineCLIP evaluation tasks, STEVE-1 achieves the shortest distance between the episode and the MineCLIP goal embedding when prompted appropriately for most cases. This shows the strong general performance of STEVE-1 I across wide variety of short-horizon tasks, Right: We embed the visual goal loops (Figure 14) with MineCLIP video encoder. Across our 11 visual MineCLIP evaluation tasks, STEVE-1 achieves the shortest distance between the episode and the MineCLIP goal embedding when prompted appropriately except for two cases, where it mixes up digging and dirt and swimming and going underwater. This shows the strong general performance of STEVE-1 across a wide variety of short-horizon tasks.
Conditioned Prompt
underwater
inventory
leaves
wood
seeds
nower
esplore
swim
359
dirt
sky
dig
dirt
0.00
1.00
8.00
8.30
0.90
9.00
0.30
0.00
Evaluation Prompt
sky
0.00
8.00
1.00
0.50
0.0
0.00
9.10
0.10
0.00
0.00
0.50
0.8
leaves
0.00
6.00
9.50
0.70
8.60
0.30
9.00
0.20
0.10
0.10
0.00
0.00
0.00
0.00
0.10
8.00
0.00
0.00
0.6
seeds
flower
0.4
explore 0.80
swim 0.00
0.90
0:49
630
4.90
9,70
0.00
0.70
0.2
underwater 0.00
6:5
0.00
0.00
3.00
0.00
9.10
0.00
0.40
0.00
inventory 0.29
6.50
1.00
0.90
4.09
0.20
920
0.9
0.10
1.00
Figure 13: Visual Evaluation Success-Rare Matrix. We manually reviewed the same videos used for the MineCLIP Visual Evaluation matrix in Figure 3b order to verify that the MineCLIP scores correspond well to human judgment. Thus, the values in this matrix are subject to human error and subjectivity. Each cell value shows how often the agent achieves the Evaluation Prompt when conditioned on the Conditioned Prompt (success-rate). The dotted cell(s) is/are the maximum value in the row. Across the tasks, STEVE-1 achieves the highest success-rate when prompted appropriately except in three cases, where it breaks wood more than leaves, explores more than it swims, and swims more than it goes underwater. This shows the strong general performance of STEVE-1 across a wide variety of early-game short-horizon tasks.
25

---

dig
dirt
sky
leaves
wood
seeds
flower
explore
swim underwater inventory
Figure 14: Sample frames from each of the 11 visual goals. Note that the text overlaid on the frame is not present when we encode the 16-frame clip with the MineCLIP video encoder, and is only present for the figure visualization.
Shortened Name
Conditioning Variant Name
Text Prompt
dig
dig far as possible get dirt
dig as far possible
dirt
get dirt
sky
look at the sky
look the sky
leaves
break leaves
break leaves
wood
chop tree
chop tree
seeds
collect seeds
collect seeds
flower
break flower
break flower
explore
go explore
go explore
swim
go swimming
go swimming
underwater
go underwater
go underwater
inventory
open inventory
open inventory
dirt (engineered)
get dirt
get dirt, dig hole, dig dirt,
gather a of dirt, collect dirt
wood (engineered)
chop down the tree
chop down the tree, gather wood, pick up wood, chop down, break tree
seeds (engineered)
break tall grass...
break tall grass, break grass, collect seeds, punch the ground, run around circles getting seeds from bushes
Table 3: A summary of the different ways we refer to the 11 early-game evaluation task prompts. "Shortened Name" is the way we refer the prompts in any success-rate matrix MineCLIP matrix. This is also how we refer to the visual prompts which have no actual text. "Conditioning Variant " is the name used to refer to the "Text Prompts" in Figure 16, since not all text prompts fit in the figure. A "Conditioning Variant Name" with that does not in the figure. Also, reference to the experiments Appendix C.5, the bolded and italicized prompts in the "Text Prompt" column are those that were present verbatim in the text-video pair dataset.
indicates that this is an engineered text prompt
26

---

Seed Collection with Conditioning
Travel Distance with Conditioning
VPT (tex))
300
(text)*
VPT (uncond)
Trwed Distance (Blocko))
(urcond))
Serds Collected
200
160
100
dig hide
broak loavos
furosle wond
Sonee
go explore
swinmmingg
underwater
Invertecyy
000 didt
break Jeares
brook wood
collect seeds
brooks Blowes
10 explone
go swimming
go walewaler
open Inventory
look the
dig
look
Conditioning (Visual)
Couditioning (Visual)
(a)
(b)
Log Collection with Conditicaing
Dirt Collection with Conditicning
VPT [tssj
VPT [text)*
Woodlen Logs Collected
VPT (uscome)
VPT (uncone)
85
Ditt Collected
30
40
20
dig hole
get diet
break Inaoess
break wood
callect scends
braak Bommer
DD oxplore
go miitaming
mnderoontes
mommooy
hole
dirt
look at the dey
broak Ieaosss
break wood
brokk foomrr
explore
go oiiaminn
go mnderoster
open noveloty
Conditioning (Viuul)
Conditioning (Visual)
(c)
(d)
Figure 15: Conditioning with Visual Goals. We plot the performance of the programmatic metrics, along with their mean values and 95% confidence intervals, across different goal conditioning. See Figure 14 for visualization of these visual prompts. Plots are similar to Figure 20 the VPT appendix [5]. Each conditioning variant is run with 10 trials, each with different environmental seed and with an episode length 3000 time steps (2.5 minutes gameplay). We use the policy that was trained using the hyperparameters specified in Table and with conditional scaling values > 7. The dashed horizontal lines refer to an unconditional VPT agent ('VPT (uncond)") and a text-conditioned agent from the VPT appendix ("VPT (text)*") that was conditioned on the relevant text, for the purpose of comparison.
27

---

Travel Distance with Conditioning
Seed Collection with Couditioning
300
VPT [lext)*
15.0
VPT [beat)
Trieel Distance (Blocke)
VPT (uncund]
VPT (unoand)
250
Seeds Collected
200
150
7.5
100
5.0
2.5
dig bl
8P9 dirt
look at the sky
callect, seeds
break a Slower
explore
seimmiing
open Invertoory undermnuer
dis as poesible
20t diee
break Iroves
dlop tree
collect seeds
break flower axplare
go svimming
open Insontory
darr
dlop down the toee beeak tall gross
at the
get dut
broak
clop
break tall
50
Conditioning (Text)
Couditioning (Textt)
(a)
Log Collection with Conditioning
Dirt. Collection with Condit
VPT (texet)
Wooden Collected
VPT (uncoed)
VPT (text))
VPT (uncond)
10
Dirt Collected
1.5
10
possible
gat dirt
look at thke sky
Leoks Jeaves
callect neeks
tevak flowerr
go explece
Bo winnmiin
go underwater hovensery
805 dits
chop down the tree
broak (grees
dig oc for as possble
gest diet
broak Jeaves
dlop troe
collect seeds
flowe
50 exples
setamniing
mnderwaterr
open inverstary
get dirrt
beeak tall ptross
chop
as
chop
Conditioning (Text)
Conditioning (Text)
(c)
Figure 16: Conditioning with Text goals. See Table 3 for the exact text string used for each
conditioning variant. We use the same policy model but with a conditional scaling value = 6. We observe strong steerability which outperforms ext-connitioned VPT in Appendix of [5], and observe that prompt-engineering can improve performance.
Toxel Disucee (Dlocles)
M
M
Wooden Logs Collessed
20
(
Scerie Collecte
Die: Collected
200
10
5
20
10
0
6
10
0
2
10
2
10
Conditional Scale A
The conditional scale À in classifier-free guidance [23] can be tuned to improve perfor- mance in each of the programmatic tasks. By tuning À to classifier-free guidance at inference time, STEVE-1 1 is able to collect 7.5x more dirt, 15x more wood, 23x more seeds, and travel 1.7x further than
Figure 17:
(no guidance).
28

---

Task
Prompts
Precondition
dig a hole, get dirt, look at the sky, break leaves, get wood, get seeds, break flower, go explore, go swimming, go underwater, open inventory
Use the best-performing prompts Table in the Appendix.
For grass and flowers, it works best when grass and flowers are in the current biome. We found breaking flowers to be less reliable than the others.
make a tower
"build a tower"
building blocks hotbar. To obtain building blocks, you can use the dig hole or get wood prompts from Table 3.
craft wooden planks
"make wooden planks, craft wooden planks" (*)
wooden logs. obtain wooden logs, use the get wood prompt from Table 3.
place torches, place crafting table, place wooden planks
"place [torches/a crafting table/wooden planks]"
[torches/crafting table/wooden planks] in the hotbar
make a crafting table
"make a crafting table"
wooden planks.
break stone
"mine stone, go mining, get stone" (*)
get cobblestone
"mine stone, go mining, get cobblestone" (*)
pickaxe in hotbar.
create wooden pickaxe
"craft wooden pickaxe, make wooden pickaxe" (*)
already looking placed crafting table, has necessary materials.
hit sheep, hit pig, hit cow
"kill a [sheep/pig/cow]"
agent close to and looking at the sheep/pig/cow. (See Appendix B.
Table 4: Examples of tasks that STEVE-1 1 is able to achieve. (+) denotes that this is single prompt-engineered prompt. Note that this list represents only a subset of the capabilities STEVE- I due to the difficulty associated with evaluating and describing the capabilities of open open-ended models, which is an open research problem itself since capability depends strongly on preconditions, prompt engineering, and our own ability to come up with varied and challenging tasks. Please see Appendix B.4 for further discussion.
Model
Simple Prompt
Complex Prompt
Stable Diffusion [50]
steampunk market interior
steampunk market interior, colorful,
3D scene, Greg Rutkowski, Zabrocki, Karlkka, Jayison Devadas, trending ArtStation, 8K, ultra-wide-aggle zenith view, pineushion lens effect [20]
STEVE-1
collect seeds
break tall grass, break grass, collect seeds, punch the ground, run around in circles getting seeds from bushes
Table 5; Example of evolving simple prompts into more complex ones for various models.
29

---

Frame #52
Frame # #1000
Build a Tower
Agent Height Position
Dirt in Inventory
2
30-
Framu #2015
Frame #2250
1000 2000
Episode Timestep
Figure 18: Build a Tower task. (Left) We track the amount of dirt in the inventory and the agent's height position (y-axis) throughout the episode. In the first 1500 timesteps, the agent is conditioned on the visual get dirt goal, then the agent conditioned on the visual build tower goal for the final 1500 timesteps. Vertical dotted lines with numbers indicate the corresponding frames on the right. (Right) The agent's observation frames 4 different points in the episode. First the agent collects dirt, then begins to build the tower using the dirt blocks.
1: Frame #220
2: Frame #876
Make Wooden Planks
Planks in Annoo Soottne A
Logs Inventory
15
Frame #1500
Frame #1140
10
500
1000
1560
2000
2500
3000
Episode Timestep
Figure 19: Make Wooden Planks task. (Left) We track the number of logs and planks in the inventory. In the first 1500 timesteps, the agent is conditioned on the visual break wood goal, then the agent is conditioned on crafting the visual wooden planks goal for the final 1500 timesteps. Similarly to Figure 18, a vertical dotted line annotated with number indicates the corresponding frame to the right. (Right) The agent's 's observation frames 4 different points in the episode. First the agent breaks trees to collect wooden logs, then opens the inventory and crafts wooden planks.
30


Reviews:
[{"review_id": "MUXEtDkOZK", "reviewer": "Reviewer_3aRk", "summary": "Paper presents a method to create instruction following agents in Minecraft. It starts with collecting trajectories using OpenAI’s VPT Minecraft agents, which cannot be controlled through instructions. Then some intermediate visual observations are randomly selected as visual goals. These visual goals are then encoded by the visual encoder of a pretrained text-video contrastive foundation model in the Minecraft domain called MineCLIP. VPT is then fine-tuned to achieve these visual goals (specified as MineCLIP visual embeddings) using the same trajectories. Finally, a CVAE is trained to produce textual embedding that matches the visual goal embeddings, therefore the agents can be piloted by natural language instructions. Experiments highlight the performances of presented method on some entry-level tasks including dirt, log and seed collection. Additional results on prompt chaining, scaling, prompt engineering and other hyperparameters confirm the effectiveness of the proposed approach.", "questions": "See [Weaknesses]", "limitations": "One thing that could be missing in the limitation statement is how well does the cVAE model perform on producing goal embedding and whether it could be a bottleneck to the overall performances on more challenging tasks like long-horizon planning, etc. The authors seem to blame MineCLIP for this but since in the original unCLIP paper, diffusion is used instead of a cVAE, more comparisons might be necessary. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "+The paper is overall clear and well-written. The presentation should be very friendly to readers without RL or LfD background by foundation model in general. The research problem here is relevant to the scope of NeurIPS and its proximity to many emerging topics including open-endedness, generalist agents and large models should be of interest to large audience of this conference.\n\n+I find the approach presented here is technically sound with good results. Although the original VPT agent did offer some preliminary results on instruction following, but no enough details were provided so the method here seems to make solid contributions on building the first open-ended language-piloted agents in Minecraft. Some tricks like baseline subtraction seems helpful. Thanks for bringing this to the attention of the community.\n\n+The authors did a good job highlighting go/no-go about their method. Just list a few that I find most interesting:\n\n- MineCLIP evaluation, which help explain why their method works better on short-horizon task\n\n- Prompt-chaining and metrics over time, which clearly demonstrate how the agent progresses under different prompt conditions.\n\n- Prompt-engineering, which showcases how different instruction can have significant impact at the agent.\n", "weaknesses": "At this point, I don’t have any major concerns but here are a few suggestions:\n\n-Baselines; in the main paper, I couldn’t find any baselines or ablations other than the main ”Steve-1” model and VPT(without instruction-following). I agree that there might not be many proper counterparts available but some single-task RL and imitation learning baseline, ex. [10] could still help with better sense on the actual gain the proposed model has in Minecraft. Moreover, the baseline subtraction trick should also be ablated in the main paper with more evidences on other tasks, it seems to be a working technique, but it must be better validated.\n\n-Error bar; In Figure 6, I am not sure if the numbers in parentheses indicate error range. If they are not, error bar is needed for these results.\n\n-Scaling: please elaborate more on “we see evidence of tasks that do not require much data for STEVE-1 to learn, tasks that steadily get more reliable as the agent is trained longer, and tasks where capability suddenly spikes after the agent reaches some threshold.” If not all tasks can benefit from scaling, why “Put together, this suggests that further scaling would likely significantly improve the agent”\n\n-After reading the paper, it’s still unclear to me how the $60 budget is allocated. Please provide detailed explanation to justify this point.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and kind words. We are so glad that you found our work an approachable and valuable contribution to the community.\n\n> Baselines; in the main paper, I couldn’t find any baselines or ablations other than the main ”Steve-1” model and VPT(without instruction-following). I agree that there might not be many proper counterparts available but some single-task RL and imitation learning baseline, ex. [10] could still help with better sense on the actual gain the proposed model has in Minecraft.\n> \n\nThe two most interesting baselines from our perspective are VPT [6] with text-conditioning (Appendix I in [6]) and the multi-task RL experiment done in the MineDojo paper [18]. As you mentioned, the VPT paper did not include many details and did not perform well. MineDojo evaluates on slightly different tasks and the authors appear to find very limited zero-shot generalization ability after training on 12 tasks. We’ll explicitly mention these two baselines in a new section of the paper and include a copy of their results. Note that Appendix E.2 does include a comparison to the VPT with text-conditioning results from Appendix I in [6], but we will include a more clear comparison in the main paper for future versions. Also, we have included a number of ablations in Section 4 and Appendix B that show the importance of our design choices, including experiments on the use of classifier-free guidance, pretraining, chunk size for packed hindsight relabeling, and VAE design choices. Thanks for pointing out [10]. We’ll explore it further.\n\n> Moreover, the baseline subtraction trick should also be ablated in the main paper with more evidences on other tasks, it seems to be a working technique, but it must be better validated.\n> \n\nWe have updated the baseline subtraction trick figure (classifier-free guidance; Figure 5) to include our two other programmatic tasks of “travel distance” and “seeds collected”. We find that it does not make much of a difference for travel distance and improves the seed collection task. Please find the updated Figure C in our rebuttal PDF. Also, note that Appendix B.1 shows an ablation on the effect of using classifier-free guidance during finetuning training at all, which is different to testing the effect of the conditional scale parameter at inference time (Figure C in the rebuttal PDF).\n\n> Error bar; In Figure 6, I am not sure if the numbers in parentheses indicate error range. If they are not, error bar is needed for these results.\n> \n\nThese are 95% confidence intervals. We have updated the figure caption to make note of this.\n\n> Scaling: please elaborate more on “we see evidence of tasks that do not require much data for STEVE-1 to learn, tasks that steadily get more reliable as the agent is trained longer, and tasks where capability suddenly spikes after the agent reaches some threshold.” If not all tasks can benefit from scaling, why “Put together, this suggests that further scaling would likely significantly improve the agent”\n> \n\nThanks for the question. At a high level, we mean to say that since we see non-decreasing performance with scale (travel distance and seeds stay roughly constant, dirt and logs increase), then we posit that further scaling will help improve performance. We suspect that for tasks which didn’t benefit from scaling in our experiments, we are either close to optimal performance or we didn’t reach the critical amount of scale required to see strong performance. There is evidence of this type of sudden emergence in capability in both our dirt and logs task as well as in the literature [58]. We will improve the scaling section in the paper to clarify.\n\n> After reading the paper, it’s still unclear to me how the $60 budget is allocated. Please provide detailed explanation to justify this point.\n> \n\nThe $60 cost we reported in the paper corresponds to the cost of renting a 8xA10g node using spot instances on AWS for 12 hours using our instances prices at the time. We are considering translating this into more standard numbers using on-demand pricing for A100s for the final paper release since we realized that spot prices can fluctuate and be misleading.\n\n> One thing that could be missing in the limitation statement is how well does the cVAE model perform on producing goal embedding and whether it could be a bottleneck to the overall performances on more challenging tasks like long-horizon planning\n> \n\nGood question. We don’t believe that the cVAE is a bottleneck to long-term planning since using STEVE-1 with visual instructions is equivalent to just bypassing the cVAE entirely and this doesn’t improve long-term planning abilities. However, a better cVAE model will likely improve text performance and we will add a discussion about this to the limitations section. Thanks for the suggestion."}, {"review_id": "vXCGUXIbnL", "reviewer": "Reviewer_Q7hP", "summary": "The paper introduces STEVE-1, a sequential decision-making agent designed to follow textual instructions and accomplish goals in the Minecraft environment. The authors utilize two pre-trained models, VPT (Video Pretraining Transformer) and MineCLIP, to facilitate this process. VPT is a transformer model trained to predict action sequences from aligned video sequences, while MineCLIP aligns consecutive video timesteps with corresponding transcripts in Minecraft.\n\nTo finetune VPT, the authors employ self-supervised behavioral cloning conditioned on latent visual goals. They generate goal-conditioned data by randomly selecting timesteps from episodes and using hindsight relabeling to set intermediate goals. MineCLIP is used to map textual goals to visual goal embeddings in the unCLIP-based approach. This mapping is achieved through a conditional Variational Autoencoder (VAE) with Gaussian prior and posterior, conditioned on MineCLIP text representations.\n\nThe training dataset comprises Minecraft gameplay data, combined with the OpenAI contractor dataset, and additional data generated using VPT. The authors curate 2,000 instruction-labeled trajectory segments, each consisting of 16 frames, and augment this dataset by identifying similar gameplay segments. Additionally, 8,000 additional instructions are generated using GPT-3.5 Turbo. Classifier-free guidance is employed to balance the logits between unconditional and conditional behavior.\n\nThe results demonstrate that STEVE-1 successfully solves various short-horizon open-ended text and visual tasks, with a training cost of only $60. The performance of the agent is evaluated using programmatic evaluation and MineCLIP evaluation, revealing significant improvements with goal conditioning, especially visual. The authors also observe that prompt chaining, as opposed to direct prompting, proves advantageous for complex tasks like building towers or creating planks. A thorough ablation study is conducted on the classifier-free guidance hyperparameter, pretrained VPT weights, and prompt engineering, providing valuable insights into the optimal settings for these components.\n\nOverall, the paper showcases the effectiveness of STEVE-1 in achieving goals based on textual instructions in the Minecraft environment. The combination of VPT and MineCLIP, along with the conditioning techniques and ablation studies, contribute to a comprehensive understanding of the agent's performance and its potential applications.", "questions": "- Can you provide insights into why random timesteps from episodes are selected as goals? Are there alternative approaches that could be considered, and why is this method deemed the most effective?\n- Does randomly resetting the agent's memory and turning the agent to a random direction during data generation provide necessary benefits? What would be the implications if this step were omitted?\n- How does the utilization of pre-trained VPT contribute to the approach when the input is modified with conditional goal embeddings? Can you elaborate on the specific advantages and improvements gained from incorporating VPT in this manner?\n- Could you explain the process of generating additional text instructions based on GPT 3.5? How are these instructions generated and integrated into the training process?\n- It is not clear why the graph from Baker et al. is not included in the text-conditioning results. Can you provide clarification or discuss the reasoning behind its exclusion?\n- In Figure 4 (right), what is \"relevant\" and \"irrelevant\" in the graph? Clarifying the interpretation of these components would improve the understanding of the presented results.", "limitations": "- The authors extensively discuss the limitations of the approach, including challenges with multiple steps of reasoning, prompt engineering, and potential negative societal impacts. While further discussion on these aspects could be beneficial, there are no explicit omissions or significant limitations that require specific mention. The paper adequately addresses and acknowledges its limitations.\n\n\n\n", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This unique adaptation - STEVE-1 - of the unCLIP method demonstrates the versatility and effectiveness of the approach in the context of sequential decision making in Minecraft.\n- The experiments and analysis conducted in the paper are highly novel and insightful. The authors provide valuable findings, such as the benefits of prompt chaining, the potential for scaling to improve certain metrics, and the limitations observed in complex tasks. These insights enhance our understanding of the proposed approach and its implications.\n- The paper demonstrates that prompt chaining is effective in accomplishing complex tasks, such as building towers or making wooden planks. The results show that success rates and programmatic evaluation metrics plateau after a certain number of frames, highlighting the potential and limitations of the approach. Additionally, the comparison to direct prompting reveals the superiority of prompt chaining in achieving satisfactory performance.\n- The thorough ablation study conducted on various components, including the classifier-free guidance hyperparameter, pretrained VPT weights, and prompt engineering, provides valuable insights into the optimal settings for these elements.\n- The inclusion of additional ablations, such as goal chunk sizes, VAE variants, and text augmentation, in the appendix further enhances reproducibility and facilitates a deeper understanding of the approach.", "weaknesses": "- The paper would benefit from clearer explanations regarding certain aspects, such as the distinction between packed hindsight relabeling and hindsight relabeling. Additionally, providing a more detailed explanation of the interpretation of the heatmap, specifically the meaning of the 1/0 values, would enhance understanding.\n- Some questions (see section) during the review could be addressed in the main work or in an appendix to provide a more comprehensive understanding of the approach.\n- The proposed approach combines existing ideas from well-established approaches, such as CVAE, MineCLIP, and VPT, which may limit the novelty of the work. However, the extensive experiments and detailed study conducted in the paper contribute to its significance and overall value.\n- The scalability of the approach to other environments or datasets is not discussed or addressed in the paper. Considering the dependency on pretrained video-text aligned models and large-scale pretrained transformers like VPT, the applicability of the approach to environments without such models is unclear. Addressing this limitation and discussing potential scalability issues would enhance the practical relevance and broader applicability of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive questions and feedback, and for recognizing the versatility, novelty, and significance of our work on STEVE-1. Please see below for responses to your comments and questions:\n\n> Clearer explanation on the distinction between packed hindsight relabeling and hindsight relabeling.\n> \n\nHindsight relabelling was introduced in the Hindsight Experience Replay (HER) work [4]. With this technique, trajectories are relabelled with imagined goals, and these goals can be chosen using different strategies. *Packed hindsight relabeling* is our specific implementation of hindsight relabeling (see Algorithm 1 in Appendix), which “**packs” multiple relabeled goal sequences into a single sequence**. Specifically, we split a trajectory into multiple chunks and pick the last timestep in each chunk as the relabelled goal for that chunk (see Figure 2).\n\n> A more detailed explanation of the interpretation of the heatmap, specifically the meaning of the 1/0 values\n> \n\nThanks for raising this. The ideal performance with the MineCLIP heatmap would be where the minimum values of each row lie on the diagonal. That is, the agent performs a specific task best when it is asked to perform that specific task. The numbers (1/0) on the diagonal correspond to the ranking that the diagonal has in its row. A 1 represents that the diagonal is the second lowest value in the row and 0 means it is the lowest. \n\n> The scalability of the approach to other environments or datasets is not discussed or addressed in the paper.\n> \n\nWe designed STEVE-1 for Minecraft due to the availability of two key ingredients: 1) a strong behavioral prior (VPT), and 2) a powerful visual-language model which maps text and video to a joint embedding space (MineCLIP). However, the method used to create STEVE-1 is not specific to the Minecraft domain. Given the rapid development of generative models, we expect that similar models to VPT and MineCLIP will become available in many other domains. As these models become available, future work could investigate the applicability of the STEVE-1 approach to these other domains. Thanks for raising this key point, we will include a discussion of this in the final version of our paper.\n\n> Can you provide insights into why random timesteps from episodes are selected as goals?\n> \n\nWe chose to randomly select future timesteps from episodes as goals primarily due to the simplicity of the approach, but also to ensure a diverse and unbiased coverage of potential goals achievable within the short horizon. By avoiding a specific heuristic strategy, we aim to prevent any potential biases in goal selection that might influence the agent's training, thereby promoting a more generalized performance. Future works can investigate the effects of alternative approaches that filter for semantically interesting timesteps as goals.\n\n> Does randomly resetting the agent's memory and turning the agent to a random direction during data generation provide necessary benefits?\n> \n\nThanks for asking this question. These implementation details of the VPT dataset were chosen as heuristics to increase the diversity of the generated trajectories. Unfortunately, it’s very compute-intensive to get concrete answers about the effects that these decisions have. We will consider running additional experiments to investigate this question in the future.\n\n> How does the utilization of pre-trained VPT contribute to the approach when the input is modified with conditional goal embeddings? Can you elaborate on the specific advantages and improvements gained from incorporating VPT in this manner?\n> \n\nGreat question. Yes, due to how we modify the inputs to the transformer, the input distribution is different from what VPT expects. However, since we finetune the modified VPT architecture on our gameplay dataset with relabelled goal embeddings, then the model learns to adapt to this new distribution. Empirically, in Figure 5 (left) we found that using the VPT pretrained weights improves performance as compared to finetuning from scratch. Note that this conditioning method via a bias to the transformer input was also explored in Baker et al.’s Appendix I. However, future works can study other mechanisms for conditioning.\n\n> Could you explain the process of generating additional text instructions based on GPT 3.5? How are these instructions generated and integrated into the training process?\n> \n\nWe’ll update the appendix to include the GPT-3.5-turbo system prompt and user query prompt. Following the creation of approximately 8,000 additional examples from GPT, we further find potential matching clips from the gameplay dataset as described in Appendix D.2. Specifically, for each of our text labels, we find the top 5 closest (cosine similarity) timesteps in 2000 episodes from our gameplay dataset using MineCLIP’s similarity score. These 50,000 automatically-mined text-video pairs are added to the original 2,000 hand-labeled examples to form the final dataset used for training the prior. We will add an algorithm box to make this more precise in the final version.\n\n> It is not clear why the graph from Baker et al. is not included in the text-conditioning results.\n> \n\nWe showed this comparison in Appendix E.2 (Figures 13 & 14) and we will add the results from Baker et al. to Figure 3 (left). The caveats with this comparison is that we have some differences in the experimental setup: (1) we only used half the episode length, and (2) it’s unclear exactly how Baker et al. defined their item count metric. Despite being evaluated for half the episode length, STEVE-1 achieves much stronger steerability performance.\n\n> In Figure 4 (right), what is \"relevant\" and \"irrelevant\" in the graph?\n> \n\n“Relevant” refers to the single relevant prompt for the task (”collect seeds” for task “Seeds Collected”, etc.) and “irrelevant” corresponds to the performance averaged over 10 other irrelevant prompts. We will clarify in the final version."}, {"review_id": "qWYMCkSg6H", "reviewer": "Reviewer_vziy", "summary": "This paper proposed an instruction-tuned model for Minecraft, which turns previous RL model like VPT into goal-conditioned model. The experiment shows that the proposed method can follow nearly any short-horizon open-ended text and visual task in Minecraft.", "questions": "I am curious about how is the prompt designed in this paper? The detail of the prompt should be explained in the main text of the paper.", "limitations": "The limitation is adequately addressed in the paper. No discussion about negative social impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a novel technique for instruction-tuning video game RL-agent which is inspired by previous unCLIP method in the text-to-image generation field. This idea is interesting and the method can turn an RL agent into a goal-conditioned one which shows great potential for designing a more general RL agent.\n\n2. The experiment part shows that the tuned agent can follow short-horizon goal instructions very well. When given the appropriate textual instruction, the proposed method collects 66x more dirt, 4.5x more wood, 28x more seeds, and travels 3x further than the unconditional agent. ", "weaknesses": "This method is not applicable for long-horizon tasks in the open world. For example, in Minecraft, if the goal is obtaining a diamond, the instruction may be too difficult for the proposed method since it is not trained for long-horizon trajectories. Regarding this, it is not sure if this method can be applied to challenging tasks in the open world which may contain extremely long planning steps. In this paper, no complicated tasks like obtaining diamond are studied. Thus I am concerned the scalability of the proposed method to much complicated tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments. We’re glad to hear that you found our work interesting and that you see it as a potential method for designing more general sequential decision-making agents. Please see below for responses to your comments and questions. \n\n> This method is not applicable for long-horizon tasks in the open world. For example, in Minecraft, if the goal is obtaining a diamond, the instruction may be too difficult for the proposed method since it is not trained for long-horizon trajectories. … In this paper, no complicated tasks like obtaining diamond are studied. Thus I am concerned the scalability of the proposed method to much complicated tasks.\n> \n\nWe agree that the STEVE-1 agent is currently not capable of accomplishing long-horizon tasks like obtaining a diamond. This is a limitation of our work and something that we think is an interesting direction for future work, as solving long-horizon tasks while taking actions using low-level mouse/keyboard controls is a very challenging and exciting research direction. \n\nBased on our experience, there are a few things that we think could help improve long-horizon performance:\n\n**1) Scaling:** Our scaling results indicate that as we train the agent on more data, more tasks become achievable. So scaling up the amount of data could enable the agent to complete longer-horizon tasks. \n\n**2) Finetuning with RL:** We also think that finetuning the STEVE-1 pre-trained agent with RL is another very interesting avenue. For example, VPT finetuned their pretrained agent with RL to learn to mine diamonds. So we think it’s likely that RL finetuning on top of the pretrained STEVE-1 could enhance long-horizon task completion abilities. \n\n**3) Using LLMs or VLMs:** Using LLMs or Visual-Language Models (VLMs) to automatically provide prompt chains to the STEVE-1 agent could be an effective way to improve long-horizon performance. We started to investigate this possibility with our prompt chaining experiments and we think that this is a very exciting direction for follow-up work.\n\n> I am curious about how is the prompt designed in this paper? The detail of the prompt should be explained in the main text of the paper.\n> \n\nThanks for asking this question. We answer both how the prompts were designed and what the exact prompts are.\n\n- **Exact prompts:** Table 3 in the appendix includes a list of the full text prompts used for each task, but we agree that the exact prompts should be made more clear in the main text of the paper. We have included a modified version of the table that hopefully provides more clarity in exactly which text prompt corresponds to which evaluated task (see below). We will include this figure in future versions of the paper.\n\n| Figures 3 (Right) & 11 Label | Figures 13 & 14 Label | Text Prompt |\n| --- | --- | --- |\n| dig | dig as far as possible | dig as far as possible |\n| dirt | get dirt | get dirt |\n| sky | look at the sky | look at the sky |\n| leaves | break leaves | break leaves |\n| wood | chop a tree | chop a tree |\n| seeds | collect seeds | collect seeds |\n| flower | break a flower | break a flower |\n| explore | go explore | go explore |\n| swim | go swimming | go swimming |\n| underwater | go underwater | go underwater |\n| inventory | open inventory | open inventory |\n| dirt... | get dirt ... | get dirt, dig hole, dig dirt, gather a ton of dirt, collect dirt |\n| wood... | chop down the tree ... | chop down the tree, gather wood, pick up wood, chop it down, break tree |\n| seeds... | break tall grass ... | break tall grass, break grass, collect seeds, punch the ground, run around in circles getting seeds from bushes |\n- **Prompt design:** In our experiments we used both short and longer prompts. The short prompts are either taken from previous literature (e.g., the language-conditioning experiment in the VPT appendix) or they were simply the first thing we tried. The longer prompts were created by taking inspiration from the prompt engineering methods used with text-to-image models such as Stable Diffusion [47]. To design these prompts, we simply strung together a lot of terms related to our task in order to increase the specificity of the prompts. We were excited to discover that this style of prompt design inspired by the prompt engineering community works well in STEVE-1.\n\nThank you again for the positive review. We are excited about the simplicity, scalability, and strong performance of STEVE-1 and the many future research opportunities that it unlocks. If you have any additional questions or needs for clarification, please don’t hesitate to ask."}, {"review_id": "aTtJS3XSRY", "reviewer": "Reviewer_7QfK", "summary": "This paper proposes a generative pretraining method to learn an instruction-following agent in Minecraft by fine-tuning the VPT agent. It first trains an image-goal-conditioned policy and then leverages the foundation model MineCLIP as the bridge to map the language instruction into the image-goal space. Using this method, this paper claims that it can solve any short-horizon open-ended text and visual task in Minecraft. ", "questions": "As stated in the weakness part. ", "limitations": "The main limitations include (1) prompt design being too sensitive and (2) chain-of-thought prompting is not automated. \nThe authors have adequately stated the limitations. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) This paper explores the feasibility of jointly using the foundation models (such as the vision-language model MineCLIP and pre-trained policy model VPT) for solving decision-making problems and shows a preliminary conclusion. \n\n(2) Using unCLIP approach to solve instruction-following decision-making problems is interesting. \n\n(3) This paper proposes to improve the instruction sensitivity with the classifier-free guidance technique, which is interesting and reasonable. \n\n(4) The paper is well-written and easy to follow. ", "weaknesses": "(1) The paper overstates its performance by claiming that \"STEVE-1 can follow nearly **any short-horizon** open-ended text and visual task in Minecraft.\"\n\nTo verify this, I downloaded the provided code and ran the attached checkpoints on some basic short-horizon tasks like \"kill sheep\", \"kill cow\", and \"kill pig\". Before testing these tasks, I had already summoned 10 \"pigs\", \"cows\", and \"sheep\" nearby. While I found that the agent actually killed some animals, it did not seem aware of which particular animal it was targeting. The behavior was more like \"kill cow if there is a cow\" rather than following the specific instruction. I tried several other prompt variants such as \"hunt cow\" and \"hunt cow in Minecraft\", but they were unsuccessful. Additionally, I tested STEVE-1 on some short-horizon crafting tasks with sufficient materials in the initial inventory, like \"craft stick\", \"craft oak plank\", and \"craft torch\". The agent merely opened its inventory and then acted randomly.\n\nGiven these observations, I do not believe that STEVE-1 has solved all short-horizon tasks, as it cannot even differentiate between \"cows\" and \"sheep\". This could lead others to misconstrue the extent of the research in the Minecraft environment.\n\n(2) This paper lacks generalization experiments on unseen text instructions. \n\nAs the primary aim of this method is to support open-ended text instructions in STEVE-1, it is crucial to demonstrate its performance on unseen text instructions. The authors collected approximately 10,000 instruction-trajectory pairs, with 2,000 of them hand-labeled and an additional 8,000 instructions generated by GPT. It remains unclear whether the tasks (\"dig dirt\", \"build a tower\", \"make wooden planks\") in the experiments were already included in these instructions. If so, the experiments may not be particularly convincing, as neural networks can easily map the text instructions to corresponding visual embeddings in the training set. That is, pre-training STEVE-1 on visual instructions already achieves this goal and the later generative model training may be not important, which is not impressive. So, I strongly recommend including generalization experiments if the paper wants to show support for open-ended text instructions. \n\n(3) The evaluation metric is insufficient.\n\nThe MineCLIP evaluation metric is not convincingly effective in measuring whether a trajectory corresponds to a given task. This is because the MineCLIP latent space cannot distinguish some trajectories well. For instance, we have found that it was unable to differentiate between two trajectories where one involved approaching a tree and the other involved moving away from it (MineCLIP was not sensitive to distance).\n\nTo address this issue, I recommend that the paper incorporates a success-rate-based confusion matrix, in which the values in Figure 3 (b) are replaced with the task success rate, conditioned on the given instruction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the great questions and for engaging deeply with the work. We’re glad that you found the approach interesting and reasonable, and the paper easy to follow.  We address your questions and comments below.\n\n> The paper overstates its performance by claiming that \"STEVE-1 can follow nearly **any short-horizon** open-ended text and visual task in Minecraft.\"\n> \n\nWe agree that this overstates STEVE-1’s capabilities. Shortly after submitting we actually changed this language ourselves to say “follow a wide range of short-horizon text and visual instructions in Minecraft”. We hope you agree that this language more accurately describes the performance of STEVE-1.\n\nRegarding the specific examples listed (hitting a cow, crafting a stick), the behavior you noted seems about right for the weights that we uploaded to OpenReview. The phenomenon of the agent performing a seemingly related task rather than the intended one is something we noticed too and is related to the concept of goal misgeneralization (Langosco et al. 2022). Generalization can be helpful when the task we assign the agent is impossible to achieve from the current state and the agent instead performs a closely related action, but harmful when the task is achievable. We note two things: first, the powerful generalization ability of STEVE-1 probably comes from the MineCLIP embeddings and it especially improves the ability of STEVE-1 to follow visual instructions when the exact items or blocks nearby are not available in the current environment, which is an extremely common scenario. Second, we notice that the tendency of the agent to misgeneralize decreases with scale. For example, with a model trained on fewer data we find that asking the agent to look up and punch a tree to get a log often resulted in the agent looking in the air and punching nothing; training the model on more data results in the agent first walking over to a nearby tree and looking up to get a log. \n\nWe are including (via the AC) an updated codebase with the ability to run the agent in an interactive mode where new instructions can be presented to the agent in real-time. This updated code also includes a new snapshot with updated hyperparameters (see general response) that shows a modest improvement in performance. We will add a section on goal misgeneralization and its relationship with scaling to the appendix of our updated paper, and we hope that future research can further investigate these issues and how to improve misgeneralization. We stress that for a wide range of tasks, including but not limited to the tasks included in our evaluations, STEVE-1 shows strong performance on both the original weights and the updated weights.\n\n> The [MineCLIP] evaluation metric is insufficient.\n> \n\nPlease see the general response section titled “Reviewer 7QfK raised a concern regarding the strength of the MineCLIP evaluation.”\n\n> This paper lacks generalization experiments on unseen text instructions.\n> \n\nThanks for pointing this out; we strongly agree that generalization experiments would improve the paper. During the rebuttal period we have performed a set of simple generalization experiments which we hope helps to answer these important questions.\n\n**Instruction Training Set Contamination:** Please refer to Table A in our rebuttal PDF. Among our evaluation instructions, the bolded instructions show up in the instruction-trajectory dataset. While some instructions do show up, most of the instructions do not show up in our training set (verbatim).\n\n**Training Set Decontamination:** To measure the effect on performance of removing a concept from the training set, we ran an experiment where we removed every instruction with the words “dirt” or “dig” in them and retrained the VAE model. This corresponds to around 10% of the instructions. We found that even without training on the concept of dirt or digging at all, STEVE-1 can still be instructed to dig holes and get dirt. This demonstrates clearly that STEVE-1 can generalize to unseen text instructions (see Figure B in rebuttal PDF) — likely because most of the text-understanding comes from the pretrained MineCLIP model which was trained on a highly diverse dataset of YouTube videos and captions. The prior VAE only needs to learn a simple mapping between the text and visual MineCLIP embeddings. Note that there is a slight decrease in performance across all tasks likely due to the smaller VAE training set (~10% less).\n\nThe instruction-following capability of STEVE-1 is shared between: the policy, which learns to follow instructions in the visual MineCLIP embedding space; the MineCLIP text-encoder, which is trained to align well with the visual embeddings and performs most of the text-understanding; and our prior VAE model, which learns a simple function to translate between text and visual embeddings. While this means that the VAE doesn’t hold the most important role in giving instruction-following capabilities, we disagree with the reviewer that this is unimpressive. Contrary to this viewpoint, it is precisely our modeling setup which lets us fully exploit pretrained models such as MineCLIP to gain impressive language understanding without relying on having our own large datasets or compute.\n\nWe conclude by stressing that while STEVE-1 is not perfect, it represents a large step towards a general recipe for creating generalist agents by building on pretrained models. As future generative visual-language/action models get more powerful and general, we think that approaches like STEVE-1, which can exploit and steer their capabilities, will provide great value and enable exciting future research.\n\nThanks again for the thorough review. We hope that some of the additional experiments and explanations we have conducted in response to the issues that you raised go a long way towards improving the paper and changing your opinion. Please don’t hesitate to ask if you have any additional questions or require clarification."}]

Author rebuttal:
Thanks to all the reviewers for your time and effort during the review process. We appreciate that you found our work well written, insightful, and novel, and we’re glad that there is excitement about our approach to creating an open-ended agent by building on pretrained models.

We have responded to each reviewer individually, uploaded a rebuttal PDF, updated our code (shared with the AC), and collected the below response to general concerns. If you find our answers responsive to your concerns, we would be grateful if you considered increasing your score, and if you have additional questions, we’re happy to engage further.

 

> Reviewer 7QfK raised a concern regarding the strength of the MineCLIP evaluation.
> 

While we agree that MineCLIP is imperfect, our experimentation found that it performs impressively at understanding a wide variety of texts and relating them to Minecraft videos. Following the suggestion of reviewer 7Qfk, we have created a success-rate-based confusion matrix in order to demonstrate how MineCLIP evaluation aligns well with our human intuitions about task completion. This matrix is constructed using the exact same episodes used in the MineCLIP evaluation (Figure 3), but rather than use MineCLIP to evaluate task completion, we manually inspect each episode. We find that the overall pattern remains and STEVE-1 completes the instructed task the vast majority of the time (see the figure for the numbers). The success-rate-based matrix can be found in Figure A in the rebuttal PDF and we are considering replacing the MineCLIP evaluation matrix in the main paper with this new figure to increase clarity. Thanks for the suggestion!

> Reviewer 7QfK requested to see generalization experiments to unseen instructions.
> 

We have included a new table showing the degree to which evaluation prompts appear in the training set for the VAE and we have also conducted a decontamination experiment where all instructions containing the words “dirt” or “dig” were removed from the training dataset. We find that the new model can still achieve tasks like digging and getting dirt even with this new setup, suggesting that the generalization capability of our agent to new instructions is strong and that most of the language-understanding capability comes from the pretrained MineCLIP model. We will include a discussion of these results in the updated version of the paper. Please see the response to reviewer 7QfK for more details.

> We have committed to making the following changes to enhance the paper in response to helpful comments by the reviewers. Many of these points are elaborated upon in responses to individual reviewers.
> 
- Include in the main paper the VPT text-conditioning baseline and MineDojo multi-task RL baseline along with a discussion of limitations and differences.
- Include more tasks showing the effect of classifier-free guidance (baseline subtraction trick) in the appendix.
- Update the Figure 6 caption to indicate that the values in the parentheses are a 95% confidence interval.
- Improve the scaling section of the paper to make it more clear why we expect further scaling will improve agent performance.
- Add a section to the appendix indicating how we calculated the total cost of training STEVE-1.
- Add a discussion to the limitations section about the difference in performance between text and visual instructions.
- Add clarification regarding the differences between hindsight relabeling and packed hindsight relabeling.
- Add a more detailed explanation of the MineCLIP evaluation matrix.
- Add a section in the limitations section that summarizes what is needed to use our method in a new domain.
- Add a discussion of selecting goals in training randomly versus with a more sophisticated strategy.
- Add the prompts used to generate the GPT instructions for the prior training dataset.
- Update Figure 3 (left) to include VPT text-conditioning results as a baseline.
- Clarify “relevant” and “irrelevant” prompts in Figure 4 (right).
- Add a discussion on how the performance of STEVE-1 could be improved on longer horizon tasks.
- Add discussion about how the prompts were designed to the paper.
- Update Table 3 in the appendix to clarify which prompts correspond to which tasks in the MineCLIP evaluation matrix.
- Update “STEVE-1 can follow nearly any short-horizon open-ended text and visual task in Minecraft” to “STEVE-1 can follow a wide range of short-horizon text and visual instructions in Minecraft”.
- Add a discussion of goal misgeneralization to the main text.
- Add a human-labeled success-rate-based evaluation matrix to replace or in addition to the MineCLIP evaluation matrix in Figure 3 (right).
- Add discussion and experiments on training set contamination and generalization to new instructions to the appendix.

> Updated code and new model snapshots.
> 

We have updated the code to fix a few bugs that we found and to facilitate running an interactive session with the agent. An interactive session makes it easy to test out and change prompts within an episode and to record videos. We also have included a slightly updated snapshot of STEVE-1 (new links in `download_weights.sh`). The main difference is that we trained the agent until the validation loss stopped going down (2 epochs → 3 epochs) and changed the VAE hyperparameters by increasing its size and tuning the $\beta$ hyperparameter. The performance is marginally better than the previous snapshot. This is the version of the agent and code that we will be releasing to the public as well as in the final NeurIPS supplemental materials. We’ve shared this anonymously with the AC.

We again thank the reviewers for their engagement and we appreciate all the suggestions that we believe will make the paper significantly stronger!

Meta-review: The reviews are unanimous in favor of acceptance. We have also decided recommending it for spotlight. Considering that embodied agent is appealing to a broad community, we all agree that this paper is about some potentially very impactful topic. And one fundamental challenge in embodied agent is the text-to-action generation. The reviews all appreciate this paper’s conceptually simple formulation accompanied with thorough and careful analysis of prompt changing and potential of scaling. The reviews further appreciate the thorough ablation studies in the appendix and appreciate the careful documentation to facilitate reproducibility. However considering the diversity of complex tasks in Minecraft, we would also recommend this paper to include even more discussion about its limitation. Have we observed certain patterns when the proposed method certainly does not work? After careful discussion, one concern does remain. Considering this paper does require certain prior work at ready (e.g., VPT and mineclip), how a similar proposed method could work in other domains remains a question.
Decision: Accept (spotlight)