Title: STEVE-1: A Generative Model for Text-to-Behavior in Minecraft
Venue: NeurIPS 2023
Accepted: yes
Keywords: minecraft

instruction following

foundation models

sequence models

reinforcement learning

sequential decision making

goal conditioned reinforcement learning

text conditioned reinforcement learning

transformers

deep learning
Abstract: Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces a methodology, inspired by unCLIP, for instruction-tuning generative models of behavior without relying on a large dataset of instruction-labeled trajectories. Using this methodology, we create an instruction-tuned Video Pretraining (VPT) model called STEVE-1, which can follow short-horizon open-ended text and visual instructions in Minecraft. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, reducing the need for costly human text annotations, and all for only $60 of compute. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines and robustly completing 12 of 13 tasks in our early-game evaluation suite. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.

STEVE-1: A Generative Model for Text-to-Behavior in Minecraft

Abstract

Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces a methodology, inspired by unCLIP, for instruction-tuning generative models of behavior without relying on a large dataset of instruction-labeled trajectories. Using this methodology, we create an instruction-tuned Video Pretraining (VPT) model called STEVE-1, which can follow short-horizon open-ended text and visual instructions in Minecraft™. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, reducing the need for costly human text annotations, and all for only $60 of compute. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines and robustly completing 12 of 13 tasks in our early-game evaluation suite. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.

1 Introduction

The ability to use text instructions to control and interact with powerful AI models has made these models accessible and customizable for the masses. Such models include ChatGPT [41], which can respond to messages written in natural language and perform a wide array of tasks, and Stable Diffusion [50], which turns natural language into an image. While those models cost anywhere from hundreds of thousands to hundreds of millions of dollars to train, there has been an equally exciting trend whereby powerful open-source foundation models like LLaMA [59] can be finetuned with surprisingly little compute and data to become instruction-following (e.g., [58, 13]).

In this paper, we study whether such an approach could be applicable to sequential decision-making domains. Unlike in text and image domains, diverse data for sequential decision-making is very expensive and often does not come with a convenient 'instruction' label like captions for images. We propose to instruction-tune pretrained generative models of behavior, mirroring the advancements seen in recent instruction-tuned LLMs like Alpaca [58], without relying on a large dataset of instruction-labeled trajectories.

Text

Encoder

'chop a  tree'

CVAE

Video

encoder

CLIP   Objective

Transformer

(VPT)

MineCLIP

linear

Prior

Policy

ResNet

ResNet

ResNet

Mouse/Keyboard

//

In the past year, two foundation models for the popular open-ended video game Minecraft™ were released: a foundation model for behavior called VPT [5] and a model aligning text and video clips called MineCLIP [17]. This has opened up an intriguing avenue to explore fine-tuning for instruction-following in the sequential decision-making domain of Minecraft. VPT was trained on 70k hours of Minecraft gameplay, so the agent already has vast knowledge about the Minecraft environment. However, just as the massive potential of LLMs was unlocked by aligning them to follow instructions, it is likely that the VPT model has the potential for general, controllable behavior if it is finetuned to follow instructions. In particular, our paper demonstrates a method for fine-tuning VPT to follow short-horizon text instructions with only $60 of compute and around 2,000 instruction-labeled trajectory segments.

Our method draws inspiration from unCLIP [48], the approach used to create the popular text-toimage model DALL·E 2. We decompose the problem of creating an instruction-following Minecraft agent into two models: a VPT model finetuned to achieve visual goals embedded in the MineCLIP latent space, and a prior model that translates text instructions into MineCLIP visual embeddings. We finetune VPT using behavioral cloning with self-supervised data generated with hindsight relabeling [3], avoiding the use of expensive text-instruction labels in favor of visual MineCLIP embeddings. We apply unCLIP with classifier-free guidance [23] to create our agent called STEVE-1, which sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming the baseline set by Baker et al. [5].

Our main contributions are as follows:

We introduce a methodology, inspired by unCLIP [48], for instruction-tuning generative models of behavior without relying on a large dataset of expensive instruction labels.

We apply this methodology to create STEVE-1, a Minecraft agent that can follow shorthorizon open-ended text and visual instructions with a high degree of accuracy, all for only $60 of compute. We perform extensive evaluations of our agent, showing that it can robustly complete 12 of 13 goal-conditioned control tasks in our early-game evaluation suite in Minecraft. For long-horizon tasks 1 like crafting and building, we show that a basic version of prompt chaining can dramatically improve performance.

We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, data scaling, prompt-engineering, and other design choices. In particular, we show that unCLIP [48] and classifier-free guidance [23] translate well to sequential decision making and are essential for strong performance.

We release model weights for STEVE-1 as well as training scripts and evaluation code to help foster more research into instructable, open-ended sequential decision-making agents. 2

2 Related Work

Minecraft as a Test-bed for AI Minecraft has gained popularity as a benchmark for AI research due to its complex and dynamic environment, making it a rich test-bed for reinforcement learning and other AI methods (e.g., [26, 19, 17, 21, 40, 62, 38, 9]). We leverage the MineRL environment [19] to research the creation of agents that can follow open-ended instructions in complex visual environments using only low-level actions (mouse and keyboard). We build STEVE-1 on top of two recent foundation models. In order to align text and videos, we use MineCLIP [17], a CLIP [47] model trained on paired web videos of Minecraft gameplay and associated captions. To train STEVE-1's policy, we fine-tune VPT [5], a foundation model of Minecraft behavior that is pretrained on 70k hours of web videos of Minecraft along with estimated mouse and keyboard actions. Several prior works [61, 62] have explored the use of LLMs in creating instructable Minecraft agents. These works typically use LLMs to make high-level plans that are then executed by lower-level RL [40, 62] or scripted [46] policies. Since STEVE-1 is a far more flexible low-level policy, the combination of STEVE-1 with LLMs is a promising direction for future work. Fan et al. [17] introduced an agent trained using RL with MineCLIP as a shaping reward on 12 different tasks and conditioned on MineCLIP-embedded text-prompts. However, this agent failed to generalize beyond the original set of tasks without further RL finetuning using the MineCLIP reward function. Cai et al. [9] proposed a Goal-Sensitive Backbone architecture for goal-conditioned control in Minecraft which is trained on a fixed set of goals, while STEVE-1 learns goal-reaching behavior from a large dataset in a self-supervised way without training on an explicit set of tasks.

Foundation Models for Sequential Decision-Making Foundation models which are pretrained on vast amounts of data and then finetuned for specific tasks have recently shown great promise in a variety of domains including language [8, 14, 59], vision [48, 10, 47], and robotics [7, 53, 25, 39, 65]. GATO [49] and RT-1 [7] have demonstrated the potential of training transformers to perform both simulated and real-world robotic tasks. With the exception of Kumar et al. [30], which uses Qlearning, the vast majority of cases [32, 7, 49] where deep learning has been scaled to large, multitask offline-RL datasets have used supervised RL. Supervised RL (e.g., [42, 18, 12]) works by framing the sequential decision-making problem as a prediction problem, where the model is trained to predict the next action conditioned on some future outcome. While these approaches are simple and scale well with large amounts of compute and data, more work is needed to understand the trade-offs between supervised RL and Q-learning or policy gradient-based methods [43, 44, 6, 55]. Recent works explore the use of hindsight relabeling [3] using vision-language models [47, 2] to produce natural language relabeling instructions. DIAL [65] finetunes CLIP [47] on human-labeled trajectories, which is then used to select a hindsight instruction from a candidate set. Sumers et al. [56] uses Flamingo [2] zero-shot for hindsight relabeling by framing it as a visual-question answering (VQA) task. In contrast, STEVE-1 relabels goals using future trajectory segment embeddings given by the MineCLIP [17] visual embedding.

Text-Conditioned Generative Models There has been a recent explosion of interest in text-to-X models, including text-to-image (e.g., [48, 51, 50]), text-to-3D (e.g., [27, 35]), and even text-to-music (e.g., [1]). These models are typically either autoregressive transformers modeling sequences of discrete tokens [60, 8] or diffusion models [24]. Most related to our work is unCLIP, the method used for DALL·E 2 [48]. unCLIP works by training a generative diffusion model to sample images from CLIP [47] embeddings of those images. By combining this model with a prior that translates text to visual CLIP embeddings, unCLIP can produce photorealistic images for arbitrary text prompts. unCLIP and many other diffusion-based approaches utilize a technique called classifier-free guidance [23], which lets the model trade-off between mode-coverage and sample fidelity post-training. We utilize the basic procedure of unCLIP and classifier-free guidance for training STEVE-1.

3 Method

Inspired by the rapid recent progress in instruction-tuning Large Language Models (LLMs), we choose to leverage the recently released Video Pretraining (VPT) [5] model as a starting point for our agent. Since VPT was trained on 70k hours of Minecraft gameplay, the agent already has vast knowledge about the Minecraft environment. However, just as the massive potential of LLMs was unlocked by aligning them to follow instructions, it is likely that the VPT model has the potential

z

1

z

2

z

2

z

3

z

4

z

5

z

6

z

7

z

2

z

7

z

7

z

7

z

7

z

7

for general, controllable behavior if it is finetuned to follow instructions. In this work, we present a method for finetuning VPT to follow natural, open-ended textual and visual instructions, which opens the door for a wide range of uses for VPT in Minecraft.

Our approach is inspired by unCLIP, the method behind the recent text-to-image generation model, DALL·E 2 [48]. Our goal is to create a generative model of behavior in Minecraft conditioned on text instructions y . To do so, we utilize a dataset of Minecraft trajectory segments, some of which contain instruction labels y : [( τ 1 , y 1 ) , ( τ 2 , y 2 ) , . . . , ( τ n , ∅ )] where τ is a trajectory of observations and actions. We also employ a pretrained CLIP model called MineCLIP [17], which generates aligned latent variables z τ t : t +16 , z y , where z τ t : t +16 is an embedding of any 16 consecutive timesteps from the trajectory. MineCLIP is trained using a contrastive objective on pairs of Minecraft videos and transcripts from the web. For simplicity of notation, we refer to the MineCLIP embedding of the last 16 timesteps of a trajectory segment as z τ goal . Like unCLIP [48], we utilize a hierarchical model consisting of a prior and a policy:

A prior p ( z τ goal | y ) that produces a latent variable z τ goal conditioned on a text instruction y .

A policy p ( τ | z τ goal ) that produces a trajectory conditioned on a latent variable z τ goal .

These two models can then be combined to produce a generative model of behaviors conditioned on text instructions:

3.1 Policy

To learn our policy, we finetune VPT, a foundation model of Minecraft behaviors p θ ( τ ) trained on 70k hours of Minecraft gameplay videos. Specifically, VPT consists of a ResNet [22] that processes frames of dimension 128 × 128 × 3 , and a Transformer-XL [15] which processes the frame representations and autoregressively predicts the next action using the joint hierarchical action space described in Baker et al. [5]. In order to modify the architecture to condition on goal information, we add an affine transformation of z τ goal to the output of the ResNet before passing it to the transformer:

[+ Conditioning on MineCLIP Embedding Goal]:

TransformerXL θ ( x t , . . . , x t + T ) → a t + T

In order to finetune VPT to condition on goals, we finetune the model using a method inspired by supervised RL approaches like Decision Transformer [12], GLAMOR [42], and GCSL [18]. We use a modification of hindsight relabeling which we call packed hindsight relabeling (see Figure 2) to generate a new dataset of trajectories with goals pulled from future states that periodically switch. In contrast with hindsight relabeling, packed hindsight relabeling packs multiple relabeled sequences into a single sequence. Specifically, our method to generate this dataset consists of two steps:

Given a trajectory τ with T timesteps, randomly generate indices to select goals from: i 1 , i 2 , . . . , i n . These indices are chosen by starting at the first timestep and repeatedly sampling a new timestep by adding a random value to the previous timestep. This ensures that the data reflects that some goals may take longer to achieve than others.

For each chosen goal at timestep i j , set the goals for timesteps i j -1 +1 , . . . , i j to be the goal at timestep i j , denoted z τ i j .

Our final dataset D relabeled consists of observation sequences ( o 1 , . . . , o T ) , action sequences ( a 1 , . . . , a T ) , and packed hindsight relabeled goals ( z 1 , . . . , z T ) . We then finetune VPT on this dataset using a supervised loss to predict each action autoregressively using a causal attention mask:

3.2 Prior

In order to condition not only on embeddings of visual goals but on latent goals, we need the prior, a model that produces a latent variable z τ goal conditioned on a text instruction y . Our model is a simple conditional variational autoencoder (CVAE) [54, 29] with a Gaussian prior and a Gaussian posterior. Rather than learn to condition directly on text, we choose to condition on frozen text representations from MineCLIP z y . Thus, the prior learns a function to translate from a text embedding z y to a visual embedding z τ goal (see Appendix C.5). Both the encoder and decoder of our CVAE are parameterized as two-layer MLPs with 512 hidden units and layer normalization [4]. We train the model on our dataset, for which we have text labels D labels using the following loss:

3.3 Datasets

To train our policy, we gather a gameplay dataset with 54M frames ( ≈ 1 month at 20FPS) of Minecraft gameplay along with associated actions from two sources: contractor gameplay and VPT-generated gameplay. To train our prior, we use a small dataset of text-video pairs gathered by humans and augmented using the OpenAI API gpt-3.5-turbo model [41] and MineCLIP. See Appendix D for more detailed dataset information.

OpenAI Contractor Dataset We use 39M frames sourced from the contractor dataset which VPT [5] used to train its inverse dynamics model and finetune its policy. The dataset was gathered by hiring human contractors to play Minecraft and complete tasks such as house building or obtaining a diamond pickaxe. During gameplay, keypresses and mouse movements are recorded. We use the same preprocessing as VPT, including filtering out null actions.

VPT-Generated Dataset We generate an additional dataset of 15M frames by generating random trajectories using the various pretrained VPT agents. The diversity of this dataset is improved by randomly switching between models during trajectories [44], randomly resetting the agent's memory, and randomly turning the agent to face a new direction.

Text-Video Pair Dataset To train our prior model, which learns a mapping between text embeddings and visual embeddings, we also manually gather a small dataset of 2,000 text instructions paired with 16-frame video segments (less than a second) from our gameplay dataset. This dataset corresponds to less than 30 minutes of gameplay and takes just a few hours to collect. We augment this dataset by using the alignment between text and video embeddings from MineCLIP. For each text instruction, we find the top k most similar gameplay segments in our dataset and use the corresponding 16-frame segment as additional training data. For augmentation, we also add 8,000 text-instructions generated by the OpenAI API gpt-3.5-turbo model [41], in addition to our 2,000 hand-labeled instructions.

3.4 Inference

At inference time, we use the prior to sample a latent goal z τ goal from the text instruction y . We then use the policy to autoregressively sample actions a t conditioned on the observation history o 1 ...t and the latent goal z τ goal . Similar to the observation in Appendix I of Baker et al. [5], even with conditioning the policy often fails to follow its instruction and simply acts according to its prior behavior. To mitigate this, we borrow another trick used in image generation models: classifier-free guidance. Specifically, during inference we simultaneously compute logits for the policy conditioned on the goal f ( o t , . . . , o t +1 , z τ goal ) and for the unconditional policy f ( o t , . . . , o t +1 ) . We then compute a combination of the two logits using a λ parameter to trade-off between the two:

By setting a higher value of λ , we can encourage the policy to follow actions that are more likely when conditioned on the goal and, as demonstrated in Section 4.5, this significantly improves performance. Also, in order to train the policy to generate these unconditional logits, we occasionally dropout the goal embedding z τ goal from the policy's input (with probability 0.1). This lets us generate both the conditional and unconditional logits using the same model with batch processing at inference time.

3.5 Evaluation

Evaluating the performance of our agent is a challenging task due to the wide variety of instructions that are possible and the difficulty of evaluating whether the agent has successfully achieved its task. We use a combination of programmatic evaluation metrics and automatic MineCLIP evaluation metrics to get a sense of the agent's capability level. We collectively refer to all of our evaluation tasks including the 11 evaluation tasks from Figure 3 and the two prompt chaining tasks from Section 4.3 as our early-game evaluation suite .

Programmatic Evaluation We compute programmatic evaluation metrics by monitoring the MineRL [19] environment state throughout each evaluation episode. As done in VPT [5], we compute multiple programmatic metrics including travel distance and early-game item collection. The travel distance is the maximum displacement of the agent along on the horizontal (X-Z) plane, measured from the initial spawn point. For early-game inventory counts, we store the maximum number of log, seed, and dirt items seen in the agent's inventory during the episode.

MineCLIP Evaluation We explore the use of text-visual alignment in MineCLIP latent space between trajectories and text or visual goals to evaluate our agent over a wider variety of tasks where programmatic evaluation isn't practical. To determine the degree to which a task has been completed at all during an evaluation episode, we record the minimum cosine distance between the (text or visual) goal embedding and the visual MineCLIP embedding at any timestep during an episode.

4 Results

In our experiments, we aim to answer the following questions:

How well does STEVE-1 perform at achieving both text and visual goals in Minecraft?

How does our method scale with more data?

What choices are important for the performance of our method?

4.1 Training Setup

We base our implementation off of the official VPT codebase 3 . The main STEVE-1 is trained using Pytorch [45] distributed data parallel on four A40 GPUs for 160M frames, or just under three epochs of our gameplay dataset. Hyperparameters are selected to match those in Baker et al. [5] with the exception of learning rate, which we set to 4e-5. Our models are optimized using AdamW [37]. See Table 1 for a full list of hyperparameters.

4.2 Performance on Textual and Visual Goals

Due to the hierarchical nature of our model, we can evaluate the performance of our agent at achieving either text or visual goals simply by choosing whether to use the prior to condition on text or bypass the prior and condition on a MineCLIP video embedding directly. We first tested our model on a set of 11 tasks that are achievable within the first 2.5 minutes of gameplay and which do not require multiple steps to complete (e.g., chop a tree or dig a hole, but not build a house). A complete list of the tasks and prompts we used for evaluation can be found in Table 3 in the appendix. To select visual goals for testing each of the evaluation tasks, we implemented a tool that searches through

VPT

VPT

(Text)*

STEVE-1

(Text)

STEVE-1

(Visual)

0

100

200

300

400

Travel Distance (Blocks)

VPT

VPT

(Text)*

STEVE-1

(Text)

STEVE-1

(Visual)

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

Seeds Collected

VPT

VPT

(Text)*

STEVE-1

(Text)

STEVE-1

(Visual)

0

5

10

15

20

VPT

VPT

(Text)*

STEVE-1

(Text)

STEVE-1

(Visual)

0

20

40

60

(a) Programmatic Evaluation

dig

dirt

sky

leaves

wood

seeds

flower

explore

swim

underwater

inventory

Conditioned Prompt

dig

dirt

sky

leaves

wood

seeds

flower

explore

swim

underwater

inventory

Evaluation Prompt

1

0

0

0

0

0

0

0

0

1

0

0.02

0.04

0.06

0.08

0.10

0.12

(b) MineCLIP Evaluation

10% of our gameplay dataset by finding the closest 16-frame videos to a given text prompt. We then manually selected a 16-frame video that clearly demonstrates the task being completed and use the corresponding MineCLIP video embedding as the goal embedding for that task. Screenshots of these visual goals can be found in Figure 14 in the appendix.

In Figure 3, we compare the performance of our text and visual-conditioned agents with the unconditional VPT agent and text-conditioned VPT agent (Appendix I in [5]) across our programmatic tasks. We find that when given the relevant text instruction, STEVE-1 collects 75 × more dirt, 4 . 9 × more wood, 22 × more seeds, and travels 4 . 3 × farther than the unconditional agent, and STEVE-1 collects 3 . 3 × more dirt, 4 . 4 × more wood, 8 . 1 × more seeds, and travels 2 . 2 × farther than the textconditioned VPT agent. This represents a significant improvement over the reported performance of text-conditioned VPT, which collects several times fewer resources despite having twice as long of an episode to do so. We also run an automatic evaluation using MineCLIP embedding distances by measuring the minimum distance of a goal embedding to any frame in the episode. As shown in Figure 3b, the distance between the goal and the episode is significantly lower when the agent is conditioned on the corresponding visual goal than otherwise. Full results for STEVE-1 with both text and visual goals can be found in Appendix F.

In addition to our evaluations of STEVE-1, we also recorded several sample interactive sessions we had with the agent (controlling it in real-time by giving it written text instructions or specific visual goals). These sessions demonstrate STEVE-1's ability to responsively follow instructions in real-time in a variety of situations. We believe that such use-cases, where humans give an agent natural instructions that it can follow to complete tasks, will become increasingly important and have practical uses in the creation of instructable assistants and virtual-world characters. These videos, as well as videos of our agent performing our evaluation tasks, can be found at https://sites.google.com/view/steve-1.

4.3 Prompt Chaining

We also experiment with longer horizon tasks that require multiple steps, such as crafting and building. We explore two different prompting methods: directly prompting with the target goal, and a simple form of prompt chaining [11, 64, 16] where the task is decomposed into several subtasks and the

0

1000

2000

3000

Episode Timestep

0

5

10

15

20

25

30

Dirt in Inventory

0

1000

2000

3000

Episode Timestep

0

2

4

6

8

Logs in Inventory

20M

40M

60M

80M

100M

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Build Tower Success

Prompt Chaining

Direct Prompting

20M

40M

60M

80M

100M

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Crafting Success

55

60

65

70

75

80

85

Agent Height Position

Build a Tower

0

2

4

6

8

10

12

14

Planks in Inventory

Make Wooden Planks

Number of Frames

100

200

300

Travel Distance (Blocks)

0

2

4

6

Seeds Collected

relevant

irrelevant

20M

40M

60M

80M

100M

0

5

10

15

20

20M

40M

60M

80M

100M

0

20

40

60

Number of Frames

prompts are given sequentially for a fixed number of steps. We explore prompt chaining with visual goals for two tasks: 1) building a tower and 2) making wooden planks. When using prompt chaining, we first prompt STEVE-1 to gather dirt before building a tower, and to gather wooden logs before crafting wooden planks. Figure 4 shows that directly prompting STEVE-1 with the final tasks results in near-zero success rates. However, prompt chaining allows STEVE-1 to build a tower 50% of the time and craft wooden planks 70% of the time. For the tower building task, STEVE-1 immediately starts collecting dirt until the prompt switches, at which point its average height starts increasing rapidly and its dirt decreases as it builds a tower. Similarly, for the crafting wooden planks task, STEVE-1 immediately starts collecting a large amount of wooden logs until the prompt switches and it rapidly converts these wooden logs into wooden planks (causing the amount of wooden logs in its inventory to immediately decrease and the number of wooden planks to increase as it crafts more). Figure 4 visualizes the average item counts and agent height for the prompt chaining episodes. See Figure 18 and Figure 19 in the appendix for visualizations of specific prompt chaining episodes.

4.4 Scaling

Recent works in language modeling have found that scaling up pretraining FLOPs, by training on more data or by training a model with more parameters, can improve performance on downstream tasks [28, 57, 63]. In certain cases when measuring performance with metrics such as exact-match [52], performance improvement may appear to be 'emergent' [63], appearing suddenly as the model is trained with more compute. Here, we aim to gain a basic understanding of how the performance of STEVE-1 on various tasks scales by training with more data (learning rate schedule is chosen appropriately).

To assess performance gain, we first isolated the performance of the policy from the prior, measuring performance of the agent through training on programmatic tasks (travel distance, seeds, logs, dirt) with visual goals. Due to compute constraints, we chose to use the 2x VPT model, which has 248M parameters. We found that both seed collection and travel distance did not improve significantly past 20M frames. From inspecting gameplay, we suspect that travel distance is a relatively easy task since it is close to VPT's default behavior of running around and exploring. For seed collection, performance remains suboptimal, suggesting that further scaling may be beneficial. This hypothesis is supported by the observation that performance on log and dirt collection remained roughly level until 60M frames when it began to rapidly improve. Figure 4 shows the scaling curves for STEVE-1 on each programmatic task when conditioning on relevant vs. irrelevant visual prompts for that task. Since we do not observe regression on any tasks as we train the model with more compute, we expect the model to continue to perform better as we train larger models on larger datasets.

We also evaluated the scaling properties of STEVE-1 for our multi-step tasks with and without prompt chaining. Without prompt chaining, the tasks remain challenging for STEVE-1 throughout training. However, we note that after 60M frames, STEVE-1 learns to gather wooden logs and build a small tower when told to build a tower. This is likely because our visual prompt for tower building shows a video of a tower being built out of wooden logs. With prompt chaining, the performance of STEVE-1 steadily increases with more data. We conjecture that this is because the success of a chained prompt requires the success of each element in the chain. Since different abilities emerge at different scales, one would expect chained prompts to steadily get more reliable as these subgoals become more reliably completed. In the case of building wooden planks, we note that crafting is one such task that gets significantly more reliable as the agent is trained on more data. Figure 4 shows the scaling curves for STEVE-1 on the prompt chaining tasks.

In summary, we see evidence of tasks that do not require much data for STEVE-1 to learn, tasks that steadily get more reliable as the agent is trained longer, and tasks where capability suddenly spikes after the agent reaches some threshold. Put together, this suggests that further scaling would likely significantly improve the agent, although we leave the task of predicting exactly how much performance there is to gain to future studies.

4.5 What Matters for Downstream Performance?

Pretraining Baker et al. [5] finds that by pretraining a behavioral prior with imitation learning on internet-scale datasets for Minecraft, the learned policy can be effectively finetuned to accomplish tasks that are impossible without pretraining. In this section, we demonstrate that pretraining is also massively beneficial for instruction-tuning in Minecraft. We hypothesize that due to the strong performance of STEVE-1 and the relatively small amount of compute ( ≈ 1% additional compute) used for instruction finetuning, most of the capabilities of our agent come from the pretraining rather than the finetuning. To test this hypothesis, we finetune several varients of STEVE-1 from various pretrained weights: foundation-2x , bc-early-game-2x , rl-from-foundation-2x , and with randomly initialized weights. In this experiment, each model was finetuned on 100M frames.

Figure 5 shows the performance of these models on our programmatic tasks with visual goals. Note that while an agent trained on our dataset from scratch can accomplish basic tasks like dirt collection fairly well, it is unable to find and chop down trees, in contrast to the pretrained agents. This demonstrates that the abilities present in the agent due to pretraining are successfully transferred to the finetuned agent. Out of all the pretrained weights we tried, we noticed that rl-from-foundation-2x performed the best, having qualitatively better performance at tasks like crafting and chopping down trees. Indeed, Figure 5 shows that this model has strong performance, likely due to the massive amount of compute it was trained with during its RL training [5].

Classifier-Free Guidance Baker et al. [5] observed that when conditioning the agent on text, it tended to ignore its instruction and instead perform the prior behavior learned during pretraining. As discussed in section 3.4, classifier-free guidance gives a knob for trading off between goal-conditioned and prior behaviors. Figure 5 shows the effect of this parameter λ on the log and dirt collection tasks. The performance of the agent reaches its maximum around λ = 5 . 0 to λ = 7 . 0 , after which it starts to drop off. These results demonstrate the importance of classifier-free guidance, which improves the performance of STEVE-1 by orders of magnitude.

Prompt Engineering Prompt engineering as a discipline has rapidly emerged over the last year due to the observation that the quality of the output of text-to-X models can dramatically change depending on the prompt [67]. For example, Table 5 in the appendix shows how a prompt for Stable Diffusion [50] might be written. By listing out the various attributes of the image such as visual medium, style, and the phrase 'trending on ArtStation', the user is able to get a higher quality image [20, 36]. In this section, we explore how this same style of prompt engineering can improve the performance of STEVE-1. Figure 6 shows how a simple prompt of 'get dirt' might be changed in order to more accurately specify the type of behavior that is desired. Just like in image generation models, the performance of STEVE-1 significantly improves by modifying the prompt in this fashion. By changing to more complicated prompts, STEVE-1 is able to collect 1 . 6 × more wood, 2 × more dirt, and 3 . 3 × more seeds.

scratch

fd

bc

rl

0

5

10

scratch

fd

bc

rl

0

25

50

75

0

2

4

6

8

10

0

10

20

0

2

4

6

8

10

20

40

60

Conditional Scale

λ

Pretrained Weights

5 Limitations and Conclusion

In this paper, we present a methodology for creating instruction-following foundation models of behavior. Specifically, by leveraging two existing pretrained foundation models: a behavioral prior (VPT [5]) and a domain-specific CLIP model (MineCLIP [17]), we create a powerful Minecraft agent that can follow short-horizon open-ended text and visual instructions, all for only $60 of compute. The resulting foundation model, STEVE-1, sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines and robustly completing 12 of 13 tasks in our early-game evaluation suite. We note that generalist agents such as STEVE-1 can have potential negative effects on society. We include a thorough discussion of these issues in Appendix A.

STEVE-1 is a significant advancement in creating generative models of text-to-behavior, but it has several limitations, as described in Appendix B. First, STEVE-1 is mostly proficient at achieving short-horizon tasks while struggling with longer-horizon tasks. While prompt chaining is a promising approach for improving performance on complex tasks, more can be done in future work to improve performance. Another limitation we observe is that prompt engineering, as with other generative models, can be unintuitive and time-consuming. Future work should investigate improving the steerability of STEVE-1 through a better understanding of natural language prompts. Additionally, we note that evaluating and describing the capabilities of open-ended generalist agents is an open research problem itself since capability depends strongly on preconditions, prompt engineering, and our own ability to come up with varied and challenging tasks. Finally, since our approach is not specific to the Minecraft domain, we hope that the method used to create STEVE-1 can inspire future work in creating powerful generalist agents in other domains and environments.

References

Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse H. Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matthew Sharifi, Neil Zeghidour, and Christian Havnø Frank. Musiclm: Generating music from text. CoRR , abs/2301.11325, 2023. doi: 10.48550/arXiv.2301.11325. URL https://doi.org/10.48550/ arXiv.2301.11325 .

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35: 23716-23736, 2022.

Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5048-5058, 2017. URL https://proceedings.neurips.cc/ paper/2017/hash/453fadbd8a1a3af50a9df4df899537b5-Abstract.html .

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems , 35:24639-24654, 2022.

David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? arXiv preprint arXiv:2206.01079 , 2022.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .

Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. arXiv preprint arXiv:2301.10034 , 2023.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021.

Harrison Chase. Langchain, 2022. URL https://github.com/hwchase17/langchain .

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/ .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR , abs/2204.02311, 2022. doi: 10.48550/arXiv.2204.02311. URL https://doi.org/10.48550/arXiv.2204.02311 .

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pages 2978-2988. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1285. URL https://doi.org/10.18653/v1/p19-1285 .

David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-dickstein, Kevin Murphy, and Charles Sutton. Language model cascades, 2022.

Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 18343-18362. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 74a67268c5cc5910f64938cac4526a90-Paper-Datasets_and_Benchmarks.pdf .

Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations , 2021. URL https://openreview. net/forum?id=rALA0Xo6yNJ .

William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440 , 2019.

Gustavosta. Stable-diffusion-prompts. Hugging Face, 2023. URL https://huggingface. co/datasets/Gustavosta/Stable-Diffusion-Prompts . Hugging Face Datasets.

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.90. URL https://doi.org/10.1109/CVPR.2016.90 .

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR , abs/2207.12598, 2022. doi: 10.48550/arXiv.2207.12598. URL https://doi.org/10.48550/arXiv.2207.12598 .

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HsuanTien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html .

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094 , 2022.

Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In Ijcai , pages 4246-4247, 2016.

Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. CoRR , abs/2305.02463, 2023. doi: 10.48550/arXiv.2305.02463. URL https://doi.org/10.48550/ arXiv.2305.02463 .

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR , abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361 .

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014. URL http://arxiv.org/abs/1312.6114 .

Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=4-k7kUavAj .

Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 12004-12019. PMLR, 17-23 Jul 2022. URL https: //proceedings.mlr.press/v162/langosco22a.html .

Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances in Neural Information Processing Systems , 35:27921-27936, 2022.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval , 2023.

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. CoRR , abs/2211.09110, 2022. doi: 10.48550/ARXIV.2211.09110. URL https://doi.org/10. 48550/arXiv.2211.09110 .

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Vivian Liu and Lydia B. Chilton. Design guidelines for prompt engineering text-to-image generative models. In Simone D. J. Barbosa, Cliff Lampe, Caroline Appert, David A. Shamma, Steven Mark Drucker, Julie R. Williamson, and Koji Yatani, editors, CHI '22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 - 5 May 2022 , pages 384:1-384:23. ACM, 2022. doi: 10.1145/3491102.3501825. URL https://doi.org/ 10.1145/3491102.3501825 .

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7 .

Federico Malato, Florian Leopold, Amogh Raut, Ville Hautamäki, and Andrew Melnik. Behavioral cloning via search in video pretraining latent space. arXiv preprint arXiv:2212.13326 , 2022.

Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 , 2022.

Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep?: Embodied decision making using language guided world modelling. arXiv preprint arXiv:2301.12050 , 2023.

OpenAI. Introducing ChatGPT, Nov 2022. URL https://openai.com/blog/chatgpt .

Keiran Paster, Sheila A McIlraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv preprint arXiv:2012.02419 , 2020.

Keiran Paster, Sheila A. McIlraith, and Jimmy Ba. You can't count on luck: Why decision transformers and rvs fail in stochastic environments. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=atb3yifRtX .

Keiran Paster, Silviu Pitis, Sheila A. McIlraith, and Jimmy Ba. Return augmentation gives supervised RL temporal compositionality. In NeurIPS 2022 Foundation Models for Decision Making Workshop , 2022. URL https://openreview.net/forum?id=q5olkWCt7nl .

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 8024-8035, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ bdbca288fee7f92f2bfa9f7012727740-Abstract.html .

PrismarineJS and Others. Mineflayer. https://github.com/PrismarineJS/mineflayer , 2023. URL https://github.com/PrismarineJS/mineflayer . GitHub repository.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ ec795aeadae0b7d230fa35cbaf04c041-Abstract-Conference.html .

Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? CoRR , abs/2304.15004, 2023. doi: 10.48550/arXiv.2304.15004. URL https://doi.org/10.48550/arXiv.2304.15004 .

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning , pages 894-906. PMLR, 2022.

Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 3483-3491, 2015. URL https://proceedings.neurips.cc/paper/2015/hash/ 8d55a249e6baa5c06772297520da2051-Abstract.html .

Miroslav Strupl, Francesco Faccio, Dylan R. Ashley, Jürgen Schmidhuber, and Rupesh Kumar Srivastava. Upside-down reinforcement learning can diverge in stochastic environments with episodic resets. CoRR , abs/2205.06595, 2022. doi: 10.48550/arXiv.2205.06595. URL https: //doi.org/10.48550/arXiv.2205.06595 .

Theodore Sumers, Kenneth Marino, Arun Ahuja, Rob Fergus, and Ishita Dasgupta. Distilling internet-scale vision-language models into embodied agents. arXiv preprint arXiv:2301.12507 , 2023.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. CoRR , abs/2210.09261, 2022. doi: 10.48550/arXiv.2210.09261. URL https://doi.org/10.48550/arXiv.2210.09261 .

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR , abs/2302.13971, 2023. doi: 10.48550/arXiv.2302.13971. URL https://doi.org/10.48550/arXiv.2302.13971 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html .

Ryan Volum, Sudha Rao, Michael Xu, Gabriel DesGarennes, Chris Brockett, Benjamin Van Durme, Olivia Deng, Akanksha Malhotra, and Bill Dolan. Craft an iron sword: Dynamically generating interactive game characters by prompting large language models tuned on code. In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022) , pages 25-43, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.wordplay-1.3. URL https://aclanthology.org/2022. wordplay-1.3 .

Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560 , 2023.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. CoRR , abs/2206.07682, 2022. doi: 10.48550/arXiv.2206.07682. URL https://doi. org/10.48550/arXiv.2206.07682 .

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/ forum?id=_VjQlMeSB_J .

Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint arXiv:2211.11736 , 2022.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview. net/forum?id=92gvk82DE-.

A Broader Impact

With the increasing capability level of artificial intelligence comes many potential benefits and also risks. On the positive side, we anticipate that the techniques that used to create STEVE-1 could be applied to the creation of helpful agents in other sequential decision making domains, including robotics, video games, and the web. Our demonstration of such a low cost approach to creating a powerful, instruction-following model also has the potential to improve the democratization of AI. However, on the negative side, agents pretrained on large internet datasets reflect the biases of the internet and, as suggested by our experiments, these pretraining biases can potentially remain after instruction-tuning. If not addressed carefully, this could lead to devastating consequences for society. We hope that while the stakes are low, works such as ours can improve access to safety research on instruction-following models in sequential decision-making domains.

B Limitations and Future Work

B.1 Goal Misgeneralization

One of the most common mistakes that STEVE-1 makes during evaluation is to overgeneralize. For instance, if we prompt STEVE-1 with a video of someone punching a cow, it may simply run to the nearest animal and punch that animal instead. This is related to the concept of goal misgeneralization [31]. Generalization can be helpful when the task we assign the agent is impossible to achieve from the current state and the agent instead performs a closely related action, but harmful when the task is achievable. We note two things: first, we believe the powerful generalization ability of STEVE-1 probably comes from the MineCLIP embeddings and it especially improves the ability of STEVE-1 to follow visual instructions when the exact items or blocks nearby are not available in the current environment, which is an extremely common scenario. Second, we notice that the tendency of the agent to misgeneralize decreases with scale. For example, with a model trained on less data, we find that asking the agent to look up and punch a tree to get a wooden log often resulted in the agent looking in the air and punching nothing; training the model on more data results in the agent first walking over to a nearby tree and looking up to get a wooden log. Future work should look to measure more closely the relationship between misgeneralization and scale.

B.2 Random Selection of Hindsight Goals

In this work, we choose to randomly select future episode timesteps as hindsight goals, rather than use a more sophisticated strategy. This is primarily due to the simplicity of the approach, but also to ensure a diverse and unbiased coverage of potential goals achievable within the short horizon. Future works can investigate the effects of alternative approaches that filter for semantically interesting timesteps as goals.

B.3 Difference in Performance Between Text and Visual Goals

In our experiments, we observed that STEVE-1 often performed better when conditioned with visual goals compared to text goals converted through the prior. There are several potential factors that could contribute to this performance gap between text and visual conditioning. First, the prior model may not be accurately capturing the full meaning of the text prompt. Training the prior on more data or using a more powerful model architecture could potentially improve the quality of the sampled latent goals. Second, the visual goals can provide more precise demonstrations of the desired behavior. Text goals are inherently more ambiguous. Providing additional information such as the observation context to the prior and further prompt-engineering to make text prompts more detailed and less ambiguous, could help close this gap. Because text conditioning provides more flexibility and potential for generalization, closing the performance gap between text and visual conditioning is an important direction for future work.

B.4 Challenges in Evaluating STEVE-1

See Table 4 for a non-exhaustive list of tasks that the STEVE-1 agent is able to achieve. It is worth mentioning that evaluating and describing the capabilities of open-ended generalist agents is an open

research problem itself since capability depends strongly on preconditions, prompt engineering, and our own ability to come up with varied and challenging tasks. For example, there are many recent works on the evaluation of LLMs (e.g., [34, 33, 66]) which highlight these challenges.

That being said, there are a number of tasks which STEVE-1 is unable to accomplish. As previously mentioned, long-horizon tasks such as obtaining a diamond or building a house are currently beyond the capability of STEVE-1. Further, STEVE-1 also struggles with more complex crafting tasks like crafting an enchanting table, bookcase, or boat. Again, the virtually limitless and open-ended nature of tasks in Minecraft makes it very difficult to test generalist agents in this domain. We hope future works develop more sophisticated methods to evaluate the performance of generalist agents on short and long-horizon tasks (potentially through an extension of our MineCLIP evaluation method).

It is also worth noting that it is currently not possible to test STEVE-1 or VPT [5] in the MineDojo [17] environment, which is meant for generalist agent evaluation, since the action spaces are not compatible. We believe that bridging this gap could be greatly beneficial to the generalist agent community and we hope future works investigate this further.

B.5 Towards Improved Long-Horizon Performance

STEVE-1 is a significant advancement in creating generative models of text-to-behavior, but it has several limitations. First, STEVE-1 is mostly proficient at achieving short-horizon tasks while struggling on longer-horizon tasks like obtaining a diamond. Solving long-horizon tasks while taking actions using low-level mouse/keyboard controls is a very challenging and exciting research direction and, while prompt chaining is a promising approach for improving performance on complex tasks, more can be done in future work to improve performance.

One potential bottleneck is the fact that during packed hindsight relabeling, the hindsight goals are limited to at most 200 timesteps in the future ( ∼ 10 seconds). Thus, tasks which require more than 200 timesteps to complete are technically out-of-distribution for STEVE-1. Although sampling hindsight goals from farther into the future could theoretically enhance long-horizon performance, our experiments in Appendix C.4 indicate that the performance tends to decrease if we increase this hyperparameter too much. We suspect that while increasing this hyperparameter may be able to improve long-horizon performance, it also increases noise and comes at the cost of reducing performance on short-horizon goals. Investigating whether it is possible to achieve a better tradeoff is an important avenue for future work. We also suspect that the long-horizon capabilities of STEVE-1 could be improved through scaling or finetuning with reinforcement learning, or leveraging LLMs or VLMs to automatically provide prompt chains to the STEVE-1 agent.

B.6 Applying the STEVE-1 Approach to Other Domains

We designed STEVE-1 for Minecraft due to the availability of two key ingredients: (1) a strong behavioral prior (VPT [5]), and (2) a powerful visual-language model which maps text and video to a joint embedding space (MineCLIP [17]). However, the method used to create STEVE-1 is not specific to the Minecraft domain. Given the rapid development of generative models, we expect that similar models to VPT and MineCLIP will become available in many other domains. As these models become available, future work could investigate the applicability of the STEVE-1 approach to these other domains.

C Additional Ablations

In this section, we describe additional ablations on design choices for our method, including the use of classifier-free guidance during training, text augmentation strategies, different V AE variants, and varying chunk sizes during finetuning. We use programmatic evaluation metrics to compare the performance of the various ablations.

C.1 Classifier-Free Guidance During Training

We examine the importance of using classifier-free guidance during training by finetuning a model with no guidance which does not drop out the goal embedding z τ goal from the policy's input (i.e., p uncond = 0 . 0 ) and comparing it to the version which uses guidance ( p uncond = 0 . 1 ). The chunk

no guidance (

λ

= 0)

guidance (

λ

= 0)

guidance (

λ

= 3)

0

100

200

300

Travel Distance (Blocks)

no guidance (

λ

= 0)

guidance (

λ

= 0)

guidance (

λ

= 3)

0

1

2

3

Seeds Collected

no guidance (

λ

= 0)

guidance (

λ

= 0)

guidance (

λ

= 3)

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

no guidance (

λ

= 0)

guidance (

λ

= 0)

guidance (

λ

= 3)

0

20

40

60

size is set to the range 15 to 50 and we train each policy for 100M frames. In Figure 7, we compare the performance of using visual goals (MineCLIP video embedding) on the no guidance model using conditional scale λ = 0 and the guidance model using conditional scales λ = 0 and λ = 3 . We observe that while the no guidance model slightly outperforms the guidance model at λ = 0 across a few metrics, the agent with guidance outperforms the no guidance agent by a factor of 2 to 3 times for the inventory collection tasks when we increase the conditional scale to λ = 3 (which we cannot do for the no guidance model). For the travel distance metric, both of the guidance versions perform similarly to the no guidance version.

C.2 Text Augmentation

During finetuning, instead of using only self-supervision with future MineCLIP video embedding as the goal, we considered using the text embeddings from the 2,000 human labeled trajectory segments as goal embeddings, either solely or in addition to the self-supervised video embeddings. In order to more fairly compare with the CVAE prior approach, we augment the human-labeled data with additional text-gameplay pairs generated as described in Appendix E.2. We implement this experiment by replacing the visual embeddings used for relabeling in Algorithm 1 with text embeddings, when available, with a 90% probability. To experiment with not using visual embeddings at all, we can replace the visual embeddings with zeros in the same way. In Figure 8, we observe that using only the visual embeddings during training, in combination with the CVAE, can outperform using MineCLIP text embeddings directly in the other two baselines. In this experiment, the chunk size is set to the range 15 to 50 and we train each policy for 100M frames.

C.3 VAE Variants

We study the dataset used to train the CV AE prior model. In Figure 9, we observe that augmentation helps in some programmatic tasks, including the dirt and seed collection tasks, but slightly hurts the wooden log collection and travel distance metrics. In this experiment, we use the same policy with each CVAE variant and we tune the conditional scale λ for each variant. The chunk size is set to the range 15 to 200 and we train the policy for 100M frames.

text (raw text)

text + visual (raw text)

visual (text VAE)

0

100

200

300

Travel Distance (Blocks)

text (raw text)

text + visual (raw text)

visual (text VAE)

0

2

4

6

Seeds Collected

text (raw text)

text + visual (raw text)

visual (text VAE)

0

2

4

6

8

10

text (raw text)

text + visual (raw text)

visual (text VAE)

0

2

4

6

8

10

Human

Human + Aug

0

100

200

300

Travel Distance (Blocks)

Human

Human + Aug

0

5

10

15

Seeds Collected

Human

Human + Aug

0

5

10

15

20

Human

Human + Aug

0

10

20

30

C.4 Chunk Size

During finetuning, we compare different goal chunk sizes by varying the max_btwn_goals=[100,200,300,400] , while keeping the min_btwn_goals=15 . See Algorithm 1 for more details. A larger max_btwn_goals introduces more noise, with actions that led to achieving the further away goal being less correlated to the actions present in that goal chunk. In Figure 10, we observe that the best max_btwn_goals chunk size is around 200, and increasing the chunk size beyond that causes a drop in performance. We train each policy for 160M frames and tune the conditional scale for each.

100

200

300

400

150

200

250

300

350

Travel Distance (Blocks)

100

200

300

400

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

Seeds Collected

100

200

300

400

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

20

.

0

22

.

5

100

200

300

400

20

30

40

50

60

70

noDirtorDig

normal

0

100

200

300

Travel Distance (Blocks)

noDirtorDig

normal

0

2

4

6

Seeds Collected

noDirtorDig

normal

0

5

10

15

noDirtorDig

normal

0

10

20

30

40

50

C.5 Generalization to Novel Text Instructions

We train the prior on a dataset of human and GPT-generated instructions designed to be representative of the tasks that appear in our gameplay dataset. Here, we have performed a set of simple generalization experiments to measure the degree to which the prior can generalize to unseen instructions.

Instruction Training Set Contamination The 'Text Prompt' column in Table 3 shows which of the prompts used in evaluation show up in our training dataset. Among our evaluation instructions, the bolded and italicized instructions show up in the instruction-trajectory dataset. While some instructions do show up, most of the instructions do not show up in our training set (verbatim).

Training Set Decontamination To measure the effect on performance of removing a concept from the training set, we ran an experiment where we removed every instruction with the words 'dirt' or 'dig' in them and retrained the VAE model. This corresponds to around 10% of the instructions. As shown in Figure 11, we found that even without training on the concept of dirt or digging at all, STEVE-1 can still be instructed to dig holes and get dirt. This demonstrates clearly that STEVE-1 can generalize to unseen text instructions - likely because most of the text-understanding comes from the pretrained MineCLIP model which was trained on a highly diverse dataset of YouTube videos and captions. The prior VAE only needs to learn a mapping between the text and visual MineCLIP embeddings. Note that there is a slight decrease in performance across all tasks likely due to the smaller VAE training set ( ∼ 10% less).

The instruction-following capability of STEVE-1 is shared between: the policy, which learns to follow instructions in the visual MineCLIP embedding space; the MineCLIP text-encoder, which is trained to align well with the visual embeddings and performs most of the text-understanding; and our prior VAE model, which learns a simple function to translate between text and visual embeddings. This modeling setup lets us fully exploit pretrained models such as MineCLIP to gain impressive language understanding without relying on having our own large datasets or compute.

D Dataset Details

D.1 Gameplay Dataset

Our gameplay dataset consists of two types of episodes: 7,854 episodes (38.94M frames) of a contractor dataset made available from Baker et al. [5] and 2,267 episodes (14.96M frames) of gameplay generated by running various pretrained VPT agents.

OpenAI Contractor Dataset The majority of our data comes from the contractor data used to train VPT [5]. OpenAI released five subsets of contractor data: 6.x, 7.x, 8.x, 9.x, and 10.x. We use an equal

mix of 8.x, 9.x, and 10.x, which correspond to 'house building from scratch', 'house building from random starting materials', and 'obtain diamond pickaxe'. Contractors were given anywhere from 10 to 20 minutes to accomplish these goals to the best of their abilities while their screen, mouse, and keyboard were recorded.

VPT-Generated Dataset We generated additional data by generating episodes using various pretrained VPT agents. In order to increase the diversity of data as well as to get data of the agent switching tasks randomly throughout the middle of episodes, we added random switching between the different pretrained agents during episodes. Specifically, at the beginning of an episode we randomly sample two VPT agents from ( foundation_model_2x , bc_early_game_2x , bc_house_3x , rl_from_foundation_2x , rl_from_house_2x ) and switch between them at each timestep with a probability of 1 / 1000 . Since the RL agents all act quite similarly, we avoid sampling two RL agents at once. Additionally, with a probability of 1 / 750 each timestep, we cause the agent to spin a random number of degrees. This adds more data where the agent spontaneously changes tasks, increasing downstream steerability.

D.2 Text-Video Pair Dataset

We gathered a small dataset of 2,000 human-labelled trajectory segments (text-video pairs) by manually labeling gameplay from our datasets. We used a simple web app that presented a video of 16 frames to the user from a randomly sampled episode. This only corresponds to 32,000 frames of labeled data, which corresponds to labeling 0.06% of the full dataset, or 27 minutes of labeled data. However, as discussed in Appendix E.2, combining this with automatically labeled data using gpt-3.5-turbo and MineCLIP results in a strong prior model.

D.3 Prompt Design

In our experiments we used both short and longer prompts. The short prompts are either taken from previous literature (e.g., the language-conditioning experiment in the VPT appendix [5]) or they were simply the first prompt we tried. The longer prompts were created by taking inspiration from the prompt engineering methods used with text-to-image models such as Stable Diffusion [50]. To design these prompts, we simply strung together a lot of terms related to our task in order to increase the specificity of the prompts. We were excited to discover that this style of prompt design inspired by the prompt-engineering community works well in STEVE-1.

E Training Details

E.1 Policy Training

STEVE-1 was trained using distributed data parallel in PyTorch [45]. During training, segments of 640 timesteps were sampled from the dataset. Due to memory constraints, these segments were further broken up into chunks of 64, which are processed sequentially. Since VPT uses a Transformer-XL [15], this sequential processing lets the policy attend to previous batches up to the limit of its context length. We optimized the weights using AdamW [37] with a maximum learning rate of 4e-5 and a linear warmup for the first 10M frames followed by a cosine learning rate decay schedule that decays to 10% of the original learning rate. See Table 1 for an exhaustive list of hyperparameters used during training.

During training, we sample data using packed hindsight relabeling (Figure 2). This involves sampling a segment of an episode, randomly selecting some timesteps at which to change goals, and then filling in the corresponding goal embeddings for the entire episode with the embeddings from the corresponding goal segments. See Algorithm 1 for a detailed explanantion of packed hindsight relabelling.

E.2 Prior Training

The prior model is a simple CV AE [54] that conditions on MineCLIP [17] text embeddings and models the conditional distribution of visual embeddings given the corresponding text embedding. This model is trained on a combination of around 2,000 hand-labeled trajectory segments and augmented

with additional data by automatically searching for text-gameplay pairs from our gameplay dataset. This is done using the following steps:

Combine the 2,000 text labels with 8,000 additional labels generated by querying gpt-3.5-turbo .

For each of these 10,000 text labels, search through 1,000 episodes sampled from the gameplay dataset to find the top 5 closest visual MineCLIP embeddings to the text embedding of the text label.

These 50,000 automatically-mined text-video pairs are added to the original 2,000 hand-labeled examples to form the final dataset used for prior training.

We noticed when prompting STEVE-1 using visual goals that when the visual goal showed the agent hitting a block but not following through and breaking it that STEVE-1 actually avoided breaking blocks. Unfortunately, many of the automatically discovered text-gameplay clips include gameplay of this kind. In order to prevent this issue, we added an offset to the embeddings found in this manner. By selecting embeddings from a timestep offset steps after the originally-selected timestep, the agent is much more likely to follow through with breaking blocks.

We trained our prior model for 50 epochs on this dataset and used early-stopping with a small validation set. An exhaustive list of hyperparameters used for creating the prior model can be found at Table 2.

E.3 Training Costs

The $60 cost we report corresponds to the cost of renting a 8xA10g node using spot instances on AWS for 12 hours using our instances prices in May 2023.

F Additional Visualizations

F.1 MineCLIP Evaluation

We ran MineCLIP evaluation on both text and visual prompts. The MineCLIP evaluation results can be found in Figure 12.

F.2 Steerability with Programmatic Metrics

Similar to Figure 20 in the VPT appendix [5], we plot the programmatic metric performances (mean and 95% confidence intervals) across the different goal prompt conditioning, both using visual prompts (Figure 15) and text prompts with CVAE prior (Figure 16) conditioning, on our policy trained with hyperparameters in Table 1 and using conditional scaling λ = 7 (for visual prompts) and λ = 6 . 0 (for text prompts with CVAE prior). Each conditioning variant is run with 10 trials, each trial with a different environmental seed and with an episode length of 3000 timesteps (2.5 minutes gameplay). Across the conditioning variant, we use the same set of environmental seeds. For comparison, we also plot the metrics for an unconditional VPT ( early_game ) agent (' VPT (uncond )') and the text-conditioned agent investigated in VPT appendix [5] (' VPT (text)* ') when conditioned on the relevant text. When using visual goal conditioning, we the use MineCLIP video encoder to embed a 16-frame clip of the agent performing the desired task taken from our training dataset. An example frame from each of the visual goals is illustrated in Figure 14. When using text VAE goal conditioning, we use the MineCLIP text encoder to encode the text prompts (Table 3) and use the CVAE prior to sample the goal embedding from the MineCLIP text embedding.

We note several differences in our experimental setup compared to that in VPT [5]. We only run our evaluation episodes for 3000 timesteps, equivalent to 2.5 minutes of gameplay, compared to 5 minutes in the VPT paper. Due to a limited computational budget, we generate 10 episodes per conditioning variant, and 110 episodes for the unconditional (' VPT (uncond) '), compared to VPT's 1000 episodes. Lastly, when measuring the inventory count, we log the maximum inventory count seen throughout the episode, which is a lower bound on the potential number of items collected since the agent can later throw out, place, or use these items to craft. As a result of these caveats, we denote the ' VPT (text)* ' legend in Figure 15 and Figure 16 with an asterisk as we use the results reported in [5] directly for comparison.

We make several observations. First, we observe that our agents is more steerable : when conditioned to collect certain items (in bold), the agent collects (relatively) many more of those items than when conditioned on other instructions unrelated to that item, as well as compared to the unconditional VPT. When conditioned on tasks unrelated to the item (e.g. break a flower when interested in measuring logs collected), we also observe that the agent pursues that item less than the unconditional agent. Second, we observe that for the bolded instructions which we expect to stand out, we outperform VPT performance (dashed blue line) [5], even with half the amount of time in the episode rollout. This suggests that our agent is both more steerable relative to the unconditioned VPT agent and the text-conditioned VPT agent investigated in the VPT appendix [5].

F.3 Prompt Chaining Visualization

We visualize two specific episodes from the prompt chaining experiments in Section 4.3 in Figure 18 (building a tower) and Figure 19 (crafting wooden planks).

(a) MineCLIP Text Evaluation

dig

dirt

sky

leaves

wood

seeds

flower

explore

swim

underwater

inventory

Conditioned Prompt

dig

dirt

sky

leaves

wood

seeds

flower

explore

swim

underwater

inventory

Evaluation Prompt

1

0

0

0

0

0

0

0

0

1

0

0.02

0.04

0.06

0.08

0.10

0.12

(b) MineCLIP Visual Evaluation

Conditioned Prompt

aves

dig

07

wood

seeds

flower

explore

swim

underwater

inventory

0.14

dirt

01

0.12

Prompt

sky

0

leaves

0

0.10

wood

1

Evaluation l

seeds

07

0.08

flower

1

0.06

explore

1

swim

1

0.04

underwater

0

inventory

0

0.02

dig

dirt

sky

leaves

wood

seeds

flower

explore

swim

underwater

inventory

Conditioned Prompt

dig

dirt

sky

leaves

wood

seeds

flower

explore

swim

underwater

inventory

Evaluation Prompt

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.90

1.00

0.10

0.10

0.10

0.90

0.00

0.00

0.00

0.20

0.00

0.00

0.00

1.00

0.80

0.30

0.00

0.10

0.10

0.00

0.00

0.50

0.00

0.20

0.10

0.70

0.90

0.30

0.00

0.20

0.10

0.10

0.00

0.00

0.30

0.00

0.40

1.00

0.10

0.00

0.00

0.00

0.10

0.00

0.00

0.20

0.00

0.10

0.20

0.50

0.30

0.00

0.10

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.20

0.00

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.00

0.10

1.00

1.00

0.70

0.00

0.00

0.50

0.30

0.40

0.20

0.30

0.70

0.90

0.80

0.70

0.00

0.00

0.10

0.00

0.00

0.00

0.20

0.10

0.20

0.50

0.40

0.00

0.10

0.30

1.00

0.30

0.20

0.10

0.20

0.20

0.20

0.10

1.00

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.90

1.00

0.10

0.10

0.10

0.90

0.00

0.00

0.00

0.20

0.00

0.00

0.00

1.00

0.80

0.30

0.00

0.10

0.10

0.00

0.00

0.50

0.00

0.20

0.10

0.70

0.90

0.30

0.00

0.20

0.10

0.10

0.00

0.00

0.30

0.00

0.40

1.00

0.10

0.00

0.00

0.00

0.10

0.00

0.00

0.20

0.00

0.10

0.20

0.50

0.30

0.00

0.10

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.20

0.00

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.00

0.10

1.00

1.00

0.70

0.00

0.00

0.50

0.30

0.40

0.20

0.30

0.70

0.90

0.80

0.70

0.00

0.00

0.10

0.00

0.00

0.00

0.20

0.10

0.20

0.50

0.40

0.00

0.10

0.30

1.00

0.30

0.20

0.10

0.20

0.20

0.20

0.10

1.00

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.90

1.00

0.10

0.10

0.10

0.90

0.00

0.00

0.00

0.20

0.00

0.00

0.00

1.00

0.80

0.30

0.00

0.10

0.10

0.00

0.00

0.50

0.00

0.20

0.10

0.70

0.90

0.30

0.00

0.20

0.10

0.10

0.00

0.00

0.30

0.00

0.40

1.00

0.10

0.00

0.00

0.00

0.10

0.00

0.00

0.20

0.00

0.10

0.20

0.50

0.30

0.00

0.10

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.20

0.00

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.00

0.10

1.00

1.00

0.70

0.00

0.00

0.50

0.30

0.40

0.20

0.30

0.70

0.90

0.80

0.70

0.00

0.00

0.10

0.00

0.00

0.00

0.20

0.10

0.20

0.50

0.40

0.00

0.10

0.30

1.00

0.30

0.20

0.10

0.20

0.20

0.20

0.10

1.00

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.90

1.00

0.10

0.10

0.10

0.90

0.00

0.00

0.00

0.20

0.00

0.00

0.00

1.00

0.80

0.30

0.00

0.10

0.10

0.00

0.00

0.50

0.00

0.20

0.10

0.70

0.90

0.30

0.00

0.20

0.10

0.10

0.00

0.00

0.30

0.00

0.40

1.00

0.10

0.00

0.00

0.00

0.10

0.00

0.00

0.20

0.00

0.10

0.20

0.50

0.30

0.00

0.10

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.20

0.00

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.00

0.10

1.00

1.00

0.70

0.00

0.00

0.50

0.30

0.40

0.20

0.30

0.70

0.90

0.80

0.70

0.00

0.00

0.10

0.00

0.00

0.00

0.20

0.10

0.20

0.50

0.40

0.00

0.10

0.30

1.00

0.30

0.20

0.10

0.20

0.20

0.20

0.10

1.00

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.90

1.00

0.10

0.10

0.10

0.90

0.00

0.00

0.00

0.20

0.00

0.00

0.00

1.00

0.80

0.30

0.00

0.10

0.10

0.00

0.00

0.50

0.00

0.20

0.10

0.70

0.90

0.30

0.00

0.20

0.10

0.10

0.00

0.00

0.30

0.00

0.40

1.00

0.10

0.00

0.00

0.00

0.10

0.00

0.00

0.20

0.00

0.10

0.20

0.50

0.30

0.00

0.10

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.20

0.00

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.00

0.10

1.00

1.00

0.70

0.00

0.00

0.50

0.30

0.40

0.20

0.30

0.70

0.90

0.80

0.70

0.00

0.00

0.10

0.00

0.00

0.00

0.20

0.10

0.20

0.50

0.40

0.00

0.10

0.30

1.00

0.30

0.20

0.10

0.20

0.20

0.20

0.10

1.00

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.90

1.00

0.10

0.10

0.10

0.90

0.00

0.00

0.00

0.20

0.00

0.00

0.00

1.00

0.80

0.30

0.00

0.10

0.10

0.00

0.00

0.50

0.00

0.20

0.10

0.70

0.90

0.30

0.00

0.20

0.10

0.10

0.00

0.00

0.30

0.00

0.40

1.00

0.10

0.00

0.00

0.00

0.10

0.00

0.00

0.20

0.00

0.10

0.20

0.50

0.30

0.00

0.10

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.20

0.00

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.00

0.10

1.00

1.00

0.70

0.00

0.00

0.50

0.30

0.40

0.20

0.30

0.70

0.90

0.80

0.70

0.00

0.00

0.10

0.00

0.00

0.00

0.20

0.10

0.20

0.50

0.40

0.00

0.10

0.30

1.00

0.30

0.20

0.10

0.20

0.20

0.20

0.10

1.00

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.90

1.00

0.10

0.10

0.10

0.90

0.00

0.00

0.00

0.20

0.00

0.00

0.00

1.00

0.80

0.30

0.00

0.10

0.10

0.00

0.00

0.50

0.00

0.20

0.10

0.70

0.90

0.30

0.00

0.20

0.10

0.10

0.00

0.00

0.30

0.00

0.40

1.00

0.10

0.00

0.00

0.00

0.10

0.00

0.00

0.20

0.00

0.10

0.20

0.50

0.30

0.00

0.10

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.20

0.00

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.00

0.10

1.00

1.00

0.70

0.00

0.00

0.50

0.30

0.40

0.20

0.30

0.70

0.90

0.80

0.70

0.00

0.00

0.10

0.00

0.00

0.00

0.20

0.10

0.20

0.50

0.40

0.00

0.10

0.30

1.00

0.30

0.20

0.10

0.20

0.20

0.20

0.10

1.00

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.90

1.00

0.10

0.10

0.10

0.90

0.00

0.00

0.00

0.20

0.00

0.00

0.00

1.00

0.80

0.30

0.00

0.10

0.10

0.00

0.00

0.50

0.00

0.20

0.10

0.70

0.90

0.30

0.00

0.20

0.10

0.10

0.00

0.00

0.30

0.00

0.40

1.00

0.10

0.00

0.00

0.00

0.10

0.00

0.00

0.20

0.00

0.10

0.20

0.50

0.30

0.00

0.10

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.20

0.00

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.00

0.10

1.00

1.00

0.70

0.00

0.00

0.50

0.30

0.40

0.20

0.30

0.70

0.90

0.80

0.70

0.00

0.00

0.10

0.00

0.00

0.00

0.20

0.10

0.20

0.50

0.40

0.00

0.10

0.30

1.00

0.30

0.20

0.10

0.20

0.20

0.20

0.10

1.00

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.90

1.00

0.10

0.10

0.10

0.90

0.00

0.00

0.00

0.20

0.00

0.00

0.00

1.00

0.80

0.30

0.00

0.10

0.10

0.00

0.00

0.50

0.00

0.20

0.10

0.70

0.90

0.30

0.00

0.20

0.10

0.10

0.00

0.00

0.30

0.00

0.40

1.00

0.10

0.00

0.00

0.00

0.10

0.00

0.00

0.20

0.00

0.10

0.20

0.50

0.30

0.00

0.10

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.20

0.00

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.00

0.10

1.00

1.00

0.70

0.00

0.00

0.50

0.30

0.40

0.20

0.30

0.70

0.90

0.80

0.70

0.00

0.00

0.10

0.00

0.00

0.00

0.20

0.10

0.20

0.50

0.40

0.00

0.10

0.30

1.00

0.30

0.20

0.10

0.20

0.20

0.20

0.10

1.00

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.90

1.00

0.10

0.10

0.10

0.90

0.00

0.00

0.00

0.20

0.00

0.00

0.00

1.00

0.80

0.30

0.00

0.10

0.10

0.00

0.00

0.50

0.00

0.20

0.10

0.70

0.90

0.30

0.00

0.20

0.10

0.10

0.00

0.00

0.30

0.00

0.40

1.00

0.10

0.00

0.00

0.00

0.10

0.00

0.00

0.20

0.00

0.10

0.20

0.50

0.30

0.00

0.10

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.20

0.00

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.00

0.10

1.00

1.00

0.70

0.00

0.00

0.50

0.30

0.40

0.20

0.30

0.70

0.90

0.80

0.70

0.00

0.00

0.10

0.00

0.00

0.00

0.20

0.10

0.20

0.50

0.40

0.00

0.10

0.30

1.00

0.30

0.20

0.10

0.20

0.20

0.20

0.10

1.00

1.00

1.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.90

1.00

0.10

0.10

0.10

0.90

0.00

0.00

0.00

0.20

0.00

0.00

0.00

1.00

0.80

0.30

0.00

0.10

0.10

0.00

0.00

0.50

0.00

0.20

0.10

0.70

0.90

0.30

0.00

0.20

0.10

0.10

0.00

0.00

0.30

0.00

0.40

1.00

0.10

0.00

0.00

0.00

0.10

0.00

0.00

0.20

0.00

0.10

0.20

0.50

0.30

0.00

0.10

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.20

0.00

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.10

0.00

0.10

1.00

1.00

0.70

0.00

0.00

0.50

0.30

0.40

0.20

0.30

0.70

0.90

0.80

0.70

0.00

0.00

0.10

0.00

0.00

0.00

0.20

0.10

0.20

0.50

0.40

0.00

0.10

0.30

1.00

0.30

0.20

0.10

0.20

0.20

0.20

0.10

1.00

0.0

0.2

0.4

0.6

0.8

1.0

！

dig

dirt

leaves

wood

seeds

flower

explore

swim

underwaterinventory

dig a hole

get dirt

look at the sky

break leaves

break wood

collect seeds

break a flower

go explore

go swimming

go underwater

open inventory

Conditioning (Visual)

0

50

100

150

200

250

300

Travel Distance (Blocks)

Travel Distance with Conditioning

VPT (text)*

VPT (uncond)

(a)

dig a hole

get dirt

look at the sky

break leaves

break wood

collect seeds

break a flower

go explore

go swimming

go underwater

open inventory

Conditioning (Visual)

0

2

4

6

8

Seeds Collected

Seed Collection with Conditioning

VPT (text)*

VPT (uncond)

(b)

dig a hole

get dirt

look at the sky

break leaves

break wood

collect seeds

break a flower

go explore

go swimming

go underwater

open inventory

Conditioning (Visual)

0

5

10

15

20

Log Collection with Conditioning

VPT (text)*

VPT (uncond)

(c)

dig a hole

get dirt

look at the sky

break leaves

break wood

collect seeds

break a flower

go explore

go swimming

go underwater

open inventory

Conditioning (Visual)

0

10

20

30

40

50

60

70

Dirt Collection with Conditioning

VPT (text)*

VPT (uncond)

(d)

dig as far as possible

get dirt

look at the sky

break leaves

chop a tree

collect seeds

break a flower

go explore

go swimming

go underwater

open inventory

get dirt ...

chop down the tree ...

break tall grass ...

Conditioning (Text)

0

50

100

150

200

250

300

Travel Distance (Blocks)

Travel Distance with Conditioning

VPT (text)*

VPT (uncond)

(a)

dig as far as possible

get dirt

look at the sky

break leaves

chop a tree

collect seeds

break a flower

go explore

go swimming

go underwater

open inventory

get dirt ...

chop down the tree ...

break tall grass

...

Conditioning (Text)

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

Seeds Collected

Seed Collection with Conditioning

VPT (text)*

VPT (uncond)

(b)

dig as far as possible

get dirt

look at the sky

break leaves

chop a tree

collect seeds

break a flower

go explore

go swimming

go underwater

open inventory

get dirt ...

chop down the tree

...

break tall grass ...

Conditioning (Text)

0

2

4

6

8

10

12

Log Collection with Conditioning

VPT (text)*

VPT (uncond)

(c)

dig as far as possible

get dirt

look at the sky

break leaves

chop a tree

collect seeds

break a flower

go explore

go swimming

go underwater

open inventory

get dirt

...

chop down the tree ...

break tall grass ...

Conditioning (Text)

0

5

10

15

20

25

30

Dirt Collection with Conditioning

VPT (text)*

VPT (uncond)

(d)

0

2

4

6

8

10

100

200

300

Travel Distance (Blocks)

0

2

4

6

8

10

0

2

4

6

8

Seeds Collected

0

2

4

6

8

10

0

5

10

15

20

0

2

4

6

8

10

20

40

60

Conditional Scale

λ

0

500

1000

1500

2000

2500

3000

Episode Timestep

0

5

10

15

20

25

30

35

Dirt in Inventory

1

2

3

4

70

75

80

85

90

95

100

105

Agent Height Position

Build a Tower

1: Frame #52

3: Frame #2015

2: Frame #1500

4: Frame #2540

0

500

1000

1500

2000

2500

3000

Episode Timestep

0

1

2

3

4

5

6

7

Logs in Inventory

1

2

3

4

0

5

10

15

20

25

30

35

Planks in Inventory

Make Wooden Planks

1: Frame #220

3: Frame #1500

2: Frame #876

4: Frame #1540

R

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generative pretraining method to learn an instruction-following agent in Minecraft by fine-tuning the VPT agent. It first trains an image-goal-conditioned policy and then leverages the foundation model MineCLIP as the bridge to map the language instruction into the image-goal space. Using this method, this paper claims that it can solve any short-horizon open-ended text and visual task in Minecraft.\n\nSTRENGTHS:\n(1) This paper explores the feasibility of jointly using the foundation models (such as the vision-language model MineCLIP and pre-trained policy model VPT) for solving decision-making problems and shows a preliminary conclusion. \n\n(2) Using unCLIP approach to solve instruction-following decision-making problems is interesting. \n\n(3) This paper proposes to improve the instruction sensitivity with the classifier-free guidance technique, which is interesting and reasonable. \n\n(4) The paper is well-written and easy to follow.\n\nWEAKNESSES:\n(1) The paper overstates its performance by claiming that \"STEVE-1 can follow nearly **any short-horizon** open-ended text and visual task in Minecraft.\"\n\nTo verify this, I downloaded the provided code and ran the attached checkpoints on some basic short-horizon tasks like \"kill sheep\", \"kill cow\", and \"kill pig\". Before testing these tasks, I had already summoned 10 \"pigs\", \"cows\", and \"sheep\" nearby. While I found that the agent actually killed some animals, it did not seem aware of which particular animal it was targeting. The behavior was more like \"kill cow if there is a cow\" rather than following the specific instruction. I tried several other prompt variants such as \"hunt cow\" and \"hunt cow in Minecraft\", but they were unsuccessful. Additionally, I tested STEVE-1 on some short-horizon crafting tasks with sufficient materials in the initial inventory, like \"craft stick\", \"craft oak plank\", and \"craft torch\". The agent merely opened its inventory and then acted randomly.\n\nGiven these observations, I do not believe that STEVE-1 has solved all short-horizon tasks, as it cannot even differentiate between \"cows\" and \"sheep\". This could lead others to misconstrue the extent of the research in the Minecraft environment.\n\n(2) This paper lacks generalization experiments on unseen text instructions. \n\nAs the primary aim of this method is to support open-ended text instructions in STEVE-1, it is crucial to demonstrate its performance on unseen text instructions. The authors collected approximately 10,000 instruction-trajectory pairs, with 2,000 of them hand-labeled and an additional 8,000 instructions generated by GPT. It remains unclear whether the tasks (\"dig dirt\", \"build a tower\", \"make wooden planks\") in the experiments were already included in these instructions. If so, the experiments may not be particularly convincing, as neural networks can easily map the text instructions to corresponding visual embeddings in the training set. That is, pre-training STEVE-1 on visual instructions already achieves this goal and the later generative model training may be not important, which is not impressive. So, I strongly recommend including generalization experiments if the paper wants to show support for open-ended text instructions. \n\n(3) The evaluation metric is insufficient.\n\nThe MineCLIP evaluation metric is not convincingly effective in measuring whether a trajectory corresponds to a given task. This is because the MineCLIP latent space cannot distinguish some trajectories well. For instance, we have found that it was unable to differentiate between two trajectories where one involved approaching a tree and the other involved moving away from it (MineCLIP was not sensitive to distance).\n\nTo address this issue, I recommend that the paper incorporates a success-rate-based confusion matrix, in which the values in Figure 3 (b) are replaced with the task success rate, conditioned on the given instruction.\n\nQUESTIONS:\nAs stated in the weakness part.\n\nLIMITATIONS:\nThe main limitations include (1) prompt design being too sensitive and (2) chain-of-thought prompting is not automated. \nThe authors have adequately stated the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper proposed an instruction-tuned model for Minecraft, which turns previous RL model like VPT into goal-conditioned model. The experiment shows that the proposed method can follow nearly any short-horizon open-ended text and visual task in Minecraft.

STRENGTHS:
1. This paper proposes a novel technique for instruction-tuning video game RL-agent which is inspired by previous unCLIP method in the text-to-image generation field. This idea is interesting and the method can turn an RL agent into a goal-conditioned one which shows great potential for designing a more general RL agent.

2. The experiment part shows that the tuned agent can follow short-horizon goal instructions very well. When given the appropriate textual instruction, the proposed method collects 66x more dirt, 4.5x more wood, 28x more seeds, and travels 3x further than the unconditional agent.

WEAKNESSES:
This method is not applicable for long-horizon tasks in the open world. For example, in Minecraft, if the goal is obtaining a diamond, the instruction may be too difficult for the proposed method since it is not trained for long-horizon trajectories. Regarding this, it is not sure if this method can be applied to challenging tasks in the open world which may contain extremely long planning steps. In this paper, no complicated tasks like obtaining diamond are studied. Thus I am concerned the scalability of the proposed method to much complicated tasks.

QUESTIONS:
I am curious about how is the prompt designed in this paper? The detail of the prompt should be explained in the main text of the paper.

LIMITATIONS:
The limitation is adequately addressed in the paper. No discussion about negative social impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces STEVE-1, a sequential decision-making agent designed to follow textual instructions and accomplish goals in the Minecraft environment. The authors utilize two pre-trained models, VPT (Video Pretraining Transformer) and MineCLIP, to facilitate this process. VPT is a transformer model trained to predict action sequences from aligned video sequences, while MineCLIP aligns consecutive video timesteps with corresponding transcripts in Minecraft.\n\nTo finetune VPT, the authors employ self-supervised behavioral cloning conditioned on latent visual goals. They generate goal-conditioned data by randomly selecting timesteps from episodes and using hindsight relabeling to set intermediate goals. MineCLIP is used to map textual goals to visual goal embeddings in the unCLIP-based approach. This mapping is achieved through a conditional Variational Autoencoder (VAE) with Gaussian prior and posterior, conditioned on MineCLIP text representations.\n\nThe training dataset comprises Minecraft gameplay data, combined with the OpenAI contractor dataset, and additional data generated using VPT. The authors curate 2,000 instruction-labeled trajectory segments, each consisting of 16 frames, and augment this dataset by identifying similar gameplay segments. Additionally, 8,000 additional instructions are generated using GPT-3.5 Turbo. Classifier-free guidance is employed to balance the logits between unconditional and conditional behavior.\n\nThe results demonstrate that STEVE-1 successfully solves various short-horizon open-ended text and visual tasks, with a training cost of only $60. The performance of the agent is evaluated using programmatic evaluation and MineCLIP evaluation, revealing significant improvements with goal conditioning, especially visual. The authors also observe that prompt chaining, as opposed to direct prompting, proves advantageous for complex tasks like building towers or creating planks. A thorough ablation study is conducted on the classifier-free guidance hyperparameter, pretrained VPT weights, and prompt engineering, providing valuable insights into the optimal settings for these components.\n\nOverall, the paper showcases the effectiveness of STEVE-1 in achieving goals based on textual instructions in the Minecraft environment. The combination of VPT and MineCLIP, along with the conditioning techniques and ablation studies, contribute to a comprehensive understanding of the agent's performance and its potential applications.\n\nSTRENGTHS:\n- This unique adaptation - STEVE-1 - of the unCLIP method demonstrates the versatility and effectiveness of the approach in the context of sequential decision making in Minecraft.\n- The experiments and analysis conducted in the paper are highly novel and insightful. The authors provide valuable findings, such as the benefits of prompt chaining, the potential for scaling to improve certain metrics, and the limitations observed in complex tasks. These insights enhance our understanding of the proposed approach and its implications.\n- The paper demonstrates that prompt chaining is effective in accomplishing complex tasks, such as building towers or making wooden planks. The results show that success rates and programmatic evaluation metrics plateau after a certain number of frames, highlighting the potential and limitations of the approach. Additionally, the comparison to direct prompting reveals the superiority of prompt chaining in achieving satisfactory performance.\n- The thorough ablation study conducted on various components, including the classifier-free guidance hyperparameter, pretrained VPT weights, and prompt engineering, provides valuable insights into the optimal settings for these elements.\n- The inclusion of additional ablations, such as goal chunk sizes, VAE variants, and text augmentation, in the appendix further enhances reproducibility and facilitates a deeper understanding of the approach.\n\nWEAKNESSES:\n- The paper would benefit from clearer explanations regarding certain aspects, such as the distinction between packed hindsight relabeling and hindsight relabeling. Additionally, providing a more detailed explanation of the interpretation of the heatmap, specifically the meaning of the 1/0 values, would enhance understanding.\n- Some questions (see section) during the review could be addressed in the main work or in an appendix to provide a more comprehensive understanding of the approach.\n- The proposed approach combines existing ideas from well-established approaches, such as CVAE, MineCLIP, and VPT, which may limit the novelty of the work. However, the extensive experiments and detailed study conducted in the paper contribute to its significance and overall value.\n- The scalability of the approach to other environments or datasets is not discussed or addressed in the paper. Considering the dependency on pretrained video-text aligned models and large-scale pretrained transformers like VPT, the applicability of the approach to environments without such models is unclear. Addressing this limitation and discussing potential scalability issues would enhance the practical relevance and broader applicability of the approach.\n\nQUESTIONS:\n- Can you provide insights into why random timesteps from episodes are selected as goals? Are there alternative approaches that could be considered, and why is this method deemed the most effective?\n- Does randomly resetting the agent's memory and turning the agent to a random direction during data generation provide necessary benefits? What would be the implications if this step were omitted?\n- How does the utilization of pre-trained VPT contribute to the approach when the input is modified with conditional goal embeddings? Can you elaborate on the specific advantages and improvements gained from incorporating VPT in this manner?\n- Could you explain the process of generating additional text instructions based on GPT 3.5? How are these instructions generated and integrated into the training process?\n- It is not clear why the graph from Baker et al. is not included in the text-conditioning results. Can you provide clarification or discuss the reasoning behind its exclusion?\n- In Figure 4 (right), what is \"relevant\" and \"irrelevant\" in the graph? Clarifying the interpretation of these components would improve the understanding of the presented results.\n\nLIMITATIONS:\n- The authors extensively discuss the limitations of the approach, including challenges with multiple steps of reasoning, prompt engineering, and potential negative societal impacts. While further discussion on these aspects could be beneficial, there are no explicit omissions or significant limitations that require specific mention. The paper adequately addresses and acknowledges its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper presents a method to create instruction following agents in Minecraft. It starts with collecting trajectories using OpenAI’s VPT Minecraft agents, which cannot be controlled through instructions. Then some intermediate visual observations are randomly selected as visual goals. These visual goals are then encoded by the visual encoder of a pretrained text-video contrastive foundation model in the Minecraft domain called MineCLIP. VPT is then fine-tuned to achieve these visual goals (specified as MineCLIP visual embeddings) using the same trajectories. Finally, a CVAE is trained to produce textual embedding that matches the visual goal embeddings, therefore the agents can be piloted by natural language instructions. Experiments highlight the performances of presented method on some entry-level tasks including dirt, log and seed collection. Additional results on prompt chaining, scaling, prompt engineering and other hyperparameters confirm the effectiveness of the proposed approach.\n\nSTRENGTHS:\n+The paper is overall clear and well-written. The presentation should be very friendly to readers without RL or LfD background by foundation model in general. The research problem here is relevant to the scope of NeurIPS and its proximity to many emerging topics including open-endedness, generalist agents and large models should be of interest to large audience of this conference.\n\n+I find the approach presented here is technically sound with good results. Although the original VPT agent did offer some preliminary results on instruction following, but no enough details were provided so the method here seems to make solid contributions on building the first open-ended language-piloted agents in Minecraft. Some tricks like baseline subtraction seems helpful. Thanks for bringing this to the attention of the community.\n\n+The authors did a good job highlighting go/no-go about their method. Just list a few that I find most interesting:\n\n- MineCLIP evaluation, which help explain why their method works better on short-horizon task\n\n- Prompt-chaining and metrics over time, which clearly demonstrate how the agent progresses under different prompt conditions.\n\n- Prompt-engineering, which showcases how different instruction can have significant impact at the agent.\n\nWEAKNESSES:\nAt this point, I don’t have any major concerns but here are a few suggestions:\n\n-Baselines; in the main paper, I couldn’t find any baselines or ablations other than the main ”Steve-1” model and VPT(without instruction-following). I agree that there might not be many proper counterparts available but some single-task RL and imitation learning baseline, ex. [10] could still help with better sense on the actual gain the proposed model has in Minecraft. Moreover, the baseline subtraction trick should also be ablated in the main paper with more evidences on other tasks, it seems to be a working technique, but it must be better validated.\n\n-Error bar; In Figure 6, I am not sure if the numbers in parentheses indicate error range. If they are not, error bar is needed for these results.\n\n-Scaling: please elaborate more on “we see evidence of tasks that do not require much data for STEVE-1 to learn, tasks that steadily get more reliable as the agent is trained longer, and tasks where capability suddenly spikes after the agent reaches some threshold.” If not all tasks can benefit from scaling, why “Put together, this suggests that further scaling would likely significantly improve the agent”\n\n-After reading the paper, it’s still unclear to me how the $60 budget is allocated. Please provide detailed explanation to justify this point.\n\nQUESTIONS:\nSee [Weaknesses]\n\nLIMITATIONS:\nOne thing that could be missing in the limitation statement is how well does the cVAE model perform on producing goal embedding and whether it could be a bottleneck to the overall performances on more challenging tasks like long-horizon planning, etc. The authors seem to blame MineCLIP for this but since in the original unCLIP paper, diffusion is used instead of a cVAE, more comparisons might be necessary.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}