A Generative Model for Text Control in Minecraft
Shalev Lifshitz * 1 2 Keiran Paster * 1 2 Harris Chan † 1 2 Jimmy Ba 1 2 Sheila McIlraith 1 2

Abstract

ChatGPT (OpenAI, 2022), which can respond to messages
written in natural language and perform a wide array oftasks, and Stable Diffusion (Rombach et al., 2022), whichturns natural language into an image. While those models
cost anywhere from hundreds of thousands to hundreds ofmillions of dollars to train, there has been an equally exciting
trend whereby powerful open-source foundation models
like LLaMA (Touvron et al., 2023) can be finetuned withsurprisingly little compute and data to become instructionfollowing (e.g., (Taori et al., 2023; Chiang et al., 2023)).

Constructing AI models that respond to text instructions is challenging, especially for sequential
decision-making tasks. This work introduces aninstruction-tuned Video Pretraining (VPT) model
for Minecraft™ called S TEVE -1, demonstratingthat the unCLIP approach, utilized in DALL•E 2,
is also effective for creating instruction-following
sequential decision-making agents. S TEVE -1 istrained in two steps: adapting the pretrained VPTmodel to follow commands in MineCLIP’s latent space, then training a prior to predict latentcodes from text. This allows us to finetune VPTthrough self-supervised behavioral cloning andhindsight relabeling, bypassing the need for costly
human text annotations. By leveraging pretrainedmodels like VPT and MineCLIP and employing
best practices from text-conditioned image generation, S TEVE -1 costs just $60 to train andcan follow a wide range of short-horizon openended text and visual instructions in Minecraft.
S TEVE -1 sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs,
far outperforming previous baselines. We provide
experimental evidence highlighting key factors
for downstream performance, including pretraining, classifier-free guidance, and data scaling. All
resources, including our model weights, training
scripts, and evaluation tools are made available
for further research.

In this paper, we study whether such an approach could
be applicable to sequential decision-making domains. Unlike in text and image domains, diverse data for sequential
decision-making is very expensive and often does not comewith a convenient “instruction” label like captions for images. We propose to instruction-tune pretrained generativemodels of behavior, mirroring the advancements seen in recent instruction-tuned LLMs like Alpaca (Taori et al., 2023).
In the past year, two foundation models for the popular openended video game Minecraft™ were released: a foundation
model for behavior called VPT (Baker et al., 2022) and amodel aligning text and video clips called MineCLIP (Fanet al., 2022). This has opened up an intriguing avenue toexplore fine-tuning for instruction-following in the sequential decision-making domain of Minecraft. VPT was trainedon 70k hours of Minecraft gameplay, so the agent alreadyhas vast knowledge about the Minecraft environment. However, just as the massive potential of LLMs was unlocked byaligning them to follow instructions, it is likely that the VPTmodel has the potential for general, controllable behavior
if it is finetuned to follow instructions. In particular, ourpaper demonstrates a method for fine-tuning VPT to follow
short-horizon text instructions with only $60 of computeand around 2,000 instruction-labeled trajectory segments.

1. Introduction
The ability to use text instructions to control and interact
with powerful AI models has made these models accessible
and customizable for the masses. Such models include

Our method draws inspiration from unCLIP (Ramesh et al.,2022), the approach used to create the popular text-to-image
model DALL•E 2. In particular, we decompose the problem
of creating an instruction-following Minecraft agent into
two models: a VPT model finetuned to achieve visual goals
embedded in the MineCLIP latent space, and a prior modelthat translates text instructions into MineCLIP visual embeddings. We finetune VPT using behavioral cloning with
self-supervised data generated with hindsight relabeling(Andrychowicz et al., 2017), avoiding the use of expensive*
Equal contribution , † Core contribution, 1 Department of Computer Science, University of Toronto, Toronto, Canada 2 Vector Institute for Artificial Intelligence, Toronto, Canada. Correspondence
to: Shalev Lifshitz <shalev.lifshitz@mail.utoronto.ca>, Keiran
Paster <keirp@cs.toronto.edu>.

Interactive Learning with Implicit Human Feedback Workshop at
ICML 2023.

1

A Generative Model for Text Control in Minecraft

text-instruction labels in favor of visual MineCLIP embeddings. We apply unCLIP with classifier-free guidance (Hoand Salimans, 2022) to create our agent called S TEVE -1,
which sets a new bar for open-ended instruction following
in Minecraft with low-level controls (mouse and keyboard)and raw pixel inputs, far outperforming the baseline set by
Baker et al. (2022).

of web videos of Minecraft along with estimated mouse andkeyboard actions. Several prior works (Volum et al., 2022;
Wang et al., 2023) have explored the use of LLMs in creating instructable Minecraft agents. These works typicallyuse LLMs to make high-level plans that are then executedby lower-level RL (Nottingham et al., 2023; Wang et al.,2023) or scripted (PrismarineJS and Others, 2023) policies.
Since S TEVE -1 is a far more flexible low-level policy, thecombination of S TEVE -1 with LLMs is a promising direction for future work. Fan et al. (2022) introduced an agent
trained using RL on 12 different tasks and conditioned on
MineCLIP-embedded text-prompts. However, this agent
failed to generalize beyond the original set of tasks.

Our main contributions are as follows:• We create S TEVE -1, a Minecraft agent that can follow
open-ended text and visual instructions with a high degree of accuracy. We perform extensive evaluations of
our agent, showing that it can perform a wide range ofshort-horizon tasks1 in Minecraft. For longer-horizon
tasks like crafting and building, we show that a basic
version of prompt chaining can dramatically improve
performance.

Foundation Models for Sequential Decision-Making
Foundation models which are pretrained on vast amountsof data and then finetuned for specific tasks have recently
shown great promise in a variety of domains including language (Brown et al., 2020; Chowdhery et al., 2022; Touvron
et al., 2023), vision (Ramesh et al., 2022; Caron et al., 2021;
Radford et al., 2021), and robotics (Brohan et al., 2022;
Shridhar et al., 2022; Jiang et al., 2022; Nair et al., 2022;
Xiao et al., 2022). GATO (Reed et al., 2022) and RT-1
(Brohan et al., 2022) have demonstrated the potential of
training transformers to perform both simulated and realworld robotic tasks. With the exception of Kumar et al.
(2023), which uses Q-learning, the vast majority of cases(Lee et al., 2022; Brohan et al., 2022; Reed et al., 2022)
where deep learning has been scaled to large, multitask
offline-RL datasets have used supervised RL. Supervised
RL (e.g., (Paster et al., 2020; Ghosh et al., 2021; Chen et al.,2021)) works by framing the sequential decision-making
problem as a prediction problem, where the model is trained
to predict the next action conditioned on some future outcome. While these approaches are simple and scale wellwith large amounts of compute and data, more work isneeded to understand the trade-offs between supervised RL
and Q-learning or policy gradient-based methods (Pasteret al., 2022a;b; Brandfonbrener et al., 2022; Strupl et al.,2022). Recent works explore the use of hindsight relabeling(Andrychowicz et al., 2017) using vision-language models(Radford et al., 2021; Alayrac et al., 2022) to produce natural
language relabeling instructions. DIAL (Xiao et al., 2022)
finetunes CLIP (Radford et al., 2021) on human-labeled trajectories, which is then used to select a hindsight instruction
from a candidate set. Sumers et al. (2023) uses Flamingo(Alayrac et al., 2022) zero-shot for hindsight relabeling byframing it as a visual-question answering (VQA) task. In
contrast, S TEVE -1 relabels goals using future trajectory segment embeddings given by the MineCLIP (Fan et al., 2022)
visual embedding.

• We describe our method for creating S TEVE -1 using
only $60 of compute, showing that unCLIP (Ramesh
et al., 2022) and classifier-free guidance translate wellto the sequential decision-making domain and are essential for strong performance.
• We release model weights for S TEVE -1 as well as
training scripts and evaluation code in order to foster
more research into instructable, open-ended sequential
decision-making agents.2

2. Related Work
Minecraft as a Test-bed for AI Minecraft has gained
popularity as a benchmark for AI research due to its complex and dynamic environment, making it a rich test-bed forreinforcement learning and other AI methods (e.g., (Johnson
et al., 2016; Guss et al., 2019; Fan et al., 2022; Hafner et al.,2023; Nottingham et al., 2023; Wang et al., 2023; Malato
et al., 2022; Cai et al., 2023)). We leverage the MineRL
environment (Guss et al., 2019) to research the creation of
agents that can follow open-ended instructions in complex
visual environments using only low-level actions (mouseand keyboard). We build S TEVE -1 on top of two recent
foundation models. In order to align text and videos, we
use MineCLIP (Fan et al., 2022), a CLIP (Radford et al.,2021) model trained on paired web videos of Minecraft
gameplay and associated captions. To train S TEVE -1’s policy, we fine-tune VPT (Baker et al., 2022), a foundation
model of Minecraft behavior that is pretrained on 70k hours
1
Short-horizon tasks require few steps: e.g., go to a tree andchop it down, dig a hole. Long-horizon tasks take many steps: e.g.,
craft complex recipes from scratch, build a house.
2
Model weights, training code, videos, and an interactive demo script are hosted on our project webpage athttps://sites.google.com/view/steve-1.

Text-Conditioned Generative Models There has beena recent explosion of interest in text-to-X models, includ2

A Generative Model for Text Control in MinecraftVideo
encoder

CLIP Objective

ResNet//ResNetlinear

CVAE

Text
Encoder

ResNetPolicyTransformer (VPT)

Prior
Gaussian Prior“chop a tree”
MineCLIP

Mouse/Keyboard(Frozen)Figure 1. Like unCLIP (Ramesh et al., 2022), our approach involves two models. First, we train the policy by finetuning VPT to achievegoals given by pretrained MineCLIP (Fan et al., 2022) visual embeddings using our gameplay dataset. Second, for the prior model, wetrain a CVAE (Sohn et al., 2015) to sample MineCLIP visual embeddings given a text prompt. The combination of these two models
enables our agent to follow text and visual instructions.

ing text-to-image (e.g., (Ramesh et al., 2022; Saharia et al.,2022; Rombach et al., 2022)), text-to-3D (e.g., (Jun andNichol, 2023; Lin et al., 2023)), and even text-to-music(e.g., (Agostinelli et al., 2023)). These models are typicallyeither autoregressive transformers modeling sequences of
discrete tokens (Vaswani et al., 2017; Brown et al., 2020)
or diffusion models (Ho et al., 2020). Most related to ourwork is unCLIP, the method used for DALL•E 2 (Ramesh
et al., 2022). unCLIP works by training a generative diffusion model to sample images from CLIP (Radford et al.,2021) embeddings of those images. By combining thismodel with a prior that translates text to visual CLIP embeddings, unCLIP can produce photorealistic images forarbitrary text prompts. unCLIP and many other diffusionbased approaches utilize a technique called classifier-free
guidance (Ho and Salimans, 2022), which lets the model
trade-off between mode-coverage and sample fidelity posttraining. We utilize the basic procedure of unCLIP andclassifier-free guidance for training S TEVE -1.

which opens the door for a wide range of uses for VPT in
Minecraft.
Our approach is inspired by unCLIP, the method behind therecent text-to-image generation model, DALL•E 2 (Ramesh
et al., 2022). Our goal is to create a generative modelof behavior in Minecraft conditioned on text instructions
y. To do so, we utilize a dataset of Minecraft trajectory segments, some of which contain instruction labelsy: [(τ1 , y1 ), (τ2 , y2 ), . . . , (τn , ∅)] where τ is a trajectory ofobservations and actions. We also employ a pretrained CLIPmodel called MineCLIP (Fan et al., 2022), which generatesaligned latent variables zτt:t+16 , zy , where zτt:t+16 is an embedding of any 16 consecutive timesteps from the trajectory.
MineCLIP is trained using a contrastive objective on pairsof Minecraft videos and transcripts from the web. For simplicity of notation, we refer to the MineCLIP embedding of
the last 16 timesteps of a trajectory segment as zτgoal . Like
unCLIP (Ramesh et al., 2022), we utilize a hierarchicalmodel consisting of a prior and a policy:

3. Method• A prior p(zτgoal |y) that produces a latent variable zτgoal
conditioned on a text instruction y.

Inspired by the rapid recent progress in instruction-tuning
Large Language Models (LLMs), we choose to leverage
the recently released Video Pretraining (VPT) (Baker et al.,2022) model as a starting point for our agent. Since VPT
was trained on 70k hours of Minecraft gameplay, the agent
already has vast knowledge about the Minecraft environment. However, just as the massive potential of LLMs
was unlocked by aligning them to follow instructions, it
is likely that the VPT model has the potential for general,
controllable behavior if it is finetuned to follow instructions.
In this work, we present a method for finetuning VPT tofollow natural, open-ended textual and visual instructions,• A policy p(τ |zτgoal ) that produces a trajectory conditioned on a latent variable zτgoal .
These two models can then be combined to produce a generative model of behaviors conditioned on text instructions:p(τ |y) = p(τ, zτgoal |y) = p(zτgoal |y)p(τ |zτgoal )(3.1)

3.1. Policy
To learn our policy, we finetune VPT, a foundation model of
Minecraft behaviors pθ (τ ) trained on 70k hours of Minecraft3

A Generative Model for Text Control in Minecraft

z1

z2

z3

z4

z5

z6

z7

z2

z2

z7

z7

z7

z7

z7

Figure 2. To create goal-conditioned data for finetuning, we randomly select timesteps from episodes and use hindsight relabeling to set
the intermediate goals for the trajectory segments to those visual MineCLIP embeddings. This self-supervised data teaches the agent
which actions lead to which states.

3.2. Prior

gameplay videos. Specifically, VPT consists of a ResNet(He et al., 2016) that processes frames of dimension 128 ×
128 × 3, and a Transformer-XL (Dai et al., 2019) which
processes the frame representations and autoregressively
predicts the next action using the joint hierarchical action
space described in Baker et al. (2022). In order to modify
the architecture to condition on goal information, we add
an affine transformation of zτgoal to the output of the ResNet
before passing it to the transformer:
Process Frames:

In order to condition not only on embeddings of visual goals
but on latent goals, we need the prior, a model that produces
a latent variable zτgoal conditioned on a text instruction y.
Our model is a simple conditional variational autoencoder(CVAE) (Sohn et al., 2015; Kingma and Welling, 2014)
with a Gaussian prior and a Gaussian posterior. Rather thanlearn to condition directly on text, we choose to condition
on frozen text representations from MineCLIP zy . Both theencoder and decoder of our CVAE are parameterized as twolayer MLPs with 512 hidden units and layer normalization(Ba et al., 2016). We train the model on our dataset, forwhich we have text labels Dlabels using the following loss:
Lprior (ϕ) =E(zτgoal ,zy )∼Dlabels KL(qϕ (zτgoal |zy )∥p(zτgoal ))i−Ec∼qϕ (zτgoal |zy ) log pϕ (zτgoal |c, zy )ResNetθ (ot ) → xt[+ Goal Embedding]: xt → xt + Wθ zτgoal + bθ
Predict Actions:

TransXLθ (xt , . . . , xt+T ) → at+T

In order to finetune VPT to condition on goals, we finetune the model using a method inspired by supervised RL
approaches like Decision Transformer (Chen et al., 2021),
GLAMOR (Paster et al., 2020), and GCSL (Ghosh et al.,2021). We use a modification of hindsight relabeling which
we call packed hindsight relabeling (see Figure 2) to generate a new dataset of trajectories with goals pulled from future states that periodically switch. Specifically, our method
to generate this dataset consists of two steps:

3.3. Datasets
To train our policy, we gather a gameplay dataset with
54M frames (≈ 1 month at 20FPS) of Minecraft gameplay
along with associated actions from two sources: contractor
gameplay and VPT-generated gameplay. To train our prior,
we use a dataset of text-video pairs gathered by humans
and augmented using the OpenAI API gpt-3.5-turbo
model (OpenAI, 2022) and MineCLIP. See Appendix C for
more detailed information about our datasets.

1. Given a trajectory τ with T timesteps, randomly generate indices to select goals from: i1 , i2 , . . . , in . These
indices are chosen by starting at the first timestep andrepeatedly sampling a new timestep by adding a random value to the previous timestep. This ensures thatthe data reflects that some goals may take longer toachieve than others.
2. For each chosen goal at timestep ij , set the goals fortimesteps ij−1 + 1, . . . , ij to be the goal at timestep ij ,denoted zτij .

OpenAI Contractor Dataset We use 39M frames
sourced from the contractor dataset which VPT (Baker et al.,2022) used to train its inverse dynamics model and finetune
its policy. The dataset was gathered by hiring human contractors to play Minecraft and complete tasks such as housebuilding or obtaining a diamond pickaxe. During gameplay,
keypresses and mouse movements are recorded. We use
the same preprocessing as VPT, including filtering out nullactions.

Our final dataset Drelabeled consists of observation sequences(o1 , . . . , oT ), action sequences (a1 , . . . , aT ), and packed
hindsight relabeled goals (z1 , . . . , zT ). We then finetune
VPT on this dataset using a supervised loss to predict eacg
action autoregressively using a causal attention mask:
Lpolicy (θ) = EDrelabeled [− log pθ (at |o1...t , z1...t )]VPT-Generated Dataset We generate an additional
dataset of 15M frames by generating random trajectories us-

(3.2)
4

A Generative Model for Text Control in Minecraft

ing the various pretrained VPT agents. The diversity of thisdataset is improved by randomly switching between models
during trajectories (Paster et al., 2022b), randomly resetting
the agent’s memory, and randomly turning the agent to facea new direction.

game item collection (log, seed, and dirt). We also computeautomatic MineCLIP metrics to evaluate the agent’s capability level by recording the minimum cosine distance between
the (text or visual) goal embedding and the MineCLIP visual embedding at any timestep during an episode. See
Appendix D for more details about our evaluation metrics.

Text-Video Pair Dataset To train our prior model, we
also manually gather a dataset of 2,000 text instructions
paired with 16-frame videos sampled from our gameplay
dataset. We augment this dataset by using the alignment between text and video embeddings from MineCLIP. For eachtext instruction, we find the top k most similar gameplay
segments in our dataset and use the corresponding 16-frame
video as additional training data. For augmentation, we also
add 8,000 text-instructions generated by the OpenAI API
gpt-3.5-turbo model (OpenAI, 2022), in addition to
our 2,000 hand-labeled instructions.

4. Results
In our experiments, we aim to answer the following:
1. How well does S TEVE -1 perform at achieving bothtext and visual goals in Minecraft?
2. How does our method scale with more data?
3. What choices are important for the performance of our
method?

3.4. Inference

4.1. Training Setup

At inference time, we use the prior to sample a latent goalzτgoal from the text instruction y. We then use the policy
to autoregressively sample actions at conditioned on theobservation history o1...t and the latent goal zτgoal . Similar
to the observation in Appendix I of Baker et al. (2022),
even with conditioning the policy often fails to follow its
instruction and simply acts according to its prior behavior.
To mitigate this, we borrow another trick used in image
generation models: classifier-free guidance. Specifically,
during inference we simultaneously compute logits for thepolicy conditioned on the goal f (ot , . . . , ot+1 , zτgoal ) and forthe unconditional policy f (ot , . . . , ot+1 ). We then computea combination of the two logits using a λ parameter to tradeoff between the two:

We base our implementation off of the official VPT codebase3 . The main S TEVE -1 is trained using Pytorch (Paszke
et al., 2019) distributed data parallel on four A40 GPUs
for 160M frames, or just under three epochs of our gameplay dataset. Hyperparameters are selected to match thosein Baker et al. (2022) with the exception of learning rate,
which we set to 4e-5. Our models are optimized using
AdamW (Loshchilov and Hutter, 2019). See Table 3 for a
full list of hyperparameters.
4.2. Performance on Textual and Visual Goals
Due to the hierarchical nature of our model, we can evaluate
the performance of our agent at achieving either text orvisual goals simply by choosing whether to use the prior
to condition on text or bypass the prior and condition on
a MineCLIP video embedding directly. We first tested ourmodel on a set of 11 tasks that are achievable within the first
2.5 minutes of gameplay and which do not require multiple
steps to complete (e.g., chop a tree or dig a hole, but notbuild a house). A complete list of the tasks and prompts weused for evaluation can be found in Table 4 in the appendix.
To select visual goals for testing each of the evaluation tasks,
we implemented a tool that searches through 10% of our
gameplay dataset by finding the closest 16-frame videos toa given text prompt. We then manually selected a 16-frame
video that clearly demonstrates the task being completed
and use the corresponding MineCLIP video embedding as
the goal embedding for that task. Screenshots of these visual
goals can be found in Figure 14 in the appendix.

logits = (1+λ) fθ (ot , . . . , ot+1 , zτgoal ) −λ fθ (ot , . . . , ot+1 )
|
{z
}
|
{z
}
conditional logits

unconditional logits

By setting a higher value of λ, we can encourage the policy to follow actions that are more likely when conditionedon the goal and, as demonstrated in Section 4.5, this significantly improves performance. Also, in order to train
the policy to generate these unconditional logits, we occasionally dropout the goal embedding zτgoal from the policy’s
input (with probability 0.1). This lets us generate both theconditional and unconditional logits using the same modelwith batch processing at inference time.
3.5. Evaluation
Evaluating the performance of our agent is a challenging
task due to the wide variety of instructions that are possible and the difficulty of evaluating whether the agent has
successfully achieved its task. We compute programmatic
metrics by monitoring the agent’s travel distance and early-

In Figure 3, we compare the performance of our text andvisual-conditioned agents with the unconditional VPT agent
across our programmatic tasks. We find that when given
35

https://github.com/openai/Video-Pre-Training

100

VPT

STEVE-1(Text)

dig 1
0
dirt
0
sky
0
leaves
0
wood
0
seeds
0
flower0
explore0
swim
1
underwater0
inventory

5
0

STEVE-1(Visual)

dig
dir
t
sk
y
lea
woves
o
see d
flo ds
w
ex er
plo
sw re
i
un m
de
inv rwen ate
tor r
y

10

VPT

STEVE-1(Text)

Evaluation Prompt

Seeds Collected

200

0

Conditioned Prompt

15

300

STEVE-1(Visual)20
60

Dirt Collected

Wooden Logs Collected

Travel Distance (Blocks)

A Generative Model for Text Control in Minecraft

15
10
5
0

VPT

STEVE-1(Text)STEVE-1(Visual)

40
20
0

VPT

STEVE-1(Text)STEVE-1(Visual)

0.12
0.10
0.08
0.06
0.04
0.02

Figure 3. Left: In our programmatic evaluations, S TEVE -1 performed far better than the unconditional VPT agent early-game-2x
when prompted appropriately. On some tasks, visual outperforms text-based prompting, creating a gap that can likely be bridged throughbetter prompt engineering. Right: Across our 11 MineCLIP evaluation tasks, S TEVE -1 achieves the shortest distance between theepisode and the MineCLIP goal embedding when prompted appropriately except for in two cases, where it mixes up digging and dirt andswimming and going underwater. This shows the strong general performance of S TEVE -1 across a wide variety of short-horizon tasks.
See Figure 14 for sample frames from each of the 11 visual goals.

4.3. Prompt Chaining

the relevant text instruction, S TEVE -1 collects 75x moredirt, 4.9x more wood, 22x more seeds, and travels 4.3x
further than the unconditional agent. This also represents
a significant improvement over the reported performanceof text-conditioning in Appendix I of Baker et al. (2022),
which collects several times fewer resources despite having
twice as long of an episode to do so. We also run an automatic evaluation using MineCLIP embedding distancesby measuring the minimum distance of a goal embedding
to any frame in the episode. As shown in Figure 12, thedistance between the goal and the episode is significantly
lower when the agent is conditioned on the corresponding
visual goal than otherwise. Full results for S TEVE -1 withboth text and visual goals can be found in Appendix F.

We also experiment with longer horizon tasks that require
multiple steps, such as crafting and building. We explore
two different prompting methods: directly prompting withthe target goal, and a simple form of prompt chaining(Chase, 2022; Wei et al., 2022b; Dohan et al., 2022) where
the task is decomposed into several subtasks and the prompts
are given sequentially for a fixed number of steps. We explore prompt chaining with visual goals for two tasks: 1)
building a tower and 2) making wooden planks. When using prompt chaining, we first prompt S TEVE -1 to gather
dirt before building a tower, and to gather wooden logs before crafting wooden planks. Figure 4 shows that directly
prompting S TEVE -1 with the final tasks results in near-zerosuccess rates. However, prompt chaining allows S TEVE -1
to build a tower 50% of the time and craft wooden planks
70% of the time. For the tower building task, S TEVE -1
immediately starts collecting dirt until the prompt switches,at which point its average height starts increasing rapidly
and its dirt decreases as it builds a tower. Similarly, forthe crafting wooden planks task, S TEVE -1 immediately
starts collecting a large amount of wooden logs until the
prompt switches and it rapidly converts these wooden logs
into wooden planks (causing the amount of wooden logs
in its inventory to immediately decrease and the numberof wooden planks to increase as it crafts more). Figure 4
visualizes the average item counts and agent height for the
prompt chaining episodes. See Figure 17 and Figure 18 in
the appendix for visualizations of specific prompt chaining

In addition to our evaluations of S TEVE -1, we also recorded
several sample interactive sessions we had with the agent(controlling it in real-time by giving it written text instructions or specific visual goals). These sessions demonstrate
S TEVE -1’s ability to responsively follow instructions inreal-time in a variety of situations. We believe that such usecases, where humans give an agent natural instructions thatit can follow to complete tasks, will become increasingly important and have practical uses in the creation of instructableassistants and virtual-world characters. These videos, aswell as videos of our agent performing our evaluation tasks,
can be found at https://sites.google.com/view/steve-1.

6

75

15

70

10

65

5

60
0

1000

2000

3000

55

12
10

4

8
6

2

4
2

0

0

Episode Timestep

2000

3000

0

0.4
0.2
0.0

0.8
0.6
0.4
0.2
0.0

20M

40M

60M

80M

100

Wooden Logs Collected

0.6

Wooden Planks
Crafting Success

1.0

Prompt Chaining
Direct Prompting

0.8

200

4
2
0

Episode Timestep

1.0

Build Tower Success

1000

relevant
irrelevant

6

300

100M

20M

40M

60M

80M

100M

20

60

Dirt Collected

0

14
6

Seeds Collected8020Make Wooden Planks

Travel Distance (Blocks)

25

8

Planks in Inventory

85

Logs in Inventory

Build a Tower

30

Agent Height Position

Dirt in Inventory

A Generative Model for Text Control in Minecraft

15
10
5
0

20

0
20M

Number of Frames

40

40M

60M

80M 100M

20M

40M

60M

80M 100M

Number of Frames

Figure 4. Top left: By sequentially chaining visual prompts like “get dirt” and “build a tower”, S TEVE -1 successfully gathers dirt andthen uses this dirt to build a tower. The prompts switch at the dotted vertical line. Bottom left: The success rates of the chained prompts
improve steadily as we train S TEVE -1 on more data. Right: The performance of S TEVE -1 in different tasks scales in different ways whenconditioning on relevant visual prompt for the metric (e.g. break wood for the “Wooden Logs Collected” metric) versus other irrelevant
visual prompts. For instance, in the wood-collection and dirt-collection tasks, performance starts increasing after training on 60M framesof gameplay.

episodes.

began to rapidly improve. Figure 4 shows the scaling curves
for S TEVE -1 on each programmatic task when conditioning
on relevant vs. irrelevant visual prompts for that task.

4.4. Scaling

We also evaluated the scaling properties of S TEVE -1 for our
multi-step tasks with and without prompt chaining. Without
prompt chaining, the tasks remain challenging for S TEVE 1 throughout training. However, we note that after 60M
frames, S TEVE -1 learns to gather wooden logs and build asmall tower when told to build a tower. This is likely because
our visual prompt for tower building shows a video of a
tower being built out of wooden logs. With prompt chaining,
the performance of S TEVE -1 steadily increases with moredata. We conjecture that this is because the success of achained prompt requires the success of each element in the
chain. Since different abilities emerge at different scales,
one would expect chained prompts to steadily get morereliable as these subgoals become more reliably completed.
In the case of building wooden planks, we note that crafting
is one such task that gets significantly more reliable as theagent is trained on more data. Figure 4 shows the scaling
curves for S TEVE -1 on the prompt chaining tasks.

Recent works in language modeling have found that scaling up pretraining FLOPs, by training on more data or bytraining a model with more parameters, can improve performance on downstream tasks (Kaplan et al., 2020; Suzgun
et al., 2022; Wei et al., 2022a). In certain cases when measuring performance with metrics such as exact-match (Schaeffer et al., 2023), performance improvement may appear
to be “emergent” (Wei et al., 2022a), appearing suddenly
as the model is trained with more compute. Here, we aim
to gain a basic understanding of how the performance of
S TEVE -1 on various tasks scales by training with more data
(learning rate schedule is chosen appropriately).
To assess performance gain, we first isolated the performance of the policy from the prior, measuring performanceof the agent through training on programmatic tasks (travel
distance, seeds, logs, dirt) with visual goals. Due to compute constraints, we chose to use the 2x VPT model, whichhas 248M parameters. We found that both seed collection
and travel distance did not improve significantly past 20M
frames. From inspecting gameplay, we suspect that travel
distance is a relatively easy task since it is close to VPT’sdefault behavior of running around and exploring. For seedcollection, performance remains suboptimal, suggesting thatfurther scaling may be beneficial. This hypothesis is supported by the observation that performance on log and dirt
collection remained roughly level until 60M frames when it

In summary, we see evidence of tasks that do not require
much data for S TEVE -1 to learn, tasks that steadily get
more reliable as the agent is trained longer, and tasks where
capability suddenly spikes after the agent reaches somethreshold. Put together, this suggests that further scaling
would likely significantly improve the agent, although weleave the task of predicting exactly how much performance
there is to gain to future studies.

7

10
5
0

scratch

fd

bc

rl

50
25
0

scratch

fd

bc

rl

20

Dirt Collected

Wooden Logs Collected

75

Dirt Collected

Wooden Logs Collected

A Generative Model for Text Control in Minecraft

10

60
40
20

0
0

2

4

6

8

10

0

2

4

6

8

10

Conditional Scale λPretrained Weights

Figure 5. Left: We trained S TEVE -1 on 100M frames starting from four different pretrained weights: random initialization (scratch),foundation-2x (fd), bc-early-game-2x (bc), and rl-from-foundation-2x (rl). The rl-from-foundation-2x
agent is generally the most performant after fine-tuning. Using pretrained weights performs better than training from scratch, especiallyfor more complicated tasks like collecting wood. Right: By using classifier-free guidance (Ho and Salimans, 2022), S TEVE -1 collects
5.8x more dirt and 9x more wood.

4.5. What Matters for Downstream Performance?

massive amount of compute it was trained with during its
RL training (Baker et al., 2022).

In this section, we show that pretraining and classifier-free
guidance both have a siginificant impact on downstream
performance. See Appendix B for additional ablations on
design choices for our method, including prompt engineering, the use of classifier-free guidance during training, text
augmentation strategies, different VAE variants, and varying
chunk sizes during finetuning.

Classifier-Free Guidance Baker et al. (2022) observedthat when conditioning the agent on text, it tended to ignore its instruction and instead perform the prior behavior
learned during pretraining. As discussed in section 3.4,classifier-free guidance gives a knob for trading off between
goal-conditioned and prior behaviors. Figure 5 shows theeffect of this parameter λ on the log and dirt collection
tasks. The performance of the agent reaches its maximum
around λ = 5.0 to λ = 7.0, after which it starts to drop off.
These results demonstrate the importance of classifier-free
guidance, which improves the performance of S TEVE -1 byorders of magnitude.

Pretraining Baker et al. (2022) finds that by pretraining a behavioral prior with imitation learning on internetscale datasets for Minecraft, the learned policy can be effectively finetuned to accomplish tasks that are impossible without pretraining. In this section, we demonstratethat pretraining is also massively beneficial for instructiontuning in Minecraft. We hypothesize that due to the strong
performance of S TEVE -1 and the relatively small amountof compute (≈ 1% additional compute) used for instruction finetuning, most of the capabilities of our agent comefrom the pretraining rather than the finetuning. To test
this hypothesis, we finetune several varients of S TEVE 1 from various pretrained weights: foundation-2x,
bc-early-game-2x, rl-from-foundation-2x,
and with randomly initialized weights. In this experiment,each model was finetuned on 100M frames.

5. Conclusion
We demonstrate the promise of our approach of applying
unCLIP (Ramesh et al., 2022) and classifier-free guidance(Ho and Salimans, 2022) to create S TEVE -1, a powerful
instructable agent that is able to achieve a wide range ofshort-horizon tasks in Minecraft. S TEVE -1 can be prompted
to follow text or visual goals, achieving strong instructionfollowing performance in Minecraft with only $60 of compute and around 2,000 instruction-labeled trajectory segments. We show that an initial version of prompt chaining is
a promising approach for improving performance on longer
horizon tasks that require multiple steps of reasoning, andmore can be done in future work to improve performance.

Figure 5 shows the performance of these models on our
programmatic tasks with visual goals. Note that while anagent trained on our dataset from scratch can accomplish
basic tasks like dirt collection fairly well, it is unable to findand chop down trees, in contrast to the pretrained agents.
This demonstrates that the abilities present in the agent
due to pretraining are successfully transferred to the finetuned agent. Out of all the pretrained weights we tried, we
noticed that rl-from-foundation-2x performed thebest, having qualitatively better performance at tasks likecrafting and chopping down trees. Indeed, Figure 5 showsthat this model has strong performance, likely due to the

Due to the generality of our approach, which operates on
raw pixels and produces low-level actions (mouse and keyboard), we hope that S TEVE -1 can spark future work in creating instructable agents in other domains and environments.
Future work should include addressing the limitations of
S TEVE -1 by improving its performance in longer-horizon
tasks, perhaps through the use of LLMs.
8

A Generative Model for Text Control in Minecraft

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger,
Tom Henighan, Rewon Child, Aditya Ramesh, Daniel
Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, MarkChen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin
Chess, Jack Clark, Christopher Berner, Sam McCandlish,
Alec Radford, Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners. In H. Larochelle,
M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,
Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
URL
https://proceedings.neurips.cc/
paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64aPaper.pdf.

Andrea Agostinelli, Timo I. Denk, Zalán Borsos,
Jesse H. Engel, Mauro Verzetti, Antoine Caillon,
Qingqing Huang, Aren Jansen, Adam Roberts, Marco
Tagliasacchi, Matthew Sharifi, Neil Zeghidour, andChristian Havnø Frank.
Musiclm: Generatingmusic from text.
CoRR, abs/2301.11325, 2023.
doi: 10.48550/arXiv.2301.11325. URL https://doi.org/10.48550/arXiv.2301.11325.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur
Mensch, Katherine Millican, Malcolm Reynolds, et al.
Flamingo: a visual language model for few-shot learning.
Advances in Neural Information Processing Systems, 35:
23716–23736, 2022.
Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas
Schneider, Rachel Fong, Peter Welinder, Bob McGrew,
Josh Tobin, Pieter Abbeel, and Wojciech Zaremba.
Hindsight experience replay.
In Isabelle Guyon,
Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach,
Rob Fergus, S. V. N. Vishwanathan, and Roman
Garnett, editors, Advances in Neural Information
Processing Systems 30: Annual Conference on Neural
Information Processing Systems 2017, December 4-9,
2017, Long Beach, CA, USA, pages 5048–5058, 2017.
URL
https://proceedings.neurips.cc/
paper/2017/hash/453fadbd8a1a3af50a9df4df899537b5Abstract.html.

Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao
Liang. Open-world multi-task control through goal-aware
representation learning and adaptive horizon prediction.
arXiv preprint arXiv:2301.10034, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.
Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on
Computer Vision (ICCV), 2021.
Harrison Chase. Langchain, 2022. URL https://github.com/hwchase17/langchain.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee,
Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind
Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances inneural information processing systems, 34:15084–15097,
2021.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
Layer normalization. arXiv preprint arXiv:1607.06450,
2016.
Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga,
Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul
Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advancesin Neural Information Processing Systems, 35:24639–
24654, 2022.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao
Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P.
Xing. Vicuna: An open-source chatbot impressing gpt-4
with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.

David Brandfonbrener, Alberto Bietti, Jacob Buckman,
Romain Laroche, and Joan Bruna. When does returnconditioned supervised learning work for offline reinforcement learning? arXiv preprint arXiv:2206.01079,
2022.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin,
Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul
Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha
Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker
Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran,
Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope,James Bradbury, Jacob Austin, Michael Isard, Guy GurAri, Pengcheng Yin, Toju Duke, Anselm Levskaya,
Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski,

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana
Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine
Hsu, et al. Rt-1: Robotics transformer for real-worldcontrol at scale. arXiv preprint arXiv:2212.06817, 2022.
9

A Generative Model for Text Control in Minecraft

Xavier Garcia, Vedant Misra, Kevin Robinson, Liam
Fedus, Denny Zhou, Daphne Ippolito, David Luan,
Hyeontaek Lim, Barret Zoph, Alexander Spiridonov,
Ryan Sepassi, David Dohan, Shivani Agrawal, Mark
Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta,
Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei,
Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, SlavPetrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311, 2022.
doi: 10.48550/arXiv.2204.02311. URL https://doi.org/10.48550/arXiv.2204.02311.

dataset of minecraft demonstrations.
arXiv:1907.13440, 2019.

arXiv preprint

Gustavosta. Stable-diffusion-prompts. Hugging Face, 2023.
URL
https://huggingface.co/datasets/
Gustavosta/Stable-Diffusion-Prompts.
Hugging Face Datasets.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through worldmodels. arXiv preprint arXiv:2301.04104, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun. Deep residual learning for image recognition. In
2016 IEEE Conference on Computer Vision and Pattern
Recognition, CVPR 2016, Las Vegas, NV, USA, June
27-30, 2016, pages 770–778. IEEE Computer Society,2016. doi: 10.1109/CVPR.2016.90. URL https://doi.org/10.1109/CVPR.2016.90.

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell,
Quoc Viet Le, and Ruslan Salakhutdinov. Transformer-xl:
Attentive language models beyond a fixed-length context. In Anna Korhonen, David R. Traum, and Lluís
Màrquez, editors, Proceedings of the 57th Conference of
the Association for Computational Linguistics, ACL 2019,
Florence, Italy, July 28- August 2, 2019, Volume 1: Long
Papers, pages 2978–2988. Association for Computational
Linguistics, 2019. doi: 10.18653/v1/p19-1285. URL
https://doi.org/10.18653/v1/p19-1285.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.
CoRR, abs/2207.12598, 2022.
doi: 10.48550/arXiv.2207.12598. URL https://doi.org/10.48550/arXiv.2207.12598.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoisingdiffusion probabilistic models. In Hugo Larochelle,Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina
Balcan, and Hsuan-Tien Lin, editors, Advances in Neural
Information Processing Systems 33: Annual Conference
on Neural Information Processing Systems 2020,
NeurIPS 2020, December 6-12, 2020, virtual, 2020.
URL
https://proceedings.neurips.cc/
paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4bAbstract.html.

David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin,
David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-dickstein,
Kevin Murphy, and Charles Sutton. Language model
cascades, 2022.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar,
Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An
Huang, Yuke Zhu, and Anima Anandkumar. Minedojo:
Building open-ended embodied agents with internet-scale
knowledge. In S. Koyejo, S. Mohamed, A. Agarwal,
D. Belgrave, K. Cho, and A. Oh, editors, Advances in
Neural Information Processing Systems, volume 35,
pages 18343–18362. Curran Associates, Inc., 2022.
URL
https://proceedings.neurips.cc/
paper_files/paper/2022/file/
74a67268c5cc5910f64938cac4526a90Paper-Datasets_and_Benchmarks.pdf.

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang,
Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot
manipulation with multimodal prompts. arXiv preprint
arXiv:2210.03094, 2022.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David
Bignell. The malmo platform for artificial intelligenceexperimentation. In Ijcai, pages 4246–4247, 2016.

Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu,
Coline Manon Devin, Benjamin Eysenbach, and Sergey
Levine. Learning to reach goals via iterated supervised
learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/
forum?id=rALA0Xo6yNJ.

Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. CoRR, abs/2305.02463,
2023. doi: 10.48550/arXiv.2305.02463. URL https://doi.org/10.48550/arXiv.2305.02463.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B.
Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec
Radford, Jeffrey Wu, and Dario Amodei. Scaling laws forneural language models. CoRR, abs/2001.08361, 2020.
URL https://arxiv.org/abs/2001.08361.

William H Guss, Brandon Houghton, Nicholay Topin,
Phillip Wang, Cayden Codel, Manuela Veloso, andRuslan Salakhutdinov.
Minerl: A large-scale
10

A Generative Model for Text Control in Minecraft

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 1416, 2014, Conference Track Proceedings, 2014. URL
http://arxiv.org/abs/1312.6114.

Embodied decision making using language guided worldmodelling. arXiv preprint arXiv:2301.12050, 2023.
OpenAI. Introducing ChatGPT, Nov 2022. URL https://openai.com/blog/chatgpt.
Keiran Paster, Sheila A McIlraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv
preprint arXiv:2012.02419, 2020.

Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George
Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes. In The
Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/
forum?id=4-k7kUavAj.

Keiran Paster, Sheila A. McIlraith, and Jimmy Ba. You
can’t count on luck: Why decision transformers and rvs
fail in stochastic environments. In Alice H. Oh, Alekh
Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022a. URL https://openreview.net/
forum?id=atb3yifRtX.

Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang,
Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al.
Multi-game decision transformers. Advances in Neural
Information Processing Systems, 35:27921–27936, 2022.

Keiran Paster, Silviu Pitis, Sheila A. McIlraith, and Jimmy
Ba. Return augmentation gives supervised RL temporalcompositionality. In NeurIPS 2022 Foundation Models
for Decision Making Workshop, 2022b. URL https://openreview.net/forum?id=q5olkWCt7nl.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa,
Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: Highresolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer,
James Bradbury, Gregory Chanan, Trevor Killeen,
Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban
Desmaison, Andreas Köpf, Edward Z. Yang, Zachary
DeVito, Martin Raison, Alykhan Tejani, Sasank
Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai,and Soumith Chintala. Pytorch: An imperative style,high-performance deep learning library. In Hanna M.
Wallach, Hugo Larochelle, Alina Beygelzimer, Florenced’Alché-Buc, Emily B. Fox, and Roman Garnett, editors,
Advances in Neural Information Processing Systems 32:
Annual Conference on Neural Information Processing
Systems 2019, NeurIPS 2019, December 8-14, 2019,
Vancouver, BC, Canada, pages 8024–8035, 2019.
URL
https://proceedings.neurips.cc/
paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740Abstract.html.

Vivian Liu and Lydia B. Chilton. Design guidelines
for prompt engineering text-to-image generative models. In Simone D. J. Barbosa, Cliff Lampe, Caroline Appert, David A. Shamma, Steven Mark Drucker,
Julie R. Williamson, and Koji Yatani, editors, CHI’22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 5 May 2022, pages 384:1–384:23. ACM, 2022. doi:
10.1145/3491102.3501825. URL https://doi.org/
10.1145/3491102.3501825.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay
regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA,May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
Federico Malato, Florian Leopold, Amogh Raut, Ville Hautamäki, and Andrew Melnik. Behavioral cloning via
search in video pretraining latent space. arXiv preprint
arXiv:2212.13326, 2022.

PrismarineJS and Others.
Mineflayer.
https://github.com/PrismarineJS/mineflayer,
2023.
URL
https://github.com/
PrismarineJS/mineflayer.
GitHub repository.

Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea
Finn, and Abhinav Gupta. R3m: A universal visual
representation for robot manipulation. arXiv preprint
arXiv:2203.12601, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.

Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr,Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, and Roy
Fox. Do embodied agents dream of pixelated sheep?:11

A Generative Model for Text Control in Minecraft

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey
Chu, and Mark Chen. Hierarchical text-conditional
image generation with clip latents. arXiv preprint
arXiv:2204.06125, 2022.

2022. doi: 10.48550/arXiv.2205.06595. URL https://doi.org/10.48550/arXiv.2205.06595.
Theodore Sumers, Kenneth Marino, Arun Ahuja, Rob Fergus, and Ishita Dasgupta. Distilling internet-scale visionlanguage models into embodied agents. arXiv preprint
arXiv:2301.12507, 2023.

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez
Colmenarejo, Alexander Novikov, Gabriel Barth-Maron,
Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias
Springenberg, et al. A generalist agent. arXiv preprint
arXiv:2205.06175, 2022.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha
Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, andJason Wei. Challenging big-bench tasks and whether
chain-of-thought can solve them. CoRR, abs/2210.09261,
2022. doi: 10.48550/arXiv.2210.09261. URL https://doi.org/10.48550/arXiv.2210.09261.

Robin Rombach, Andreas Blattmann, Dominik Lorenz,
Patrick Esser, and Björn Ommer. High-resolution image
synthesis with latent diffusion models. In Proceedingsof the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 10684–10695, 2022.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois,
Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instructionfollowing llama model. https://github.com/
tatsu-lab/stanford_alpaca, 2023.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li,
Jay Whang, Emily L. Denton, Seyed Kamyar Seyed
Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol
Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, andMohammad Norouzi.
Photorealistic text-to-image
diffusion models with deep language understanding. In
NeurIPS, 2022. URL http://papers.nips.cc/
paper_files/paper/2022/hash/ec795aeadae0b7d230fa35cbaf04c041Abstract-Conference.html.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar,
Aurélien Rodriguez, Armand Joulin, Edouard Grave,and Guillaume Lample. Llama: Open and efficient
foundation language models. CoRR, abs/2302.13971,
2023. doi: 10.48550/arXiv.2302.13971. URL https://doi.org/10.48550/arXiv.2302.13971.

Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo.
Are emergent abilities of large language models a mirage?
CoRR, abs/2304.15004, 2023.
doi: 10.48550/arXiv.2304.15004.
URL https://doi.org/10.48550/arXiv.2304.15004.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser,
and Illia Polosukhin. Attention is all you need. In Isabelle
Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M.
Wallach, Rob Fergus, S. V. N. Vishwanathan, andRoman Garnett, editors, Advances in Neural Information
Processing Systems 30: Annual Conference on Neural
Information Processing Systems 2017, December 4-9,
2017, Long Beach, CA, USA, pages 5998–6008, 2017.
URL
https://proceedings.neurips.cc/
paper/2017/hash/
3f5ee243547dee91fbd053c1c4a845aaAbstract.html.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport:
What and where pathways for robotic manipulation. In
Conference on Robot Learning, pages 894–906. PMLR,
2022.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning
structured output representation using deep conditional
generative models. In Corinna Cortes, Neil D. Lawrence,
Daniel D. Lee, Masashi Sugiyama, and Roman Garnett,
editors, Advances in Neural Information Processing
Systems 28: Annual Conference on Neural Information
Processing Systems 2015, December 7-12, 2015,
Montreal, Quebec, Canada, pages 3483–3491, 2015.
URL
https://proceedings.neurips.cc/
paper/2015/hash/8d55a249e6baa5c06772297520da2051Abstract.html.

Ryan Volum, Sudha Rao, Michael Xu, Gabriel DesGarennes, Chris Brockett, Benjamin Van Durme, Olivia
Deng, Akanksha Malhotra, and Bill Dolan. Craft
an iron sword: Dynamically generating interactive
game characters by prompting large language models tuned on code. In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022), pages 25–43, Seattle, United States, July
2022. Association for Computational Linguistics. doi:
10.18653/v1/2022.wordplay-1.3.
URL https://aclanthology.org/2022.wordplay-1.3.

Miroslav Strupl, Francesco Faccio, Dylan R. Ashley, Jürgen
Schmidhuber, and Rupesh Kumar Srivastava. Upsidedown reinforcement learning can diverge in stochastic environments with episodic resets. CoRR, abs/2205.06595,
12

A Generative Model for Text Control in MinecraftZihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao
Liang. Describe, explain, plan and select: Interactiveplanning with large language models enables open-world
multi-task agents. arXiv preprint arXiv:2302.01560,
2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten
Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff
Dean, and William Fedus. Emergent abilities oflarge language models. CoRR, abs/2206.07682, 2022a.
doi: 10.48550/arXiv.2206.07682. URL https://doi.org/10.48550/arXiv.2206.07682.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le,
and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H.
Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun
Cho, editors, Advances in Neural Information Processing
Systems, 2022b. URL https://openreview.net/
forum?id=_VjQlMeSB_J.
Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid,
Anthony Brohan, Karol Hausman, Sergey Levine, andJonathan Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv
preprint arXiv:2211.11736, 2022.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han,
Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy
Ba. Large language models are human-level prompt
engineers. In The Eleventh International Conference
on Learning Representations, 2023. URL https://openreview.net/forum?id=92gvk82DE-.

13

A Generative Model for Text Control in Minecraft

A. Broader Impact
With the increasing capability level of artificial intelligence comes many potential benefits and also risks. On the positiveside, we anticipate that the techniques that used to create S TEVE -1 could be applied to the creation of helpful agents in othersequential decision making domains, including robotics, video games, and the web. Our demonstration of such a low cost
approach to creating a powerful, instruction-following model also has the potential to improve the democratization of AI.
However, on the negative side, agents pretrained on large internet datasets reflect the biases of the internet and, as suggested
by our experiments, these pretraining biases can potentially remain after instruction-tuning. If not addressed carefully, this
could lead to devastating consequences for society. We hope that while the stakes are low, works such as ours can improveaccess to safety research on instruction-following models in sequential decision-making domains.

B. Additional Ablations
In this section, we describe additional ablations on design choices for our method, including prompt engineering, the use ofclassifier-free guidance during training, text augmentation strategies, different VAE variants, and varying chunk sizes during
finetuning. We use programmatic evaluation metrics to compare the performance of the various ablations.
B.1. Prompt Engineering
Prompt engineering as a discipline has rapidly emerged over the last year due to the observation that the quality of theoutput of text-to-X models can dramatically change depending on the prompt (Zhou et al., 2023). For example, Table 1
shows how a prompt for Stable Diffusion (Rombach et al., 2022) might be written. By listing out the various attributes of
the image such as visual medium, style, and the phrase “trending on ArtStation”, the user is able to get a higher quality
image (Gustavosta, 2023; Liu and Chilton, 2022). In this section, we explore how this same style of prompt engineering can
improve the performance of S TEVE -1. Figure 6 shows how a simple prompt of “get dirt” might be changed in order to moreaccurately specify the type of behavior that is desired. Just like in image generation models, the performance of S TEVE -1
significantly improves by modifying the prompt in this fashion. By changing to more complicated prompts, S TEVE -1 is able
to collect 1.56x more wood, 2x more dirt, and 3.3x more seeds.
PromptDirt Collected“break a flower”“collect seeds”“dig as far as possible”“get dirt”“get dirt, dig hole, dig dirt,
gather a ton of dirt, collect dirt”

0.7 (-0.2, 1.6)
2.7 (0.9, 4.5)
3.9 (2.8, 5.0)
9.2 (5.7, 12.7)
26.7 (19.9, 33.5)

Figure 6. Similar to in image generation, switching to a longer, more specific prompt dramatically improves the performance of S TEVE -1.

Model

Simple Prompt

Complex Prompt

Stable Diffusion (Rombach et al., 2022)

steampunk market interior

steampunk market interior, colorful, 3D
scene, Greg Rutkowski, Zabrocki, Karlkka, Jayison Devadas, trending on ArtStation, 8K, ultra-wide-angle, zenith
view, pincushion lens effect (Gustavosta,
2023)

S TEVE -1

collect seeds

break tall grass, break grass, collectseeds, punch the ground, run around incircles getting seeds from bushes

Table 1. Example of evolving simple prompts into more complex ones for various models.

14

A Generative Model for Text Control in Minecraft

B.2. Classifier-Free Guidance During Training

7.5
5.0
2.5

60
40
20

guidance (λ = 3)

guidance (λ = 0)0
no guidance (λ = 0)

0.0
no guidance (λ = 0)

guidance (λ = 3)

guidance (λ = 0)

no guidance (λ = 0)

guidance (λ = 3)0
guidance (λ = 0)01

Dirt Collected

100

2

10.0

guidance (λ = 3)

Seeds Collected

200

12.5

guidance (λ = 0)

Wooden Logs Collected

3300

no guidance (λ = 0)

Travel Distance (Blocks)

We examine the importance of using classifier-free guidance during training by finetuning a model with no guidance which
does not drop out the goal embedding zτgoal from the policy’s input (i.e., puncond = 0.0) and comparing it to the version
which uses guidance (puncond = 0.1). The chunk size is set to the range 15 to 50 and we train each policy for 100M frames.
In Figure 7, we compare the performance of using visual goals (MineCLIP video embedding) on the no guidance modelusing conditional scale λ = 0 and the guidance model using conditional scales λ = 0 and λ = 3. We observe that while
the no guidance model slightly outperforms the guidance model at λ = 0 across a few metrics, the agent with guidance
outperforms the no guidance agent by a factor of 2 to 3 times for the inventory collection tasks when we increase theconditional scale to λ = 3 (which we cannot do for the no guidance model). For the travel distance metric, both of theguidance versions perform similarly to the no guidance version.

Figure 7. Ablation on Guidance. In the “no guidance” variant, we set puncond = 0, meaning that we do not drop any zτgoal from thepolicy’s input during training. The “guidance” variants set puncond = 0.1, dropping 10% of the time during training. Whereas the “noguidance” model is only compatible with λ = 0 at inference, the “guidance” model can use λ > 0, allowing for better performance.

B.3. Text Augmentation
During finetuning, instead of using only self-supervision with future MineCLIP video embedding as the goal, we consideredusing the text embeddings from the 2,000 human labeled trajectory segments as goal embeddings, either solely or in addition
to the self-supervised video embeddings. In order to more fairly compare with the CVAE prior approach, we augment
the human-labeled data with additional text-gameplay pairs generated as described in Appendix E.2. We implement thisexperiment by replacing the visual embeddings used for relabeling in Algorithm 1 with text embeddings, when available,with a 90% probability. To experiment with not using visual embeddings at all, we can replace the visual embeddings withzeros in the same way. In Figure 8, we observe that using only the visual embeddings during training, in combination with
the CVAE, can outperform using MineCLIP text embeddings directly in the other two baselines. In this experiment, thechunk size is set to the range 15 to 50 and we train each policy for 100M frames.
B.4. VAE Variants
We study the dataset used to train the CVAE prior model. In Figure 9, we observe that augmentation helps in some
programmatic tasks, including the dirt and seed collection tasks, but slightly hurts the wooden log collection and travel
distance metrics. In this experiment, we use the same policy with each CVAE variant and we tune the conditional scale λ foreach variant. The chunk size is set to the range 15 to 200 and we train the policy for 100M frames.
B.5. Chunk SizeDuring finetuning, we compare different goal chunk sizes by varying the max_btwn_goals=[100,200,300,400],
while keeping the min_btwn_goals=15. See Algorithm 1 for more details. A larger max_btwn_goals introduces
more noise, with actions that led to achieving the further away goal being less correlated to the actions present in that goal
chunk. In Figure 10, we observe that the best max_btwn_goals chunk size is around 200, and increasing the chunk size
beyond that causes a drop in performance. We train each policy for 160M frames and tune the conditional scale for each.
15

8
6
4
2

10
8
6
4
2
visual (text VAE)

text + visual (raw text)

text (raw text)

0
text (raw text)

0

visual (text VAE)

visual (text VAE)

text + visual (raw text)

text + visual (raw text)

0

Dirt Collected2

text (raw text)

0

4

10

visual (text VAE)

100

6

text + visual (raw text)

200

Wooden Logs Collected

Seeds Collected300

text (raw text)

Travel Distance (Blocks)

A Generative Model for Text Control in Minecraft

10
5

20

10

0
Human + Aug

0
Human

Human + Aug

Human

0
Human + Aug

0

5

15

Human

100

10

30

Dirt Collected

Seeds Collected

200

20

Human + Aug

Wooden Logs Collected

15
300

Human

Travel Distance (Blocks)

Figure 8. Ablation on Text Augmentation. In the “text (raw text)” ablation, we train the model using only the text labels from humanlabelled trajectory segments, and directly use the MineCLIP text embedding of the text label as the goal embedding during training
and at inference. For the “text + visual (raw text)” ablation, we use both the visual embedding in self-supervised manner and the text
embedding from the human labelled trajectory segments during training and use the MineCLIP text embedding during inference. Even
with augmentation, the dataset only contained around 2% text embeddings. The “visual (text VAE)” version is as reported in the main
method, using the CVAE to convert MineCLIP text embedding to visual embedding during inference.

Figure 9. Ablation on VAE Training Data. “Human” baseline uses only the 2,000 human-labelled trajectory segments (text-video pairs),
as training example for the CVAE prior model. “Human + Aug” baseline adds additional pairs of text-video examples as described in
Section 3.3.

C. Dataset Details
C.1. Gameplay Dataset
Our gameplay dataset consists of two types of episodes: 7,854 episodes (38.94M frames) of a contractor dataset madeavailable from Baker et al. (2022) and 2,267 episodes (14.96M frames) of gameplay generated by running various pretrained
VPT agents.
OpenAI Contractor Dataset The majority of our data comes from the contractor data used to train VPT (Baker et al.,2022). OpenAI released five subsets of contractor data: 6.x, 7.x, 8.x, 9.x, and 10.x. We use an equal mix of 8.x, 9.x, and10.x, which correspond to “house building from scratch”, “house building from random starting materials”, and “obtain
diamond pickaxe”. Contractors were given anywhere from 10 to 20 minutes to accomplish these goals to the best of theirabilities while their screen, mouse, and keyboard were recorded.
VPT-Generated Dataset We generated additional data by generating episodes using various pretrained VPT agents. In
order to increase the diversity of data as well as to get data of the agent switching tasks randomly throughout the middleof episodes, we added random switching between the different pretrained agents during episodes. Specifically, at the16

A Generative Model for Text Control in Minecraft

300
250
200

10.0
7.5
5.0
2.5

150
200

300

400

Chunk Size

70

20.0

60

17.5
15.0
12.5
10.0

100

200

300

400

40

20
100

Chunk Size

50

30

7.5

0.0
100

22.5

Dirt Collected

Wooden Logs Collected

12.5

Seeds Collected

Travel Distance (Blocks)

15.0
350

200

300

Chunk Size

400

100

200

300

400

Chunk Size

Figure 10. Ablation on Segment Chunk Size. We vary the max_btwn_goals parameter in Algorithm 1. The performance is roughly
the best at around 200, beginning to decline with greater values.

beginning of an episode we randomly sample two VPT agents from (foundation_model_2x, bc_early_game_2x,bc_house_3x, rl_from_foundation_2x, rl_from_house_2x) and switch between them at each timestep witha probability of 1/1000. Since the RL agents all act quite similarly, we avoid sampling two RL agents at once. Additionally,
with a probability of 1/750 each timestep, we cause the agent to spin a random number of degrees. This adds more data
where the agent spontaneously changes tasks, increasing downstream steerability.
C.2. Instruction Dataset
We gathered a small dataset of 2,000 human-labelled trajectory segments (text-video pairs) by manually labeling gameplay
from our datasets. We used a simple web app that presented a video of 16 frames to the user from a randomly sampled
episode. This only corresponds to 32,000 frames of labeled data, which corresponds to labeling 0.06% of the full dataset, or
27 minutes of labeled data. However, as discussed in section E.2, combining this with automatically labeled data using
gpt-3.5-turbo and MineCLIP results in a strong prior model.
Algorithm 1 Sampling Episode Segments with Packed Hindsight Relabeling
Function sample_episode_segment(T, min_btwn_goals, max_btwn_goals)
segment = sampleSegment(episode, T)
curr_timestep = segment.start
goal_switching_indices = []while curr_timestep < segment.end do
curr_timestep += uniform(min_btwn_goals, max_btwn_goals)
goal_switching_indices.append(curr_timestep)
relabeled_goal_embeds = []for n in range(1, len(goal_switching_indices)) dorelabeled_goal_embeds[in−1 :in ] = segment.goal_embeddings[in ]return segment.obs, segment.actions, relabeled_goal_embeds

D. Evaluation Details
Evaluating the performance of our agent is a challenging task due to the wide variety of instructions that are possible andthe difficulty of evaluating whether the agent has successfully achieved its task. We use a combination of programmatic
evaluation metrics and automatic MineCLIP evaluation metrics to get a sense of the agent’s capability level.
Programmatic Evaluation We compute programmatic evaluation metrics by monitoring the MineRL (Guss et al.,2019) environment state throughout each evaluation episode. As done in VPT (Baker et al., 2022), we compute multiple
programmatic metrics including travel distance and early-game item collection. The travel distance is the maximum
displacement of the agent along on the horizontal (X-Z) plane, measured from the initial spawn point. For early-gameinventory counts, we store the maximum number of log, seed, and dirt items seen in the agent’s inventory during the episode.
17

A Generative Model for Text Control in Minecraft

MineCLIP Evaluation For any trajectory τ we can use the text-visual alignment of MineCLIP embeddings (Fan et al.,2022) to roughly evaluate whether a segment of 16 frames corresponds to a given task. We explore the use of alignment in
the MineCLIP latent space between trajectories and either text or visual goals to evaluate our agent over a wider variety oftasks where programmatic evaluation isn’t practical. To determine the degree to which a task has been completed at all
during an episode, we record the minimum cosine distance between the (text or visual) goal embeddings at any timestep
during an episode.

E. Training Details
E.1. Policy Training
S TEVE -1 was trained using distributed data parallel in PyTorch (Paszke et al., 2019). During training, segments of 640
timesteps were sampled from the dataset. Due to memory constraints, these segments were further broken up into chunksof 64, which are processed sequentially. Since VPT uses a Transformer-XL (Dai et al., 2019), this sequential processing
lets the policy attend to previous batches up to the limit of its context length. We optimized the weights using AdamW(Loshchilov and Hutter, 2019) with a maximum learning rate of 4e-5 and a linear warmup for the first 10M frames followed
by a cosine learning rate decay schedule that decays to 10% of the original learning rate. See Table 3 for an exhaustive listof hyperparameters used during training.
During training, we sample data using packed hindsight relabeling (Figure 2). This involves sampling a segment ofan episode, randomly selecting some timesteps at which to change goals, and then filling in the corresponding goal
embeddings for the entire episode with the embeddings from the corresponding goal segments. See Algorithm 1 for a
detailed explanantion of packed hindsight relabelling.
Hyperparameter Name

Valuearchitecture

MLP

hidden_dim

512latent_dim

512hidden_layers

2batch_size

256

learning_rate

1e-4

β

0.001

n_epochs50

n_search_episodes

2000

k

5

offset8

Table 2. Prior Hyperparameters

E.2. Prior Training
The prior model is a simple CVAE (Sohn et al., 2015) that conditions on MineCLIP (Fan et al., 2022) text embeddings andmodels the conditional distribution of visual embeddings given the corresponding text embedding. This model is trainedon a combination of around 2,000 hand-labeled trajectory segments and augmented with additional data by automaticallysearching for text-gameplay pairs from our gameplay dataset. This is done using the following steps:
1. Combine the 2,000 text labels with 8,000 additional labels generated by querying gpt-3.5-turbo.
2. For each of these 10,000 text labels, search through 1,000 episodes sampled from the gameplay dataset to find the top 5
18

A Generative Model for Text Control in Minecraft

Hyperparameter Name

Value

trunc_t

64

T

640

batch_size

12

num_workers

4

weight_decay

0.039428

n_frames

160M

learning_rate

4e-5

optimizer

AdamW (Loshchilov and Hutter, 2019)

warmup_frames

10M

p_uncond

0.1

min_btwn_goals

15

max_btwn_goals

200

vpt_architecture

2xTable 3. Policy Hyperparameters

closest visual MineCLIP embeddings to the text embedding of the text label.
These 50,000 automatically-mined text-video pairs are added to the original 2,000 hand-labeled examples to form the final
dataset used for prior training.
We noticed when prompting S TEVE -1 using visual goals that when the visual goal showed the agent hitting a block but not
following through and breaking it that S TEVE -1 actually avoided breaking blocks. Unfortunately, many of the automaticallydiscovered text-gameplay clips include gameplay of this kind. In order to prevent this issue, we added an offset to theembeddings found in this manner. By selecting embeddings from a timestep offset steps after the originally-selected
timestep, the agent is much more likely to follow through with breaking blocks.
We trained our prior model for 50 epochs on this dataset and used early-stopping with a small validation set. An exhaustive
list of hyperparameters used for creating the prior model can be found at Table 2.

F. Additional Visualizations
F.1. MineCLIP Evaluation
We ran MineCLIP evaluation on both text and visual prompts. The MineCLIP evaluation results can be found in Figure 13.
F.2. Steerability with Programmatic Metrics
Similar to Figure 20 in the VPT appendix (Baker et al., 2022), we plot the programmatic metric performances (meanand 95% confidence intervals) across the different goal prompt conditioning, both using visual prompts (Figure 15) andtext prompts with CVAE prior (Figure 16) conditioning, on our policy trained with hyperparameters in Table 3 and using
conditional scaling λ = 7 (for visual prompts) and λ = 6.0 (for text prompts with CVAE prior). Each conditioning variant
is run with 10 trials, each trial with a different environmental seed and with an episode length of 3000 timesteps (2.5
minutes gameplay). Across the conditioning variant, we use the same set of environmental seeds. For comparison, we
also plot the metrics for an unconditional VPT (early_game) agent (“VPT (uncond)”) and the text-conditioned agent
investigated in VPT appendix (Baker et al., 2022) (“VPT (text)*”) when conditioned on the relevant text. When using visual
19

A Generative Model for Text Control in Minecraft

dig
dir
t
sk
y
lea
woves
o
see d
flo ds
w
ex er
plo
sw re
i
un m
de
inv rwen atetor ryConditioned Prompt

Evaluation Prompt

dig 1
0
dirt
0
sky
0
leaves
0
wood
0
seeds
0
flower0
explore0
swim
1
underwater0
inventory

Figure 11. MineCLIP Text Evaluation

0.12
0.10
0.08
0.06
0.04
0.02

Figure 12. MineCLIP Visual Evaluation

Figure 13. MineCLIP Evaluation. We measure the cosine distance between the goal embedding given to the agent and the MineCLIP
video embeddings throughout the episode and record the minimum across the episode. Dashed box indicates the minimum along therow, and the number in the diagonal box indicates the rank of the diagonal element (0 is minimum) in the row. Left: We use the prior
to convert the text into the goal embedding. Across our 11 text MineCLIP evaluation tasks, S TEVE -1 achieves the shortest distance
between the episode and the MineCLIP goal embedding when prompted appropriately for most cases. This shows the strong general
performance of S TEVE -1 across a wide variety of short-horizon tasks. Right: We embed the visual goal loops (Figure 14) with MineCLIP
video encoder. Across our 11 visual MineCLIP evaluation tasks, S TEVE -1 achieves the shortest distance between the episode and the
MineCLIP goal embedding when prompted appropriately except for in two cases, where it mixes up digging and dirt and swimming andgoing underwater. This shows the strong general performance of S TEVE -1 across a wide variety of short-horizon tasks.

goal conditioning, we the use MineCLIP video encoder to embed a 16-frame clip of the agent performing the desired tasktaken from our training dataset. An example frame from each of the visual goals is illustrated in Figure 14. When using text
VAE goal conditioning, we use the MineCLIP text encoder to encode the text prompts (Table 4) and use the CVAE prior to
sample the goal embedding from the MineCLIP text embedding.
We note several differences in our experimental setup compared to that in VPT (Baker et al., 2022). We only run ourevaluation episodes for 3000 timesteps, equivalent to 2.5 minutes of gameplay, compared to 5 minutes in the VPT paper. Dueto a limited computational budget, we generate 10 episodes per conditioning variant, and 110 episodes for the unconditional(“VPT (uncond)”), compared to VPT’s 1000 episodes. Lastly, when measuring the inventory count, we log the maximum
inventory count seen throughout the episode, which is a lower bound on the potential number of items collected since theagent can later throw out, place, or use these items to craft. As a result of these caveats, we denote the “VPT (text)*” legend
in Figure 15 and Figure 16 with an asterisk as we use the results reported in (Baker et al., 2022) directly for comparison.
We make several observations. First, we observe that our agents is more steerable: when conditioned to collect certain items(in bold), the agent collects (relatively) many more of those items than when conditioned on other instructions unrelated tothat item, as well as compared to the unconditional VPT. When conditioned on tasks unrelated to the item (e.g. break a flower
when interested in measuring logs collected), we also observe that the agent pursues that item less than the unconditional
agent. Second, we observe that for the bolded instructions which we expect to stand out, we outperform VPT performance(dashed blue line) (Baker et al., 2022), even with half the amount of time in the episode rollout. This suggests that our agent
is both more steerable relative to the unconditioned VPT agent and the text-conditioned VPT agent investigated in the VPTappendix (Baker et al., 2022).
F.3. Prompt Chaining Visualization
We visualize two specific episodes from the prompt chaining experiments in Section 4.3 in Figure 17 and Figure 18.

20

A Generative Model for Text Control in Minecraft

Figure 14. Sample frames from each of the 11 visual goals. Note that the text overlaid on the frame is not present when we encode the
16-frame clip with MineCLIP video encoder, and is only present for the figure visualization.

Conditioning Variant Namedig as far as possible
get dirt
look at the sky
break leaves
chop a tree
collect seeds
break a flower
go explore
go swimming
go underwater
open inventory
get dirt . . .
chop down the tree . . .
break tall grass . . .

Text Prompt
dig as far as possible
get dirt
look at the sky
break leaves
chop a tree
collect seeds
break a flower
go explore
go swimming
go underwater
open inventory
get dirt, dig hole, dig dirt, gather a ton of dirt, collect dirt
chop down the tree, gather wood, pick up wood, chop it down, break tree
break tall grass, break grass, collect seeds, punch the ground, run around in circles
getting seeds from bushes

Table 4. Strings corresponding to each conditioning variant for the text VAE conditioning. For conditioning variants without “. . . ”, thetext prompt is the same as the conditioning variant name.

21

A Generative Model for Text Control in Minecraft

Seed Collection with Conditioning

Travel Distance with Conditioning
VPT (text)*

VPT (text)*

8

VPT (uncond)

VPT (uncond)250

Seeds Collected

200
150
100

6
4
2

50
0

go underwater

go swimming

go explore

break a flower

collect seeds

break wood

Dirt Collection with Conditioning70

VPT (text)*

VPT (text)*
60

VPT (uncond)15

Dirt Collected

105

VPT (uncond)

50
40
30
20
10

Conditioning (Visual)

open inventory

go underwater

go swimming

go explore

break a flower

collect seeds

break woodbreak leaves

look at the sky

dig a holeopen inventory

go underwater

go swimming

go explore

break a flower

collect seeds

break woodbreak leaves

look at the sky

get dirt

0
dig a hole0

get dirt

Wooden Logs Collected

break leaves

Conditioning (Visual)

Log Collection with Conditioning
20

open inventory

Conditioning (Visual)look at the sky

dig a holeopen inventory

go underwater

go swimming

go explore

break a flower

collect seeds

break woodbreak leaves

look at the sky

get dirt

dig a hole0
get dirt

Travel Distance (Blocks)

300

Conditioning (Visual)

Figure 15. Conditioning with Visual Goals. We plot the performance of the programmatic metrics, along with their mean values and
95% confidence intervals, across different goal conditioning. See Figure 14 for visualization of these visual prompts. Plots are similar to
Figure 20 in the VPT appendix (Baker et al., 2022). Each conditioning variant is run with 10 trials, each with a different environmental
seed and with an episode length of 3000 time steps (2.5 minutes gameplay). We use the policy that was trained using the hyperparametersspecified in Table 3, and with conditional scaling values λ = 7. The dashed horizontal lines refer to an unconditional VPT agent (“VPT(uncond)”) and a text-conditioned agent from the VPT appendix (“VPT (text)*”) that was conditioned on the relevant text, for the purposeof comparison.

22

10

6

4

2

0

Conditioning (Text)

23
25
VPT (uncond)

20

15

10

5

0

go explore

get dirt ...

open inventory

go underwater

go swimmingbreak tall grass ...

VPT (text)*

break tall grass ...

Dirt Collection with Conditioning

chop down the tree ...

Conditioning (Text)

chop down the tree ...

get dirt ...

30

go underwater

VPT (uncond)

open inventory

VPT (text)*

go swimming

Log Collection with Conditioning

go explore

Conditioning (Text)
collect seeds

0.0
break a flower

0

break a flower

2.5

chop a tree

50

collect seeds

100

break leaves

150chop a tree

Travel Distance with Conditioning

break leaves

15.0

look at the sky

200

get dirt

dig as far as possible

250

Seeds Collected

VPT (uncond)look at the sky

8Dirt Collected

break tall grass ...

chop down the tree ...

get dirt ...

open inventory

go underwater

go swimming

go explore

break a flower

collect seeds

chop a tree

break leaves

look at the sky

VPT (text)*

get dirt

dig as far as possible

break tall grass ...

chop down the tree ...

get dirt ...

open inventory

go underwater

go swimming

go explore

break a flower

collect seeds

chop a tree

12

break leaves

get dirt

dig as far as possible

Travel Distance (Blocks)
300

look at the sky

get dirt

dig as far as possible

Wooden Logs Collected

A Generative Model for Text Control in Minecraft

Seed Collection with Conditioning

VPT (text)*

12.5
VPT (uncond)

10.0
7.5

5.0

Conditioning (Text)

Figure 16. Conditioning with text goals. See Table 4 for the text string used for each conditioning variant. We use the same policy model
but with a conditional scaling value λ = 6. We observe strong steerability which outperforms VPT text conditioning in (Baker et al.,2022), and we observe that more specific prompt engineered text can lead to improvements.

Build a Tower

Dirt in Inventory

35

1

30

2

3

105

4

100

25

95

20

90

15

85

10

80

5

75

0

0

500

1000

1500

2000

2500

3000

70

Agent Height Position

A Generative Model for Text Control in Minecraft

1: Frame #52

2: Frame #1500

3: Frame #2015

4: Frame #2540

Episode Timestep

Make Wooden Planks

Logs in Inventory

7

1

2

3

35

4

6

30

5

25

4

20

3

15

2

10

1

5

0

0

500

1000

1500

2000

2500

3000

0

Planks in Inventory

Figure 17. Build a Tower task. (Left) We track the amount of dirt in the inventory and the agent’s height position (y-axis) throughout theepisode. In the first 1500 timesteps, the agent is conditioned on the visual get dirt goal, then the agent is conditioned on the visual build atower goal for the final 1500 timesteps. Vertical dotted lines with numbers indicate the corresponding frames on the right. (Right) The
agent’s observation frames at 4 different points in the episode. First the agent collects dirt, then begins to build the tower using the dirt
blocks.

1: Frame #220

2: Frame #876

3: Frame #1500

4: Frame #1540

Episode Timestep

Figure 18. Make Wooden Planks task. (Left) We track the number of logs and planks in the inventory. In the first 1500 timesteps, theagent is conditioned on the visual break wood goal, then the agent is conditioned on crafting the visual wooden planks goal for the final
1500 timesteps. Similarly to Figure 17, a vertical dotted line annotated with a number indicates the corresponding frame to the right.
(Right) The agent’s observation frames at 4 different points in the episode. First the agent breaks trees to collect wooden logs, then opens
the inventory and crafts wooden planks.

24