SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Due to the hierarchical nature of our model, we can evaluate the performance of our agent at achieving either text or visual goals simply by choosing whether to use the prior to condition on text or bypass the prior and condition on a MineCLIP video embedding directly. We first tested our model on a set of 11 tasks that are achievable within the first 2.5 minutes of gameplay and which do not require multiple steps to complete (e.g., chop a tree or dig a hole, but not build a house). A complete list of the tasks and prompts we used for evaluation can be found in Table 3 in the appendix. To select visual goals for testing each of the evaluation tasks, we implemented a tool that searches through
$^{3}$ https://github.com/openai/Video-Pre-Training
(a) Programmatic Evaluation
Figure 3: Left: In our programmatic evaluations, STEVE-1 performed far better than the unconditional VPT agent early-game-2x and the text-conditioned VPT agent when prompted appropriately. The asterisk * in the "VPT (Text)*" indicates that this result was taken from Appendix I in [5], which had twice the episode length compared to our setting. On some tasks, visual outperforms text-based prompting, creating a gap that can likely be bridged through better prompt engineering. Right: Across our 11 MineCLIP evaluation tasks, STEVE-1 achieves the shortest distance between the episode and the MineCLIP goal embedding when prompted appropriately except for in two cases, where it mixes up digging and dirt and swimming and going underwater. This shows the strong general performance of STEVE-1 across a wide variety of short-horizon tasks. The dashed box marks minimum element along the row, and the diagonal number signifies the diagonal element's rank (0 means it is the minimum row element). See Figure 14 for sample frames from each of the 11 visual goals and Figure 13 for a success-rate version of this matrix.
(b) MineCLIP Evaluation
$10\%$  of our gameplay dataset by finding the closest 16-frame videos to a given text prompt. We then manually selected a 16-frame video that clearly demonstrates the task being completed and use the corresponding MineCLIP video embedding as the goal embedding for that task. Screenshots of these visual goals can be found in Figure 14 in the appendix.
In Figure 3, we compare the performance of our text and visual-conditioned agents with the unconditional VPT agent and text-conditioned VPT agent (Appendix I in [5]) across our programmatic tasks. We find that when given the relevant text instruction, STEVE-1 collects  $75 \times$  more dirt,  $4.9 \times$  more wood,  $22 \times$  more seeds, and travels  $4.3 \times$  farther than the unconditional agent, and STEVE-1 collects  $3.3 \times$  more dirt,  $4.4 \times$  more wood,  $8.1 \times$  more seeds, and travels  $2.2 \times$  farther than the text-conditioned VPT agent. This represents a significant improvement over the reported performance of text-conditioned VPT, which collects several times fewer resources despite having twice as long of an episode to do so. We also run an automatic evaluation using MineCLIP embedding distances by measuring the minimum distance of a goal embedding to any frame in the episode. As shown in Figure 3b, the distance between the goal and the episode is significantly lower when the agent is conditioned on the corresponding visual goal than otherwise. Full results for STEVE-1 with both text and visual goals can be found in Appendix F.
In addition to our evaluations of STEVE-1, we also recorded several sample interactive sessions we had with the agent (controlling it in real-time by giving it written text instructions or specific visual goals). These sessions demonstrate STEVE-1's ability to responsively follow instructions in real-time in a variety of situations. We believe that such use-cases, where humans give an agent natural instructions that it can follow to complete tasks, will become increasingly important and have practical uses in the creation of instructable assistants and virtual-world characters. These videos, as well as videos of our agent performing our evaluation tasks, can be found at https://sites.google.com/view/steve-1.
4.3 Prompt Chaining
We also experiment with longer horizon tasks that require multiple steps, such as crafting and building. We explore two different prompting methods: directly prompting with the target goal, and a simple form of prompt chaining [11, 64, 16] where the task is decomposed into several subtasks and the
Figure 4: Top left: By sequentially chaining visual prompts like "get dirt" and "build a tower", STEVE-1 successfully gathers dirt and then uses this dirt to build a tower. The prompts switch at the dotted vertical line. Bottom left: The success rates of the chained prompts improve steadily as we train STEVE-1 on more data. Right: The performance of STEVE-1 in different tasks scales in different ways when conditioning on a relevant visual prompt for the metric versus other irrelevant visual prompts (e.g., the break wood prompt is the relevant prompt for the "Wooden Logs Collected" metric, while the other prompts are "irrelevant"). For instance, in the wood-collection and dirt-collection tasks, performance starts increasing after training on 60M frames of gameplay. See Figure 14 for sample frames from each visual prompt.
prompts are given sequentially for a fixed number of steps. We explore prompt chaining with visual goals for two tasks: 1) building a tower and 2) making wooden planks. When using prompt chaining, we first prompt STEVE-1 to gather dirt before building a tower, and to gather wooden logs before crafting wooden planks. Figure 4 shows that directly prompting STEVE-1 with the final tasks results in near-zero success rates. However, prompt chaining allows STEVE-1 to build a tower  $50\%$  of the time and craft wooden planks  $70\%$  of the time. For the tower building task, STEVE-1 immediately starts collecting dirt until the prompt switches, at which point its average height starts increasing rapidly and its dirt decreases as it builds a tower. Similarly, for the crafting wooden planks task, STEVE-1 immediately starts collecting a large amount of wooden logs until the prompt switches and it rapidly converts these wooden logs into wooden planks (causing the amount of wooden logs in its inventory to immediately decrease and the number of wooden planks to increase as it crafts more). Figure 4 visualizes the average item counts and agent height for the prompt chaining episodes. See Figure 18 and Figure 19 in the appendix for visualizations of specific prompt chaining episodes.
Recent works in language modeling have found that scaling up pretraining FLOPs, by training on more data or by training a model with more parameters, can improve performance on downstream tasks [28, 57, 63]. In certain cases when measuring performance with metrics such as exact-match [52], performance improvement may appear to be "emergent" [63], appearing suddenly as the model is trained with more compute. Here, we aim to gain a basic understanding of how the performance of STEVE-1 on various tasks scales by training with more data (learning rate schedule is chosen appropriately).
To assess performance gain, we first isolated the performance of the policy from the prior, measuring performance of the agent through training on programmatic tasks (travel distance, seeds, logs, dirt) with visual goals. Due to compute constraints, we chose to use the 2x VPT model, which has 248M parameters. We found that both seed collection and travel distance did not improve significantly past 20M frames. From inspecting gameplay, we suspect that travel distance is a relatively easy task since it is close to VPT's default behavior of running around and exploring. For seed collection, performance remains suboptimal, suggesting that further scaling may be beneficial. This hypothesis is supported by the observation that performance on log and dirt collection remained roughly level until 60M frames when it began to rapidly improve. Figure 4 shows the scaling curves for STEVE-1 on each programmatic task when conditioning on relevant vs. irrelevant visual prompts for that task. Since we do not observe regression on any tasks as we train the model with more compute, we expect the model to continue to perform better as we train larger models on larger datasets.
We also evaluated the scaling properties of STEVE-1 for our multi-step tasks with and without prompt chaining. Without prompt chaining, the tasks remain challenging for STEVE-1 throughout training. However, we note that after 60M frames, STEVE-1 learns to gather wooden logs and build a small tower when told to build a tower. This is likely because our visual prompt for tower building shows a video of a tower being built out of wooden logs. With prompt chaining, the performance of STEVE-1 steadily increases with more data. We conjecture that this is because the success of a chained prompt requires the success of each element in the chain. Since different abilities emerge at different scales, one would expect chained prompts to steadily get more reliable as these subgoals become more reliably completed. In the case of building wooden planks, we note that crafting is one such task that gets significantly more reliable as the agent is trained on more data. Figure 4 shows the scaling curves for STEVE-1 on the prompt chaining tasks.
In summary, we see evidence of tasks that do not require much data for STEVE-1 to learn, tasks that steadily get more reliable as the agent is trained longer, and tasks where capability suddenly spikes after the agent reaches some threshold. Put together, this suggests that further scaling would likely significantly improve the agent, although we leave the task of predicting exactly how much performance there is to gain to future studies.
4.5 What Matters for Downstream Performance?
Pretraining Baker et al. [5] finds that by pretraining a behavioral prior with imitation learning on internet-scale datasets for Minecraft, the learned policy can be effectively finetuned to accomplish tasks that are impossible without pretraining. In this section, we demonstrate that pretraining is also massively beneficial for instruction-tuning in Minecraft. We hypothesize that due to the strong performance of STEVE-1 and the relatively small amount of compute ( $\approx 1\%$  additional compute) used for instruction finetuning, most of the capabilities of our agent come from the pretraining rather than the finetuning. To test this hypothesis, we finetune several variants of STEVE-1 from various pretrained weights: foundation-2x, bc-early-game-2x, rl-from-foundation-2x, and with randomly initialized weights. In this experiment, each model was finetuned on 100M frames.
Figure 5 shows the performance of these models on our programmatic tasks with visual goals. Note that while an agent trained on our dataset from scratch can accomplish basic tasks like dirt collection fairly well, it is unable to find and chop down trees, in contrast to the pretrained agents. This demonstrates that the abilities present in the agent due to pretraining are successfully transferred to the fine-tuned agent. Out of all the pretrained weights we tried, we noticed that rl-from-foundation-2x performed the best, having qualitatively better performance at tasks like crafting and chopping down trees. Indeed, Figure 5 shows that this model has strong performance, likely due to the massive amount of compute it was trained with during its RL training [5].
Classifier-Free Guidance Baker et al. [5] observed that when conditioning the agent on text, it tended to ignore its instruction and instead perform the prior behavior learned during pretraining. As discussed in section 3.4, classifier-free guidance gives a knob for trading off between goal-conditioned and prior behaviors. Figure 5 shows the effect of this parameter  $\lambda$  on the log and dirt collection tasks. The performance of the agent reaches its maximum around  $\lambda = 5.0$  to  $\lambda = 7.0$ , after which it starts to drop off. These results demonstrate the importance of classifier-free guidance, which improves the performance of STEVE-1 by orders of magnitude.
Prompt Engineering Prompt engineering as a discipline has rapidly emerged over the last year due to the observation that the quality of the output of text-to-X models can dramatically change depending on the prompt [67]. For example, Table 5 in the appendix shows how a prompt for Stable Diffusion [50] might be written. By listing out the various attributes of the image such as visual medium, style, and the phrase "trending on ArtStation", the user is able to get a higher quality image [20, 36]. In this section, we explore how this same style of prompt engineering can improve the performance of STEVE-1. Figure 6 shows how a simple prompt of "get dirt" might be changed in order to more accurately specify the type of behavior that is desired. Just like in image generation models, the performance of STEVE-1 significantly improves by modifying the prompt in this fashion. By changing to more complicated prompts, STEVE-1 is able to collect  $1.6 \times$  more wood,  $2 \times$  more dirt, and  $3.3 \times$  more seeds.
Pretrained Weights
Figure 6: Similar to in image generation, switching to a longer, more specific prompt dramatically improves the performance of STEVE-1. Values in parentheses are  $95\%$  confidence intervals.
Figure 5: Left: We trained STEVE-1 on 100M frames starting from four different pretrained weights: random initialization (scratch), foundation-2x (fd), bc-early-game-2x (bc), and rl-from-foundation-2x (rl). The rl-from-foundation-2x agent is generally the most performant after fine-tuning. Using pretrained weights performs better than training from scratch, especially for more complicated tasks like collecting wood. Right: By using classifier-free guidance [23], STEVE-1 collects  $7.5 \times$  more dirt and  $15 \times$  more wood than when  $\lambda = 0$  (no guidance). See Figure 17 in the Appendix for similar results with other programmatic tasks.
Conditional Scale  $\lambda$
Prompt	Dirt Collected
“break a flower”	0.7 (-0.2, 1.6)
“collect seeds”	2.7 (0.9, 4.5)
“dig as far as possible”	3.9 (2.8, 5.0)
“get dirt”	9.2 (5.7, 12.7)
“get dirt, dig hole, dig dirt, gather a ton of dirt, collect dirt”	26.7 (19.9, 33.5)
In this paper, we present a methodology for creating instruction-following foundation models of behavior. Specifically, by leveraging two existing pretrained foundation models: a behavioral prior (VPT [5]) and a domain-specific CLIP model (MineCLIP [17]), we create a powerful Minecraft agent that can follow short-horizon open-ended text and visual instructions, all for only $60 of compute. The resulting foundation model, STEVE-1, sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines and robustly completing 12 of 13 tasks in our early-game evaluation suite. We note that generalist agents such as STEVE-1 can have potential negative effects on society. We include a thorough discussion of these issues in Appendix A.
STEVE-1 is a significant advancement in creating generative models of text-to-behavior, but it has several limitations, as described in Appendix B. First, STEVE-1 is mostly proficient at achieving short-horizon tasks while struggling with longer-horizon tasks. While prompt chaining is a promising approach for improving performance on complex tasks, more can be done in future work to improve performance. Another limitation we observe is that prompt engineering, as with other generative models, can be unintuitive and time-consuming. Future work should investigate improving the steerability of STEVE-1 through a better understanding of natural language prompts. Additionally, we note that evaluating and describing the capabilities of open-ended generalist agents is an open research problem itself since capability depends strongly on preconditions, prompt engineering, and our own ability to come up with varied and challenging tasks. Finally, since our approach is not specific to the Minecraft domain, we hope that the method used to create STEVE-1 can inspire future work in creating powerful generalist agents in other domains and environments.
All of the authors gratefully acknowledge funding for this research from the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canada CIFAR AI Chairs Program (Vector Institute for Artificial Intelligence). SL is supported by a Vector Institute internship and by an NSERC Discovery Grant. KP is supported by an NSERC PGS-D award. HC is supported by an NSERC CGS-D award. JB acknowledges funding from the Canada CIFAR AI Chairs program, Fujitsu Japan, and an Amazon Research Award. In addition to NSERC and CIFAR (Vector Institute), SM acknowledges funding from Microsoft Research. We thank Silviu Pitis, Romi Lifshitz, Forest Yang, and Yongchao Zhou for their helpful comments; Alisa Wu and Ziming Chen for their contributions to the text-video pair dataset; and Finn Paster for the logo and graphic for the website. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute for Artificial Intelligence (www.vectorinstitute.ai/partners).
[1] Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse H. Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matthew Sharifi, Neil Zeghidour, and Christian Havnø Frank. Musicl: Generating music from text. CoRR, abs/2301.11325, 2023. doi: 10.48550/arXiv.2301.11325. URL https://doi.org/10.48550/arXiv.2301.11325.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716-23736, 2022.
[3] Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5048-5058, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/453fadbd8a1a3af50a9df4df899537b5-AAbstract.html.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[5] Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
[6] David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? arXiv preprint arXiv:2206.01079, 2022.
[7] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfbcb4967418bf8ac142f64a-Paper.pdf.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017. URL https://proceedings.neurips.cc/paper/2017/bit/3f5ee243547dee91fbd053c1c4a845aa-Abscract.html.
[61] Ryan Volum, Sudha Rao, Michael Xu, Gabriel DesGarennes, Chris Brockett, Benjamin Van Durme, Olivia Deng, Akanksha Malhotra, and Bill Dolan. Craft an iron sword: Dynamically generating interactive game characters by prompting large language models tuned on code. In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022), pages 25-43, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022(wordplay-1.3. URL https://aclanthology.org/2022(wordplay-1.3.
[62] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560, 2023.
[63] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. CoRR, abs/2206.07682, 2022. doi: 10.48550/arXiv.2206.07682. URL https://doi.org/10.48550/arXiv.2206.07682.
[64] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian richter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=_VjQlMeSB_J.
[65] Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint arXiv:2211.11736, 2022.
[66] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
[67] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=92gvk82DE-.
With the increasing capability level of artificial intelligence comes many potential benefits and also risks. On the positive side, we anticipate that the techniques that used to create STEVE-1 could be applied to the creation of helpful agents in other sequential decision making domains, including robotics, video games, and the web. Our demonstration of such a low cost approach to creating a powerful, instruction-following model also has the potential to improve the democratization of AI. However, on the negative side, agents pretrained on large internet datasets reflect the biases of the internet and, as suggested by our experiments, these pretraining biases can potentially remain after instruction-tuning. If not addressed carefully, this could lead to devastating consequences for society. We hope that while the stakes are low, works such as ours can improve access to safety research on instruction-following models in sequential decision-making domains.
B Limitations and Future Work
B.1 Goal Misgeneralization