SlowGuess/NeurIPS_2023_OCR huggingface dataset:
One of the most common mistakes that STEVE-1 makes during evaluation is to overgeneralize. For instance, if we prompt STEVE-1 with a video of someone punching a cow, it may simply run to the nearest animal and punch that animal instead. This is related to the concept of goal misgeneralization [31]. Generalization can be helpful when the task we assign the agent is impossible to achieve from the current state and the agent instead performs a closely related action, but harmful when the task is achievable. We note two things: first, we believe the powerful generalization ability of STEVE-1 probably comes from the MineCLIP embeddings and it especially improves the ability of STEVE-1 to follow visual instructions when the exact items or blocks nearby are not available in the current environment, which is an extremely common scenario. Second, we notice that the tendency of the agent to misgeneralize decreases with scale. For example, with a model trained on less data, we find that asking the agent to look up and punch a tree to get a wooden log often resulted in the agent looking in the air and punching nothing; training the model on more data results in the agent first walking over to a nearby tree and looking up to get a wooden log. Future work should look to measure more closely the relationship between misgeneralization and scale.
B.2 Random Selection of Hindsight Goals
In this work, we choose to randomly select future episode timesteps as hindsight goals, rather than use a more sophisticated strategy. This is primarily due to the simplicity of the approach, but also to ensure a diverse and unbiased coverage of potential goals achievable within the short horizon. Future works can investigate the effects of alternative approaches that filter for semantically interesting timesteps as goals.
B.3 Difference in Performance Between Text and Visual Goals
In our experiments, we observed that STEVE-1 often performed better when conditioned with visual goals compared to text goals converted through the prior. There are several potential factors that could contribute to this performance gap between text and visual conditioning. First, the prior model may not be accurately capturing the full meaning of the text prompt. Training the prior on more data or using a more powerful model architecture could potentially improve the quality of the sampled latent goals. Second, the visual goals can provide more precise demonstrations of the desired behavior. Text goals are inherently more ambiguous. Providing additional information such as the observation context to the prior and further prompt-engineering to make text prompts more detailed and less ambiguous, could help close this gap. Because text conditioning provides more flexibility and potential for generalization, closing the performance gap between text and visual conditioning is an important direction for future work.
B.4 Challenges in Evaluating STEVE-1
See Table 4 for a non-exhaustive list of tasks that the STEVE-1 agent is able to achieve. It is worth mentioning that evaluating and describing the capabilities of open-ended generalist agents is an open
research problem itself since capability depends strongly on preconditions, prompt engineering, and our own ability to come up with varied and challenging tasks. For example, there are many recent works on the evaluation of LLMs (e.g., [34, 33, 66]) which highlight these challenges.
That being said, there are a number of tasks which STEVE-1 is unable to accomplish. As previously mentioned, long-horizon tasks such as obtaining a diamond or building a house are currently beyond the capability of STEVE-1. Further, STEVE-1 also struggles with more complex crafting tasks like crafting an enchanting table, bookcase, or boat. Again, the virtually limitless and open-ended nature of tasks in Minecraft makes it very difficult to test generalist agents in this domain. We hope future works develop more sophisticated methods to evaluate the performance of generalist agents on short and long-horizon tasks (potentially through an extension of our MineCLIP evaluation method).
It is also worth noting that it is currently not possible to test STEVE-1 or VPT [5] in the MineDojo [17] environment, which is meant for generalist agent evaluation, since the action spaces are not compatible. We believe that bridging this gap could be greatly beneficial to the generalist agent community and we hope future works investigate this further.
B.5 Towards Improved Long-Horizon Performance
STEVE-1 is a significant advancement in creating generative models of text-to-behavior, but it has several limitations. First, STEVE-1 is mostly proficient at achieving short-horizon tasks while struggling on longer-horizon tasks like obtaining a diamond. Solving long-horizon tasks while taking actions using low-level mouse/keyboard controls is a very challenging and exciting research direction and, while prompt chaining is a promising approach for improving performance on complex tasks, more can be done in future work to improve performance.
One potential bottleneck is the fact that during packed hindsight relabeling, the hindsight goals are limited to at most 200 timesteps in the future ( $\sim 10$  seconds). Thus, tasks which require more than 200 timesteps to complete are technically out-of-distribution for STEVE-1. Although sampling hindsight goals from farther into the future could theoretically enhance long-horizon performance, our experiments in Appendix C.4 indicate that the performance tends to decrease if we increase this hyperparameter too much. We suspect that while increasing this hyperparameter may be able to improve long-horizon performance, it also increases noise and comes at the cost of reducing performance on short-horizon goals. Investigating whether it is possible to achieve a better tradeoff is an important avenue for future work. We also suspect that the long-horizon capabilities of STEVE-1 could be improved through scaling or finetuning with reinforcement learning, or leveraging LLMs or VLMs to automatically provide prompt chains to the STEVE-1 agent.
B.6 Applying the STEVE-1 Approach to Other Domains
We designed STEVE-1 for Minecraft due to the availability of two key ingredients: (1) a strong behavioral prior (VPT [5]), and (2) a powerful visual-language model which maps text and video to a joint embedding space (MineCLIP [17]). However, the method used to create STEVE-1 is not specific to the Minecraft domain. Given the rapid development of generative models, we expect that similar models to VPT and MineCLIP will become available in many other domains. As these models become available, future work could investigate the applicability of the STEVE-1 approach to these other domains.
C Additional Ablations
In this section, we describe additional ablations on design choices for our method, including the use of classifier-free guidance during training, text augmentation strategies, different VAE variants, and varying chunk sizes during finetuning. We use programmatic evaluation metrics to compare the performance of the various ablations.
C.1 Classifier-Free Guidance During Training
We examine the importance of using classifier-free guidance during training by finetuning a model with no guidance which does not drop out the goal embedding  $z_{\tau_{\text{goal}}}$  from the policy's input (i.e.,  $p_{\text{uncond}} = 0.0$ ) and comparing it to the version which uses guidance ( $p_{\text{uncond}} = 0.1$ ). The chunk
Figure 7: Ablation on Guidance. In the "no guidance" variant, we set  $p_{uncond} = 0$ , meaning that we do not drop any  $z_{\tau_{goal}}$  from the policy's input during training. The "guidance" variants set  $p_{uncond} = 0.1$ , dropping  $10\%$  of the time during training. Whereas the "no guidance" model is only compatible with  $\lambda = 0$  at inference, the "guidance" model can use  $\lambda > 0$ , allowing for better performance.
size is set to the range 15 to 50 and we train each policy for 100M frames. In Figure 7, we compare the performance of using visual goals (MineCLIP video embedding) on the no guidance model using conditional scale  $\lambda = 0$  and the guidance model using conditional scales  $\lambda = 0$  and  $\lambda = 3$ . We observe that while the no guidance model slightly outperforms the guidance model at  $\lambda = 0$  across a few metrics, the agent with guidance outperforms the no guidance agent by a factor of 2 to 3 times for the inventory collection tasks when we increase the conditional scale to  $\lambda = 3$  (which we cannot do for the no guidance model). For the travel distance metric, both of the guidance versions perform similarly to the no guidance version.
C.2 Text Augmentation
During finetuning, instead of using only self-supervision with future MineCLIP video embedding as the goal, we considered using the text embeddings from the 2,000 human labeled trajectory segments as goal embeddings, either solely or in addition to the self-supervised video embeddings. In order to more fairly compare with the CVAE prior approach, we augment the human-labeled data with additional text-gameplay pairs generated as described in Appendix E.2. We implement this experiment by replacing the visual embeddings used for relabeling in Algorithm 1 with text embeddings, when available, with a  $90\%$  probability. To experiment with not using visual embeddings at all, we can replace the visual embeddings with zeros in the same way. In Figure 8, we observe that using only the visual embeddings during training, in combination with the CVAE, can outperform using MineCLIP text embeddings directly in the other two baselines. In this experiment, the chunk size is set to the range 15 to 50 and we train each policy for 100M frames.
C.3 VAE Variants
We study the dataset used to train the CVAE prior model. In Figure 9, we observe that augmentation helps in some programmatic tasks, including the dirt and seed collection tasks, but slightly hurts the wooden log collection and travel distance metrics. In this experiment, we use the same policy with each CVAE variant and we tune the conditional scale  $\lambda$  for each variant. The chunk size is set to the range 15 to 200 and we train the policy for 100M frames.
Figure 8: Ablation on Text Augmentation. In the "text (raw text)" ablation, we train the model using only the text labels from human labelled trajectory segments, and directly use the MineCLIP text embedding of the text label as the goal embedding during training and at inference. For the "text + visual (raw text)" ablation, we use both the visual embedding in self-supervised manner and the text embedding from the human labelled trajectory segments during training and use the MineCLIP text embedding during inference. Even with augmentation, the dataset only contained around  $2\%$  text embeddings. The "visual (text VAE)" version is as reported in the main method, using the CVAE to convert MineCLIP text embedding to visual embedding during inference.
Figure 9: Ablation on VAE Training Data. "Human" baseline uses only the 2,000 human-labelled trajectory segments (text-video pairs), as training example for the CVAE prior model. "Human + Aug" baseline adds additional pairs of text-video examples as described in Section 3.3.
C.4 Chunk Size
During finetuning, we compare different goal chunk sizes by varying the max_btwn_goals=[100,200,300,400], while keeping the min_btwn_goals=15. See Algorithm 1 for more details. A larger max_btwn_goals introduces more noise, with actions that led to achieving the further away goal being less correlated to the actions present in that goal chunk. In Figure 10, we observe that the best max_btwn_goals chunk size is around 200, and increasing the chunk size beyond that causes a drop in performance. We train each policy for 160M frames and tune the conditional scale for each.
Figure 10: Ablation on Segment Chunk Size. We vary the max_btwn_goals parameter in Algorithm 1. The performance is roughly the best at around 200, beginning to decline with greater values.
Figure 11: Even without training the prior on the concept of dirt or digging at all, STEVE-1 can still be instructed to dig holes and get dirt. This demonstrates that STEVE-1 can generalize to unseen text instructions.
C.5 Generalization to Novel Text Instructions
We train the prior on a dataset of human and GPT-generated instructions designed to be representative of the tasks that appear in our gameplay dataset. Here, we have performed a set of simple generalization experiments to measure the degree to which the prior can generalize to unseen instructions.
Instruction Training Set Contamination The "Text Prompt" column in Table 3 shows which of the prompts used in evaluation show up in our training dataset. Among our evaluation instructions, the bolded and italicized instructions show up in the instruction-trajectory dataset. While some instructions do show up, most of the instructions do not show up in our training set (verbatim).
Training Set Decontamination To measure the effect on performance of removing a concept from the training set, we ran an experiment where we removed every instruction with the words "dirt" or "dig" in them and retrained the VAE model. This corresponds to around  $10\%$  of the instructions. As shown in Figure 11, we found that even without training on the concept of dirt or digging at all, STEVE-1 can still be instructed to dig holes and get dirt. This demonstrates clearly that STEVE-1 can generalize to unseen text instructions — likely because most of the text-understanding comes from the pretrained MineCLIP model which was trained on a highly diverse dataset of YouTube videos and captions. The prior VAE only needs to learn a mapping between the text and visual MineCLIP embeddings. Note that there is a slight decrease in performance across all tasks likely due to the smaller VAE training set ( $\sim 10\%$  less).
The instruction-following capability of STEVE-1 is shared between: the policy, which learns to follow instructions in the visual MineCLIP embedding space; the MineCLIP text-encoder, which is trained to align well with the visual embeddings and performs most of the text-understanding; and our prior VAE model, which learns a simple function to translate between text and visual embeddings. This modeling setup lets us fully exploit pretrained models such as MineCLIP to gain impressive language understanding without relying on having our own large datasets or compute.
D.1 Gameplay Dataset
Our gameplay dataset consists of two types of episodes: 7,854 episodes (38.94M frames) of a contractor dataset made available from Baker et al. [5] and 2,267 episodes (14.96M frames) of gameplay generated by running various pretrained VPT agents.
OpenAI Contractor Dataset The majority of our data comes from the contractor data used to train VPT [5]. OpenAI released five subsets of contractor data: 6.x, 7.x, 8.x, 9.x, and 10.x. We use an equal
mix of 8.x, 9.x, and 10.x, which correspond to "house building from scratch", "house building from random starting materials", and "obtain diamond pickaxe". Contractors were given anywhere from 10 to 20 minutes to accomplish these goals to the best of their abilities while their screen, mouse, and keyboard were recorded.
VPT-Generated Dataset We generated additional data by generating episodes using various pretrained VPT agents. In order to increase the diversity of data as well as to get data of the agent switching tasks randomly throughout the middle of episodes, we added random switching between the different pretrained agents during episodes. Specifically, at the beginning of an episode we randomly sample two VPT agents from (foundation_model_2x, bc_early_game_2x, bc_house_3x, rl_fromfoundation_2x, rl_from.house_2x) and switch between them at each timestep with a probability of 1/1000. Since the RL agents all act quite similarly, we avoid sampling two RL agents at once. Additionally, with a probability of 1/750 each timestep, we cause the agent to spin a random number of degrees. This adds more data where the agent spontaneously changes tasks, increasing downstream steerability.
D.2 Text-Video Pair Dataset
We gathered a small dataset of 2,000 human-labelled trajectory segments (text-video pairs) by manually labeling gameplay from our datasets. We used a simple web app that presented a video of 16 frames to the user from a randomly sampled episode. This only corresponds to 32,000 frames of labeled data, which corresponds to labeling  $0.06\%$  of the full dataset, or 27 minutes of labeled data. However, as discussed in Appendix E.2, combining this with automatically labeled data using gpt-3.5-turbo and MineCLIP results in a strong prior model.
D.3 Prompt Design
In our experiments we used both short and longer prompts. The short prompts are either taken from previous literature (e.g., the language-conditioning experiment in the VPT appendix [5]) or they were simply the first prompt we tried. The longer prompts were created by taking inspiration from the prompt engineering methods used with text-to-image models such as Stable Diffusion [50]. To design these prompts, we simply strung together a lot of terms related to our task in order to increase the specificity of the prompts. We were excited to discover that this style of prompt design inspired by the prompt-engineering community works well in STEVE-1.
E Training Details
E.1 Policy Training
STEVE-1 was trained using distributed data parallel in PyTorch [45]. During training, segments of 640 timesteps were sampled from the dataset. Due to memory constraints, these segments were further broken up into chunks of 64, which are processed sequentially. Since VPT uses a Transformer-XL [15], this sequential processing lets the policy attend to previous batches up to the limit of its context length. We optimized the weights using AdamW [37] with a maximum learning rate of 4e-5 and a linear warmup for the first 10M frames followed by a cosine learning rate decay schedule that decays to  $10\%$  of the original learning rate. See Table 1 for an exhaustive list of hyperparameters used during training.
During training, we sample data using packed hindsight relabeling (Figure 2). This involves sampling a segment of an episode, randomly selecting some timesteps at which to change goals, and then filling in the corresponding goal embeddings for the entire episode with the embeddings from the corresponding goal segments. See Algorithm 1 for a detailed explanation of packed hindsight relabelling.
E.2 Prior Training
The prior model is a simple CVAE [54] that conditions on MineCLIP [17] text embeddings and models the conditional distribution of visual embeddings given the corresponding text embedding. This model is trained on a combination of around 2,000 hand-labeled trajectory segments and augmented
Hyperparameter Name	Value
trunc_t	64
T	640
batch_size	12
num_workers	4
weight Decay	0.039428
n Frames	160M
learning_rate	4e-5
optimizer	AdamW [37]
warmup Frames	10M
p_uncond	0.1
min_btwn_goals	15
max_btwn_goals	200
vpt_architecture	2x
Table 1: Policy Hyperparameters
Hyperparameter Name	Value
architecture	MLP
hidden_dim	512
latent_dim	512
hidden_layers	2
batch_size	256
learning_rate	1e-4
β	0.001
n_epochs	50
n_search Episodes	2000
k	5
offset	8
Table 2: Prior Hyperparameters
Algorithm 1: Sampling Episode Segments with Packed Hindsight Relabeling
Function sample_episode_segment(T,min_btwn_goals,max_btwn_goals) segment  $=$  sampleSegment(episode,T) curr timestep  $\equiv$  segment.start goal SWITCHING Indices  $= []$  while curr timestep  $<  <$  segment.end do curr timestep  $+ =$  uniform(min_btwn_goals,max_btwn_goals) goal SWITCHING Indices.append(curr timestep) relabeled_goal_embeds  $= []$  for n in range(1, len(goal SWITCHING indices)) do relabeled_goal_embeds  $[i_{n - 1}:i_n] =$  segment.goal_embdings  $[i_n]$  return segment obs,segmentActions,relabeled_goal_embeds
with additional data by automatically searching for text-gameplay pairs from our gameplay dataset. This is done using the following steps:
1. Combine the 2,000 text labels with 8,000 additional labels generated by querying gpt-3.5-turbo.
2. For each of these 10,000 text labels, search through 1,000 episodes sampled from the gameplay dataset to find the top 5 closest visual MineCLIP embeddings to the text embedding of the text label.
These 50,000 automatically-mined text-video pairs are added to the original 2,000 hand-labeled examples to form the final dataset used for prior training.
We noticed when prompting STEVE-1 using visual goals that when the visual goal showed the agent hitting a block but not following through and breaking it that STEVE-1 actually avoided breaking blocks. Unfortunately, many of the automatically discovered text-gameplay clips include gameplay of this kind. In order to prevent this issue, we added an offset to the embeddings found in this manner. By selecting embeddings from a timestep offset steps after the originally-selected timestep, the agent is much more likely to follow through with breaking blocks.
We trained our prior model for 50 epochs on this dataset and used early-stopping with a small validation set. An exhaustive list of hyperparameters used for creating the prior model can be found at Table 2.
E.3 Training Costs
The $60 cost we report corresponds to the cost of renting a 8xA10g node using spot instances on AWS for 12 hours using our instances prices in May 2023.
F.1 MineCLIP Evaluation
We ran MineCLIP evaluation on both text and visual prompts. The MineCLIP evaluation results can be found in Figure 12.
F.2 Steerability with Programmatic Metrics
Similar to Figure 20 in the VPT appendix [5], we plot the programmatic metric performances (mean and  $95\%$  confidence intervals) across the different goal prompt conditioning, both using visual prompts (Figure 15) and text prompts with CVAE prior (Figure 16) conditioning, on our policy trained with hyperparameters in Table 1 and using conditional scaling  $\lambda = 7$  (for visual prompts) and  $\lambda = 6.0$  (for text prompts with CVAE prior). Each conditioning variant is run with 10 trials, each trial with a different environmental seed and with an episode length of 3000 timesteps (2.5 minutes gameplay). Across the conditioning variant, we use the same set of environmental seeds. For comparison, we also plot the metrics for an unconditional VPT (early_game) agent ("VPT (uncond)") and the text-conditioned agent investigated in VPT appendix [5] ("VPT(text)*") when conditioned on the relevant text. When using visual goal conditioning, we use MineCLIP video encoder to embed a 16-frame clip of the agent performing the desired task taken from our training dataset. An example frame from each of the visual goals is illustrated in Figure 14. When using text VAE goal conditioning, we use the MineCLIP text encoder to encode the text prompts (Table 3) and use the CVAE prior to sample the goal embedding from the MineCLIP text embedding.
We note several differences in our experimental setup compared to that in VPT [5]. We only run our evaluation episodes for 3000 timesteps, equivalent to 2.5 minutes of gameplay, compared to 5 minutes in the VPT paper. Due to a limited computational budget, we generate 10 episodes per conditioning variant, and 110 episodes for the unconditional ("VPT (uncond)"), compared to VPT's 1000 episodes. Lastly, when measuring the inventory count, we log the maximum inventory count seen throughout the episode, which is a lower bound on the potential number of items collected since the agent can later throw out, place, or use these items to craft. As a result of these caveats, we denote the "VPT (text)*" legend in Figure 15 and Figure 16 with an asterisk as we use the results reported in [5] directly for comparison.