SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We make several observations. First, we observe that our agents is more steerable: when conditioned to collect certain items (in bold), the agent collects (relatively) many more of those items than when conditioned on other instructions unrelated to that item, as well as compared to the unconditional VPT. When conditioned on tasks unrelated to the item (e.g. break a flower when interested in measuring logs collected), we also observe that the agent pursues that item less than the unconditional agent. Second, we observe that for the bolded instructions which we expect to stand out, we outperform VPT performance (dashed blue line) [5], even with half the amount of time in the episode rollout. This suggests that our agent is both more steerable relative to the unconditioned VPT agent and the text-conditioned VPT agent investigated in the VPT appendix [5].
F.3 Prompt Chaining Visualization
We visualize two specific episodes from the prompt chaining experiments in Section 4.3 in Figure 18 (building a tower) and Figure 19 (crafting wooden planks).
(a) MineCLIP Text Evaluation
(b) MineCLIP Visual Evaluation
Figure 12: MineCLIP Evaluation. We measure the cosine distance between the goal embedding given to the agent and the MineCLIP video embeddings throughout the episode and record the minimum across the episode. Dashed box indicates the minimum along the row, and the number in the diagonal box indicates the rank of the diagonal element in the row (0 specifies that the diagonal is the minimum element in the row). The ideal performance would be where the minimum values of each row lie on the diagonal. That is, the agent performs a specific task best when it is asked to perform that specific task. Left: We use the prior to convert the text into the goal embedding. Across our 11 text MineCLIP evaluation tasks, STEVE-1 achieves the shortest distance between the episode and the MineCLIP goal embedding when prompted appropriately for most cases. This shows the strong general performance of STEVE-1 across a wide variety of short-horizon tasks. Right: We embed the visual goal loops (Figure 14) with MineCLIP video encoder. Across our 11 visual MineCLIP evaluation tasks, STEVE-1 achieves the shortest distance between the episode and the MineCLIP goal embedding when prompted appropriately except for in two cases, where it mixes up digging and dirt and swimming and going underwater. This shows the strong general performance of STEVE-1 across a wide variety of short-horizon tasks.
Figure 13: Visual Evaluation Success-Rate Matrix. We manually reviewed the same videos used for the MineCLIP Visual Evaluation matrix in Figure 3b in order to verify that the MineCLIP scores correspond well to human judgment. Thus, the values in this matrix are subject to human error and subjectivity. Each cell value shows how often the agent achieves the Evaluation Prompt when conditioned on the Conditioned Prompt (success-rate). The dotted cell(s) is/are the maximum value in the row. Across the tasks, STEVE-1 achieves the highest success-rate when prompted appropriately except in three cases, where it breaks wood more than leaves, explores more than it swims, and swims more than it goes underwater. This shows the strong general performance of STEVE-1 across a wide variety of early-game short-horizon tasks.
Figure 14: Sample frames from each of the 11 visual goals. Note that the text overlaid on the frame is not present when we encode the 16-frame clip with the MineCLIP video encoder, and is only present for the figure visualization.
Shortened Name	Conditioning Variant Name	Text Prompt
dig	dig as far as possible	dig as far as possible
dirt	get dirt	get dirt
sky	look at the sky	look at the sky
leaves	break leaves	break leaves
wood	chop a tree	chop a tree
seeds	collect seeds	collect seeds
flower	break a flower	break a flower
explore	go explore	go explore
swim	go swimming	go swimming
underwater	go underwater	go underwater
inventory	open inventory	open inventory
dirt (engineered)	get dirt ...	get dirt, dig hole, dig dirt, gather a ton of dirt, collect dirt
wood (engineered)	chop down the tree ...	chop down the tree, gather wood, pick up wood, chop it down, break tree
seeds (engineered)	break tall grass ...	break tall grass, break grass, collect seeds, punch the ground, run around in circles getting seeds from bushes
Table 3: A summary of the different ways we refer to the 11 early-game evaluation task prompts. "Shortened Name" is the way we refer to the prompts in any success-rate matrix or MineCLIP matrix. This is also how we refer to the visual prompts which have no actual text. "Conditioning Variant Name" is the name used to refer to the "Text Prompts" in Figure 16, since not all text prompts fit in the figure. A "Conditioning Variant Name" with "... indicates that this is an engineered text prompt that does not fit in the figure. Also, in reference to the experiments Appendix C.5, the bolded and italicized prompts in the "Text Prompt" column are those that were present verbatim in the text-video pair dataset.
(d)
Figure 15: Conditioning with Visual Goals. We plot the performance of the programmatic metrics, along with their mean values and  $95\%$  confidence intervals, across different goal conditioning. See Figure 14 for visualization of these visual prompts. Plots are similar to Figure 20 in the VPT appendix [5]. Each conditioning variant is run with 10 trials, each with a different environmental seed and with an episode length of 3000 time steps (2.5 minutes gameplay). We use the policy that was trained using the hyperparameters specified in Table 1, and with conditional scaling values  $\lambda = 7$ . The dashed horizontal lines refer to an unconditional VPT agent ("VPT (uncond)") and a text-conditioned agent from the VPT appendix ("VPT ( $text$ )*") that was conditioned on the relevant text, for the purpose of comparison.
Figure 16: Conditioning with Text goals. See Table 3 for the exact text string used for each conditioning variant. We use the same policy model but with a conditional scaling value  $\lambda = 6$ . We observe strong steerability which outperforms text-conditioned VPT in Appendix I of [5], and we observe that prompt-engineering can improve performance.
Figure 17: The conditional scale  $\lambda$  in classifier-free guidance [23] can be tuned to improve performance in each of the programmatic tasks. By tuning  $\lambda$  to use classifier-free guidance at inference time, STEVE-1 is able to collect  $7.5 \times$  more dirt,  $15 \times$  more wood,  $23 \times$  more seeds, and travel  $1.7 \times$  further than at  $\lambda = 0$  (no guidance).
Task	Prompts	Precondition
dig a hole, get dirt, look at the sky, break leaves, get wood, get seeds, break a flower, go explore, go swimming, go underwater, open inventory	Use the best-performing prompts in Table 3 in the Appendix.	For grass and flowers, it works best when grass and flowers are in the current biome. We found breaking flowers to be less reliable than the others.
make a tower	“build a tower”	building blocks in hotbar. To obtain building blocks, you can use the dig a hole or get wood prompts from Table 3.
craft wooden planks	“make wooden planks, craft wooden planks” (*)	wooden logs. To obtain wooden logs, use the get wood prompt from Table 3.
place torches, place a crafting table, place wooden planks	“place [torches/a crafting table/wooden planks]”	[torches/crafting table/wooden planks] in the hotbar
make a crafting table	“make a crafting table”	wooden planks.
break stone	“mine stone, go mining, get stone” (*)
get cobblestone	“mine stone, go mining, get cobblestone” (*)	pickaxe in hotbar.
create a wooden pickaxe	“craft a wooden pickaxe, make a wooden pickaxe” (*)	already looking at a placed crafting table, has necessary materials.
hit a sheep, hit a pig, hit a cow	“kill a [sheep/pig/cow]”	agent is close to and looking at the sheep/pig/cow. (See Appendix B.1).
Table 4: Examples of tasks that STEVE-1 is able to achieve. (*) denotes that this is a single prompt-engineered prompt. Note that this list represents only a subset of the capabilities of STEVE-1 due to the difficulty associated with evaluating and describing the capabilities of open-ended models, which is an open research problem itself since capability depends strongly on preconditions, prompt engineering, and our own ability to come up with varied and challenging tasks. Please see Appendix B.4 for further discussion.
Model	Simple Prompt	Complex Prompt
Stable Diffusion [50]	steampunk market interior	steampunk market interior, colorful, 3D scene, Greg Rutkowski, Zabrocki, Karlkka, Jayison Devadas, trending on ArtStation, 8K, ultra-wide-angle, zenith view, pincushion lens effect [20]
STEVE-1	collect seeds	break tall grass, break grass, collect seeds, punch the ground, run around in circles getting seeds from bushes
Table 5: Example of evolving simple prompts into more complex ones for various models.
Figure 18: Build a Tower task. (Left) We track the amount of dirt in the inventory and the agent's height position (y-axis) throughout the episode. In the first 1500 timesteps, the agent is conditioned on the visual get dirt goal, then the agent is conditioned on the visual build a tower goal for the final 1500 timesteps. Vertical dotted lines with numbers indicate the corresponding frames on the right. (Right) The agent's observation frames at 4 different points in the episode. First the agent collects dirt, then begins to build the tower using the dirt blocks.
Figure 19: Make Wooden Planks task. (Left) We track the number of logs and planks in the inventory. In the first 1500 timesteps, the agent is conditioned on the visual break wood goal, then the agent is conditioned on crafting the visual wooden planks goal for the final 1500 timesteps. Similarly to Figure 18, a vertical dotted line annotated with a number indicates the corresponding frame to the right. (Right) The agent's observation frames at 4 different points in the episode. First the agent breaks trees to collect wooden logs, then opens the inventory and crafts wooden planks.