GROOT: Learning to Follow Instructions by Watching Gameplay Videos
Authors: Shaofei Cai
Bowei Zhang
Zihao Wang
Xiaojian Ma
Anji Liu
Yitao Liang
Venue: ICLR 2024 spotlight (2024)
Keywords: Agent
Goal-conditioned Policy
Imitation Learning
Open World
Minecraft
Primary area: reinforcement learning

Abstract: We study the problem of building a controller that can follow open-ended instructions in open-world environments. We propose to follow reference videos as instructions, which offer expressive goal specifications while eliminating the need for expensive text-gameplay annotations. A new learning framework is derived to allow learning such instruction-following controllers from gameplay videos while producing a video instruction encoder that induces a structured goal space. We implement our agent GROOT in a simple yet effective encoder-decoder architecture based on causal transformers. We evaluate GROOT against open-world counterparts and human players on a proposed Minecraft SkillForge benchmark. The Elo ratings clearly show that GROOT is closing the human-machine gap as well as exhibiting a 70% winning rate over the best generalist agent baseline. Qualitative analysis of the induced goal space further demonstrates some interesting emergent properties, including the goal composition and complex gameplay behavior synthesis.

Published as a conference paper at ICLR 2024
GROOT: LEARNING TO FOLLOW INSTRUCTIONS BY WATCHING GAMEPLAY VIDEOS
Shaofei Cai Bowei Zhang', Zihao Wang 12, " Xiaojian Ma², Anji Liu', Yitao Liang
Team CraftJarvis
Institute for Artificial Intelligence, Peking University
School of Intelligence Science and Technology, Peking University
3 "Scooo of Electronics Engineering and Computer Science, Peking University
*Computer Science Department, University of California, Los Angeles
BBijing Institute for General Artificial Intelligence (BIGAT)
(cai shacfei, zhangbowei, zwwang) @stu. www.ttu...uuuuu....... edu.cn
maaniinnnmaeeee mawucla edu, www.nniiuannss..n ucla. edu, yitaol@pku. edu. https: /ttaffrfftttaraaoao wttp:is..tttuuu........ooeeee... 1o/GROOT
ABSTRACT
We study the problem of building an agent that can follow open-ended instructions in open-world environments. We propose to follow reference videos instructions, which offer expressive goal specifications while eliminating the need for expensive text-gameplay annotations. We implement agent GROOT in simple yet effective encoder-decoder architecture based on causal transformers. We evalu- ate GROOT against open-world counterparts and human players on proposed Minecraft SkillForge benchmark. The Elo ratings clearly show that GROOT is closing the best generalist agent baseline. Qualitative analysis of the induced goal space further demonstrates some interesting emergent properties, including the goal composition and complex gameplay behavior synthesis.
gap as well as exhibiting 70% winning rate over the
food (B0%)
plant wheats (100%)
smelt food with furnace (80%)
Survive
build snow golems (60%)
enchant swords (80%)
Build
up one (30%)
Craft
build obsidian (50%)
and shear sheep for wool (90%)
lead onimols (100%)
chop collect (1005)
Collect
Tool
(100%)
diamond
GROOT
with trident rainy (100%))
Figure 1: Through the cultivation of extensive gameplay videos, GROOT has grown rich of skill fruits (number denotes success rate; skills shown above do not mean to exhaustive; kudos our artist Haowei Lin).
INTRODUCTION
Developing human-level embodied agents that can solve endless tasks in open-world environments. such as Minecraft (Johnson et al., 2016; Fan et al., 2022), has always been a long-term goal pursued in AI. Recent works have explored using Large Language Models (LLMs)) to generate high-level plans, which guide the agent to accomplish challenging long-horizor tasks (Wang et al., 2023b;a; Zhu et al., 2023). However, a major gap between these LLM-based agents and generalist agents that can
"Corresponding Author.

---

Published as a conference paper at ICLR 2024
complete endless amounts of tasks is the capability of their low-level controllers, which map the plans to motor commands. Recently developed controllers are only capable of completing a predefined and narrow set of programmatic tasks (Lin et al., 2021; Baker et al., 2022; Cai et al., 2023), which hinders LLM--based planning agents from unleashing their full potential. We attribute the limitation task
of these low-level controllers to how the goal is specified. Specifically, existing controllers indicator (Yu et al., 2019), future outcome (Chen et al., 2021; Lifshitz et al., 2023), and language (Brohan et al., 2022) represent the goal. While it is easy to learn a controller with some of these goal specifications, they may not be expressive enough for diverse tasks. Taking future outcome goals as an example, an image of desired house clearly lacks procedural information on how the house was built. One exception is language, but learning a controller that can receive language goals is prohibitively expensive as it requires a huge number of trajectory-text pairs with text that precisely depicts the full details of the gameplay, therefore preventing them from scaling up to more open-ended tasks.
Having observed the limitations of goal specification in the prior works, this paper seeks to find balance between the capacity of goal specification and the cost of controller learning. Concretely, we propose to specify the goal as reference gameplay video clip. While such video instruction is indeed expressive, there are two challenges: 1) How can the controller understand the actual goal being specified as the video itself can be ambiguous, i.e., a goal space or video instruction encoder has to be learned; 2) How to ultimately map such goal to actual motor commands? To this end, we introduce learning framework that simultaneously produces goal space and a vidco instruction following controller from gameplay videos. The fundamental idea is casting the problem as future state prediction based on past observations:
The predicting model needs to identify which goal is being pursued from the past observations, which requires a good goal space (induced by video instruction encoder); Since the transition dynamics model is fixed, policy that maps both the state and the recognized goal to action also needed by the predicting model when rolling the future state predictions.
Effectively, this results in the goal space and control policy we need. We introduce a variational learning objective for this problem, which leads to a combination of a cloning loss and a KL regularization loss. Based on this framework, we implement GROOT, an agent with an encoder- decoder architecture to solve open-ended Minecraft tasks by following video instructions. The video encoder is non-causal transformer that extracts the semantic information expressed in the video and maps it to the latent goal space. The controller policy is a decoder module implemented by a causal transformer, which decodes the goal information in the latent space and translates it into a sequence of actions in the given environment states in an autoregressive manner.
To comprehensively evaluate an agent's mastery of skills, we designed benchmark called Minecraft SkillForge. The benchmark covers six common Minecraft task groups: collect: bui 1d, survive, explore, tool, and craft, testing the agent's abilities in resource collection, struc- ture building, environmental understanding, and tool usage, in total of 30 tasks. We calculate Elo ratings among GROOT, several counterparts, and human players based on human evaluations. Our experiments showed that GROOT is closing the human- machine gap and outperforms the best baseline by 150 points (or 70% winning rate) in an Elo tournament system. Our qualitative analysis of the induced goal space further demonstrates some interesting emergent properties, including the goal composition and complex gameplay behavior synthesis.
To sum up, our main contributions are follows:
Start by maximizing the log-likelihood of future states given past ones, we have discovered the learning objectives that lead to a good goal space and ultimately the instruction controller from gameplay videos. It provides theoretical guidance for the agent architecture design and model optimization.
Based on our proposed learning framework, we implemented a simple yet efficient encoder- decoder agent based causal transformers. The encoder is responsible for understanding the goal information in the video instruction while the decoder as the policy emits motor commands. On our newly introduced benchmark, Minecraft SkillForgee, GROOT is closing the human-machinn gap and surpassing the state-of-the-an baselines by large margin in the overall Elo rating compar- ison. GROOT also exhibits several interesting emergent properties, including goal composition and complex gameplay behavior synthesis.
2

---

Published as a conference paper at ICLR 2024
2 PRELIMINARIES AND PROBLEM FORMULATION
Reinforcement Learning (RI.) concerns the problem in which an agent interacts with environment at discrete time steps, aiming to maximize its expected cumulative reward (Mnih et al., 2015; Schulman et al., 2017; Espeholt et al., 2018). Specifically, the environment is defined as a Markov Decision Process (MDP) (S, A, R, P, do). where S is the state space, A is the action space, R : S x A - R is the reward function, P S x A + S is the transition dynamics, and dg is the initial state distribution. Our goal is to learn a policy π (a]s)) that maximizes the expected cumulative reward E E 7'r,), where 7 E (0, 1] is a discount factor.
In goal-conditioned RI. (GCRL) tasks, we are additionally provided with a goal E G (Andrychowicz al., 2017; Ding et al., 2019; Liu et al., 2022; Cai et al., 2023; Jing et al., 2021; 2020; Yang et al., 2019). And the task becomes learning a coal-conditioned policy ([als,g) that maximizes the expected return E r'r"]. where r is the goal-specific reward achieved at time step t. Apart from being a new type of RL task, GCRL has been widely studied as pre-training stage toward conquering more challenging environments/tasks (Aytar et al., 2018b; Baker al., 2022: Zhang et al., 2022). Specifically, suppose we are provided with a good goal-condition policy, the goal can be viewed as a meta-action that drives the agent to accomplish various sub-tasks, which significantly simplifies tasks that require an extended horizon to accomplish. Further, when equipped with goal planners, we can achieve zero- or few-shot learning on compositional tasks that are beyond the reach of RL algorithms (Huang et al., 2022; Wang et al., 2023b;a; Zhu et al., 2023; Gong et al., 2023).
At the heart of leveraging such benefits, key requirement is to have a properly that (i) has wide coverage of common tasks/behaviors, and (ii) succinctly describes the task without including unnecessary information about the state. Many prior works establish goal spaces using guidance from other modalities such as language (Hong et al., 2020; Stone et al., 2023; Cai et al., 2023) or code (Wang et al., 2023a; Huang et al., 2023). While effective, the requirement on large-scale trajectory data paired with this auxiliary information could be hard to fulfill in practice. Instead, this paper studies the problem of simultaneously learning a rich and coherent goal space and the corresponding conlicoond policy, given a pre-trained inverse dynamic model and raw gameplay videos, i.e., sequences of states { [ J: collected using unknown policies.
goal space
3 GOAL SPACE DISCOVERY VIA FUTURE STATE PREDICTION
This section explains our learning framework: discovering "good" goal space well video instruction following the controller through the task of predicting future states given previous ones. We start with an illustrative example in Minecraft (Johnson al., 2016). Imagine that an agent is standing inside a grassland holding an axe that can be used to chop the tree in front of them. Suppose in the gameplay video, players either go straight chop the tree or bypass it to explore the territory. In order to predict future frames, it is sufficient to know (i) which goal (chop tree or bypass tree) is being pursued by the agent, and (ii) what will happen if the agent chooses a particular option (i.e.. transition dynamics). Apart from the latter information that is irrelevant to the past observations, we only need to capture the goal information, i.e., whether the agent decides to chop the tree or bypass the tree. Therefore, the task of establishing comprehensive yet succinet goal space can be interpreted as predicting future states while conditioning on the transition dynamics of the environment.
Formally, our learning objective is to maximize the log- fog-liieli of future states given past ones: log (0 Define as a latent variable conditioned on past states (think of it as the potential goals the agent is pursuing given past states), the evidence lower-bound of the objective given variational posterior ( (1:r) ) is the following (see Appendix A for derivations):
log pe (St+1:7 (1:t) = log Σ po ((5441:1.gg511))
2 Eynas (ogpossssurssss.. DKL (go(g]s1r) |l po(g)s1a)),
where (-I.(-Il)) denotes the KL-divergence. Next, we break down the first term (i.e., Po(8 (814t 9)) into components contributed by the (unknown) goal-conditioned policy mo(als,g)
3

---

Published as a conference paper at ICLR 2024
Pct
Rehavior Cloning
Environment
Video Encoder
Decoder as Policy
Video Encoder
Decoder as Policy
States
Learnable Tokens
States
Reference Video Learned Toleens
Rollour Observations
(a) Training
(b) Inference
training stage, a video encoder (non-causal transformer) learns to extract the semantic meaning and transfer the video (state sequence) into the goal embedding space. goal-conditioned policy (causal transformer) is learned to predict actions following the given instructions. We learn the agent using behavior cloning under KL constraint. Right: During inference, reference video passed into the encoder generate goal embeddings that drive the policy interact with environment.
Figure 2: GROOT agent architecture. Left:
and the transition dynamics po 20( 11:2, a+):
log (0010884411111155
log Σ To(a (81. pe p0(8r+1181:7;00 (r)
T
Eg (ar--pplarlleure log (a (81:1.9) 9) + +C],
r=t
where the constant C' contains terms that depend solely on the environment dynamics and are irrelevant to what want learn (i.e., the goal space and the goal-conditioned policy). Bring back the original objective, we have
T-1
By Sgwgstllenn (o((leerr))) [log To [810r 9)و DEL (go(g)s1:r) (g))))))))
constmintt (KL. regularization)
behaviour cloning
where ( (1())) is implemented as a video encoder that maps the whole state sequence into the latent goal space. po(-181:7-1) is the inverse dynamic model (IDM) that predicts actions required to achieve a desired change in the states, which is usually a pre-trained model, details are in Appendix C. Thus, the objective can be explained as jointly learning video encoder and goal-controller policy through behavior cloning under succinct goal space constraints.
GROOT ARCHITECTURE DESIGN AND TRAINING STRATEGY
This section illustrates how to create an agent (we call GROOT) that can understand the semantic meaning of a reference video and interact with the environment based the aforementioned learning framework. According to the discussion in Section 3, the learnable parts of GROOT include the video encoder and the goal-conditioned policy. Recently, Transformer (Vaswani al., 2017) has demonstrated effectiveness in solving sequential decision-making problems (Parisotto et al., 2019; Chen et al., 2021; Brohan al., 2022). Motivated by this, we implement GROOT with transformer- based encoder-decoder architecture, as shown in Figure 2.
4.1 VIDEO ENCODER
The video encoder includes a Convolutional Neural Network (CNN) to extract spatial information from image states $1.7 and a non-causal transformer to capture temporal information from videos. Specifically, use CNN backbone to extract visual embeddings {I1.T) for all frames. Additionally, motivated by Devlin al. (2019); Dosovitskiy et al. (2020), we construct set of learnable embeddings (or summary tokens), represented as (CI.N), to capture the semantic information present the video.
4

---

Published as a conference paper at ICLR 2024
The visual embeddings and summary tokens passed to non-causal transformer, resulting in the output corresponding to the summary tokens as {cI:N}
F1.T Backbone(surr),
(I)
N1:N f Transformer((I.n CI:N])
Similar to VAE (Kingma & Welling, 2013), assume that the latent goal space follows Gaussian distribution, hence we use two fully connected layers, μu()) and af-). to generate the mean and standard deviation of the distribution, respectively. During training. trick to sample a set of embeddings {gi.N from the distribution, where It ~ (H((() ((t)). During inference, we use the mean of the distribution as the goal embeddings, i.e. gr t u(o)).
use the reparameterization
4.2 DECODER POLICY
To introduce our policy module, we start with VPT (Baker et al., 2022), Minecraft foundation model trained with standard behavioral cloning. It built on Transformer-XI (Dai al., 2019) that can leverage long-horizon historical states and predict the next action seeing the current observation. However, the vanilla VPT architecture does not support instruction input. To condition on goal embeddings, we draw the inspiration from Flamingo (Alayrac et al., 2022), that is, to insert gated cross pross-attention dense layers into every Transformer- XL block. The keys and values these layers obtained from goal embeddings, while the queries are derived from the environment states
policy
(
1: GatedXATTN((o 9 2/1-1 1); Ot)
(1)
la f- TransformerXU(ghv a.f:1 01),
(2)
à + FeedForward(z,)) (M)).
where the policy reuses the visual embeddings extracted by the video encoder, i.e., Fitt = Tits the policy consists of M transformer blocks, Oz is the parameter of 1-th block, â, is the predicted action. Since our goal space contains information about how to complete task that is richer than previous lunguage-conditioned policy (Cai et al., 2023; Lifshitz et al., 2023), the cross-attention mechanism is necessary. It allows the GROOT to query the task progress from instruction information based on past states, and then perform corresponding behaviors to complete the remaining progress.
4.3 TRAINING AND INFERENCE
The training dataset can be a mixture of Minecraft gameplay videos and offline trajectories. For those videos without actions, an inverse dynamic model (Baker et al., 2022) can be used to generate approximate actions. Limited by the computation resources, we truncated all the trajectories into segments with fixed length of T without using any prior. We denote the final dataset D {(a1:r, (1:r) M, where M is the number of trajectories. We train GROOT in a fully self-supervised manner while the training process can be viewed as self-imitating, that is, training GROOT jointly using behavioral cloning and KL divergence loss
C(0, Φ) B )> gmaat #1:27)
log No to (s-1) g) + DED (go(gl91.r) po(olau:))
(3)
where λKL is the tradeoff coefficient, 96 is a posterior visual encoder, pe is prior video encoder with the same architecture. More details are in the Appendix D.
RESULT
5.1 PERFORMANCE ON MASTERING MINECRAFT SKILLS
Minecraft SkillForge Benchmark. In order to comprehensively evaluate the mastery of tasks by agents in Minecraft, we created a diverse benchmark called Minecraft SkillForge. It covers 30 tasks from 6 major categories of representative skills in Minecraft, including collect explore, craft, tool, survive, and build. For example, the task "dig three down and fill one up" in the build category asks the agent to first dig three blocks of dirt, then use the dirt to fill the space above; The task of "building
golem" requires the agent to sequentially stack 2 snow
5

---

Published as a conference paper at ICLR 2024
Iaald
10B
TUM
100
135
108
90
100
100
sarresve
78%
1070
1625
Tisk Categorie
ADB
TUU
87%
Suncess Male: (%)
loe
50
80
1450
epplore
40
1382
edllet
teed
AbB
87%
Ito
18e
3075
STEVE Sreve-lndeal) Hniee)) P VPT(te)
HUAIAN GROOT
STEVE- (reina))
STEVE ((eas)
ver(s)
verien
VPTee
VPT(o))
VeT(be)
STEVE- (fed))
STEVE ((viea))
HUMMAN
STEVE (ivimal) VPTiL=) GROUT
GROOT
STEVE STEVE-J (riee-)
VFT(b)) GROOr
(a) Elo Rating Comparison
(b) Winning Rate of GROOT vs. Baselines
(c) Success Rate Comparison
Figure 3: Results on Minecraft SkielForge benchmark. Leftt: Tournament evaluation of GROOT assessed by human players. GROOT performs better than state-of-the-ar Minecraft agent STEVE-I. A 150-score gap corresponds to 70% probability of winning. Middle: Winning rate of GROOT v.s. other agents specific task categories. Colors from red blue denote decrcase the winning rate. Apart from the human player, GROOT surpasses all other baselines. Right: Success rate on 9 representative tasks. GROOT champions process-oriented tasks, such as dig three down and fill one up ( (P) and build ld snow golems
We put the details of this benchmark in the Appendix H. Apart from some relatively simple or common tasks such as "collect wood" and "hunt animals", other tasks require the agent to have the ability to perform multiple steps in succession.
blocks and 1 carved pumpkin
We compare GROOT with the following baselines: (a) VPT (Baker et al., 2022), a foundation model pre-trained large-scale YouTube data, with three variants: VPT (fd). VPT(bc), and VPT(r)), indicating vanilla foundation model, behavior cloning finetuned model, and RL finetuned model; (b) STEVE-1 (Lifshitz et al., 2023), an instruction-folloviii
agent finetuned from VPT, with two variants: STEVE-1 (visual) and STEVE-1 (text) that receives visual and test instructions. More details are in Appendix F.1. It is worth noting that GROOT was trained from scratch.
Human Evaluation with Elo Rating. We evaluated the relative strength of agents by running an internal tournament and reporting their Elo ratings, as in Mnih et al. (2015). Before the tournament, each agent is required to generate 10 videos of length 600 on each task. Note that, all the reference videos used by GROOT are generated from another biome to ensure generalization. Additionally, we also invited 3 experienced players to do these tasks following the same settings. After the video collection, we asked 10 players to judge the quality of each pair of sampled videos from different agents. Considering the diversity of tasks, we designed specific evaluation criteria for every task to measure the quality of rollout trajectories. After 1500 comparisons, the Elo rating converged as in Figure 3 (left). Although there is large performance gap compared with human players, GROOT has significantly surpassed the current state-of the-art art STEVE-1 series and condition free VPT series on the overall tasks. Additional details are in Appendix G.
In Figure 3 (middle), we compare GROOT with other basclines in winning rate on six task groups We found that except for the performance on craft tasks, where STEVE-1 (visual) outperforms model, GROOT achieves state-of thethe-rrt results. In particular, GROOT greatly outperforms other baselines by large margin bui ld and tool. For bui 1d, the goal space needs to contain more detailed procedural information, which is the disadvantage of methods that use future outcomes as the goal. Moreover, such tasks are distributed sparsely in the dataset, even absent in the dataset, which requires the agent to have strong generalization ability. As for craft group, GROOT is not superior enough, especially on the "crafting table" task. We attribute this the wide task distribution in the dataset. Thus the future outcomes can prompt STEVE-1 to achieve high success rate.
Programmatic Evaluation. To quantitatively compare the performance of the agents, we selected 9 representative tasks out of 30 and reported the success rate of GROOT, STEVE-1 (visual), and VPT (bc) on these tasks in Figure 3 (right). We found that, based on the success rate on tasks such as dye and shear sheep( (C), sleep (e)), and lead animals (9), GROOT has already reached level comparable to that of human players (100%). However, the success rates for build snow golems and
enchat sword
smelt food ), use bow
bui ld obsidian di ttasks are only 60% and 50%. By observing the generated videos,
6

---

Published as a conference paper at ICLR 2024
(a) Random Initialized
(b) GROOT w/o KL
(c) GROOT w/ KL
(d) Synthesized Videos
Figure 4: t-SNE visualization of the goal space. Each color corresponds to specific video category. (Left): Space of randomly initialized video encoder. All the vidcos entangled together. Middle: Space of GROOT trained with self-supervised learning w/ and w/o KL regularization, respectively. The videos clustered based on their semantics. Visualization shows the subtle differences between the two, Right: Synthesized videos using concatenation manner. The concatenated videos lay on the position between the source videos.
found that GROOT cannot precisely identify the items in Hotbar (such as buckets, lava buckets, snow blocks, and pumpkin heads), resulting in low probability of switching to the correct item. STEVE-1 also has the same problem. This may be due to the current training paradigm lacking strong supervisory signals at the image level. Future work may introduce auxiliary tasks such as vision-question answering (VQA) to help alleviate this phenomenon. Details are in Appendix F3.
5.2 PROPERTIES OF LEARNED GOAL SPACE
This section studies the properties of learned goal space. We used the t-SNE algorithm (van der Maaten & Hinton, 2008) to visualize the clustering effect of reference videos encoded in goal space, as in Figure 4. We select kinds of videos, including craft items, combat enemies, harvest crops, hunt animals, chop trees, trade with villagers, and mine ores. These videos are sampled from the contractor data (Baker et al., 2022) according to the meta information (details are in Appendix F.2). Each category contains 1k video segments. As a control group, in Figure 4 (left), we showed the initial goal space of the video encoder (with a pre-trained EfficientNet-BO (Tan & Le, 2019) as the backbone) before training. We found that the points are entangled together. After being trained on offline trajectories. as in Figure 4 (middle), well understands reference videos and clusters them according their semantics. This proves that it is efficient to learn behavior-relevan task representations using our self-supervised training strategy. The clustering effect is slightly better with KI. regularization, though the difference is not very significant. Inevitably, there are still some videos from different categories entangled together. We attribute this to the possibility of overlap in the performed behaviors of these videos. For example, chop trees and harvest crops both rely on a sequential of "attack" actions.
Condition on Concatenated Videos. We also study the possibility of conditioning the policy on concatenated videos. First, we collect 3 kinds of source videos, including chop trees, hunt animals, and trade with villagers. We randomly sampled two videos from sources of chop trees and hunt animals, downsampled and concatenated them into synthetic video, denoted as [chop trees, hunt animals] By the same token, we can obtain [hunt animals, trade with villagers]. We visualize these videos together with the source videos in Figure 4 (right). We found that the source videos lie far away from each other while the concatenated videos are distributed between their source videos. Based on this intriguing phenomenon, we infer that concatenated videos may prompt GROOT to solve both tasks simultaneously. To verify this, we evaluate GROOT on three kinds of reference videos. i.e., chop trees, hunt animals, s, mammlsl s). We launched GROOT in the forest and in the animal and [chop trees, hunt inimal
plains, respectively. The collected wood and killed mobs are reported Figure 5. We found that although the concatenated vidco may not be effective raw video in driving agent to complete a single task (60% of the performance of raw vidco), it does possess the ability to drive the agent to perform multiple tasks. This is an important ability. As discussed in Wang al. (2023b), sometimes the high-level planner will propose multiple candidate goals, it will be efficient if the low-level controller can automatically determine which to accomplish based on the current observation.
Ablation on KL Divergence Loss To investigate the role of KL loss in training, we evaluated GROOT (w/ KL.) and its variant (w/o KL. three tasks: collect. seagrass ) lect.
7

---

Published as a conference paper at ICLR 2024
Wool Collected
11.0
Mobs Killed
2.2
Seagre Colleceed
Wood Collected
11.0
Acrona Fied
1.5
1.
chop
brudd
Concas
clop
lumt
coucad
w/ KL w//n KL.
w/ KL.
w/o KL
w/ KL
w/s KL
Figure 5: Comparison using raw and concate- nated reference videos conditions. Left: Col- lected wood in the forest biome. Right: Killed mobs the plains biome. "concat" denotes the refer- ence video is [chop trees, hunt animals]. Statistics are measured over 10 episodes.
Figure 6: Ablation study on KL loss. After being jointly trained with KL loss, GROOT more seagrass ( (a underwater and 1.5x wood in the forest while the difference is not as impressive the use bow task. Statistics measured
collect 2
over episodes.
GROOT
STEVE-1
128
690
128
600
dizi
grsalle andesite fron ore
reditens leu
diet rahhlestone
undesite granis
irun ore
64
dlamndd
64
dieite
Depth to the ground
dieclte
450
Depett goound
450
378
375
condition changed
condition changed
300
diamonds distrtbafied in there dep 16
226
Bnthonds hured in these depths
226
Cummberb
150
0k
21
46
61
5k
10k
12k
06 2k
4k
0k
SL
10k
12k
Timestep
Timestep
Figure 7: Results on solving challenging obtain diamond task. The vertical dashed lines represent the time when certain item is first obtained. Left: GROOT first dug down to the depth of 12 and then mined horizontally to obtain diamonds with average success rate 16%. Right: STEVE-1 quickly dug down to the specific depth but struggled to maintain its height.
wood and use bow (C). As shown in Figure 6, we found that introducing the constraint of KL loss improved agent performance by 2x and 1.5x in the first two tasks, whereas there was no significant effect in the use bow task. This may be because the first two tasks require the agent to generalize the corresponding skills to different terrains (e.g. locating trees in the environment for collecting wood and sinking specific locations for collecting seagrass). Therefore, it puts higher demands on the agent's ability to generalize in the goal space, and this exactly the role played by the KL loss. The use bow task is relatively simple in comparison because it only requires charging and shooting the arrow, without considering environmental factors.
5.3 COMBINING SKILLS FOR LONG-HORIZON TASKS
In this section, we explore whether GROOT can combine skills to solve long- tasks, which is key to its integration with high-level planner. Taking the task of mining diamonds as example, prior knowledge underground, and the probability of appearing in other depths is almost zero. Therefore, the agent needs to first dig down to the specified depth (12) and then maintain horizontal mining. To achieve this, we designed two reference videos, each 128 frames long. One describes the policy of starting from the surface and digging down, and the other demonstrates the behaviors of horizontal mining. We show an example in Figure 7 (left). In the beginning, GROOT quickly digs down the specified depth and then switches to horizontal mining mode. It maintains the same height for long time and found diamonds at 1 11 steps. In addition, we compared STEVE-I (visual) under the same setting in Figure (right). After switching to the horizontal mining prompt, STEVE-1 maintains its height for a short time before it stuck the bedrock layer (unbreakable in survival mode), greatly reducing the probability of finding diamonds. This indicates that our goal space is expressive enough instruct the way of mining, and the policy can follow the instructions persistently and reliably. In contrast, STEVE-1, which relies future outcomes as condition, attempts at various visual prompts. We conducted 25 experiments each on GROOT and STEVE-I, with success rates of 16% and 0% for finding diamonds. Additional details in the Appendix F.4.
that diamond ores are generally distributed between the 7th and 14th floors
unable maintain its depth. despite
8

---

Published as a conference paper at ICLR 2024
6 RELATED WORKS
Pre-train Policy on Offline Data. Pre-training neural networks on web-scale data has been demon- strated as an effective training paradigm in Nature Language Processing (Brown et al., 2020) and Computer Vision (Kirillov al., 2023). Inspired by this, researchers tried transfer the success to the field of decision-making from pre-training visual representations and directly distilling the policy from offiine data. As the former, Aytar et al. (2018a); Bruce al. (2023) leveraged temporal information present in videos as the supervision signal to learn visual representations. The representa- tions are then used to generate intrinsic rewards for boosting downstream policy learning, which still requires expensive online interactions with the environment. Schmidhuber (2019); Chen et al. (2021) leveraged scalable offline trajectories to train optimal policy by conditioning it on cumulated rewards. Laskin et al. (2022) proposed to learn an in-context policy improvement operator that can distill an RL algorithm in high data efficiency. Reed et al. (2022) learned a multi-task agent Gato by doing behavior cloning on large-scale expert dataset. By serializing task data into a flat of sequence, they use the powerful transformer architecture to model the behavior distribution. However, these methods either require elaborated reward functions or explicit task definitions. This makes it hard to be applied to open worlds, where tasks are infinite while rewards are lacking. Another interesting direction is to use pre-trained language models for reasoning and vision language models for discrimination, to guide the policy in life-long learning in the environment (Di Palo et al., 2023).
Condition Policy on Goal Space. Researchers have explored many goal modalities, such as language (Khandelwal et al., 2021). I), image (Du et al., 2021), and future video (Xie et al., 2023), to build a controllable policy. Brohan et al. (2022) collected a large-scale dataset of trajectory-text pairs and trained a transformer policy to follow language instructions. Despite the language being a natural instruction interface, the cost of collecting paired training data is expensive, As a solution, Majumdar et al. (2022) sorted to use hindsight relabeling to first train a policy conditioned on the target image, then aligned text to latent image space, which greatly improves training efficiency. Lifshitz et al. (2023) moved big step this paradigm by replacing the target image with a 16-frame future video and reformulating the modality alignment problem into training a prior of latent goal given text.
Build Agents in Minecraft. As a challenging open-world environment, Minecraft is attracting an increasing number of researchers to develop Al agents on it, which can be divided into plan-oriented (Wang et al., 2023b;a) and control-oriented methods (Baker et al., 2022; Cai et al., 2023; Lifshitz et al., 2023) based on their emphasis. Mlan-oriented agents aim to reason with Minecraft knowledge and decompose the long-horizon task into sub-tasks followed by calling low-level controller. Control-oriented works follow the given instructions and directly interact with the environments using low-level actions (mouse and keyboard). Baker al. (2022) pre-trained the first foundation model VPT in Minecraft using internet-scale videos. Although it achieves the first obtaining diamond milestone by fine-tuning with RL, does not support instruction input. Lifshitz et al. (2023) created the first agent that can solve open- open-ended tasks by bridging VPT and MineCLIP (Fan et al., 2022). However, its goal space is not expressive enough and prevents it from solving multi-step tasks.
7 LIMITATIONS AND CONCLUSION
Although GROOT has demonstrated powerful capabilities in expressing open-ended tasks in the form of video instructions, training such a goal space remains highly challenging. We found that GROOT is quite sensitive to the selection of reference videos, which we attribute to the fact that the goal space trained from an unsupervised perspective may not be fully aligned with the human intention for understanding the semantics of the reference video. Therefore, it would be a promising research direction in the future to use SFT (supervised fine-tuning, Sanh et al. (2021)) and RLHF (Ziegler et al., 2019) to align the pre-trained goal space with human preference.
We propose a paradigm for learning to follow instructions by watching gameplay videos. We prove that video instruction is good form of goal space that not only expresses open-ended tasks but can be trained through self-imitation (once the IDM is available to label pseudo actions for raw gameplay videos). Based this, we built an encoder-decoder transformer architecture agent named GROOT in Minecraft. Without collecting any text-video data, GROOT demonstrated extraordinary instruction- following ability and crowned the Minecraft SkillForge benchmark. Additionally, we also demonstrate its potential as a planner downstream controller in the challenging obtain diamond task. We believe that this training paradigm can be generalized in other complex open-world environments.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work is funded in part by the National Key R&D Program of China #20222D0160301, a grant from CCF-Tencent Rhino-Bird Bird Open Research Fund. We thank Haowei Lin for contributing the thumbnail figure of GROOT.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko- laj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204. 2022. URL https: /tppp/pp...a// www.enttisccootoooeooo
org/ :ttp://orps:::::::::0 :484777::2444771111 5, 18
Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Joshua Tobin, P. Abbeel, and Wojciech Zaremba. Hindsight experience replay. ArXiv, abs/1107.014955 2017. URL https::////// / api. semanticscholat org CorpusID: 3532908. 3
Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyun Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In Neural Information Processing Systems, 2018a.
URL httpt:ttt///o..t api www.mmantcscchhaaa. org/ CorpusID wtt..ooo......r/oooousot.... : 44061126. 9
Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyun Wang. and Nando de Freitas. Playing hard exploration games by watching youtube. In Neural Information Processing Systems, 2018b.
URL https::t//// /api.app...// wew.nticscholeeeee....e.
org/ CorpusID: 44611226. 3
Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Bran- don Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learinn to act by watching unlabeled online videos. ArXiv, abs/2206. 11795, 2022. URL https: //aip. wttg/orrguseeeeooooooeee http/CorpussDDD. 24995399367 3, 5, 6, 7, 9, 16, 17, 18, 19, 20, 21
semanticscho
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Anand Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Ho Vuong, F. Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale. ArXiv, abs/2212.068111 2022. URL https: /tpp//ppp...// semant wessthccschrrr 254591260. 2, 9, 17
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020. URL https: //ppi...../// semant icscholar Trgg/orpussDD 218971783. 9
Jake Bruce, Ankit Anand, Bogdan Mazoure, and Rob Fergus. Learning about progress from experts. In International Conference on Learning Representations, 2023. URL https://appt
: 2252998700 9
Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 117.3 13734-13744, 2023, URL https:/ //tp:://pp.i.. www.nticschiieeec.ee ar. htt.aaa..roggoop...o0255555/
2, 3, 16, 32
10

---

Published as a conference paper at ICLR 2024
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via se- quence modeling. In Neural Information Processing Systems, 2021. URL https://api. orgg semant cscholas arr wwt//oorgs//oo.o. Corpusrrpppppcccco 235229222229999 2, 4,4,
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguisties, Jan 2019. doi: 10.18653/ v1/p19- 1285. URL http: / / dx. org.oor//o........o......oooeoo. 10 18653886833 /v11 wwtpl////////aaaaaaaaaaeeeee 1285. 5, 18
Jacob Devlin, Ming-
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019. URL https: / /app.. semant icscholaa ar. Org/CCrppi ID: 52967399. 4
Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever, Markus Wulfmeier, Nicolas Heess, and Martin Riedmiller. Towards a unified agent with foundation models. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023. 9
Yiming Ding, Carlos Florensa, Mariano Phielipp, and P. Abbeel. Goal-conditioned imitation learning. ArXiv, abbs/190.005833 2019. URL https:/ /apii. wwwpaaatiiccccoa.ooo//oorreeeeee 189762519.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. ArXiv, abs/2010. 11999, 2020. URL httpt:tpp///.. appi.... semant i cschhlarr www.oorg/ooouooo rpuusID::
225039882. 4
Heming Du, Xin Yu, and Liang Zheng. Vinet: Visual transformer network for object goal navigation. ArXiv, abs/21055009447, 2021. URL https: semant commaniccsn icscholar org/ Corpus 234790212.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-leamer architectures. ArXiv, abs/1802.01561, 2018. URL http:::::/ //api. wtw.//pp....aiiiiip/aeea.aaata.a semant reemnntccchoollt org/CorpusID: 3645060.
Linxi (Jim) Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. ArXív, abs/2206.08853, 2022. URL https:
//api. semant wwwccccccaaaaaa... wwtpllaarrrg//oooououeo
htt.aichhh.a....oooooooooooaaaaa DI:.IID::22444282 1, 9, 16, 20, 31
Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Feii al. Mindagent: Emergent gaming interaction. arXiv preprint arXiv:2309.09977 2023. 3
William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela M. Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstra- tions. In International Joint Conference on Artificial Intelligence, 2019. URL https: //api. www.etettscceeeeoeeeeeeee
otg/Corpus ID: 1e11999000111 16
Yicong Hong. Qi Wu, Yuankai Qi, Cristian Rodriguez-Oppzz, and Stephen Gould. VIn-bert: recur- rent vision-and- lan-lauuuee bert for navigation. 2021 JEÉE/CVF Conference on Computer Vision and Paterrn Recognition (CVPR), pp. 1643-1653, 2020. URL htLhttps: / //ppppppp..///// semanticschol.a
org/Corpus 227228335 3
Wenlong Huang, F. Xia, Ted Xiao, Harris Chan, Jacky Liang, Peter R. Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning. 2022. URL https: /api....... semanticscho www..rr...ooeeeeeeeeeeeeeeeeeeee ortg//orppssT::2555555555
11

---

Published as a conference paper at ICLR 2024
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. ArXiv, 0aap/2307.0597/ 2023. URL https: / api www.nteeaatttt.. wwwchhlar.hcccccccccccccaacaa..t org/ htt.:/ola......oooooooo..aaaaaa. CCtpposI:
Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Chao Yang, Bin Fang, and Huaping Liu. Reinforcement learning from imperfect demonstrations under soft expert guidance. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5109-5116, 2020. 3
Mingxuan Jing, Wenbing Huang, Fuchun Sun, Xiaojian Ma, Tao Kong, Chuang Gan, and Lei Li. Adversarial option-aware hierarchical imitation learning. In International Conference on Machine Learning, pp. 5097-5106. PMLR, 2021.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In International Joint Conference Artificial Intelligence, 2016. URL httttps//api..smmantiiiio
org/ Corpus 9953039. 1, 16
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 14809-14818, 2021. URL http::: /ttp://appp... scmant
ID: 244344010. 9
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
URL https:tp///mm //e::///a /app..p..... semant www.tttaasccccaccc.oooocoaaaaaaa wwt.oorg/////.ooo Cttpg/CrrpssD:::/ : 216078090. 5
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. anythinn.. ArXiv, abs/2304.0266 2023. httpst /app www.mttiisccoo....o..
org/Corpus 257 952310.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. ArXiv, abs/2210.14215. 2022. URL https: //api semant eschollaclllccc. ar.org/
CorpusID:2531077 613. 9
Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila A. Mcllraith. Steve-1: A generative model for text-to-behavior in minecraft. ArXiv, abs/2306.00953 2023. URL https: Cor//rrrssIIDD. ID 258999562993 2. 5, 6, 9, 16, 18, 19,
//api. //api.semante semant wes.antcccchllaa.
Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. Juewu-me: Playing minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint ar2iv:2112.04907, 2021. 2
Minghuan Liu, Menghui Zhu, and Weinan Zhang, renpiconddiioneedeiineeeeeeeee learning: Problems and solutions. ArXiv, abs/2201.002999 2022. URL https:////// /appi. semanticscchoar..org Corpus :4::::4666600 2460638855
Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. ArXiv, abs/2206. 12403, 2022. URL https: / http://ppp.aaaatiiiiiieemm www.ettsscooolls...o.o.ccoo cot.:iaurrror/ooCoioiiiiiii 20000000:25505....
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human- control through deep reinforce- ment learning. Nature, 518:529-533, 2015. URL https::////. / //ppi./pp///////// http//pp..eaaeee.eee..
org/
Corpus 2002222222222 4204. 3, 6
Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Çaglar Gülçehre, Siddhant M. Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Manfred Otto Heess, and Raia Hadsell. Stabilizing transformers for rein- forcement learning. In International Conference on Machine Learning, 2019. URL https: //api. semammmatttt www.tticcchooo..coccccc cor.aarrr/c//rpsst:
12

---

Published as a conference paper at ICLR 2024
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez. Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint wrXiv:2205.06175, 2022. 9
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization, 2021. 9
Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards - just map them to actions. ArXiv, abs/1912.00287, 2019. URL https: / //api. semanticscholan org/ CorpusID: 200BED::2288600000
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proxi- mal policy optimization algorithms. ArXiv, abs/1707.06347. 2017. URL https://api.
ID:
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, L. Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever. Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484 489, 2016. URL https: //api. semanticscholes org Barg//oozee :Ct///ooo::::::5333 515925.. 21
Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang Huei Lee, Quan Ho Vuong, Paul Wohlhart, Brianna Zitkovich, F. Xia, Chelsea Finn, and Karol Hausman. Open-world object manipulation using pre-trained vision-language models. ArXiv, abs/2303.00905, 2023. URL
bttps::/appe semant wescccccsaaaac wttp/orppccoo.o..oooooooo ott.eeaeennaaaa.....oo.ooooooooo 257280290.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks, ArXiv, abs/1905. 1946, 2019. URL /tttps://apt www.nticsccolar.oooom
CorpusID: :11ouii::172112..... 7, 17
Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Stu- dio: Data labeling software, 2020-2022. URL https: //tgttgbtbb//t///////////ttghhtt www/hearteexabb.....o.aaaaa stuiio. Open source software available from http::/gtttbbboommmcccrrxaaaaaa studio. 22
Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2603 2008. URL https://api. www.steesmmonllea........ org CorpusID: CorpusiD: 5855042.7
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. URL https:
//api. semantics www.chollar...mm /org//CrrusIDD ID: 13756448.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi (Jim) Fan, and Anima Anandkumar. Voyager: An onen-ended embodied agent with large language models. ArXiv, abs/2305. abs/2305.16291, 2023a. UtLhttps: /api. semant icscholar org/ CorpusID: 258887849, 1, 9, 16
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. ArXiv, abs/2302.015600 2023b. URL //api. semant ieraanticcccnnn iescholar. orggCorppsSDD 256598146, 3, 7, 9, 16,
Zhihui Xie, Zichuan Lin, Deheng Ye, Qiang Fu, Wei Yang, and Shuai Li. Future-conditioned unsupervised pretraining for decision transformer. In International Conference on Machine Learning, 2023. URL https:/ //ppi. semanticscholass wxtp.//oooo.ps..222244444442
13

---

Published as a conference paper at ICLR 2024
Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. Ad- vances in neural information processing systems, 32, 2019. 3
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan C. Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. org/ CorpusID:
ArXiv, 0bss/1910.10999 2019. URL https::///// haapss//appii /tpp.
204852201.2
Qihang Zhang, Zhenghao Peng, and Bolei Zhou. Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining. In European Conference on Computer Vision, 2022. URL https: api wemantissccootteeeeeeeii.iio wow.roorg///e ozg/CorpusID: 2201127722222220 3, 17
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyuan Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Y. Qiao, Zhaoxiang Zhang, and Jifeng Dai. Ghost in the minecraft: Generally capable agents for open-world environments via large language mod- els with text-based knowledge and memory. ArXiv, abs/2305.17144, 2023. URL https: //api. wee.nticcccooll.... owo.or///oooooooo CtteCCrrpssSii ID: 258959262. 1.3
Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, hbss/999,00593,, 2019. URL https: / /api. semant wwwaaantccchoaa....o wttp:/ooaao.oo/ooooouuoo..oo 202660943. 9
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A DERIVATION
In this section, we detail how we derive the final objective. Recall that the goal is maximize the log- rigllikeihhood of future states given past ones: log (8g4((1177)1)))) Using Bayes' theorem and the Jensen's inequality, we have:
log p(st+1:7(51.) log Σ ((
(4)
p(s ((2[51T)
log Σ
(5)
((2(51:7)
E (e
log ((84412,=/11a) - log ((=1>1.7)]
(6)
R
log (1 =) + log p(2(81=) log q (=
log (8 =)] E B ( log p(a)s1a)
Bx-g(sleur))
(8)
g(2($17T)
-=zg(=)arr)) [log p(51417|s1 =)] DRL. (1(2151:7) p(=(s1=))
We break down p(s++1:1| 2) into components: goal-conditioned policy It (a (s1:+1)) and the transition dynamics p($r41 $11:; ar), we have
2( 2) II T=s
Σ Taarls007
(10)
s0.r, Furthermore, using Jensen's inequality, log log.(ss:: 1(5:+1:7 1s 2) can be written as
log p(1 11441//81131 2)
log Σ क 7aa131177 2) p(8 181r, ur);
(11)
a,
log Σ x(ar131172)) plar|5177 9741) p(5r+11511))
(12)
p(a-(s1:7)
pla log x(a (sar 2) + +
(13)
where the constant C 1(g (81:)) log p(ar((a)) describes the dataset distribution and irrelevant to what we want to learn (i.e., the goal space and the goal-conditioned policy), we have:
Ezog(sleum)) [log p(scs1e((snaa 2 E
B. Sar-pparluuna (1) [log n (ar//strr. (14)
bewd [lo x xa (s1.,
(15)
Thus, we derived the evidence lower-bound of log p( ((84117//1511 as follows
log ((5441/7 (ta) 2
log (a, |81. 2)]
Dxi. (g(2(81a)) p(z]saa)
(16)
B MINECRAFT ENVIRONMENT
Minecraft is extremely popular sandbox game that allows players to freely create and explore their world. This game has infinite freedom, allowing players to change the world and ecosystems through building, mining, planting, combating, and other methods (shown in Figure 8). It is precisely because
15

---

Published as a conference paper at ICLR 2024
P89 UA
BI WAL
Figure 8: Examples of Minecraft environment. Tasks from top to bottom, from left to right are building houses, planting wheat, fishing, brewing potion, mining diamond ores, and combating the ender dragon, respectively.
of this freedom that Minecraft becomes an excellent AI testing benchmark (Johnson et al., 2016; Baker et al., 2022; Fan et al., 2022; Cai al., 2023; Lifshitz al., 2023; Wang et al., 2023b;a). In this game, AI agents need to face situations that are highly similar to the real world. making judgments and decisions to deal with various environments and problems. Therefore, Minecraft a very suitable environment to be used as an Al testing benchmark. By using Minecraft, AI researchers can more conveniently simulate various complex and diverse environments and tasks, thereby improving the practical value and application of AI technology.
We use the combination of 1.16.5 version MineRL (Guss et al., 2019) and MCP-Reborn' as our testing platform, which is consistent with the environment used by VPT (Baker et al., 2022) and STEVE-1 (Lifshitz et al., 2023). Mainly because this platform preserves observation and action space that is consistent with human players to the fullest extent. On the one hand, this design brings about high challenges, as agents can only interact with the environment using low-level mouse and keyboard actions, and can only observe visual information like human players without any in in-game privileged information. Therefore, the AI algorithms developed on this platform can have higher generalization ability. On the other hand, this also presents opportunities for to conduct large-scale rre-training on internet-scale gameplay videos.
B.1 OBSERVATION SPACE
The visual elements included in our observation space are completely consistent with those seen by human players, including the Hotbar, health indicators, player hands, and equipped items. The player's perspective is in the first person with a field of view of 70 degrees. The simulator first generates RGB image with dimensions of 640 360 during the rendering process. Before inputting to the agent, we resize the image to 224 x 224 to enable the agent to clearly see item icons in the inventory and important details in the environment. When the agent opens the GUI, the simulator also renders the mouse cursor normally. The RGB image is the only observation that the agent can obtain from the environment during inference. see more clearly in extremely dark environments, we have added a night vision effect for the agent, which increases the brightness of the environment during nighttime.
worth noting that help the agent
B.2 ACTION SPACE
Our action space is almost identical to that of humans, except for actions that involve inputting strings. It consists of two parts: the mouse and the keyboard. The mouse movement is responsible for changing the player's camera perspective and moving the cursor when the GUI is opened. The left and right buttons are responsible for attacking and using items. The keyboard is mainly responsible for controlling the agent's 'ss movement. We list the meaning of each action in the Table
1. To avoid predicting null action, we used the same joint hierarchical action space as Baker et al. (2022), which consists of button space and camera space. Button space encodes all combinations of keyboard operations and flag indicating whether the mouse is used, resulting in total of 8461
16

---

Published as a conference paper at ICLR 2024
Table 1: Action space descriptions from Minecraft wiki (ttp:///miiicattttaadoooooooooua
Index
Action
Human Action
Description
1
Forward
key W
Move forward.
2
Back
key S
Move backward.
3
Left
key A
Strafe left.
4
Right
key
Strafe right.
5
Jump
key Space
Jump. When swimming, keeps the player afloat. Slowly move in the current direction of movement. When used in conjunction with the attack function in the GUI, it can swap items between inventory and Hotbar. When used with the craft function, it crafts the maximum possible number of items instead of just one.
6
Sneak
key left Shift
7
Sprint
key left Ctrl left Button right Button
Move quickly in the direction of current motion. Destroy blocks (hold down); Attack entity (click once). Put down the item being held or interact with the block that the player is currently looking at. Within the GUL, pick up stack of items or place single item from the stack that being held by the mouse.
8
Attack
9
Use
10
hotbar.[1-9]
keys 1 1
Selects the appropriate hothar item. When in the inven- tory GUI, swap the contents of the inventory slot under the mouse pointer and the corresponding hothar slot. Turning: aiming; camera movement. Ranging from 180 to +180.
11
Yaw
move Mouse X
12 Pitch
move Mouse
Turning: aiming; camera movement. movement.Ranging Ranging from 180 to +180.
candidate actions. The camera space discretizes the range of one mouse movement into 121 actions. Therefore, the action head of the agent is a cultt-classification network with 8461 dimensions and multi-classification network with 121 dimensions.
C INVERSE DYNAMIC MODEL
According to the theory in Section 3, we know that our training paradigm relies on the inverse dynamic model (IDM) which generates pseudo action labels for raw gameplay videos calculate the behavior cloning loss. Therefore, in this section, we introduced the background knowledge of IDM.
IDM is non-causal model that aims uncover the underlying action that caused changes in the current step by observing historical and future states, and it can be formally represented as p(aylor, 1)+ Compared with traditional policies learned via behavior cloning, IDM is more accurate in predicting actions because it can observe the changes between past and future frames. OpenAl (Baker et al., 2022) developed the first inverse dynamic model in the Minecraft domain. By extending the length of the observable sequence to 128 and modeling p(a, (0.-64:4+44) with non-causal transformer, the IDM achieved the accuracy of action prediction to over 95% with only 2k hours of game trajectories. This makes it possible for training paradigm to utilize the large-scale Minecraft data available on the Internet. Moreover, Zhang et al. (2022) has also trained an accurate IDM model with small amount of data in real autonomous driving environment, which further provides a basic guarantee for our training method to generalize to other complex environments.
D IMPLEMENTATION DETAILS
D.1 MODEL ARCHITECTURE
The video encoder consists of convolutional neural network backbone and non-causal transformer. Inspired by Brohan et al. (2022), we adopted the EfficientNet (Tan & Le, 2019) as the backbone. Specifically, use its variant EfficientNet-BD for efficiency, which takes in images of size 224 x 224
17

---

Published as a conference paper at ICLR 2024
and extracts a feature vector of shape 7 7 x 1280, where 7 x 7 denotes the spatial dimensions. In order to adaptively enhance the important visual information, we use a shallow transformer to pool the feature map along spatial channels. To fuse global visual features, we construct another learnable embedding [sp], concatenate with the 49 features in space, and obtain a token sequence of length 50. After being processed by the transformer, the output for the [sp] token corresponds to the pooled visual feature, whose dimension is daid = 1024. To capture the temporal features of the video, we remove the code related to the casual mask in the minGPT2 and obtain non-causal transformer. The policy decoder consists of 4 identical blocks, where each block contains a Flamingo gated-attention dense layer (Alayrac al., 2022) and a Transformer-Xe XL block(Dai et al., 2019). The Transformer-Xi XL block maintains a recurrence memory of past 128 key-value pairs to memory long-horizon history states. We directly use the Transformer-XI. implementation in Baker et al. (2022) with simple modification, i.e., before passing states into the policy decoder, we add the previous action to the state embedding at each timestep. Notably, We find this modification very useful especially when we need to train the policy from scratch. As it not only accelerates the training process but makes the predicted action more consistent and smooth. Additional hyperparameters can be found in Table 2.
Table 2: Hyperparameters for training GROOT
Hyperparameter
Value
Optimizer
Adam
Weight Decay
0.001
Learning Rate
0.0000181
Warmup Steps
2000
Number of Workers
4
Parallel Strategy
ddp
Type of GPUs
NVIDIA RTX 4090Ti, A40
Parallel GPUs
8
Accumulate Gradient Batches
8
Batch Size/GPU (Total)
2 (128)
Training Precision
b
Input Image Size
224 x 224 EfficientNer-B00
CNN Backbone
Encoder Transformer
minGPT (w/o causal mask) TransformerXL
Decoder Transformer
Number of Encoder Blocks
8
Number of Decoder Blocks
4
Hidden Dimension
1024
Number of Condition Slots
Trajectory Chunk size
128
Attention Memory Size
256
Weight of KL Loss
0.01
D.2 INFERENCE
To generate reference vidcos, we invited three human players to play each task according to the task description. Each person was asked produce two videos, so we could prepare six videos for each task in total. Then, we selected the most relevant video to the task description from the six videos and cropped the first 128 frames into new video, which was used to instruct GROOT to complete this task. In addition, we selected 16-frame segment that best expressed the task information as the visual prompt for STEVE-1 (visual) from these six videos. This ensures fairness in comparison.
During inference, we found that in some tasks, such as build obsi diar behavior mixed with the intention of traveling around. We believe this a bias introduced during training. We draw the inspiration from STEVE-1 (Lifshitz et al., 2023) and subtract this bias in the action logits space before sampling the action. Specifically, we infer two models at the same time, where one model's condition is specific task video and the other model's condition is 28-frrmme
GROOT's
18

---

Published as a conference paper at ICLR 2024
Ore
Bulld Oballln
Enchont Sword
0.6
0.8
Dalld Snow
0.4
0.4
0.4
Dig Tireee
64
0.5
0.3
0.1
0.1
A-00
>=10
A=15
X-2.0
A=0.0
>=10
A=15
Figure Ablation on the condition scale X.
video of traveling freely in the environment. The input observations for the two models are exactly the action logits of the previous model to subtract certain proportion of the action logits predicted by the latter model before using the Gumbel-Soffi to sample the action. The logits calculation equation is directly borrowed from Lifshitz et al. (2023)
the same. At each time step,
trick
logits, (1 + fo ggoal) Afe(O1.t, Ibias)
(17)
where fo(O1.: Igoal) and (0 9bias) are two kinds of action logits generated by feeding forward two reference videos goal and bias to GROOT, À is a trade-off parameter. As illustrated in Figure 9, we find that this trick improve the success rate of tasks such as build obsidian build snow golem enchant sword
and dig three down and fill up (P) with the À = 1.5. Interestingly, observe that the effective À scale (approximately 1.5) in our model is much smaller than the scale (approximately 6.5) used in STEVE-1, We speculate that this may be because STEVE-1 fine-tunes the foundation VPT gain steerability, but VPT does receive goal conditions for demonstrations during behavior cloning. This may cause VPT to learn overly smooth behavior distributions, requiring the use of larger lambda scales to activate goal-specific behaviors. Although this technique is effective at inference time, it still requires hyperparameter tniing in practice. In the future, will meaningful to directly remove biased behaviors from the training process.
D.3 ABLATION ON NUMBER OF CONDITION SLOTS
In this section, we explore the impact of the number of condition slots (number learnable tokens) on the final performance. We compared the performance of the model on programmatic tasks with = 1 and = 3 condition slots and computed quantitative metrics for each task. As shown in Table we find that increasing the number of condition slots leads significant decrease in the model's performance on most tasks, except for the "explore run" task. We speculate that having more condition slots may result in higher number of dimensions in the goal space, which in turn reduces the generalization ability of the learned goal space. Therefore, we suggest that when applying GROOT other environments, the hyperparameters should be carefully chosen based on the characteristics of the environment or using other parameter selection methods.
Table Ablation on the number of condition slots.
Task Name (Metric)
explore run f (distance)
build pillar (height of pillar)
collect grass
collect sengruss (num seagruss)
collect dirt (num of dirt)
mine (num stones)
(num grass)
N
54.0
37.6 13.3
23.8
3.3
6.2
12.2
N
3
59.0
5.6
0.9
5.4
11.2
DATASET DETAILS
E.
CONTRACTOR DATA
The contractor data is Minecraft offline trajectory dataset provided by Baker et al. (2022) 3, which annotated by professional human players and used for training the inverse dynamic model. this
19

---

Published as a conference paper at ICLR 2024
dataset, human players play the game while the system records the image sequence {$1:T) J M. action sequence {a1:T JM and metadata {e1-Th M generated by the players. Excluding frames containing empty actions, the dataset contains 1.6 billion frames with duration of approximately 2000 hours. The metadata records the events triggered by the agent in the game at each time step, including three types: cra ft i item, pi ckup, and mine bl block, which represent the agent's activities of crafting items using the GUI, picking up dropped items and destroying blocks at the current time step, respectively. In the process of training GROOT, we use all trajectories provided by the contractor data, but without including any metadata. We only use the metadata to retrieve relevant trajectory segments during the visualization of the goal space.
F EXPERIMENTAL SETUP
SETUP DETAILS
F.1 BASELINE DETAILS
VPT is the first foundation model in the Minecraft domain developed by Baker et al. (2022). Its architecture consists of ImpalaCNN and TransformerXI. Using behavior cloning algorithms to pre- train on large-scale YouTubb demonstrations, they obtained the first checkpoint of VPT(fd) which can freely explore the environment. To further enhance the agent's s abilities in carly-game environments, they constructed an "earlygame" dataset and fine-tuned the pre-trained foundation model on that dataset, resulting in the VPT(bc) checkpoint. This model significantly improved performance on basic tasks such as "crafting table" and "collecting wood". Based VPT(be), they used online reinforcement learning with a carefully designed reward shaping to obtain the checkpoint (PT(rl) capable of obtaining diamonds entirely from scratch. It is noteworthy that the models' architectures of all three checkpoints are consistent and do not support instruction inpul. That's why their rankings on the Minecraft SkillForge benchmark are low. We also observed that the performance of VPT( surpasses that of VPT(rl) due to the "earlygame" dataset's exploratory nature, making it perform better on explore tasks. VPT(rl) is tailored specifically for diamond mining tasks and has thus lost the capability of most tasks outside diamond mining path. No matter where you place it, the first thing (Pl(r does to look for trees and prepare to mine diamonds.
STEVE-1 is Minecraft agent that can follow open-ended text and visual instructions built on MineCLIP (Fan al., 2022) and VPT. It can perform a wide range of short-horizon tasks that can be expressed by a 16-frame future video clip. The training of STEVE-1 can be described in two steps. The first step is to train a future-video conditioned policy with packed hindsight relabeling trick. With the frozen MineCLIP visual encoder to embed the visual instruction, they finetune the VPT(bc) on the contractor data to obtain instruction into visual instruction. By training conditional variational autoencoder (CVAE) on the collected video-text pairs, they created a variant STEVE- (text) that understands text instructions. This baseline performs well on many simple tasks in the Minecraft SkillForge benchmark, such as "explore run," "collect grass," and "collect wood." " However, it struggles with multi-step and less common tasks, like "build snow golems" and "dig three down and fill one up."
The second step is to learn a model that translates textual
Please note that all baselines, including GROOT, were not fine-tuned for tasks in Minecraft SkillForge.
F.2 T-SNE VISUALIZATION DETAILS
This section details how the videos are sampled to do visualization. The selected videos are categorized into seven groups: craft items, combat enemies, harvest crops, hunt animal chop tress, trade with vi villagers, and mine ores. Generally, each group contains two types of videos, each with 1000 data points sampled. The sampling method retrieves the time when a certain event occurs in the metadata and goes back 128 frames from that time to obtain video segment that is 128 frames long. We illustrate video configurations in Table 4. For example, in the combat enemi es task, taking "combat zombies" as an example, we retrieve all the moments when the event "pickup:rotten flesh" occurs, because after killing zombies, they will drop rotten flesh, which can then be picked up by players. Through sampling observations, we found that this method can sample videos that are consistent with the descriptions.
20

---

Published as a conference paper at ICLR 2024
Table 4: Sample videos from the contractor data (Baker al., 2022) for the goal space visualization.
Group
Vidco Description
Event in Metadata
craft items
craft wodden pickaxcppeeeee with crafting. table
craft item: pickaxe -ockac craft item:iron pickaxe pickup:rotten_fee flesh pickup:spider, eye
craft items
craft iron Bainn___paeeeee with table
decmiron_ppckuxe
combat enemies
combat zombies
combat enemies
combat spiders
harvest crops
harvest wheat
mine_block:whett
harvest crops
harvest melon
mine_block:melon
hunt animals
hunt sheep
pickup:mutton
hunt animals chop trees
hunt cow
pickup:beef mine mine_block:ooa block:oak log
chop oak trees chop birch trees
chop trees
mine_ blonk_bbockbbbrch.... log_
trade with villagers
trade with villagers for emerald trade with villagers for enchanted book mine coal ores with pickaxe mine iron ores with pickaxe
craft_item:emerald craft item:enchanted _boo book
trade with villagers
mine ores
mine mine_blockkcool block:coal _ore
mine ores
mine_block:nng
F.3 PROGRAMMATIC EVALUATION DETAILS
In this section, we elaborated how each episode is regarded successful. For the dye and shear sheep task, dyeing the sheep and shearing its wool must be successfully performed to be considered a success. For the use bow (C)) task, firing the arrow after charging it to the maximum degree is required to be successful. For the sleep ( task, placing the bed and spending the night on it are required to be successful. For the smelt
task, placing the furnace and dragging coal and mutton into the designated slots are required to be successful. For the lead (S) task, successfully tethering at least one animal is considered success. For the bui id obsidian
task, pouring water bucket and lava bucket to fuse them is required to be task, placing the enchantment table, putting diamond sword
successful. For the enchant and lapis lazuli into the slots, and clicking the enchanting option are required to be successful. For the dig down three fill one up (f) task, the agent must first vertically break three dirt blocks below and then use one dirt block to seal the area above. For the bu l snow go gol ems task, placing 2 snow blocks and carved pumpkin head in order and triggering the creation of snow golem are required to be successful.
F.4 COMBINING SKILLS EXPERIMENTAL DETAILS
First, we introduce the experimental environment selected for our study. The agent is summoned on the plains biome, holding diamond pickaxe, and granted the night vision status enable the agent to see the various ores underground. At the beginning of each episode, we set the agent's condition to dig down. When the agent descends to a depth below 12 layers, the condition automatically switches to hor 1 zontal mining. Each round of episodes lasts for 12,000 frames, which is equivalent to 10 minutes in the real world. For GROOT, both the reference videos of dig down and hori zontal mining were recorded by a human player. For STEVE-1, we invited the same player to carefully record the prompt vidcos. It is worth noting that while we could easily prompt to dig down, it was difficult to keep it in the horizontal mining condition. This made STEVE-1 prone to falling into the bedrock layer and getting stuck. Finally, we did not observe STEVE-1 finding any diamonds in the 25 experiments, which can be attributed to the inability of its goal space to encode details such as horizontal mining.
G RATING SYSTEM
G.
ELO RATING
The ELO rating system is widely adopted for evaluating the skill levels of multiple players in two- player games, such Chess and Go (Silver et al., 2016). In this section, elaborate how
21

---

Published as a conference paper at ICLR 2024
Which agent performed better the task: build__olle oobiidin?
Buid Cosiaan
Pour wnteertess -our
Pour imo mold make dosldlan
Agest
Agent
Agent 2
C07/
0.04
Figure 10: Example of the annotating system for human evaluation.
introduce human evaluation and use the ELO Rating system to measure the relative performance of agents on the Minecraft SkillForge benchmark.
In the ELO rating system, each agent's skill level is represented by numerical rating. We repeatedly let agents play against each other in pairs. Specifically, in each game, we sample a task and two agents, denoted Agent A and Agent B. Then, we randomly sample trajectory for each agent corresponding to the designated task. The two trajectories are assigned to human annotator, who selects the most task-relevant one. We implement the annotating system with Label Studio (Tkachenko et al., 2020-2022), as shown in Figure 10. We consider the agent that produced this trajectory to be the winner, let's assume it is Agent After each round, we update the scores of Agent A and Agent B as follows
RA RA K.
+ 10(RA Ra)/100
(18)
RB + RB
1 + R0(RA-Ra)/100
where K is the update factor and we set it to 8. After calculating the score of the agent, we use VPT (bc) 1500 points and shift the scores of other agents accordingly. Based the ELO ratings, can easily measure the relative winning rate for each paired agent. The win rate of Agent A over Agent B can be represented as 1+ 37400 For example, the win rate ratio between two agents with score difference of 100 scores is 64% 36%. A score difference of 200 scores implicit 76% 24%.
G.2 TRUESKILL RATING
We also report the comparison results using TrueSkill 'rating system, which is used by gamers to evaluate their skill level. It was developed by Microsoft Research and is currently used on Xbox LIVE for matchmaking and ranking services. Different from ELO, it can also track the uncertainty of the rankings. This system utilizes the Bayesian inference algorithm quantify player's true skill points. TrueSkill, rating modeled Gaussian distribution which starts from (25,, 252) where is an average skill of player, and is confidence of the guessed rating. A real skill of player is between f + 2a with 95% confidence. After conducting 1500 updates, the TrueSkill scores converged in Table 5. We found that the ranking order of the baseline methods is consistent with that obtained using ELO rating: HUMAN > GROOT > STEVE- (visuul) STEVE ((text)((ttt))) > VPT(bc) VPT(fd) VPT(r)).
Table 5: TrueSkill rating comparison on the Minecraft SkillForge benchmark.
Baseline
HUMAN
GROOT
STEVE-I(tx)) STVvE-I(visul))
VPT(bc)
VPT(fd)
VPT(r)
J nto
1.0
0.9
25.8 4
24.6 + 0.8
0.8
0.8
0.9
22

---

Published as a conference paper at ICLR 2024
G.3 HUMAN PARTICIPATION
We recruited 15 students with varying degrees of Minecraft game experience, ranging from a few hours to several years, from the Minecraft project group to conduct the evaluation. They are all familiar with the basic operations of Minecraft. Each employee was asked to label 100 matches for ELO Rating or TrueSkill Rating, for a total of 1500 matches. For each employee who required to collect or assess gameplay videos, we ask them to first read the description of each task in the Minecraft SkillForge Benchmark completely, as well as the evaluation criteria for task completion quality, see Appendix H. Taking the task of building snow golem as example, the evaluation criteria are as follows: Build snow golem. > Place both kinds of blocks. > Place at least one kind of block. Place no block. This enables employees to quantify video quality and ensures that all employees evaluate task completion consistently. All these employees were explicitly informed that the collected data would be used for Al research.
H MINECRAFT SKILLFORGE BENCHMARK
this section, we detail the benchmark titled "Minecraft SkillForge" which meticulously incorporates wide spectrum of tasks prevalent within Minecraft. Our aim is ensure that every task provides meaningful evaluation of a specific skill that an AI agent might possess. We categorize these tasks into six groups: col collect, explors ore, craft, tool, survive, and build. ld. In the following subsections, we will provide a detailed introduction to each of them. The "Description" field provides brief description of the task. the "Precondition" field outlines the initial settings of the testing environment for the task, the "SkillAssessed"" field indicates which aspect(s) of the agent' 's ability are being assessed by the task, and the "Evaluation" field describes the quality evaluation metrics for task completion (based on which human players judge the quality of two rollout videos).
H.1 COLLECT
The tasks categorized under the collect collect section of our benchmark are specifically designed to evaluate an AI agent's capability in resource acquisition proficiency and spatial awareness. This means the agent should not only be adept identifying and gathering specific resources but also possess the acumen to navigate through varied environments while being aware its surroundings and the available tools at its disposal.
Task: collest dist
Description: Collect dirt from faceace.
Precondition: Spawn the player in the plains biome. skillassessed; 3asic terrain understanding and the ability to differentiate
between surface-level blocks.
Evaluation: Run away. < Look down. Dig down. < Break the dirt on the surface.
Task: collest grass
Description: Remove weeds on the surface.
Precondition: Spawn the player in the plains biome. SkillAssessed; Surface navigation
comprehension vegetation ocks..
Evaluation: away. Break the grass block. Break large field grass blocks.
Task: collest woad
Description: Cut down trees to collect wood.
Precondition: Spawn the player in the forest biome with an iren_axe in its hand. SkillAssessed: Recognition of tree structures, efficient utilization of tcols, and block harvesting capability.
Evaluation: Run away, Approach trees. Chop the tree and collect logs.
Task: collect seagrass
Description: Dive into the water and col lact seagrass.
Precondition: Spawn the player near the sea. skillAssessed: Water navigation, diving mechanics understanding, and underwater block interaction.
Evaluation: Walk on the land. < Swim on the water Dive into the water. < Break seagrass blocks.
Task: colleat woo!
23

---

Published as a conference paper at ICLR 2024
collect dirt from the surface
remove weeds on the surface
chop trees to collect wood
dive ve into the water and collect seagrass
W
dye the sheep then shear the sheep
Figure 11: Examples of tasks in collect category.
Description: Dye and shear the sheep for wool.
Precondition: Spawn the player in the plains biome with a shear (mainhand) and a stack of blae_dye leffhand), sheep neaz the player. SkillAssessed: Interaction with entities, zool and item application, and sequential
execution.
Evaluation: Ignore the sheep. Dye the sheep. Shear the sheep. First dye then shear the sheep.
Listing 1: The environment configuration evaluation metric for collect lect series tasks.
H.2 EXPLORE
The tasks encompassed within the explore category of our benchmark are intricately devised to evaluate an AI agent's navigation proficiency, understanding of diverse environments, and intrinsic motivation for exploration. Through these tasks, we gauge an agent's ability to actively traverse, understand, and interact with varied elements of the Minecraft world, and propensity unravel mysteries and challenges posed by the environment.
Task: rur and explore
Description: Run and explore.
Precondition: Spawn the player in the plains biome. SkillAssessed: Stamina utilization and distance-besse expl onpporation.
Evaluation: Exploring as far as possible.
Task: climb mountain
Description: Climb the mountain.
Precondition: Spawn the player in the stone shore bi ome and the mountain.
SkillAssessed; Vertical navigation, terrain adaptation, and gcal-oriented movemans.
Evaluation: Run away and ignore the mountain. Approach the mountain. < Climbing the mountain, Climb to the top of the mountain.
Task: mine horizontally
Description: hori aorizentally underground.
24

---

Published as a conference paper at ICLR 2024
run and expl
travel: on
on a wooden boat through water
clin climb the moun tain
open a chest and acquire the treasure
mine horizontally underground
Figure 12: Examples of tasks in explore category.
Precondition: Spawn player in deep cave Iron pickaxe the hand. SkillAssessed: Underground navigation, tool utilization, and spatial reasoning in confined spaces.
Evaluation: Run away. Break the stone. Dig down. Mine herizontally.
Task: travel by boat
Description: Travel on a wooden boat through water.
Precondition: Spawn the player near the sea with a wooden boat in the hand. SkillAssessed: Aquatic travel, toal placement, and boat naneuverability. Evaluation: Did not place the bost. Place the boat on the water. Board the boat. in the
Task: explore the treasure
Description: Rush into villager's home and open chest and acquire the treasure.
Precondition: Spawn the player in front of villager's s house.
SkillAssessed: Interaction with structures, curiosity-driven exploration, and object acquisition
Evaluation: Ignore the house and run away. Open the door. Enter the house. Open the chest. Acquire the treasure.
Listing 2: The environment configuration and evaluation metric for explore series tasks.
H.3 CRAFT
The tasks under the craft category in our benchmark have been designed to shed light AI agent's prowess in item utilization, the intricacies of Minecraft crafting mechanics, and the nuances of various game mechanic interactions. These tasks provide detailed examination of an agent's capability to convert materials into functional items and harness the game's various crafting and enhancement mechanics.
Task: craft the crafting_table Description: Open irventory and craft fiing table.
25

---

Published as a conference paper at ICLR 2024
open inventory and craft a crafting table
place the crafting table and open it to craft ladders
place furnace and use it to smelt food
place an enchanting table and enchant a diamond sword
place aj stonecutter and use it to cut the stone
Figure 13: Examples of tasks in craft category.
Precondition:
Spawn the player in the plains biome with stack of oak_planks in the inventory.
skillassessed: Inventory management and basie crafting.
Evaluation: Open the inventory. < Click on the recipe button. < Click on the c:afting_table. Brag the crafting_table into the inventery.
Task: craft ladders
the crafting table open Precondition: Spawn the player in the plains biome with crafting_table in its main hand and stack of pak_planks in the inventory.
Description:
craft ladders.
skillAssessed; Advarced crafting using crafting stations and recipe navigation.
Evaluation: Place the crafting_table on the surface. < Open the crafting Click on the recipe book. < Click on the ladder. Drag the ladder into the inventory.
Task: enchant aword Description:
enchanting ng table use enchart diamand Precondition: Spawn player in the plains biome with enchanting table ita main hand, 3 diamond sworda, and 3 stacks of lapis_lazuli in the inventory. skillAssessed: Tool enhancement using enchantnent stations and decision-naking in choosing enchantments.
Evaluation: Place the enchanting tabble on the surface. < Open the enchanting table. < Place the lapis_lazuli or diamond aword, & Place the lapis_lazuli and diamond sword. < Cheose any enchantment.
Task: smelt food Description: Place furnace and it smelt food.
Precondition: Spawn player in the plains biome with furnace table in its
main hand, stacks of mutton, and stacks of coal in the inventory. skillAssessed: Food processing using smelting furnace, raw material to product conversion, and patience in awaiting outcomes.
Evaluation: Place the furnace on the surface. Open the furnace. Place raw meat or coal. < Place both raw meat and coal. Wait for the raw meat to be cooked. Take eut cooked meat.
Task: cut stone
26

---

Published as a conference paper at ICLR 2024
Description: Place stenecutter and Precondition: Spawn the player in the plains biome with stonecutter in its main hand, 6 stacks of stones in the inventory.
stones.
Skilllssessed; Tool enhancement using enchantmert stations and decision-making in choosing enchantments.
Evaluation: Place the stonecutter on the aurface. Open the stonecutter. Place the stones. Select a target type of stone. brag stones to the inventory.
Listing 3: The environment configuration and evaluation metric for craft series tasks.
H.4 TOOL
The tasks within the Tool category of our benchmark designed deeply investigate AI agent's capabilities tool utilization, precision in tool handling, and contextual application of various tools to carry out specific tasks. This category provides insights into the agent's 's skill in wielding, using, and exploiting tools optimally within different Minecraft scenarios.
draw a bow and shoot
set fires on the forest
use rein to tie up the animals
place the pumpkins and carve pumpking ns with shears Sosten
fly the trident on a rainy day
Figure 14: Examples of tasks in tool category.
Task: use bow
Description: Draw a bow and shoot.
Precondition: Spawn the player in the plains biome with a bow in the mainhand and a stack of arrows in the inventory.
SkillAssessed: Precision, tool hendling, and projecti nastery.
Evaluation: Just run. Draw the bow and shoot the arrow. < Hold the bow steady and charge up the shot before releasing the arrow.
Task: fires
Description: Set fires the trees.
Precondation: Spawn the player in the forest biome with flint_and_steel in its main hand.
skillAssessed: Environment manipulation and controlled chaos creation. Evaluation: Atzack the tree. < Start a fire with the flint_ stee_l. Go wild with the fire.
Task: animals
27

---

Published as a conference paper at ICLR 2024
Description:
rein tie up the animals.
Precondition: Spawn the player in the plains blome with stack of leads in its main hand, Spawn 5 sheep and cows near the player's position.
SkillAssessed; Entity interaction, tool application on moving entities, and livestock
Evaluation: Ignore the animals and run away, < Use the rein to tie animals.
Task:
carve pumpkins
Description: Place the pumpkins and carve pumpkins with shears.
Precondition: Spawn the player in plains biome with shear in main hand and stack of pumpkins the inventory.
SkillAssessed; 3lock placement, black modification, and crafting interaction.
Evaluation: Just rut, Place the pumpkin on the surface. < Use the shear to carve it. Get carved pumpkin.
Task: use trident
Description: Fly the trident on rainy day,
Precondition: Spawn the player in the plains biome with a trident in the main hand, which enchanted with riptide. The weather rain.
SkillAssessed; Weather-adaptive toel utilization, motion dynamica, and advanced weapon handling,
Evaluation: Just
Use the trident to break the black. < Use the trident for quick movement. Charge to throw the trident farther.
Listing The environment configuration and evaluation metric tool series tasks.
H.5 SURVIVE
The tasks embedded within the survi ve category of our benchmark aim to analyze an AI agent's ability to ensure its own survival, adeptness in combat scenarios, and its capability to interact with the environment in order to meet basic needs. Survival, being core aspect of Minecraft gameplay. necessitates intricate balance of offensive, defensive, and sustenance-related actions. This category is structured to ensure thorough evaluation of these skills.
hunt animals on the plains
fight the enemy spider
use shield to ward off zombies
use the hoe to till the land and plant wheat seeds
a the bed on the and sleep
Figure 15: Examples of tasks in survive category.
28

---

Published as a conference paper at ICLR 2024
Task: hurt animals
Description: llunt animals the plains.
Precondition: Spawn the player in the plains biome with iron aword in the main hand. Spawn 5 sheep and 5 cows near the pl ayer's position.
skillassessed: Predator instinets, combat efficiency, and sustenance acquisition.
Evaluation: Ignoze animals and run away. < Burt animals. Kill animals.
Task: combat enemies
Description: Fight the enemy spider.
Precondition: Spawn player in plains biome with diarend sword its main hand and suite diamond equipment. Spawn 3 spiders in front of the player.
skillAssessed: Self-defense, offensive combat strategy, and equipment utilization. Evaluation: Ignore spiders and run away. < Burt spiders. Kill spiders.
Task: use shield
Description: Use shield to ward off zombies.
Precondition: Spawn the player in the plains biome with a shield its main hand and suite of diarend equipment. Spawn zombies front the player.
SkillAssessed; Defensive tactics, tcol application in combat, and strategic protection.
Evaluation: Tgnore zombies and away. Use the shield d protect itself.
Task:
plant wheats
Description: Use an iron_hoe to till the land and then plant wheat seeds. Precondition: Spawn the player in the plains biome with an iran hoe in its main hand, and a stack of wheat seeda the off hand.
SkillAssessed: Land cultivation, planting proficiency, and sustainable resource creation.
Evaluation: Just away. Till the land. Flant the wheats.
sleep the bed Description: Place the bed the surface and sleep. Precondition: Spawn the player in the plains biome with a white bed in its main hand.
Task:
skillAssessed: Sel seif-preservationn understanding of day-night cycle impl icat ions, and use of atilities for rest.
Evaluation: Just rur away, Place the bed on the surface. Sleep the
Listing 5: The environment configuration and evaluation metric for survive series tasks.
H.6 BUILD
The tasks within the build category of our benchmark are devised to evaluate an Al agent's aptitude in structural reasoning, spatial organization, and its capability interact with and manipulate the environment to create specific structures or outcomes. Building is an integral component of Minecraft gameplay, requiring an intricate interplay of planning, creativity, and understanding of block properties.
Task: build pillar Description: Build pillar with dirt.
Precondition: Spawn the player in the plains blome with stack of dirt in the main hand.
SkillAssessed: Vertical construction and basic structure formation.
Evaluation: Just run away. < Look down. < Jump and place the dirt. < pile the dir: into a few pillars. < Make a really high pillar.
Task: dig three down and fill one up
Description: Dig three dirt blocks and fill the hole above. player in SkillAssessed; Ground manipulation and depth perception.
Precondition: Spawn
plains blome.
Evaluation: Just away. < Look down. Dig down three dirt blocks. Raise the head. < Raise the head and use eirt to fill the hole.
Task: build gate Description: Build an archway gate.
Precondition: Spawn the player in the plains blome with stack of oak_planks the main hand.
skillAssessed: Synmetry, planning, and aeschetic construction.
29

---

Published as a conference paper at ICLR 2024
build a pillaz with dirt
build an archway gate
make obsidian by pouring water and lava buckets
build snow golems by placing snow
locks and pumpk in
dig three and fill one
Figure 16: Examples of tasks in bui category.
Evaluation: Place plank. Build 1 pillar. Build pillars. Build an archway gate.
Task: build absidian Description: Make obsidian by pouring a waber bucket and a lava backet. Precondition: Spawn the player in plains ome with two water buckets and two lava buckets the Hotbar.
SkillAssessed; Material cransformation, understarding in-game chemistry, and precise pouring.
Evaluation: Just run away. < Four water lava. < Pour beth liguids. Pour into nold to make obsidian.
Task: build snow golems
Description: Build snow golens by placing two snow blocks and one carved pumpkin. Precondition: Spawn the player in the plains bi ome with two stacks snew blocks twa stacks carved pumpkins in
SkillAssessed; Entity creation, sequential block placement, comsination multiple materials.
Evaluation: Place bleck. Place least kind of block. Place beth kinds of blocks. Build snow golem.
Listing The environment configuration evaluation
series tasks.
30

---

Published as a conference paper at ICLR 2024
1 TEXT CONDITIONING
Behavior Cloning
Environment
Text Encoder
Decoder as Policy 8gg
Text Encoder
Decoder as Policy
block ook jok_Jog"
"mine block grass"
Instruction
Text Instruction
Rollout Observations
(a) Fine-tuning
(b) Inference
Figure 17: Fine-tuning GROOT to understand text instructions. We replace the original video encoder with a text encoder to embed text instructions. The text encoder fine-tuned align with the learned goal space. Left: During the fine-tuning, we freeze the learned decoder to provide the supervisory signal to train the text encoder through hehavior cloning. Right: During the inference, we can feed forward the text instructions to drive the policy to interact with the environment.
Although video instruction has strong expressiveness, it still requires preparing at least one gameplay video for a new task. For most common tasks, such as collecting wood or stones, using natural language to specify goal is more natural approach. In this section, explore the possibility of aligning the pre-trained goal space with other modal instructions, such as text instructions.
Aligning text instructions with visual instructions in goal space has been validated as feasible by Lifshitz et al. (2023). They train a conditional variational autoencoder to project text into video space after collecting 10,000 diversified text-video pairs, similar to what unCLIP did. However, the success of this alignment method depends on the pre-alignment of visual and text spaces through large-scale contrastive pre-training (Fan et al., 2022). During the training process of GROOT, we did not leverage the MineCLIP visual encoder to encode videos, instead trained goal space from scratch. On the one hand, this is because MineCLIP can only handle short videos (only 16 frames); on the other hand, is to free our goal space from the expressiveness bounded by pre-trained VLM.
According the above discussion, we choose to replace the video encoder in the GROOT architecture with text encoder, BERT, and directly optimize it through behavior cloning, as shown in Figure 17. In order keep the original goal space, we freeze the decoder and regard it as a gradient generator that extracts high-level behavioral semantics from the demonstrations. We utilize the meta information in the contractor data to generate text-demonstration pairs. For example, in the task of "collect wood", identify the moment t when event "mine block:oak log" is triggered in the video, and we capture the frames within the range of [t 127, t] to form a video clip, with "mine block oak log" assigned its text, thus constructing a sample. Having been fine-tuned on these data, demonstrated some steerabilities in the text instruction space, as shown Table 6. We find that the agent fine-tuned on the -exonsssmsttttionn dataset shows basic understanding of text instructions. Our method exhibits progress in tasks such as "mine grass", "mine wood". "'min stone", "mine seagrass", "pickup beef"' and "mine dirt". However, falls short in successfully completing tasks such as "mine seagrass". We speculate that this may be related to the distribution of the data, as there much less data available for "mine seagrass" compared the other tasks (about 300 trajectories).
model
We emphasize that this experiment is very preliminary. In this experiment, the steerability of the agent fine-tuned on text instructions is still weak and it is hard solve practical tasks. Given the limited diversity of text instructions in the provided contractor data, we don't anticipate the model to possess any significant level of generalization with regard to language instructions. To further verify this point, one needs to collect more diverse and higher-quality text-demonstration pairs data. Anyway, this experimental result still indicates the possibility of optimizing the upstream instruction generator by leveraging the pre- traitrednd decoder. This creates possibilities for developing more interesting applications on GROOT. Additional discussions on text-conditionin are beyond the scope of this paper, and will leave them for future work.
31

---

Published as a conference paper at ICLR 2024
Table 6: Text conditioning results resource collection tasks, Each episode lasts 30 seconds (600 frames). Statisties are measured over episodes. The term "baseline" refers to model before being fine-tuned, while "fine-tuned" refers to the final model after line-tuning.
Variant mine grass
mine wood
mine stone
mine seagrass
pickup beef
mine dirt t
baseline
3.9 17.3 (4.4x)
0.4 3.7 (9.3x)
1.8 11.5 (6.4x)
1.3 1.2 (92%)
0.0
0.0
fine-tuned
0.1
1.3
J POTENTIAL APPLICATIONS AND INTEGRATION WITH PLANNER
GROOT is specialized in short-horizon instruction-following tasks with its goal being video clip while LLM has demonstrated the ability to plan for ang-horizon tasks in an open-world environment. For example, DEPS Wang al. (2023b) utilizes lext-conditioned policy from Cai al. (2023) to accomplish tasks such as mining diamonds from scratch. By integrating GROOT into the DEPS framework, it can act controller and assist with long-sequence tasks. However, while LLM can output language as the current goal, GROOT requires a specified video clip as its goal. Therefore, the visual language model CLIP select
when combining GROOT with DEPS, is necessary the most suitable video clip based on the language goal produced by LLM.
The proposed approach involves preparing pre-existing library of video clips V that contains various actions performed by the agent in Minecraft (e.g., "chopping trees" or "mine iron ore"). When given a long- Long-horizon instruction by LLM's Planner, is decomposed into series of short-horizon language tasks {g:}. During task execution, the CLIP model is utilized to calculate the similarity between each short-horizon clip 2; in the library V and task 9is selecting the most similar video clip GROOT's s interaction goal with the environment. Additionally, accessing video library of Minecraft content effortless due the abundance of available video data on the internet.
While GROOT mainly relies videos for input goals, LLM uses both input and output language modalities. These modalities can be aligned using visual language model, allowing us to combine GROOT short-horizon control policy with an LLM-based Planner to complete long-sequence tasks.
32


Reviews:
[{"review_id": "F9vdRxOarg", "reviewer": "Reviewer_anV3", "summary": "The authors propose GROOT, a video-goal-based instruction following agent. GROOT consists of an encoder-decoder architecture, with a video encoder for goal-conditioning and a causal transformer for the action policy. This architecture enables the agent to be conditioned on videos as flexible multi-task goals, rather than the more limited spaces of language or final states as studied in prior work. GROOT shows impressive performance when tested against prior methods for training agents in Minecraft using the proposed Minecraft SkillForge benchmark.", "questions": "It’s interesting that conditioning on multiple videos concatenated together can drive the agent to accomplish multiple tasks simultaneously, and the embedding space seems to be qualitatively meaningful. Do the authors think there would also be a benefit to constructing synthetic clips concatenating multiple tasks and training on them as well? Even if the source video settings look very different (potentially making goal-space learning harder), it might be an interesting way of generating large amounts of interesting synthetic, multi-task data.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The overall paper is well written and the method is clearly presented. The proposed method is simple, and providing the ability to condition more flexibly on multi-step video-based goals is a meaningful contribution in the space of developing generalist agents, as there is an abundance of video-based data available online to learn from.\n- The experiments seem thorough, with clear improvements compared to prior methods on a suite of different complex tasks within Minecraft. The paper considers both quantitative skill improvements as well as qualitative goal space analyses, showing that the learned goal space has likely captured some meaningful semantics.", "weaknesses": "- While the multi-step video representation for goals is more expressive than other prior Minecraft agents (that use language or outcome videos), having to provide a video to condition on can be difficult to do in practice if we do not already have access to similarly representative videos, and as the authors note, training the video-based goal space is challenging. On the other hand, it is much easier to describe a desired goal in language. It would be interesting to see if this method can be adapted flexibility to condition on either visual or text based goals, as in STEVE-1.\n- Nit: confusing notation – changing from $p_\\theta$ to $p_\\psi$ in the KL term", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PATI9z078s", "reviewer": "Reviewer_f4yG", "summary": "This paper proposes a method that utilizes reference videos as instructions to reduce the cost of text-gameplay annotations in constructing controllers that follow open-ended instructions. Drawing inspiration from the concept of Variational Autoencoders (VAE), the authors employ an encoder to map reference videos to the goal space, followed by a decoder acting as a goal-conditioned policy. The paper establishes a benchmark in the Minecraft, and demonstrating the effectiveness of video-based instructions compared to textual and visual instructions.", "questions": "Please refer to the detailed questions in the Weaknesses section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Significance: This paper addresses an interesting problem, that is, using gameplay videos to train instruction-following controllers, and validates the effectiveness of videos compared to visual information and text.\n\nOriginality: The main innovation of this paper lies in proposing a method for learning the representation of video instructions (i.e., goal space) and providing relevant theoretical derivations. Through ablation experiments, the paper demonstrates that imposing constraints on the goal space can improve performance. The visualization of the goal space is also intuitive.\n\nQuality: This paper provides a theoretical basis for the main innovation and conducts a fair comparison with baseline methods.\n\nClarity: The description of the methods in this paper is clear, but some symbols lack descriptions, and some implementation details are insufficiently described, which makes the reading a bit confusing.", "weaknesses": "1. The modeling method for video instruction in this paper is innovative and can improve performance. However, since the low-level controller still needs to select task-related videos as instructions when performing specific tasks (Section 5.1), it cannot be well adapted to LLM-based planning agents. Although the video instructions used are generated from other biomes, it only proves the controller's generalization ability in specific tasks, rather than the generalization ability for video instructions.\n- Can the authors validate the controller's generalization ability for diverse tasks by providing relevant video instructions for unseen tasks and skills? This would be of great significance for the integration of LLM-based planning agent capabilities.\n- If GROOT is combined with a high-level planner, how would it construct video instructions?\n- For the results in Figure 7, how are the two video instructions of GROOT specifically input to the controller? Is it directly combined and input, or is the video switched manually after reaching the specified depth?\n\n2. Although the paper often emphasizes learning from gameplay videos, the inverse dynamic model (IDM) is based on VPT, and there is a lack of discussion on IDM. Therefore, I believe that the essence of this paper is to use task-related state sequences as instructions and train a goal-conditioned policy. Using state sequences as instructions/goals for training agents is not uncommon. Methods such as GATO[1], AD[2] directly input demonstrations as token sequences into transformers, while [3] uses VLM to extract video information. Even without imposing constraints on the goal space, agents can exhibit good performance. The authors should compare or discuss these methods.\n\n3. As shown in Figure 5, if the video contains multiple goals, the performance may be affected. Would this lead to more stringent requirements for video quality during application?\n\n4. In Figure 6, when the KL loss is ablated, the performance decreases. Can the authors provide a visualization of the goal space without KL loss to explain its contribution to the agent's generalization in the goal space? As a significant part of the main contribution, the KL loss needs to be analyzed in more detail.\n\n5. Additionally, there are several unclear descriptions in the paper that may affect the understanding of the method.\n- In Section 3, symbols like $\\theta$ lack explicit definitions.\n- In Equation 13, why does log $p(a_τ|s_{0:τ})$ depend solely on the environment dynamics?\n- In Section 4.1, what does \"a set of learnable embeddings\" specifically refer to?\n- How is the segment length T set? Is there only a single goal within each segment?\n- In Equation 3, is the policy goal-conditioned?\n- There is a lack of detailed description for the baselines, as well as an explanation for why VPT(rl) would achieve the worst results.\n\n---\n\n[1] Reed, Scott, et al. A generalist agent. 2022.\n\n[2] Laskin, Michael, et al. In-context reinforcement learning with algorithm distillation. 2023.\n\n[3] Di Palo, Norman, et al. Towards a unified agent with foundation models. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a4eP8D9Rdw", "reviewer": "Reviewer_9vuS", "summary": "This work proposes a new agent architecture GROOT that learns goals from videos. The encoder-decoder transformer-based agent, learns from supervised videos with actions and is able to replicate the goals in reference videos. The agent GROOT is evaluated on a fairly comprehensive benchmark Minecraft SkillForge which covers a wide range of activities. Results are competitive and open up for more follow-up works in this domain.", "questions": "Figure 3 bottom rows have both the baselines not achieving any of the task, with the first plot in row 2 having the weaker baseline outperforming the VPT fine-tuned in the experiment. Is there a particular explanation towards this?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The newly designed benchmark is a nice addition to help the evaluation of the proposed agent. It covers a wide range of different activities in the Minecraft environment, including some long-horizon tasks, building tools. Release of the evaluation benchmark is able to help the community.\n\nThe design of the model architecture intuitively makes sense.", "weaknesses": "There are not enough training details disclosed in the paper. Ablation on the KL loss is nice. More ablation studies on for example the number of learnable tokens would be appreciated. These experiments will further validate the robustness of the model for the task.\n\nThe training of the model still requires action input. This means that for raw video, GROOT relies on inverse dynamics model to generate pseudo action labels. The idea of an agent learning from video might have oversold the novelty of the architecture/algorithm. The attempt to sell the algorithm as unsupervised learning is repeated in the conclusion section.\n\nThe position of the figures are out of order with the experiment/result section, making it hard to navigate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9MnWI7hZ4c", "reviewer": "Reviewer_WJvG", "summary": "The paper focuses on creating agents that complete tasks specified by a demonstration clip: agent is placed in the world and is given a demonstration clip of the goal, and agents' goal is to complete the task specified in the video. The paper proposes a new method for training such policy on unsupervised data (only videos, no skill/task labels needed). The results show that the proposed method outperforms the baselines by a significant margin.", "questions": "1) Can you provide more details on  the human participants? Mainly following parts: how were they recruited (e.g., asked inside the lab, advertised on an online community). Did they have previous experience with Minecraft? Did they get to practice Minecraft? What instructions were they given for data collection / evaluation, during and prior the data collection? Were human participants made aware how the data collected will be used (i.e., for AI research)?\n2) What are the error bars in Figure 3a? I understood the ELO rating is deterministic (given the set of comparisons/results), and does not model the uncertainty.\n3) Figure 5 and 6, what are the error bars (and can you report N in the caption, I believe it is 25)? While there is trend that one is higher than the other, the scale of noise makes me wary of the results.\n4) Was the night vision effect also added when VPT and STEVE agent played? They were not trained with this effect enabled, which might have skewed their results (but I do not expect huge change).\n5) Do you have ablation results for the logit-substraction trick described in Appendix C.2? You highlight it as an important change, but it is hard to tell how important it is given there are no results reported for this change.\n6) How did you determine 1500 pairwise to be sufficient for the ELO rankings? I assume the ranking did not change towards the end of the collection. However, I'd recommend you to also see if other metrics like TrueSkill would paint the similar picture. TrueSkill is much like ELO, but also tracks uncertainty of the rankings. See this for an open-source implementation: https://trueskill.org/\n\n\n## Update 20th Nov\n\nI have read authors' rebuttal and acknowledge they have satisfied my questions. I have kept my original review score the same (8).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Overall, considering the inclusion of a promising method and a new benchmark in a challenging environment, I recommend this paper for acceptance in ICLR. I have some hesitations around the results and comparisons (see weaknesses), which is why I steer away from strong accept, but I believe these results and contributions to be useful for ICLR community as a whole.\n\n### Originality\n\nThe proposed idea is new, and a surprisingly simple to implement. The work done for evaluation is also novel in this exact setup. The idea of using demonstration-conditioned policies is not novel per-se, but is under-explored compared to using reward signals or simple human feedback.\n\n### Quality\n\nThe results are solid: the paper presents a new (large) benchmark for testing models in the studied setting, and compare the proposed method against baselines, outperforming them with a clear margin. The experiments also push the method's limits with longer horizon tasks and ablate results, and also study the learned embeddings/model.\n\nCompared to some other Minecraft results in recent years, all results were done in the \"hardest\" Minecraft version (one used by OpenAI VPT), without using additional helper information.\n\n### Clarity\n\nPaper is well structured and readable, however the amount of content makes it slow to digest (see weaknesses).\n\n### Significance\n\nI believe many readers in this field (conditioned agents in complex environments) would find these results useful, and would build upon the proposed method. The included benchmark is useful for future exploration to this field as well.", "weaknesses": "### Flawed comparisons to baselines\n\nWhile STEVE-1 text/visual comparison is valid, being also goal conditioned policy, VPT models are not conditioned in any way. While it provides a decent \"unconditioned baseline\" result, it is not surprising that conditioning the policy improves the results. However, this is somewhat minor weakness, as there are no other baselines in the field.\n\n### (Minor) Some uncertainty around the results\n\nFigures 5 and 6 have rather high error bars which could indicate that results are not significantly different (see questions). Also, there are some questions around the evaluation and inference setup (see questions), which reduces my confidence in the results. However, overall, I believe the results hold and find the proposed architecture beneficial.\n\n### (Minor) Generalization\n\nMethod was only tested on one (but complex) environment, and the tasks provide initial tools already (and the tasks are rather short-horizon). However these are more of ideas for future work than real weaknesses of the work.\n\n### (Minor) Clarity\n\nThe paper is slightly hard to digest, but I feel this is mostly due to the amount of content (new model, human evaluation and new benchmark). Some details were kept in the appendix or missing (see questions).\n\nI'd suggest proof-reading the paper for typos (e.g., Figure 5, \"Comparision\" -> \"Comparison\", or Page 9, \"Minecraft is attracting increasing researchers\". This should probably be \"Minecraft is attracting _an_ increasing _number_ of researchers\").", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)