GROOT-2: Weakly Supervised Multimodal Instruction Following Agents
Authors: Shaofei Cai
Bowei Zhang
Zihao Wang
Haowei Lin
Xiaojian Ma
Anji Liu
Yitao Liang
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Open-world Agent
Weakly Supervised Learning
Goal-Conditioned Policy
Primary area: reinforcement learning

Abstract: Developing agents that can follow multimodal instructions remains a fundamental challenge in robotics and AI. Although large-scale pre-training on unlabeled datasets has enabled agents to learn diverse behaviors, these agents often struggle with following instructions. While augmenting the dataset with instruction labels can mitigate this issue, acquiring such high-quality annotations at scale is impractical. 
To address this issue, we frame the problem as a semi-supervised learning task and introduce \agent, a multimodal instructable agent trained using a novel approach that combines weak supervision with latent variable models. Our method consists of two key components: constrained self-imitating, which utilizes large amounts of unlabeled demonstrations to enable the policy to learn diverse behaviors, and human intention alignment, which uses a smaller set of labeled demonstrations to ensure the latent space reflects human intentions. \agent’s effectiveness is validated across four diverse environments, ranging from video games to robotic manipulation, demonstrating its robust multimodal instruction-following capabilities.

Published as a conference paper at ICLR 2025
GROOT-2: WEAKLY SUPERVISED MULTIMODAL IN- STRUCTION FOLLOWING AGENTS
Shaofei C Bowei Zhang Zihao Wang 12 Haowei Lin 1.2 Xiaojian Ma², Anji Liu", Yitao Liang"
Institute for Artificial Intelligence, Peking University Schooo of Intelligence Science and Technology, Peking University School of Electronics Engineering and Computer Science, Peking University *Computer Science Department. University California, Los Angeles
Beijing Institute for General Artificial Intelligence (BIGAI)
(caishaofei, zhangbowei, i, zhwang 8stu. pku. edu. cn, xiaoj ian wwa.aaaaaaaaaammmm ma@ucla. edu (linhaowei, yitaol/ @pku. edu. en, liuanji@es wula. edu
ABSTRACT
Developing agents that can follow multimodal instructions remains a fundamen- tal challenge in robotics and AI. Although large-scale pre-training on unlabeled datasets (no language instruction) has enabled agents to learn diverse behaviors, these agents often struggle with following instructions. While augmenting the dataset with instruction labels can mitigate this issue, acquiring such high-quality annotations scale impractical. To address this issue, frame the problem a semi-supervised learning task and introduce GROOT-2, a multimodal instructable agent trained using novel approach that combines weak supervision with latent variable models. Our method consists of two key components: constrained self- imitating, which utilizes large amounts of unlabeled demonstrations to enable the policy learn diverse behaviors, and human intention alignment, which smaller set of labeled demonstrations to ensure the latent space reflects human intentions. GROOT-2' effectiveness is validated across four diverse environ- ments, ranging from video games to robotic manipulation, demonstrating its robust multimodal instruction-folowing capabilities.
Craft
Build
Collect
GROOT
GROOT-2
Figure 1: By feeding mixture of demonstrations and some multimodal labels, leam GROOT-2, human- aligned agent capable of understanding multimodal instructions and adaptable various environments, ranging from video games to robot manipulation, including Atari, Minecraft, Language Table, Simpler Env.
INTRODUCTION
Developing policies that can follow multimodal instructions to solve open-ended tasks in open- world environments is a long-standing challenge in robotics and AI research. With the advancement of large-scale pretraining (Brown et al.) 2020 Baker et al. 2022 Brohan et al.) 2022), the research paradigm for -nslroction-folloiing policies has shifted from reinforcement learning to supervised
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
Naive Action Copying
Rough Trajectory
Behavior Semantic
Condition Agnostic
2
2
M
BC
Mechanical Imitation
Posterior Collapse
Figure 2: The ELBO Objective of the VAE and Latent Space Spectrum. We define a spectrum based on R = BC-KL: where R corresponds to "mechanical imitation" and R to "posterior collapse." At low R. latent vector z directly outputs action sequences without considering observations ( BC + 0). As H increases, : represents high-level task information, such as specific object interactions. At R = 1, provides no beneficial information for decision-making.
BC
learning. In a supervised learning approach, researchers collect large amounts of demonstration data and annotate each demonstration with multimodal instructions-such as videos (Duan al. 2017
Jang et al. 2022), texts (Padalkar et al. 2023 Lynch et al. (023), and episode returns (Chen et al. 2021) using hindsight relabeling. In theory, the instruction-followim capability of such policies
improves as the dataset grows. However, annotating demonstrations with high-qualig multimodal
labels is prohibitively expensive, making it challenging scale these methods in practice.
Another line of work (Lynch et al. 2020c Ajay et al., 2020: Cai et al. 20236) avoids the need for
additional human annotations by learning from demonstration-on!! data a sulf-supervised manner.
These approaches leverage latent variable generative models (Kingma & Welling (0033) to jointly learn an encoder and a latent-conditioned policy. The resulting policy capable of completing multiple tasks specified by a reference video (Cai et 2023b). While reference video is generally expressive enough to represent various tasks, the inherent ambiguity in videos can lead to a learned latent space that is misaligned with human intention. For example, the encoder module may capture the dynamics between adjacent frames in a video, thereby learning a latent representation of the action sequence- process refer to "mechanical imitation." " While this latent space accurately reconstructs the target action sequence, the resulting latent representation is difficult for human users leverage during policy deployment. Another potential issue is "posterior collapse," " where the latent space collapses to a single point and loses its influence over the policy during inference. We attribute this mismatch between training and inference to the absence of direct supervision for aligning the latent space with human intention. As illustrated in Figure 2 an ideal controllable latent- induced policy space must strike a balance between these two extremes.
We present GROOT-2 (refer to Figure D. multimodal instructable agent developed using latent variable model under weak supervision. To unify the training pipeline, we encode instructions from all modalities distributions over the latent space. The training objectives consist of two key compo- nents: (1) constrained self-imitating, which utilizes large amounts of unlabeled demonstrations to enable the latent-conditioned policy to learn diverse behaviors; and (2) human intention alignment, which uses relatively small sets of multimodal labels to align the latent space with human intentions. Specifically, apply the maximum log- liggliikehhood method in the latent space for alignment. The un- derlying principle is that the latent embedding encoded by multimodal labels should also be sampled from the distribution learned from the corresponding video. Our approach is both general and flexible, as demonstrated through evaluations across four diverse environments rragging from video games to robotic manipulation-innluue
Atari Games (Bellemare et al. 2013). Minecraft (Johnson et al.) 2016). Language Table Lynch et al. (2023), and Simpler Env (Li et al. 2024). These experiments highlight GROOT-2 's robust ability to follow multimodal instructions, with extensive tests showing that scaling up either unlabeled or labeled demonstrations further enhances performance.
2 BACKGROUND AND PROBLEMS
2.1 LATENT VARIABLE MODELS ENABLE CONTROLLABLE BEHAVTOR GENERATION
In recent years, the GPT series (Radford 2018 Radford et al. 2019 Brown et al. 2020 has demonstrated impressive capabilities in controllable text generation. Its success can be attributed to
2

---

Published as a conference paper at ICLR 2025
self superviede pretraining and the advantageous properties of natural language. A natural language paragraph contains rich dependencies between sentences. For instance, the title of an article sets the central theme for its body, and the response in a ceasstion-nsswe dialogue is highly correlated with the preceding text. This characteristic enables large language models, trained via next-token prediction, achieve controllable text generation through prompts during inference. Unfortunately, such strong correlations do not exist between low-level actions. A desired behavior may not have necessary preceding trajectory segment. Thus, it isn' 't easy to prompt pre-trained policy model to generate desired behavior. Instead, the generation of actions depends on an underlying latent intention variable. natural approach employ latent variable generative models to jointly model trajectory data and the latent variables that drive them, allowing for controllable behavior generation by manipulating the latent variables during inference. Next, we will elaborate on how latent variable models model trajectory data.
As a classic latent variable generative model, Variational Autoencoder (VAE, Kingma & Welling (2013)) has been widely used in fields such as image generation and text generation. With the development of the offline pretraining paradigm, recent years have seen an increasing number of works utilizzig VAE to model trajectory data. Typically, its architectures consist of three components: a posterior encoder, a prior encoder, and a policy decoder. The posterior encoder, (2][T), encodes specific behavioral trajectory = (oI:N, a1:N) and generates a posterior distribution over the latent space. When the action sequence can be accurately inferred from the observation sequence 2022 Zhang et al.] [222)_i.e.. when the inverse dynamics model of the environment PIDM a 0 1 is easily lcarned-the action sequence can be excluded from the posterior's 's input (Cai et al. 2023b), thus reducing the distribution condition to O1:N. The prior encoder, p(z (01:k). generates a distribution over the latent space based on the history of observations, where lc denotes the length of the observation window. When O, the prior distribution is independent of historical observations and is typically assumed to follow a standard normal distribution N(; 1). The decoder, π (a, o =). is generally a latent-conditioned policy that takes in the environment's observations along with specific latent variable to predict the next action to be executed. According to variational inference theory, we can optimize the VAE's modeling capabilities by maximizing the Evidence Lower Bound (ELBO)
(Baker et al.
N
LELBO = E [(
((gl(aa) 2) + -XL(((z)oNN)) p(z(o<a)).
(I)
There are generally three main objectives for using VAE to model trajectory data: (1) Modeling multimodal behaviors (Lynch et al. 2020a Mees et al.) 2022): For instance, when trajectory data collected from different individuals, the variations in action sequences across different behavior modes can be substantial. Directly applying naive behavior cloning algorithm may result in poor modeling performance. Introducing an additional latent variable to differentiate between behavior modes can help mitigate conflicts between them during training.
(2) Skill discovery (Xu et al.) 2023 Gupta et al. 2019): Complex trajectory data is often composed of various skills. A VAE can abstract action sequences in self-supervised manner, enabling skill reuse in downstream tasks, such as accelerating the exploration process in reinforcement learning (Pertsch et al., 2021] Ajay et al., 2020). (3) Following reference videos to complete open-ended tasks (also known as one-shot demonstration learning. Cai et al. (202bb)) This approach aims to leverage the learned posterior encoder to recognize the underlying intention behind reference video and encode it as a latent, which can then drive a policy to complete the specified task in novel deployment. It points to a way to pre-train cooructtoo-following policies using unlabeled trajectory data. We primarily focus on the third objective in the following paragraphs.
2.2 MODELING BEHAVIORS WITH VAE LEADS TO AMBIGUOUS LATENT SPACE
Several studies on VAE Alemm et al. 2018 Abeer et al. 2024 have pointed out that the Pareto frontier of the ELBO contains an infinite number of solutions for the latent space, a phenomenon refer to as latent space ambiguity. To facilitate understanding, we provide an informal illustration in Figure 2 which shows several possible latent spaces when VAE is used to model behaviors, BC all having similar ELBO values. We differentiate these latent spaces using the ratio R = KT where R = and R = 1 represent two extremes of the latent space. When R approaches 0, the latent condition contains much information, nearly dictating every action of the policy's 's behavior. We
3

---

Published as a conference paper at ICLR 2025
Reference Video
Env
Rollout Trajectory
Go stroight and fall into the lava.
Go straight and mine the diamond
Digging around the lava reach the diamond mise.
Figure 3: Comparison of Policies with Different Latent Spaces. The reference video depicts digging for diamonds. A policy that mechanically imitates the trajectory falls into lava, while one aligned with human intention avoids lava and successfully reaches the diamonds.
refer to this as mechanical imitation, where the VAE effectively degenerates into an Autoencoder (AE). Conversely, when R approaches 1, the latent loses its ability to control the policy's 's output, phenomenon known as posterior collapse (Fang et al., 2019; Pagnoni et al. 2018), in which the R increases, the information encoded
VAE reduces to Auto-regressive (AR) model. Intuitively., in the latent space becomes more high-level, and the policy relies more on environmental feedback (observations) make decisions that align with the dataset's distribution. On the other hand, when R is smaller, the policy tends to down-weight the environment's 's observations.
Not all latent spaces effectively support following reference video. As shown in Figure the gap between the environment state in the reference video and during policy deployment requires the posterior encoder to extract intentions independent of environmental dynamics. For instance, in the Minecraft task "mining a diamond underground," a reference video may show player walking forward and mining diamond. If the latent encodes only the trajectory sketch, the policy might fail by colliding with obstacles in the deployment environment. This mismatch occurs because humans interpret the video as "mining the diamond" rather than copying specific actions. Aligning the latent space with human intentions is critical for improving policy steerability.
3 ALIGNING POLICY LEARNERS WITH WEAK SUPERVISION
We explore the development of instructable agents based on latent variable models. To avoid "latent space ambiguity", we introduce human intention knowledge into the generative pretraining process of the policy model to assist in shaping the latent space. As multimodal labels associated with demonstrations carry rich human intention details, we propose a weakly supervised policy learning algorithm to leverage large amounts of unlabeled demonstration data to learn the latent space while using a small amount multimodal labeled data to align the latent space with human intention. Ultimately, this enables instructions from all modalities to be unified within the same latent space. Next, we will elaborate on the dataset collection, training pipeline, and inference procedure.
Dataset Collection. We collect two types of training data from the web: large set of unlabeled demonstrations D Daem = {(01:N, a1:)) }} and a relatively small set of annotated demonstrations Diub = ((0 aI:N, WI-M)), where is the image observation provided by the environment, is the action taken by the policy, w is the word token, N is the length of a demonstration, AM is the length of annotation sentence. The annotation sentence can be multimodal, such as a language sentence (with M 2 1) or scaler of the episode return (with M = 1), which explains the behavior or outcome of the demonstration from human's 's perspective. Since the annotation data is expensive (Daeml-
to collect, we have iDianl <>
Training Pipeline. Our goal is to learn a shared latent space 2, per-modal instruction encoders e(2(c). and latent-conditionel policy π Ta, ((a. 2). Leveraging past observations is essential for policy to make decisions a partially observable environment such as Minecraft (Johnson al. 2016). We call the learned policy model GROOT-2, whose training pipeline is shown in Figure For an unlabeled demonstration (O1:N, a1.N)) ), we use the encoder module to produce a prior distribution e(2 [1) and a posterior distribution e(2) ((2)01:.). Using the reparameterization trick (Kingma & Welling 2013), sample the latent v from the posterior distribution e(2)o).n) and train the policy model, conditioned on and 01 to reconstruct the entire action sequence causally. To limit the information presented in
4

---

Published as a conference paper at ICLR 2025
Constrained Self Imiitating
Human Infention Aligament
P
Lncoder
Encoder
RusinOriuall v0te3))
Policy
Tevetion hpppltta)))
Policy
Encoder
Mi-u
Iroge Takes
Toxt Returns Token
Adtiaa Tikes
Part Oloeraaiies
Consitian Deartbation
Strapled Conditoon Latere
Step Quakleet Opesstion
Figure 4: Pipeline for Constructing a Training Batch for GROOT-2. Each batch includes two sample types: (1) demonstration-onig samples for learning latent-condittined policy (Constrained Self-Imitating:: and (2) labeled samples (text expected returns) for aligning the latent space with human intentions (Human Intention Alignment). The sample ratio varies by dataset distribution.
the latent space, we introduce an auxiliary KL divergence term in the objective:
Lidem a) = (ezee()oo
log π (a. 2) + B1 DEL (el(e( e(z
This allows the model to leverage demonstration-only data to enhance the complexity of the la- tent space, process refer (OI:,, &i:,, WI.M), we pass the label WI.M through the encoder module to obtain latent distribu- tion and train the policy model reconstruct the action sequence based the latent sampled from this distribution ((=[W:M)). This allows human knowledge to be modeled in the latent space. Further, make the encoder understand demonstration O1:N just like humans. we introduce auxiliary MLE term: maximize the log-likeliban likkliiheood of ((2(O1.N] given the latent sampled from e(2 (2/[).M)).. Unlike the prior behavior cloning term, the aligning term can be quickly calculated closed form. This process is referred to "human intention alignment":
"constrained self-imitating." For labeled demonstration
(o, a, w) = Ezwe(s Wash))
log π (a.lo1:1, 2) -B pgllesagge [log e(=/o1))
where sg].] denotes stop gradient operation. The MLFE-based alignment objective ensures that the latent sampled from the lahel-conditioned distribution e(=]w):as) can also be sampled from its video-conditioned distribution e(2(o1.)). The final loss function combines the two objectives:
((Ddem; Diah) = (a) [Leem (o, a)] HE (o...a.) [Ciah (o, a, w)]
Specific implementation details, such as the model design choice, can be found in the Appendix
Inference Procedure. GROOT-2 supports two types of instructions during inference: (1) visual- based instruction the user can either retrieve demonstration from the dataset reference video or manually record reference video to serve as the condition for the policy; (2) label-based instruction the user can input a text sentence or specify an expected return as the condition (depending on label modality used during the model's training). We tested them in the following experiments.
4 CAPABILITIES AND ANALYSIS
We aim to address the following questions: (1) How does GROOT-2 perform in open-world video games and robotic manipulation? (2) Can GROOT-2 follow instructions beyond language and video? (3) What insights scale with labeled and unlabeled trajectories? (5) What the impact of backbone initialization on performance? (6) How do language and video losses influence performance?
be gained from visualizing the leamned latent space? (4) How does GROOT-2
Environment and Benchmarks. We conduct experiments across four types of representative environments: classical 2D game-playing benchmarks on Atari (Bellemare al. (203)). 3D open- world gameplaying benchmarks on Minecraft (Johnson et al. 2016]]Lin et al. 2003), and Robotics benchmarks Language Table simulator (Lynch et al. 2023 and Simpler Eny simulator[Li et al.
5

---

Published as a conference paper at ICLR 2025
(a) Atari
Minecraft
(c) Language Table
(d) Simpler Env
Figure 5: Diverse visual environments used in the experiments. We test our GROOT-2 on both video games (simple Atari games and the complex Minecraft game) and robotic manipulation environments (Language Table and Simpler Env). Minecraft is a partially observable open-ended environment, while others are fully observable.
Table I:
Results on the Open- Minecraft Benchmark. This benchmark includes 8 task families and 100 tasks. Each task is evaluated 30 times across three seeds, and the average success rate is calculated per task family, For examplc, Combat (16) indicates 16 tasks in the Combat family.
Combat (16)
Hunt (10)
Ride
Breed
Craft (20)
Mine (20)
Interact (10)
Plant (12)
Methods
Prompt
(4)
(8)
VPT
N/A
1143
20
7+2
2+o
4+1
7+2
21
22
STEVE-1
lang
1243
9+2
5448
412
5a2
643
5319
33+8
STEVE-1
visual
1544
1043
3849
642
622
10a4
4048
4347
GROOT-1
visual
1845
28
2646
1223
1524
2227
5718
7516
GROOT-2
lang
40
43g6
4646
22
1823
37
5544
7549
GROOT-2
visual
37
4847
5144
20
27_3
3627
6346
7717
2024), illustrated in Figure 5 These four simulators are used to evaluate whether GROOT-2 can
be effectively steered by returns (Chen et al. [202 Mnih et al., 2015). reference videos CCai et al. 2023b Jang et al.) 2022). and textual instructions (Brohan et al. 2022 2023).
Results on the Open-World World Minecraft Benchmark. To evaluate policy models in Minecraft, we used the contractor dataset from Baker et al.] (2022), containing 160M frames. According to the
meta information, labeled trajectories account for approximately 35% of the total data. We extended
the Minecraft SkillForge Benchmark (Cai et al., (023b) from 30 to 100 tasks, grouped into eight families: Combat, Hunt, Ride, Breed, Craft, Mine, Interact, and Plant. Details are in the Appendix C We compared GROOT-2 with three baselines: (1) VPT (Baker al. 2022). a foundational model trained on YouTube data via imitation learning, lacking instruction-followinn (2) STEVE-1 (Lifshitz al. 2023). which supports text and future conditoond instructions; and (3) GROOT-1 (Cai et al. 2023b), self-supervised model using reference videos as instructions. Key findings from Table are as follows: (1) GROOT-2 (visual) consistently outperforms GROOT-1 across all task categories, with particularly notable gains in mob interaction tasks like Combat and Hunt. Comparing trajectories on Hunt, GROOT-1 mechanically repeats "attack" actions, while GROOT-2 actively tracks objects, showing that text data enhances object-centric understanding and better aligns with human intentions. (2) GROOT-2 (text) performs similarly to GROOT-2 (visual) across most tasks, demonstrating that language and visual modalities share task knowledge. This enables the model to leverage both modalities for improved task completion. This highlights the advantage of combining multimodal data for better alignment with human intentions and improved policy performance.
Results on the Language Table benchmark. To assess GROOT-2's multimodal instruction following capabilities in the context of Robotic Table Manipulation, we utilized the Google Language Table as our testing platform and compared it with methods such as LAVA (Lynch et al. 2023). RT-1 (Brohan et al.) (022). GROOT-1 (Cai al. 2023b). We utilize a dataset provided by Lynch al. (2023) comprising TOOM trajectories. We removed the text labels from half of the trajectories in the dataset, creating 1 1 ratio of labeled to unlabeled trajectories. Given that the Language Table environment comes with five task families, all of which are instructed solely through language, we curated five reference videos for each task with relatively clear intentions to evaluate the model's ability to comprehend video instructions. Detailed specifics are provided in the appendix (D] The experimental results are shown in the Table We observed that: (1) GROOT-2 leads by an absolute success rate of 4% following text-based instructions compared, likely due GROOT-2's s more

---

Published as a conference paper at ICLR 2025
Table 2: Results on the Language Table Benchmark. We reported success rates (in %) within 200 steps for each instruction modality, averaging over 250 rollouts. Results are averaged over seeds with mean and stderr. "_" indicates missing data. The percentages in parentheses indicate the proportion of labels used.
BC-Zero
LAVA
RT-1
GROOT-1 GROOT-2 (50%)
GROOT-2 (100%)
Task Family
lang
lang
lang
visual
lang
visual
lang
visual
block to block block to absolute loc
90
842
8429
78
864s
82
72++
10+3
70+8
68ss
76#
70
block to block relative loc
72+3
441
7449
64+7
76
62+6
block relative loc separate two blocks
64
842
8245
78+6
84
80+4
94+2
12+2
9841
96+2
98+0
98+0
Overall
72+3
7844
74
8422
8228
7627
8416
7828
Table 3: Results on the Simpler Env Benchmark. We report the success rate (in %) of the cadlooinstructton language-instrune instruction following each model task families. percentages in parentheses indicate the proportion of labels used.
Pick Coke Can
Move Near
Open/Close Drawer
Methods
Prompt
H-Pose
V-Pose
S-Pose
Avg
Avg
Open
Close
Avg
RT-1-X Octo-base
lang
57
20
70
52
29
lang
5
0
1
3
0
1
GROOT-2 (50%)
visual
42
18
52
37
35
29
30
30
lang
52
20
50
41
42
27
33
30
visual
40
22
47
36
35
27
33
30
GROOT-2 (100%)
lang
53
52
27
31
refined model architecture design. We mark the results of RT-2 in gray here, as it uses significantly more training data than ours. (2) The performance of GROOT-2 in following video instructions dropped by approximately 6% compared to text instructions, possibly due to the ambiguity of the reference videos, where "block to block" type video could be interpreted as a "block relative location" type task. (3) GROOT-1 struggled to understand the intentions conveyed by the reference videos. We observed that GROOT-1 imitated reference video's trajectory sketch rather than their colors and shapes. This further underscores the importance of introducing language annotations for some trajectory data as a crucial method to align with human intentions.
Results the Simpler Env Benchmark. We utilized the Simpler Env (Li et al. 2024) simulation of the Google Robot environment to evaluate the policy's 's capability in controlling complex robotic arms. GROOT-2 is trained the OpenX dataset (Collaboration al., 2023). We erased the text labels from half of the dataset's trajectories, achieving a 1:1 balance between labeled and unlabeled data. We evaluated three types of tasks: Pick Coke Can, Move Near, and Open/Close Drawer. Following Li et al. (2024); Brohan et al. 2023)) ), we set up multiple variants for each task. For instance, the Pick Coke Can task involved three different poses for the Coke can; in the Move Near task, the layout and types of objects varied; and in the Open/Close Drawer task, the drawer had three layers from top to bottom. We compared GROOT-2 with bascline methods such as RT-1 (Brohan al., (022), and Octo (Octo Model Team et al., 2024). Among these, RT-1-X is an efficient
transformer-bases policy trained on the entire OpenX (Collaboration el. (023) dataset, which can be considered the performance boundary that GROOT-2 can achieve. As shown in Table 3 found that GROOT-2 (lang) and GROOT-2 (visual) achieved comparable performance to the RT-I-X model across all three tasks. This indicates that method retains language control capabilities and imbues the policy with equivalent visual instruction control abilities.
Can GROOT-2 Follow Instructions Beyond Language and Video, Like Episode Returns?
We evaluated GROOT-2 's steerability and performance on four Atari games (Breakout, Demon Attack, Hero, and Name This Game). Datasets from Agarwal et al. 2020). containing approximately 10M frames per game, were used. Episode returns were normalized to 0,
For training, constructed dataset with 30% labeled trajectories (returns) and 70% unlabeled data. Using this dataset, we trained GROOT (wsl) (weakly supervised learning). For comparison, GROOT (ssl) was trained the same dataset without return labels in fully self-supervised manner.

---

Published as a conference paper at ICLR 2025
Iemekout
Gemotretlack
bam
Tarne some
20000
7307
12380
100
Vilm De10
2390
5060
vase (m-))
8300
Vileo 2010 20EE
200
Vermalard Eepected
Meraatzed Dipected
Expected
Noresited Dipetedd
Figure Comparison Weakly Supervised (WSL) and Supe-Supe (SSL) Learning on Atari Games. Policies are evaluated under return and reference video conditions. For retum conditioning, normalized returns are input into the encoder, while for video conditioning, videos with similar returns (error < 0.05) are used.
Normalized Returns
Figure 7: t-SNE Visualization Learned Latent Spaces on Four Atari Games. The first row shows results under self-supervised learning, while the second row displays GROOT-2 's performance under weakly supervised learning. Points represent reference videos, with shapes indicating games and colors denoting episode returns. The first four columns compare individual games, and the last column shows mixed-game comparison.
Both models were jointly trained across the four games. During inference, we evaluated policy perfor- mance in following reference videos sampled from the test set with normalized returns {-1, 0, +1}; using 20 samples per category. Results (Figure 6) show: (1) GROOT (ssl) can recognize behavioral quality in reference videos, constructing rough intention space even without labeled guidance. (2) Labeled data significantly improved GROOT (wsl)'s ' ability to understand video instructions, with the greatest gains in Hero and Name This Game. We also evaluated GROOT (wsl) retur-sttle instruc- tions with normalized rewards {-1,0 0, +1 The similarity between video and reward-conditioned performance suggests the video encoder and reward encoder share the same intention space. The Atari experiments aim to evaluate GROOT-2 's performance on modalities beyond language and video, rather than maximizing scores, distinguishing from traditional offline RL methods.
What Does the Visualization of the Learned Latent Space Reveal?
We applied t-SNE to visualize embeddings from randomly sampled reference videos. Each point in Figure (egure/rrepren represenns a unique video, with shapes denoting game environments and colors indicating episode returns. The first row illustrates results for GROOT (ssl), where videos in Breakout, Demon Attack, and Name This Game are classified into two categories based on episode return magnitudes, suggesting that the self-supervised algorithm distinguishes only significant score differences. In contrast, GROOT (ssl) shows poor clustering and limited steerability in the Hero game. The second row shows results for GROOT (wsl), which captures continuous variations in video behavior quality across all games. As shown in the fifth column, embeddings from different environments follow a continuous pattern aligned with reward labels, indicating a shared latent space that promotes cross-cnvironment knowledge transfer.
How Does Scaling Up Unlabeled Trajectories Impact Performance?
We trained four GROOT-2 variants with 0%, 25%, 50%, and 100% unlabeled data in Minecraft. Performance was tested on five Minecraft tasks (Chop Tree, Hunt Animals, Combat Enemies, Open Chest, Climb Mountain) and scored relative skilled human players. For example, if human collects 20.0 logs in 600 steps and GROOT-2 collects 15.0, the score is 0.75. Results (Figure g) show consistent improvement with unlabeled data, with the 100% variant achieving 5x gain in the
8

---

Published as a conference paper at ICLR 2025
25%
50%
100%
Hunt Animals
Combat Eneriles
Open
Climb Mountoii
Itman Noroslized Soom
0.8
0.75
0.77
3.87
0.53
0.57
0.60
0.60
0.50
0,50
0.40
0.40
0.35
0.40
0.30
0.10
0.15
Figure 8: Performance Comparison on Unlabeled Demonstrations. Human- normalned task scores are averaged over 20 rollouts across 5 Minecraft tasks to evaluate the agent's reference-video following ability.
Climb Mountain task over the 0% version. It is worth noting that the Climb Mountain and Open Chest tasks do not have language instructions in the training set.
How Does Scaling Up Labeled Trajectories Impact Performance?
To evaluate the impact of labeled trajectory proportions in capabilities of GROOT-2, we conducted experiments on the Language Table benchmark. The total number of trajectories remained constant across different dataset configurations, with only the propor- tion of trajectories containing text labels varying. Figure 9 reports the success rate achieved by GROOT-2 conditioned on language. At low labeled data proportions (0% - -25%). the
the training set on the instruction-followin
80
65%
73%
82%
83%
60
40
34%
20
096
Average Success
10
Labeled Data Percentage (%)
Figure 9: Ablation Study on Labeled Trajectories in the Language Table.
success rate rapidly increased from 10% to 65%, indicating that labeled data significantly influences model performance. However, as the labeled data proportion increased to 50% 80%, the success rate plateaued, rising slightly from 82% to 83%, demonstrating diminishing marginal gains from additional labeled data. Therefore, under resource constraints, a labeled data proportion of 50% may represent the optimal balance between performance and cost.
How Does Backbone Initialization Affect Performance?
We evaluated different initializations for VIT (random, ImageNet, CLIP) and BERT (random, BERT, CLIP) on the Language Table Benchmark. For randomly initialized models, both backbones were unfrozen during training. According to Table CLIP initialization yielded the best results for ViT, followed by ImageNet, with minimal difference between them, while random initialization performed worst. For BERT. CLIP and standard BERT initialization performed similarly, both surpassing random initialization. Initializing vision and language encoders with CLIP parameters improves policy performance and reduces training time.
Backbone VIT
VIT
VIT/BERT BERT BERT
Variants
Cis baseline - Lass baseline
Weights SR (in %) 76410
ImageNet
CLIP 82
BERT
Prompt SR (in %)
vision 1042
vision
lang 1243
lang
8041
79412
8148
7627
82+s
Table 4: Ablation study the backbone initialization.
Table 5: Ablation on Lus and Zaem objectives.
How Does Language and Video Losses Impact Performance?
The Liab loss significantly enhances the model's understanding of reference videos, as observed in the Language Table environment. We compared a variant without Lis loss to the full GROOT-2 model, both trained on the same scale of the Language Table dataset, and tested their ability to follow reference videos using standard evaluation scripts. As shown Table the variant without Libb loss failed to complete any tasks. Further analysis of its output videos revealed that it mechanically mimicked the arm movement trajectories in the reference videos, completely ignoring object colors and shapes, which is inconsistent with human understanding of the reference vidcos.
The Zdem loss indispensable the GROOT-2 architecture. Removing Zaem causes the pipeline to degrade into autoencoder when processing unlabeled data. Without constraints on the latent encoding, the model tends to learn the video encoder as an inverse dynamics model, encoding low- level action sequences in latent instead of high-level task information, thereby significantly reducing the behavior cloning loss. Additionally, Table 5 show that removing Zaem causes the language encoder's 's latent to collapse, leading dramatic drop in task success rates.

---

Published as a conference paper at ICLR 2025
5 RELATED WORKS
Learning Policies Across Diverse Domains. Developing policies for sequential control tasks in real and virtual environments poses significant challenges. Research spans domains such as robotic manipulation (Yu et al. 2019, Lynch et al. 2023), video games (Bellemare et al., 2013 Guss et al. 2019), and embodied navigation (Hong 2020 Savva et al. 2019 Huang et al. 2023), with approaches categorized into reinforcement learning (RL) and imitation learning (IL) based on reward
function reliance. For video games with dense rewards g, ALE platform (Bellemare et al. 2013
online RL algorithms can achieve superhuman performance (Mnih et al., 2015 [adia et al. 2020)
but suffer from low efficiency, risky interactions, and limited generalization. These challenges restrict
their applicability physical (Padalkar al. (023) or embodied environments (Guss et al. 2019).
where rewards and cheap interactions are unavailable. IL, as a supervised learning paradigm, addresses these issues through batch efficiency and scalability with large datasets, leveraging Transformer architectures (Zhang & Chail [021 Pashevich et al.) [ang] et al.. 2022). The RT-X series (Brohan al. 2022 2023] Padalkar et al. 2023) advances robotic manipulation by training Transformers on large expert demonstration datasets, achieving strong zero-shot generalization. Similarly, [Baker al. 2022) developed Transformer-based policy for Minecraft using internet-scale gameplay data, solving the diamond challenge. Building on this. (Schmidhuber (2019) frames RL as supervised learning, while Chen et al.) (20TT)] Lee al. (2022) introduce "decision transformers" to model joint distributions of rewards, states, and actions from offline data, highlighting the potential for unified policy learning within Transformers.
Learning Policies to Follow Instructions. Enabling policies to follow instructions is key to building general-purposs agents. A common approach involves using language annotations from offline
demonstrations to train language-conditioned policies (Abramson et al. [2020] Brohan et al. 2022
Reed et al. 2022; Cai et al. 2023al Huang et al. 2023; Raad et al. 2024; Wang et al. 2023a b). leveraging the compositionality of natural language for generalization. However, obtaining high- quality annotations is costly. An alternative uses anticipated outcomes as instructions. Majumdar let al. (2022) trained an image-goal conditioned navigation policy via hindsight relabeling (HER) (Andrychowicz et al. (017)) and aligned goal spaces with text. Similarly, [Lifshitz et al.) (2023) used this strategy for open-ended tasks in Minecraft. Generative latent variable models offer another solution, using label-free demonstrations to train plan-conditioned policies (Lynch et al. 20206 Ajay al. 2020p. Extending this, Cai et al. (023b) applied a posterior encoder to interpret reference videos in Minecraft. Policy learning with weak supervision remains less explored. Lynch & Sermanet (2020) proposed a shared latent space conditioned on language and -en--ggeeeette goal images, while Jang et al. (2022) replaced goal images with video labels under full supervision. Jain et al. (2024) trained robots using human videos as task representations but required extensive paired video- trajectory data. Myers et al. (023)) combined labeled and unlabeled trajectories, aligning start-goal pairs with language via contrastive learning, effective for Table Manipulation but limited in handling complex tasks or generalizing to partially observable environments like Minecraft.
6 CONCLUSIONS, LIMITATIONS AND FUTURE WORKS
This paper investigates the joint learning of a latent intention space and multimodal instruction- following policy under weak supervision. We identify the "latent space ambiguity" issue in latent variable generative models when handling text-free trajectory data, arising from the absence of direct human guidance in shaping the latent space. To address this, we propose a weakly supervised algorithm for training GROOT-2. Evaluations across four diverse environments, from video games to robotic manipulation, demonstrate GROOT-2 's generality and flexibility in following multimodal instructions. However, GROOT-2 's reliance on trajectory data for training limits its applicability to video data, which lacks action labels. Considering the abundance and diversity of video data available online compared to trajectory data, extending the weak supervision framework to leverage both play and trajectory data would be promising avenue for future work.
7 ACKNOWLEDGEMENTS
This work is funded in part by the National Key R&D Program of China #20227D01603001.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
ANM Nafiz Abeer, Nathan M Urban, M Ryan Weil, Francis J Alexander, and Byung-Jun Yoon. Multi-bbjective latent space optimization of generative molecular design models. Patterns, 2024.
Josh Abramson, Arun Ahuja, lain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, al. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672, 2020.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pp. 104-114. PMLR, 2020.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Otir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning, arXiv preprint arXiv:2010.13611, 2020.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock. Aida Nematzadch. Sahand Sharifzadeh, Miko- laj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204, 1,1222204,,1198,, 2022. URL https://api semant wrrranticccchl.ie
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken elbo. In International conference on machine learning, pp. 159-168. PMLR, 2018.
Marcin Andrychowicz. Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Joshua Tobin, P. Abbeel, and Wojciech Zaremba. Hindsight experience replay. ArXiv, abs/1707.01195 2017. URL URL htt.:httt://// /taass:/appi semant remanticscho cscho wwpoolaa..rroooo www.oorg/ooopuso 3532908
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International conference on machine learning, pp. 507-517. PMLR, 2020.
Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Bran- don Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. ArXiv, abs/2206. 11795. 2022. URL https://api
semanticschar http://ooooSS.:223333888
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang. Isabel Leal, Kuang- Huei Lee. Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Anand Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Ho Vuong, F. Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control scale. ArXiv, abs/2212.06811 2022. URL htt.httts///apl /ttppppppp//aaoo.mmtiiccccco www.eeetnnicciicem 254591260
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision- aaaioo-languag-acttion models transfer web knowledge robotic control. arXiv preprint arXiv:2307. 138.//3077.18//8 2023.
11

---

Published as a conference paper at ICLR 2025
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020. URL https: www.nticscceeeeeeeeee.e. ar www. Wwwg/Corruusiiiiee ID: 288777883
Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13734-13744, 2023a. URL
https://api. seeeenttiscchoe
wwtp/oorggsiaaaa.... 2261144112
Shaofei Cai, Bowei Zhang, Zihao Wang. Xiaojian Ma, Anji Liu, and Yitao Liang. Groot: Learning to follow instructions by watching gameplay videos. In The Twelfth International Conference on Learning Representations, 2023b.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Adiyya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via se- quence modeling. In Neural Information Processing Systems, 2021. URL https://aat //api
semanticscholi http::/ccccaaa..oo/ooooooooooeee
Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh. Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang. Chenfeng Xu, Cheng Chi. Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh. Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang. Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg. Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang. Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang. Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng. Kyle Hatch, Kylc Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du Michael Ahn, Michael C. Yip. Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo. Nur Muhammad Mahi Shafiullah. Oier Mees. Oliver Kroemer, Osbert Bastani. Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart in-Mart' in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shuran Song, Sichun Xu. Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted
12

---

Published as a conference paper at ICLR 2025
Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davehev, Tony Z. Zhao, Travis Anmstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang. Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao. Yuch-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X- Embodmmmmnt: Robotic learning datasets and RT-X models. ILTD:://aaxx..crraa///333...800 2023.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, Jan 2019. doi: 10.18653/ v1/p19- 1285. URL http: //xx wwt.:ddi...ogg.... ...00...8.6 18653/6633/ 18653/v1//11111 92
Jacob Devlin, Ming-Woe-ee Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04800 2019. URL
https: wer.dieeeannnccccccc ortp.//aa.....g//ooopssiaaaa:aa. httpa//wwaa..aaaaaaa..aoo.ooorrr
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. ArXiv, abs/2010.1 2020.
URL. https: /tas://appa /apip semant cscho ar. wwr.or///oe rpus D
225039882
Yan Duan. Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho. Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.
Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. Implicit deep latent variable models for text generation. arXiv preprint arXiv: 1527, 2019.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long- horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv: /910: 1956, 2019.
William Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela M. Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstra- tions. In International Joint Conference on Artificial Intelligence, 2019. URL https
7/api. www.emmtttttn mommanticscholar. wwwp/orpps/oooooooooooo 11: 199000710
Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. Vin-bert: A recur- rent lan-auugg bert for navigation. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1643-1653, 2020.
oxc/CorpusID::222288.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311. 2023.
Vidhi Jain, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, et al. Vid2robot: End-to-end video- conditioned policy learning with cross-attention transformers. arXiv preprint 2033 2024.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. ArXiv, 20..2202000055 2022. URL :/appp::://ppa //api. semanticscholar org/CorpusID; 237257594|
13

---

Published as a conference paper at ICLR 2025
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentatiion International Joint Conference on Artificial Intelligence, 2016.
Diederik P. Kingma and Max Welling, Auto-encooinng varero- variational bayes. CoRR, abs/1312.6114, 2013.
Kuang-Huee Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadar- rama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35:27921-27936, 2022.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies simulation. arXiv preprint arXiv:2405.05944 2024.
Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila A. Mcllraith. Steve-1: A generative model for bexh-io-b in minecraft. ArXiv, abs/2306.00%4 2023. URL https
7/api saai.ssematiicccooo... :
Haowei Lin, Zihao Wang, Jianzhu Ma, and Yitao Liang. Meu: A task-centric framework for open-ended agent evaluation minecraft. arXiv preprint drXiv:2310.88367, 2023.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172, 2023.
2023.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint 10X1::2005.076448 2020.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp. 1113-1132. PMLR, 2020a.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp. 1113-1132. PMLR, 2020b.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference robot learning, pp. 1113-1132. PMLR, 2020c.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking robots in real time. IEEE Roboties and Automation Letters, 2023.
Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. ArXiv, abs/2206. 12403. 2022. URL httpl:tss:///mm /ttp://taa/ppp..emattttcaoaaaoe htt.://a.....or//ooopppiaaa::---
Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters. 7(4): 1205- 11212, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforce- ment learning. Nature, 518:529-533 2015. httpt:tt//// api. sewant /rg/ CorpusTD:205242744
14

---

Published as a conference paper at ICLR 2025
Vivek Myers, Andre Wang He, Kuan Fang. Homer Rich Walke, Philippe Hansen- Ching-Ar An Cheng, Mihai Jalobeanu, Andrey Kolobov, Anca Dragan, and Sergey Levine. Goal representations for instruction following: A semi-supervised language interface to control. In Conference Robot Learning, pp. 3894-3908. PMLR, 2023.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023.
Artidoro Pagnoni, Kevin Liu, and Shangyan Li. Conditional variational autoencoder for neural machine translation. XrXiv preprint arXiv: 04405, 2018
Alexander Pashevich, Cordelia Schmid. and Chen Sun. Episodic transformer for vision-and- -nd-gaga navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15942-15952, 2021.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188 288. PMLR, 2021.
Maria Abi Raad, Arun Ahuja, Catarina Barros, Frederic Besse, Andrew Bolt, Adrian Bolton, Bethanie Brownfield, Gavin Buttimore, Max Cant, Sarah Chakera, et al. Scaling instructable agents across many simulated worlds. arXiv preprint arXiv:2404.10177 2024.
Alec Radford. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. URL https: //app..p.pp semant cscho ar. waw.orrg/orppuu
2315 03159144 445)
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXivv2205.066 02205.061755 15, 2022.
Manolis
Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. Proceedings of the IEEE/CVF international conference on computer vision, pp. 2019.
Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards just map them to actions. ArXiv, abs/1912.028755, 2019. URL htttt::///// //api api smm.mccccocooo.ooooooggg
Corpus :I:
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk. Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Shawn Ma, and Yitao Liang. Describe, cxplain, plan and select: interactive planning with 11ms enables open-world multi-task agents. Advances in Neural Information Processing Systems, 36, 2023a.
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng. Yaodong Yang. et al. Jarvis-l: Open-world multi-task agents with memory- augmented multimodal language models. arXiv preprint arXiv:2311.088 1.05997, 2023b.
15

---

Published as a conference paper at ICLR 2025
Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. Xskill: Cross embodiment skill discovery. In Conference on Robot Learning, pp. 3536-35555 3555. PMLR, 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan C. Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. ArXiv, abs/1910.10897, 2019. URL https: //api. semant i cccco ar org/ Corppsrrpussp 204852201
Qihang Zhang, Zhenghao Peng, and Bolei Zhou. Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining. In European Conference Computer Vision, 2022. URL : //a//aaa// /api. semant htwpaaatttscccooaaooooooo/ooraap
226
Yichi Zhang and Joyce Chai. Hierarchical task learning from language instructions with unified transformers and self- melftooni...,ng. arXiv preprint a:X1::2106.0327, 2021.
16

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A. MODEL ARCHITECTURE
This section outlines the architectural design choices employed in our approach. GROOT-2 utilizes a Transformer encoder-decoder architecture, augmented with a probabilistic latent space. We detail the components of the model in structured sequence: extract representations, encode instructions, and decode actions.
Extract Representations. This paragraph elaborates the backbone networks used to extract repre- sentations from various data modalities. We denote the modalities of image observation, language instruction, and expected returns as 01:N, WI-M, and respectively. For vision inputs, utilize a pre-trained Vision Transformer (ViT) (Dosovitskiy et al. 2020) initialized with CLIP (Radford al. 2021) weights. Specifically, the step image observation o resized 224 224 and
processed to extract 7 x 7 patch embeddings x = -1",11))" , The video representation
x" is then composed of the averages of these embeddings across the video frames, denoted as r" (avg(z?), avg(2z) (z)))) where avg(-) refers to spatial average pooling minimize com- putational overhead and N represents the video length. Textual inputs are processed using the BERT encoder (Devlin et al. 2019) of the CLIP model. Rather than utilizing the (CLS) token the
final representation, retain all word embeddings generated by BERT x" = -[ The
BERT model parameters are kept frozen during training. For the scalar-fore modality of expected returns, we employ simple Multi-Layer Perceptron (MLP) to process these values, represented as x² <- MrP(r). These embeddings are then forwarded to subsequent modules.
Encode Multimodal Instructions with Non-Causal Transformer. Recent works (Reed et al. 2022
Lu et al.) 2023 Team et al.] 2023) have demonstrated the Transformer's s effectiveness in capturing both intra-modal and inter-modal relationships, which inspires to adopt unified Transformer encoder for encoding multimodal instructions. This approach offers two significant advantages: (1) It eliminates the need for designing separate architectures and tuning hyperparameters for each modality. (2) It promotes the sharing of underlying representations across different modalities. Instructions are represented as a sequence of embeddings. Before encoding, each embedding is augmented with modality-specifee marker. For instance, video instructions are represented as ,." + [VID]), where [VID] is learnable embedding.
(x) + [VID),
... ... ,x²
Decode Action with Causal Transformer. Given a latent and a temporal sequence of perceptual observations U1 the policy aims to predict the next action de. Following prior works (Baker et al.)
2022 Cai et al., 2023b Raad et al. 2024), we employ the Transformer-XL. model (Dai et al. 2019) in our policy network, which enables causal attention to past memory states and facilitates smooth predictions. Additionally, we utilize the shared vision backbone to extract vision representations, thereby representing perceptual inputs as xf.t- A significant challenge with this approach low efficiency: each new observation x& adds up to 49 tokens to the input sequence, substantially increasing memory and computational demands. To address this issue, we introduce pre-fusion mechanism inspired by Aybrrmsooon et al. (2020); Lynch et (2023); Alayrac al. (2022). Specifically, we deploy a lightweight cross ross-attention module XATTN( = kv to perform spatial pooling on as the keys and values:
r.. using z as the query and
, F >
1 XATTN(Q kv " (611119
(5)
This pre-fusion mechanism not only reduces the sequence length but also enhances the integration of perceptual and latent representations. Utilizing the latent-fused representations vi. as the input sequence, we articulate the action decoding process in an autoregressive manner:
dt Trr
(6)
A.2 HYPER-PARAMERR
Hypee--arameters for training GROOT-2 are shown in Table
17

---

Published as a conference paper at ICLR 2025
Table 6: Hyperparameters for training GROOT-2.
Hyperparameter
Value
Optimizer
Adam W
Weight Decay
0.001
Learning Rate
0.0000181
Warmup Steps
2000
Number of Workers
4 ddp NVIDIA A800
Parallel Strategy
Type of GPUs
Parallel GPUs
Accumulate Gradient Batches
Batch Size/GPU (Total)
16 (128)
Training Precision
b
Input Image Size
224 x 224
Visual Backbone
ViT/32
Encoder Transformer
minGPT (w/o causal mask) Tra
Decoder Transformer
Number of Encoder Blocks
8
Number of Decoder Blocks
Hidden Dimension
1024
Trajectory Chunk size
128
Attention Memory Size
256
31
0.1
82
0.1
B ATARI
Environment Description. Atari 2600 games contain a lot of diverse video games, which is widespread benchmark to evaluate the decision-making capability of an agent. The Atari games do not inherently support multitasking concepts; agents are typically tasked with optimizing for the highest possible rewards. However, an advanced human player can deliberately control their gameplay level and achieve any potential score. The ability to "control scores" is generally considered a higher intelligence level compared with merely "winning the game". Therefore, this paper does not emphasize the highest absolute score agent can achieve in the Atari environment. Instead, it focuses on evaluating the agent's 's ability to follow instructions the form of videos and "desired cumulative rewards" and to perform the appropriate level. Especially when videos serve as conditions, the agent needs to infer the player's level demonstrated in the reference gameplay, which poses significant challenge for the current agents. To our knowledge, this setting has not been explored by previous works.
Observation and Action Spaces. We utilize the popular Arcade Learning Environment (ALE) as our testing platform, where the original observation image provided is 210 x 160, and the action space consists of 18 discrete actions defined by the joystick controller. Following previous works, the observation images are typically resized to 84 x 84 grayscale images. In implementing GROOT-v2, we employ the VIT/32 model initialized with OpenAl's pre-trained CLIP model. The observation image, 84x84, is resized to resolution of 224 x 224 before being fed into the model for unification. During the training process, the ViT backbone is jointly fine-tuned. The TransformerXL architecture used for the decoder has its memory set horizon of 128.
Training Dataset. We utilize the trajectories from the Replay Buffer generated during the training of agents using the DQN algorithm on Atari, provided by Google, We access this data through the interface provided in the d4rl-atari project at GitHut"] Specifically, the trajectory data for each game consists of three parts: mixed, medium, and expert, representing the environment interaction data from 0-1M steps, 9M-10M steps, and the final 1M steps of a training session, respectively. We construct training data of 10M steps for each Atari game, with the proportions of mixed, medium, and expert data being 2:5:3. During training, we use single model to
source of training data.
18

---

Published as a conference paper at ICLR 2025
Semilatt
Noreistt MEA Iedtrg
Herileed
Vorm
Suruitad vila demennaltadt
sona sa frosthie
crety-llebbe
extoro
Meralos
Iartaloddviiee qbert
Moraline a
Hirribed robetarit
Sonaa tne sa seaquest
thvereld
road rostirmmer
Sarvalow
Sunaa
Figure B.1: Distribution of episode returns for each Atari game.
fit 35 selected game environments on Atari. Considering the significant differences in absolute scores across different games, we standardize the reward scores. Specifically, we calculate the cumulative reward scores for each complete trajectory and adjust them to mean of O and standard deviation of 1 using the formula R + (R- p)/a, which represents the game level corresponding to that trajectory. Figure B.1 illustrates the episode return distributions for each Atari game. Subsequently, each trajectory is segmented into 128-step fragments with the same label.
Complete Results. We selected trajectory data from 35 Atari games, totaling 350 million frames, to train GROOT-2, with 30% of the data labeled with returns and the remaining 70% containing only image observations and actions, aligning with the setup for weakly supervised training. After the model converged, we tested GROOT-2's ' ability to follow return-format and video-format instructions across these 35 games. When testing return-formal instructions, we chose three samples within the normalized returns space: {-1, 1}. For video-formal instructions, we randomly sampled segment of 128 frames from the test data with normalized rewards within the range of (-1,0, 0, 1) allowing a deviation of +5.05. Each instruction was tested 40 times, with the results depicted in Figure B.2 We observed the following: (1) In the majority of games, GROOT-2's performance showed a clear positive correlation with the game level corresponding to the instructions. (2) In certain games (such as Pong, Seaquest, Skiing, Wizard Wor), video-format instructions yielded better control over the agent than return-format instructions. Conversely, in games like Amidar, Battle Zone, and Zaxxon, return-formin instructions demonstrated significantly superior control compared to video-format instructions.
19

---

Published as a conference paper at ICLR 2025
anider
355sult
attant25
bonk beost
DOM (20%
1230
6000
lace
Vitre
120000
06000
V9360
Vileo
10000
2000
10000
902
20000
12003
bontle 2000
beum rider
bonting
brcobous
chooper concoand
10303
(903
TaAmnn 30M (D2%
200
0003
60
150
300
2003
20
2011233
Norratand Dipectad Retome.
Komastood Diperted Setars
Norsatood Sxjerted Romies
Nicralaod Expacted Botres
Nacrollud Sapected Rotuma
Ortonn-atack
euduzo
Bshing derby
Evwibit
10300
3039
20.113
1000
prleno
#930
Tameer MON
4030
2033
Nertaaiee Expected Retume
Aerraktnn tipextei 2etenss
hocastzed Expeited Heluts
Naceatonn Exposted Hotues
Neciotore bapesbod Hinams
guebee
langaroo
Danerde 20M 12% C11
Deturee Vitre
63300
600
10000
b0000
400
20
2000
502
video
Expectad Retome
Ripeetri
Sereoited
keuil
kungy mussseer
INII:: drispune
plocsis
200g
Diaatto 120 08304 CD
5000
SANITO 1000 03334 C11
38308
Vilie
68co
10
8037
59900
4000
2000
4300
12308
2000
Nortazzedd Expected Rsture gbert
Nannakesi Expoxted Setusay
Noundieed Expeded Reiuro
Daceadiond Expoded Neturo
Necasdort Experiod Beiano
divernod
rood rennee
seaquest
rnbotaok
Jevert 300 037% C11
15003
12000
50006
1000
tD Sucti DoCK Sare
19923
43000
000
000
3320
6930
23028
40
6000
4030
10320
20/123
200
Nortaa2inn) Expec 1nd Metanm
Autraatnn) Erpexted sotecio
Noxaoleed bxpe Mouuss
PAceadiond Dape Botces
Necsstors Eapedto Mouns
Danerth 23% C11
-24003
4030
Denannk C0
300
1000
12500
300
3030
20920
2003
150
200
-20000
1030
13300
Nerra (ied Expectad Rateme.
Serrataod Prpected Neterns
Noceetzee Pape Butirss
Neveloal Papeited Betres
Nerviine Papestod Batamn
Figure B.2: IQM scores (with 95% confidence interval) of GROOT-2 which is jointly trained on 35 Atari games. GROOT-2 can understand both the returns-formal format instructions and video-formal instructions most of the games. The performance of the GROOT-2 exhibits positive correlation with the game level corresponding the provided instructions.
C MINECRAFT
Environment Deseription. Minecraft is a 3D sandbox game with global monthly active user base of 100 million. It features procedurally generated worlds of unlimited size and includes dozens of biomes such as plains, forests, jungles, and oceans. The game grants players a high degree of freedom to explore the entire world. The mainstream gameplay includes gathering materials, crafting items, constructing structures, farming land, engaging in combat mobs, and treasure hunting, among others. In this game, players need to face situations that highly similar the real world, making
20

---

Published as a conference paper at ICLR 2025
judgments and decisions to deal with various environments and problems. One easily specify task with a natural language description or a demonstration video. Therefore, Minecraft is an ideal environment to test how an agent behaves in an open-world environment.
Observation and Action Spaces. We use the combination of 1.16.5 version MineRL (Guss et al. 2019) and MCP-Reborn as our testing platform, which is consistent with the environment used by VPT (Baker et al. 2022) STEVE-1 (Lifshitz et al. 2023) and GROOT-1 (Cai et al. (023b). Mainly because this platform preserves observation and action space that is consistent with human players to the fullest extent. On the one hand, this design brings about high challenges, as agents can only interact with the environment using low-level mouse and keyboard actions, and can only observe visual information like human players without any in in-game privileged information. The Minecraft simulator first generates an RGB image with dimensions of 640 360 during the rendering process. Before inputting to the agent, we resize the image to 224 x 224 to enable the agent to see item icons in the inventory and important details in the environment. When the agent opens the GUI, the simulator also renders the mouse cursor normally. The RGB image is the only observation that the agent can obtain from the environment during inference. It is worth noting that to help the agent see more clearly in extremely dark environments, we have added a night vision effect for the agent, which increases the brightness of the environment during nighttime. Our action space is almost identical to that of humans, except for actions that involve inputting strings. It consists of two parts: the mouse and the keyboard. The mouse movement is responsible for changing the player's camera perspective and moving the cursor when the GUI is opened. The left and right buttons are responsible for attacking and using items. The keyboard is mainly responsible for controlling the agent's movement. To avoid predicting null action, we used the same joint hierarchical action space as Baker et al. (2022), which consists of button space and camera space. Button space encodes all combinations of keyboard operations and a flag indicating whether the mouse is used, resulting in total of 8461 candidate actions. The camera space discretizes the range of one mouse movement into 121 actions. Therefore, the action head of the agent is a multi-classification network with 8461 dimensions and a multi-classification network with 121 dimensions.
Training Dataset. The contractor data is a Minecraft offline trajectory dataset provided by Baker et al. (2022), which is recorded by professional human players. In this dataset, human players play the game while the system records the image sequence 01:N. action sequence @1:N. and metadata e1:N generated by the players. Excluding frames containing empty actions, the dataset contains 1.6 billion frames with a duration of approximately 2000 hours. The metadata records the 7 kinds of events triggered by the agent in the game each timestep, i.e. craft item, pickup, mine block, drop item, kill entity, use item, and custom. We augment each event with a text description using the OpenAI chatGPT service. To construct trajectory data with textual labels, we enumerate all timesteps within the trajectory where an event occurs. From this point, we count 112 frames backward and 16 frames forward to form a segment of 128 frames. The textual label for this segment is derived from the text associated with the event. It is important to note that many events occur frequently; for example, when the player is mining a tunnel, the event "mine block: cobblestone" triggered on average twice per second. To address this issue, if segment generated by an event overlaps with previously generated segment, it is skipped. Each event collects a maximum of 2000 segments, and across all 1518 events, 414,387 segments are included. It is noteworthy that significant amount of duplication persists within these segments, as a single segment may encompass multiple events.
D LANGUAGE TABLE
Environment Description. Language Table (Lynch et al. 2023) is a comprehensive evaluation suite proposed by the Google for assessing a robot's ability to follow natural language instructions to solve Table Manipulation tasks. It includes a dataset, environment, benchmarks, and a baseline policy. The evaluation benchmark encompasses over 87, 000 diverse behaviors and more than 600, 000 trajectories annotated with text instruction. In addition to data from real environments, the suite also provides a simulator akin to a real environment along with corresponding simulated data.
Observation and Action Spaces. Language Table's 's simulated environment resembles the real-world tabletop manipulation scenario, which consists of an xArm6 robot, constrained to move in 2D plane with cylindrical end-effector, in front of smooth wooden board with a fixed set of 8 plastic blocks.
21

---

Published as a conference paper at ICLR 2025
Reference Video
Rollout Trajectory
Yur Mue mooe neatie pirtoogg
'oun the cole traands the Mar cule"
grees
Figure C.3: GROOT-2 can infer the intention behind the reference video and follow it to complete tasks. The left visualizes three reference videos along with their textual descriptions. The right figure displays the policy's 's rollout trajectories when conditioned on the reference videos. The white dashed line represents arm's movement trajectory, and the red dashed circle highlights the arm's 's final position.
comprising 4 colors and 6 shapes. In both simulation and real collection, they use high-rate rate human teleoperation with a 3rd person view oline-of-sight in real). Actions are 2D delta Cartesian setpoints, from the previous setpoint to the new one. They batch collected training and inference data to 5hz observations and actions.
Training Dataset. We use the training trajectories from the official Language Table repository. An oracle script generates the trajectories and covers all task families, each containing 20M trajectories, in a total of 100M trajectories. The dataset names are: tanguugg-table-blockeee oracle-sim, language-taie table- blockkoobbockrraativeoooroee sim, language- taale--aal--bbaaee wubbl-blooctorellttiiii wtwptrrreelaivv---lle---ieeeeeee oracle-sim. language tawlttaale-eeee separate oracle-sim.
sim, language-taile teree--tabbebbbbeeeee
The evaluation benchmark consists of 5 task families (block2block, block2abs, block2rel, block2blockrel, separate), totaling 696 distinct task variants. We report the success rate of the agent within 200 steps on each task as the final metric. Considering that the Language Table inherently includes instructions in the language modality for its task families, we have curated a set of reference videos for each task family, each with relatively clear intentions, to serve as video instruction set. This is done evaluate the model's ability to comprehend video instructions. The details are in Table 7
Task Definition.
We visualize some examples when conditioning GROOT-2 on reference
videos in Figure C.3
E SIMPLER ENV
Environment Description. Simpler Env a physical simulator proposed by et (2024), efficient, scalable, and informative complements to real-world evaluations. It can be used to evaluate diverse sets of rigid-body tasks (non-articulated articulated objects, tabletop / non-tabletop tasks), with many intra-task variations (e.g., different object combinations; different object/robot positions and orientations), for each of two robot embodiments (Google Robot and WidowX).
Observation and Action Spaces. The observation and action spaces of Simpler Env are the same as the Language Table. The action sequence is expected to be a 6D end-effector pose trajectory with a gripper flag indicating the open/close status. Before feeding the image observation into the policy, resize the image to a 224 224 resolution.
22

---

Published as a conference paper at ICLR 2025
Table 7: Sampled reference videos to build video instruction set.
Task Family
Video Description
block to block
put the red moon to the blue moon put the blue moon towards the yellow slide red pentagon close the green cube slide the green star to the red moon put the green cube next to the red pentagon
block to block
block to block
block to block
block to block
block to absolute location
slide the blue cube to the upper left corner push the blue moon to the top left of the hoard the bottom left slide yellow star the right side the board push the yellow pentagon to the left side
block to absolute location
block to absolute location
the red
block absolute location
block to absolute location
block block relative location
move the green star to left side the yellow pentagon push the green star diagonally up and to the right of the green cube put the red moon to the bottom left side of the yellow star slide the yellow pentagon to the bottom left side of the red pentagon slide the blue cube to top the blue moon
block to block relative location
block to block relative location
block to block relative location
block block relative location
block relative location
push the green cube right
block to relative location
slide yellow pentagon downwards and to the right push the blue cube somewhat to the left move the blue moon to the right slide the red pentagon up
block to relative location
block to relative location
block to relative location
separate
pull the yellow pentagon apart from the blue moon
separate
pull the green star apart from the yellow star
separate
pull the blue cube apart from the blue moon and red pentagon
separate
move the blue cube away from the yellow star
separate
move the green star away from the yellow pentagon
23


Reviews:
[{"review_id": "IAY5kiFOQn", "reviewer": "Reviewer_QHCB", "summary": "The authors propose a simple VAE style architecture to combine a small amount of language/return annotated data with a large demonstration-only dataset to learn a downstream policy. The authors test the method on Atari, Robotics benchmarks (Simpler and Language Table) and MineCraft. The authors compare their work to several per domain baselines and report strong improvements in MineCraft. Robotics results show smaller/statistically insignificant gains.", "questions": "1. How was beta_1 and beta_2  tuned? How was the R ratio controlled empirically?  \n2. Does the model finetune easily? If one were to use the model for a new set of tasks, would this transfer be data efficient", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n1. The paper is well written and easy to understand\n2. The theoretical underpinnings of the work are clearly described using the R ratio diagrams. This motivates the work well.\n3. The experimental suite is large and the method seems to work on a variety of benchmarks. \n4. The question of utilizing labeled and label-free demonstrations needs to be answered in order to move the needle on generalist agents.", "weaknesses": "While I enjoyed reading the paper I have several issues with the current version of the paper. It reads more like a technical report for promotion purposes, not scientific material. I enlist my qualms below:\n1. **How much supervision?:** While the authors have the word “weakly supervised”, it's unclear how much supervision is actually being used. Are all tasks covered in the labeled demonstration dataset? Is there any amount of language generalization that occurs, for e.g., are any tasks solvable zero-shot? How good are the demonstrations? What if there was play data/suboptimal demonstrations in the unlabeled demonstration dataset? Would the method still work? None of these questions are answered and I think they should be added since the focus of the work is weak supervision.\n2. **No ablations/experiments describing model architectures?:** How does initializing the model with something other than BERT work? What about vision inputs? No ablations on ViT architectures initializations are provided. Works like LIV (https://arxiv.org/abs/2306.00958), R3M etc are available as baselines? \n3. **Returns is not language:** The choice of adding of Atari is odd - there are no language labels here. Incidentally, it is only here that details of the amount of labels are provided. What is the distribution of labels? Are they mostly high reward trajectories? If return conditioned imitation is the focus, then works like Multi-game Decision transformer, CQL and other offline methods should be added as baselines. The rest of the paper does not consider return conditioning at all, so this section feels out-of-place. \n4. **Missing baseline.** A very necessary baseline, I believe, is missing here. What if you trained a classifier on all the labeled data, which conditioned on a video would generate the textual label for the video. This labeler could be subsequently used to pseudo label all the demonstration-only data. Finally, a language conditioned imitation learning agent could be trained on this pseudo-labeled data along with the labeled demonstrations. This I believe is a natural baseline to your work. This is missing.  \n5. **Baselines are uninformative:** In the robotics section, why compare with RT-1? It’s trained on a different data distribution and it has a different architecture? This comparison generates more questions than answers. A clean experiment here would be retraining an open source Octo/OpenVLA model on all the labeled data only, i.e., only utilizing your labeled dataset.   \n6. **Missing ablation on amount of labeled data:** While I appreciate that the method scales with unlabeled data, what happens when the amount of labeled data is varied? Afterall, this is the limiting resource for this work. \n7. Table 1 is missing error bars.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u7edx5XsBa", "reviewer": "Reviewer_SgYT", "summary": "The authors propose an imitation learning method for training policies that can follow multi-modal instructions (either language or a video demonstration) using data that is only partially labeled with language. Like a prior paper, GROOT, they train a VAE on the expert data to obtain an encoder that produces a latent given a reference video as well as a latent-conditioned policy that predicts actions given an observation history and the latent. In this way the latent space models the demonstrator's intentions, and the policy can be commanded with a reference video by querying a latent from the encoder. This paper extends GROOT to the case where some of the expert data is labeled with language instructions. The authors modify the encoder to additionally accept language, and they introduce a new loss term that aligns the latents produced by language with those produced by the corresponding reference videos. In this way the language labels assist in learning a more well-structured latent space. Experiments on Minecraft, Atari, SIMPLER, and Language Table demonstrate that the proposed method, GROOT-2, outperforms GROOT and other relevant baselines when commanded with language or reference videos.", "questions": "- It seems the prior $e(z | o_1)$ is only ever input the first observation. Shouldn't the prior receive all the observations up to the current timestep $t$ like in equation 1?\n- In the SIMPLER evaluation, did you use Octo-1.5 or the original Octo model? The original model actually works better on SIMPLER.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important problem since language annotations are not always easy to acquire for imitation learning datasets. The method presented is a novel modification of an existing method which adds a non-trivial new capability. The explanation of \"latent space ambiguity\" is a nice justification for the method. The experiments cover a wide range of simulated settings.", "weaknesses": "My main concerns are some issues with the evaluation and the presentation/writing.\n\n**Evaluation**\n\n- It seems a central question is whether language loss ($L_{lab}$) improves reference video following capabilities and whether the reference video loss ($L_{dem}$) improves the language following capabilities. The cleanest way to answer this question would be to train with the GROOT-2 architecture and try removing either one of these loss components but keeping dataset size the same. However, I don't see this experiment. The atari experiment only tries removing $L_{lab}$. \n- For each evaluation setting, please report what proportion of the training data is labeled with language and whether there are language labels for the tasks that were evaluated.\n-  Only table 2 has error bars. It'd be great to have them for all the results.\n- Why are there dashes in Table 2? Are the numbers taken from the prior papers (and they didn't provide the per task breakdown)? If so, did their evaluation protocol match your evaluation protocol?\n- In the t-SNE plots there should be a legend showing what the colors mean.\n- Why was a human-normalized score used as a metric for the Minecraft experiments focused on scaling up unlabeled demos, but a different metric used in Table 1?\n- This sentence is confusing: \"This enhancement may be due to the limitations of our text-demonstration data collection method, which might not include text annotations related to the Climb Mountain task.\" If there are no text annotations for this task, then what was the 0% unlabeled data policy trained on?\n\n**Presentation/Writing**\n\n- It'd be nice to state the main questions the experiments are designed to answer at the beginning of the section.\n- The main text should clarify whether the encoder is a single model or multiple models.\n- In tables 1, 2, and 3 bold the best result for each task. This would make them easier to read. \n- The related work should discuss the relationship between prior work and the proposed method rather than just listing prior work. Additional papers to discuss/cite include GRIF which attempts to align task representations from language and initial state-goal pairs [1] as well as another work that uses videos as a task specifications [2]\n- In the conclusion, \"play data\" does not typically mean data without actions, so I would avoid using this term here.\n\n**Comments**\n\n- CALVIN [3] would be a good benchmark in which to test this method since it contains both labeled and unlabeled data.\n\n[1] https://arxiv.org/abs/2307.00117\n\n[2] https://arxiv.org/abs/2403.12943v2\n\n[3] https://arxiv.org/pdf/2112.03227", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDbVwtzzFu", "reviewer": "Reviewer_2co2", "summary": "This submission introduces GROOT-2, a weakly self-supervised instruction-following agent. It is trained in two stages, with the first being pretrained on large web data and the second being fine-tuning with small amount of human annotated data. Experiments are conducted on three different benchmarks, ranging from video games to robotics. It demonstrates competitive performance against previous approaches.", "questions": "1. L208: How web data come with action labels?\n2. What is the task horizon in each environment?\n3. Can authors show more complicated tasks? E.g., robotics tasks beyond table-top pick-and-place or real-robot experiments.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Good writing and illustrative examples, easy to follow.\n- Sufficient evaluation on three benchmarks.\n- Good analysis.", "weaknesses": "- The idea to train latent-space conditioned policy is not novel. Although authors claim that the introduced \"constrained self-imitating\" and \"human intention alignment\" help to balance two extremes on the Pareto frontier, no ablations are showed to verity that.\n- How is human annotation defined is unclear. Under the embodied AI and robotics context, human annotations usually refer to actions either collected or labeled by human. But in this work, human annotation seems to refer to high-level language instructions.\n- Following the previous point, if action labels are indeed required to train the policies, how does the proposed method scale up to domains beyond video games and robot pick-and-place, where action labels are scarce?\n- Experiment results seem not impressive enough. For those Minecraft tasks, the overall success rates are pretty low (only two tasks \"interact\" and \"plant\" succeed more than half). For robotics tasks, only the simple table-top picking-and-placing is demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MbYu52TN0I", "reviewer": "Reviewer_Mcj1", "summary": "This paper introduces GROOT-2, a multimodal instruction-following agent trained with a novel weakly supervised learning framework. GROOT-2 addresses the challenge of multimodal instruction adherence in agents by reducing the reliance on labeled data, which is often costly and time-intensive to obtain. Instead, the model utilizes a two-part strategy: *constrained self-imitating* and *human intention alignment*. The constrained self-imitating method enables GROOT-2 to learn diverse behaviors from large sets of unlabeled demonstrations, while the human intention alignment component aligns the latent space of the model with human intentions using a smaller, labeled dataset. The authors validate GROOT-2’s performance across multiple environments, including video games like Atari and Minecraft, as well as robotic manipulation tasks, demonstrating that it outperforms previous models (e.g., GROOT-1 and STEVE-1) in multimodal instruction-following tasks.", "questions": "See above sections.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Novel Semi-Supervised Framework for Multimodal Instruction-Following:** The paper introduces a novel framework for training multimodal instruction-following agents with weak supervision. By framing the challenge as a semi-supervised problem, the authors effectively reduce reliance on costly labeled data, presenting a scalable alternative to fully supervised approaches. The combination of *constrained self-imitating* for leveraging large sets of unlabeled demonstrations and *human intention alignment* to ensure the latent space aligns with human intentions is both innovative and practical. This strategy addresses common issues like *posterior collapse* and *latent space ambiguity*, which have hindered multimodal instruction adherence in prior work. The proposed weakly supervised approach extends the applicability of instruction-following agents across a broad range of tasks and environments.\n\n2. **Comprehensive Experimental Validation Across Diverse Environments:** The authors provide a rigorous empirical evaluation of GROOT-2 across multiple environments, including Atari games, Minecraft, and robotic manipulation tasks (e.g., Language Table and Simpler Env). The paper reports detailed quantitative comparisons against baseline models, such as GROOT-1 and STEVE-1, highlighting the model’s significant improvement across task families. Furthermore, the authors support the theoretical claims about their model with a thorough analysis of the learned latent space, illustrating how human intention alignment resolves ambiguities in latent representations. The experiments are comprehensive, with comparisons against varied baselines and multiple training data scales, showing the robustness and scalability of GROOT-2.\n\n3. **Clear and Structured Presentation:** The paper is well-organized and clearly presents the problem, the proposed approach, and the experimental results.\n\n4. **Broad Impact and Applicability of Weakly Supervised Instruction-Following:** The weakly supervised approach proposed by GROOT-2 has implications for the development of scalable, multimodal instruction-following agents. By reducing the reliance on labeled data, this framework broadens the feasibility of deploying such agents in real-world applications, where data annotation is often prohibitively expensive. The demonstrated effectiveness of GROOT-2 across diverse environments highlights its generalizability and adaptability. The contribution is particularly significant for applications in robotics, where the ability to follow human-like instructions across varied tasks without exhaustive labeling can accelerate the adoption of robotic agents.", "weaknesses": "1. **Scalability Limitations Due to Data Requirements**  \n   Although GROOT-2 successfully reduces the need for labeled data by using weak supervision, its current framework still relies on trajectory data, which includes both observations and actions. This dependence restricts the model’s scalability, as a vast amount of available data on the internet, such as play data or interaction data, often lacks explicit action information. The authors acknowledge this limitation but do not provide a concrete solution for incorporating such unlabeled play data. Addressing this issue could further expand the scalability of GROOT-2, potentially allowing it to leverage far larger datasets. Future work could explore methods for approximating or inferring actions from play data or combining GROOT-2’s framework with unsupervised or self-supervised approaches to handle action-less data. \n**Recommendation:** Exploring methods to extend the framework to utilize play data would address a core scalability limitation.\n\n2. **Evaluation Scope for Diverse Task Complexity:**  The paper demonstrates GROOT-2’s performance across varied environments; however, within each environment, the range of tasks presented may not fully capture the complexity of real-world, open-ended tasks. For instance, while Minecraft and Atari offer diverse scenarios, it remains unclear how well GROOT-2 would perform in environments with truly unpredictable dynamics or in tasks requiring complex, multi-step reasoning. An analysis of GROOT-2’s performance on tasks of varying complexity within each benchmark environment would provide a more granular understanding of the model’s adaptability and robustness. This could involve testing tasks with multiple dependencies or obstacles to simulate more realistic scenarios. \n**Recommendation:** Evaluating GROOT-2 on tasks of varying difficulty would yield insights into the model’s robustness and real-world applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)