OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents
Authors: Zihao Wang
Shaofei Cai
Zhancun Mu
Haowei Lin
Ceyao Zhang
Xuejie Liu
Qing Li
Anji Liu
Xiaojian Ma
Yitao Liang
Venue: NeurIPS 2024 poster (2024)
Keywords: open-world
multimodal language model
decision making
generalist agents
Primary area: reinforcement_learning
TL;DR: We propose a novel VLA models, OmniJARIVS, by jointly modeling vision, language and action in multimodal interaction data.

Abstract: This paper presents OmniJARVIS, a novel Vision-Language-Action (VLA) model for open-world instruction-following agents in Minecraft. Compared to prior works that either emit textual goals to separate controllers or produce the control command directly, OmniJARVIS seeks a different path to ensure both strong reasoning and efficient decision-making capabilities via unified tokenization of multimodal interaction data. First, we introduce a self-supervised approach to learn a behavior encoder that produces discretized tokens for behavior trajectories $\tau = \{o_0, a_0, \dots\}$ and an imitation learning policy decoder conditioned on these tokens. These additional behavior tokens will be augmented to the vocabulary of pretrained Multimodal Language Models. With this encoder, we then pack long-term multimodal interactions involving task instructions, memories, thoughts, observations, textual responses, behavior trajectories, etc into unified token sequences and model them with autoregressive transformers. Thanks to the semantically meaningful behavior tokens, the resulting VLA model, OmniJARVIS, can reason (by producing chain-of-thoughts), plan, answer questions, and act (by producing behavior tokens for the imitation learning policy decoder). OmniJARVIS demonstrates excellent performances on a comprehensive collection of atomic, programmatic, and open-ended tasks in open-world Minecraft. Our analysis further unveils the crucial design principles in interaction data formation, unified tokenization, and its scaling potentials. The dataset, models, and code will be released at https://craftjarvis.org/OmniJARVIS.

OmniJARVIS
Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents
Zihao Wang', Shaofei Cai', Zhancun Mu?, Haowei Lin', Ceyao Zhang", Xuejie Liul Qing Liš, Anji Liu', Xiaojian Ma², Yitao Liang!
Team CraftJJarvis
Institute for Artificial Intelligence, Peking University
Yuanpei College, Peking University
Beijing Institute for General Artificial Intelligence (BIGAI)
Universityy of California, Los Angeles
{ 2hwwngg, ca caishaofei) @stu. pku. edu, cn
xiaojian, www.:aaaaaaalaaacccca..a. lasae,,,iiaaa Liuan jiecs. ucla.edie wtt....yyllllllkuuuuuuu
Abstract
This paper presents OmniJARVIs, novel Dision-Languagc-ARe (VLA) model for open-world instruction-folloiing agents in Minecraft. Compared to prior works that either emit textual goals to separate controllers produce the control command directly, OmniJARVIS seeks a different path to ensure both strong reasoning and efficient seeisionnmmking capabilities via unified tokenization of multimodal interaction data. First, we introduce a self-supervised approach to learn behavior encoder that produces discretized tokens for behavior trajecto- ries = foo, ......) } and an initation learning policy decoder conditioned on these tokens. These additional behavior tokens will be augmented the vocab- ulary of pretrained Multimodal Language Models. With this encoder, we then pack long-term multimodal interactions involving task instructions, memories, thoughts, observations, textual responses, behavior trajectories, etc. into unified token sequences and model them with autoregressive transformers. Thanks the se- mantically meaningful behavior tokens, the resulting VLA model, Omn MiAAAJRRISS can reason (by producing chain-of siciin-o--hhuseee of-tthogghts), plan, answer questions, and act (by producing behavior tokens for the IL policy decoder). Omni iJRVIS demon- strates excellent performances on comprehensive collection of atomic, program- matic, and open-ended tasks in open-world Minecraft. Our analysis further unveils the crucial design principles interaction data formation, unified tokenization, and its scaling potentials. The dataset, models, and code will be released at hattps://craf:: wttpafffjaavvsss..oooooii..o.
Introduction
Upon the success of pretrained Large Language Models (LLMs) 7 35 40
and Multimodal Langauge Models (MLMs) [31|20] 20 53 33 some recent works have been venturing into developing 38) promising pathway towards the ultimate goal of building autonomous agents that can follow and even self gelf-generated instructions to fulfill various reasoning and acting tasks in open world environments. Among them, two most prominent
17 13
Vision- Vision-LangaageActtonn (VLA) models
"Corresponinng Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

[II onturction on
MMj-emory
[A] icttion
C-llt and use jooks to gogiltes,
coftt items and Mlocss
Minol stone, mined out Inavee, mire agt. cratted planks, auaied crulted
The player circedy na tools for Given the recent menerry of cuin the wasden Hemos, the player should focus atiiting the woode So should be
Koyboard W Monuee 1+S
cornume utkee fureace, coltting bei,
interleaved obs act (behavior trajectory)
Expend and Process the RAW interactive data
interleaved obs act (behavior trajectory)
interleaved obs act
Conteet:
t=1 1
tokenization
Behavior Token
Behavior Token
Context
1
t=
Visian Modality
Language Modality
Action Modality
Figure 1: Illustration multi-modal interaction data for decision-making. A canonical interaction sequence depicting the human seciisionmmaking process starts from a given task instruction and memory, followed by series of sub-task completion which involves initial observations. chain-of-thought reasoning, and behavior trajectories. Our proposed VLA model OmniJARVIS jointly models the visiin (observations), language (instructions, memories, thoughts), and actions (behavior trajectories) unified autoregressive sequence prediction. A self-supervised behavior encoder (detailed in [Section aFnuur converts the actions into behavior tokens while the other modalities are tokenized following the practices of MLMs BJ
architectures have been proposed: 1) Combining an off-the-shet MLM 31 with separate goal- conditioned controllers 28 10 91. where MLM reasons, plans and pilots the controllers by producing textual goal instructions, egg. DEPS (46), JARVIS-1 [47]. voyager [4); 2) Tuning a pretrained MLM into producing control commands directly, while maintaining the reasoning and language capabilities, e.g. RT-2 [6. LEO [22]. However, these two designs could still have significant drawbacks when comes to open-world environments. First, open world (e.g.. Minecraft) usually teams up with an infinite number of complex and highly contextualized tasks [16] [9]. and it can be fairly challenging depict them in text only. Therefore, VLA models that solely depend on text communicate with the text-conditioned policies [47] 46) may fail to correctly pilot these controllers. On the other side, emitting the control command directly 6 22 without invoking separate controllers could alleviate the aforementioned communication problem but given the long- hong-horizon nature of open-world tasks, it is less practical to perform long-term control with large VLA model as the context length requirement, computation cost and inference efficiency could become unaffordable.
In this paper, we aim tackle the aforementioned issues of existing VLA models when facing open-world environments: complex & context-dependent tasks and long-term tasks. Our key insight originates from the observation of human decision-makinn making: Given these open-world tasks, humans can make informed decisions via multi-round mental, verbal, and physical interactions (an illustration can be found Figure I}. Therefore, the VLA model can manage learn from such interaction data, it may master the underlying human decision-making procedures. However, modeling interaction data is non-trivial: is multi-modal, encloses vision (mostly observations), language (instructions, thoughts, etc.), and actions (behavior trajectories). Compared to the fruitful explorations on jointly tokenizing vision and language B 43
into sequences for autoregressive modeling 77). tokenizing behavior trajectories (actions) is hard due to the following reasons. On the one hand, directly using low-level actions from the environment would pose huge challenges to the model's 's ability to process long sequences, which significantly hurts performance. It also hinders us from leveraging the planning ability of generative models. On the other hand, language-level action tokens require significantly more supervision and cannot accurately describe all possible actions.
To this end, we propose OmniJARVI and actions in interaction data with unified tokenization. Omn i JARVIS comprises two key ideas: 1) Behavior Tokenization. We introduce a self-supervised approach to learn a behavior encoder
novel VLA model that jointly models vision, language.

---

Tokenizer
De Dolkenizer
Discrete Behavior Tokens
Behavikr Clontnng
C
C
@2
Behavior Encoder
Finite Scalar Quanlizer
Decoder as Policy
Observations
Learnable Tokens
Observations
Figure 2: Self-supervised learning for behavior tokenizer of OmniJARVIS. We modify the VAE-based self-supervised learning of behavior trajectories in [10] to train the behavior tokenizer and de-tokenizer in S. Specifically, we adopt the auto-cncoding objective but replace the Gaussian latent with a discrete representation based on Finite Sealar Quantizer [34]. The encoder will then be used the bchavior tokenizer produce discrete tokens from the actions (behavior trajectories) in multimodal interaction data, while the behavior tokens emitted by Cmni JARVIS will be sent to policy decoder to perform motor control.
that produces discretized tokens for actions (bchavior trajectories) and an imitation learning policy decoder conditioned these tokens (Section 2); 2) Autoregressive Modeling. By augmenting these behavior tokens into the vocabulary of pretrained MLMs, we pack the multimodal interaction data into unified token sequences and learn a transformer these sequences with an autoregressive modeling objective. We conduct comprehensive evaluations in the spendworld Minecraft Universe [29]. Omn MARVISS demonstrates impressive performances open-ended Minecraft tasks. Our analysis confirms several critical design choices in data formation, tokenization, and the scaling potential of Omn iARVIS. Our contributions are follows:
wide range of atomic, programmatic, and
We propose Omn i JARVIS, novel VLA model capable of following instructions to reason, plan, and act in open-world environments by jointly modeling vision. language, and actions in multimodal interaction data for decision-nin -ocision-maaing....
We propose supervupee approach to learn behavior encoder to tokenize actions and an imitation learning policy decoder to produce control commands from behavior tokens emitted by Omn i JARVIS, allowing joint learning VLA and smooth action readout.
We conduct extensive evaluations open-world Minecraft to demonstrate Omni JARVIS's profi- ciency across various tasks and present in-depth analyses to reveal valuable insights.
2 A Tokenizer for Behaviors
As illustrated in Section I key challenge for VLA is the mismatch between the action modality and other modalities such as the language instructions. key insight is that a good amount knowledge about the effects of actions can be learned directly from behavior trajectories (7(0).. We propose learn behavior tokenizer in addition to the well-studied vision and language tokenizers to achieve unified tokenization of the vision. language and actions in multimodal interaction data (Figure 1). We pose two main requirements to the behavior tokens. First, they should be able to express complete trateetoriie trajectories. Further, the tokens should contain semantic information so that they are compatible with the other modalities, which enables the reasoning and planning ability of L.MMs ge.g., by conducting chain-of-thought reasoning).
and diverse behavior from (short)
bby from a behavior trajectory {oo, ao. ..... }. Further, de-tokenizer is needed to map these tokens back an action rollout in the environment that reproduces the goal achieved in r. GROOT [10] explores a VAE-based approach jointly learn latent representation of behavior trajectories and an imitation learning policy decoder that conditions the latent as goal. However, the continuous latent cannot be used as the behavior tokens as they can be more difficult to learn and decode with the existing discrete tokens of pretrained MLMs [22 32). Therefore, we replace the Gaussian latent in GROOT with an improved vector quantized discrete latent called Finite Scalar Quantization (FSQ) 34
Specifically, we aim at producing set N discrete behavior tokens .by .-.-, 8
We adopt
quantization configuration of [8, 8, 8, 6, 5). which means code with a length=5 and a codebook size of 8 x 8 x
15360 produced. The configuration is selected by simple grid search.
Overall, the behavior tokenizer (bchavior encoder) (p(or))) and the de-tokenizer (IL policy decoder)

---

The player's current inventory indudes vanety items, rotably... thought
Interaction
The player alneadl has base wooden took and resourses lik osk planes and stide...
interaction
Decoder Policy
Decoder as Pollcy
thought
Omnilarvis
Start progress collecting resdurces...
Mined
Tank Instruction
Figure 3: Architecture and Inference JARVIS. The main body of OInRVVS multimodal language model (MLM) augmented with additional behavior tokens. Given a task instruction, initial memory, and observation, Omn MiniJARVIS will iteratively perform chain-of of thought reasoning and produce behavior tokens means of control via the decoder policy (hehavior de-tokenizer). Every 128 steps, Omni JARVIS TS is forced reason again and produce behavior tokens with the latest ohservation. (Not shown above) Cmn JARV can also make textual responses, e.g. answering questions.
(a (O1:e) ) learned with the following objective:
argmin ET-D (0.0)
Σ logg fle
(I)
where f()) denotes the finite scalar quantizer. We choose a non-causal (bidirectional) transformer and causal transformer to parameterize the encoder es (01:T) and the policy decoder (a (1 respectively. In practice, we set 128 as the trunk size of the behavior trajectory to be encoded. We will discuss how to handle trajectories longer than 128 in the next section.
Compared our behavior tokenization, most prior work in VLA models, either represents the behavior trajectories in interaction data as a textual goal description and invokes a separate goal- conditioned controller 47 [6]. or represents the state-action sequence {ou, ag.... directly in 38 6). Our approach offers more compact but still informative Decision Transformers (DT) 22
representation of the actions part in multimodal interaction data. Moreover, the action readout, i.e. simply sending the behavior tokens to the policy decoder, is also more efficient than the D'T-style direct control from VLA models
38
3 Multimodal Interaction Data and OmniJARVIS
canonical multimodal interaction data comprises vision (observations), language (instructions, memories, thoughts), and actions (behavior trajectories). However, can be difficult to directly collect such interaction data from human annotators. Therefore, we propose to convert an existing Minecraft gameplay dataset [2] into the multimodal interaction data required by Omni IJRVISS. We begin with formal definition of the interaction data, followed by our approach for data conversion and augmentation from existing datasets, and finish up with the architecture, formulation of learning on such interaction data, and inference procedure of Omn i JARVIS. An overview Omni i JARVIS architecture and inference can be found Figure
As illustrated in Figure
3.1 Data Formation
An interaction sequence of decision-making ID (DrYT comprises segments. Each segment D be sentence of text words [ f.e. the language part such instructions Dinst memory Dmmm or thoughts Dthtt D, can also an image I, i.e. the vision part such observations D Finally, D; may belong to the action (behavior trajectory) part, i.e. Dbbr = {op, ao.... We assume these segments follow the ordering below (Figure T):
Dinsa 3 Dmmm
Dbbs
Dhh bb
Dahr Dth Dbhr
Context
sub-task
sub-task 2
We tokenize such sequence of segments into series of tokens (so...., 8M using the vision and language tokenizer offered by a pretrained ML.M and the behavior tokenizer introduced in Section

---

3.2 Preparing Multimodal Interaction Data
In reality, many segments of the multimodal interaction D can be missing in public datasets. We consider the Minecraft contractor data released by OpenAl [2] and it only contains behavior trajec- tories D Therefore, we need to properly augment the data with the additional textual segments including instructions Di memory Di and thoughts D We follow the prior practices [22 31 to synthesize the required text using LLMs. Below, we detail how each type of segment is constructed. More details can be found in appendix.
Synthesis of instruction Dlast The instruction is high-level description of what task is being performed in the current interaction sequence, The considered OpenAl Minecraft data includes meta information of each gameplay video, which depicts fundamental events that happened during in Minecraft gameplay, egg. what block was just destroyed, what entity was just killed, what item was just crafted, etc. Such inta-inforrmttion can provide basic overview of what the player has been through in the gameplay. We therefore prompt an LLM into summarizing the gameplay with the meta information. The summary will be used as the instruction D of the current trajectory.
Synthesis of memory Dimem The memory is the summary of what agents have finished in the previous interaction sequences, Due to the limited sequence length that the auto-regressive model can handle, the model needs to learn to summarize key information related to the task in historical interactions and ignore behaviors unrelated to instructions. The memory will be updated based on the results of each episode trunk and used for subsequent episode trunks. We therefore prompt an L.MM into summarizing the gameplay with the meta information. The summary will then be used as the memory Dmsem of the current interaction trajectory. The memory prompt can be found in Appendix F
Synthesis of thought DIM The thought is the agent's 's reasoning and explanation of its own decisions. Previous methods have confirmed that using understand decision -making [21]. Compared to labeling thoughts by humans [50] 6). we assume that thought an intermediate variable that can be determined by the actions taken and observations made before and after the action, which is similar to an Inverse Dynamics Model [2]. We therefore
enhanced interaction data helps language models
prompt an LLM into estimating the thought of decisions with in-context learning, which will then be used as the thought Dt of the current behavior. Details can be found in Appendix E
3.3 Architecture, Training, and Inference of Omn MARiJSRVIS
As illustrated in Figure 3] OmniJARVIs IS is built upon a pretrained MLM. We augment the original vocabulary of the MLM with additional tokens from the behavior tokenizer. Specifically, as adopted the [a, b, c] FSQ configuration (Section 2). we augment with a + b + c new tokens as each behavior comprises n behavior tokens Bhee she corresponding to n FSQ levels. We formulate the learning objective of Omni JARVIS following [7] in prefix language modeling fashion. For a batch B of token sequence 8, we optimize Omn i JARVIS via:
18
T
L( = Σ
(0.00 (0.00) (s00 log (o (8
(b.1)
(3)
ppeelix -.y pretix
b=1
where Sprelix denotes the prefix token, which is tokenized from the segments that served context for reasoning and decision-making, i.e. instruction Dins, memory D.mem and observation Dehs within the interaction sequence (Equation 2). The remaining tokens (tokenized from thought Dtarr and behavior trajectory D. will be predicted in an autoregressive fashion. From high level, Omn i JARVIS is trained to reason (producing thought tokens) and act (producing behavior tokens) from contexts with task instructions, memory, and current observations. During inference, we begin with the feeding OmniJARVIS with a task instruction, an empty memory, and an initial observation. Omn iiRARIIRIS will produce a chain-of-thought as a means of reasoning and subsequently, emit behavior tokens for control. Every N steps, it is forced to reason again to produce new behavior tokens with the latest observation. We empirically set N 32.
4
Capabilities and Analysis
4.1 Overview
Training details and Datasets. The training of the the first step, we use a self-supervised training method to train a Behavior Tokenizer, including the Encoder and Decoder jointly. We use FSQ as quantization method and build a codebook
is divided into two stages. In
5

---

with 8*8*8*655 discrete codes. The training data for Behavior Tokenizer comes from Contractor Dataset 22. which is collection of Minecraft gameplay videos. The training parameters and details remain consistent with GROOT, which can be found in Appendix A
In the second stage, use this behavior tokenizer process Minecraft offline trajectories obtain behavior token sequences. We add 35 (8+8+8+6+5) additional tokens to the MLM tokenizer as behavior tokens for unified representation, so each time the VLA needs output a continuous sequence of 5 tokens to represent a complete behavior. We use GPT-3.5 to synthesize thought, memory, and instruction to raw offline datasets to build complete interaction data. The specific prompt can be found in Appendix E These data collectively constitute the embodied instruction- following dataset of OwwiJARVia mcluding 600k trajectories and about 900M tokens.
The training dataset of Omn JARVTS further includes a large amount of QA data about Minecraft. We generate a large number of seed questions about these texts using web pages on the Minecraft wiki. Then, we use the self-instruct method to generate a large number of creative questions and instructions. This constructed QA dataset consists of 300k conversations with about 90M tokens. During the training process, the QA data and instruction-followw about IT tokens, to train OmniJARVIS, In specific, we SFT (supervised finetune) VA-V (3I. The details can be found in Appendix A To further demonstrate the generalizability of the method, we also fine-tune LLaVA VA at different scales and VLM Fuyu-8B with different architectures. The relevant results are presented in Section 4.5 and Section 4.6
data are mixed, with total of
Experimental Setups. We conduct experiments in the complex and open-world environment of Minecraft, a voxel-based 3D video game that has garnered significant attention from real-life research due to its popularity and diverse mechanics [18] [6]. We first evaluate Omn JARVIS with atomic tasks, which are skill-level tasks, testing VLAs ability to follow simple and straightforward i instructions. Then we evaluate iJAVIS with programmatic tasks, which require the agent to obtain an item starting from an empty inventory. The success of these tasks requires VLAS to decompose the provided instruction into atomic-level subtasks, and hence tests VLAs' complex reasoning ability. Finally, we test imniJJAVIS with open-world embodied question- -answerngn benchmarks and creative free-form Instruction-follovii Omn MAnVIJASSS with different behavior tokenizers, different training dataset formats, and different vision tokenizations. Finally, we explore the generalization abilities of Omn i JARVIS of Atari Games and the scaling potential of Omn i iJAVIS with different models and data scales.
We also conduct ablation experiments of
4.2 Main Results I: Short-horizon Atomic Tasks
Atom tasks are various simple skills that agents in Minecraft need to master. They are basic tasks yet are fundamental skills that agents need to master during the learning process. We first evaluate OmniJARVIS with our learned behavior tokenizer on these tasks.
We select "chopping trees"* "digging dirt" "mining stones* and "collecting wheat seeds" 4. as the evaluation tasks. We directly take those short task descriptions as instructions for agents. We use text-conditioned VPT 2]. Open-world Control 19. STEVE-I (28), and video- instructed GROOT [0] as baselines. We compute the average rewards of different agents on every task in Table T across 10 runs. By observing the environment and adjusting action tokens dynamically, Omn i JARVIS effectively follows straightforward instructions across various scenarios. It consistently achieves a high average reward with minimal standard deviation.
4.3 Main Results II: Long-horizon Programmatic Tasks
To further verify the ability of OmniJARVIS to complete tasks with long sequences, we use 30 programmatic tasks to evaluate the performance of different agents. These tasks require the agent to start from an empty inventory in a new world until obtaining the final required items, which is usually chain of atom tasks. These tasks divided into five groups hased difficulty: wooden, food, stone, iron, and diamond. For example, the prompt for task "Obtain diamond pickaxe" 70 is "Give you nothing in the inventory, obtain a diamond pickaxe." " This task requires more game time and more complex planning for up 10 different intermediate items 2 We list all programmatic tasks and its corresponding instructions in the Appendix C.I
Baselines are divided into two types: 1) directly outputs actions, namely the native behavior tokenizer, including STEVE-I (28 and GROOT [00. 2) using pretrained L.MM as a planner to output language goals and connect the STEVE-I to execute these goals, including Zero-Sho Planner (GPT) (23).

---

Table I: Evaluation results (rewards) on short-horizon tasks, The text-conditioned VPT 2 ("VPT
Table 2: Results open-ended instruction following.
Instruction
VPT
STEVE-1
Voyager
DEPS 929.5
Ours
(text) ") is from Appendix of its paper.
Following
972.7
932.1
886.2
Method
Condition
421
Table 3: Results on open-ended question answering.
[PTT
STEVE
28
2
Language
Languaee
11.0436
2,
10,ges
9 9.2501
3,2414
5.1k26
We use LLM-as-judge (35] to evaluate the accuracy.
GROOT
Omn JARVIS
10
Languge
Video
14.3
10,9:59
20.3
19.7
18.0
25,8455
82233
7.3486
QA1
Vicuma-BB
2.34
Vicuma-1 138
2.85
LLaMA2-70B
2.50
GPT-3.5
7.50
Ours
8.40
Table 4: Success rate of different agents on long-horizon programmatic tasks.
Method
Action Tokenizer
Wooden (10)
Food (5)
Stone (5)
Iron (5)
Diamond (5)
Average
STEVE-I
28
Native
0.04
0.00
0.00-1..0 0.00-10.00
0.00.10.00
0.01
GROOT
10
Native
0.05
0.00
0.00
0.00
0.00
0.02
GPT 23
Language
0. 0.26
0.08
0.24
0.04
0. 0.00
0.15
ReAct 50
Language
0.44
0.12
0.30
0.06=20005
0.0020.00
0.23
DEPS 46
Language
0.78+0.11
0.12+0.04
0. 0.68
0.16
0.04
0.43
OmniJARvis
FSQ GROOT
0.95
0.44
0. 0.82t00.08
0.32
0.0g
0.59
ReAct [50], and DEPS [46]. We use success rate to evaluate the completion of tasks, that is, whether the task is completed within the specified time. The experimental results are listed in Table 6
Programmatic Tasks usually require complex reasoning for planning. While STEVE-I and GROOT can only finish short skill-level tasks in atom tasks, is difficult to finish these programmatic tasks. Agents based on Language behavior tokenizer can complete complex tasks including diamond group ones, but with a low success rate. This is because these in-context learning methods leverage the pretrained LLM which may lack the necessary knowledge about this world. is worth noting that in the Food group, agents based on Language Tokenizer have an average success rate of around 10%, as this set of tasks does not require complex reasoning. This indicates that Language-conditionel conditioned trajectories as supervision for training while there was less such data available during STEVE-I's training phase leading to significant performance gaps. Mcanwhile, Omn iJARVIS uses self-supervised trained behavior tokenizer which does not require extra language labels and hence receives more training resulting in good performance across a wider range of tasks. We will further prove this in the next set of Creative Task experiments.
Tokenizers need additional lunguage-conditioned
4.4 Main Results III: Open-ended Arestion-Answering and Instruction Following Tasks
The open-ended tasks differ from programmatic tasks due to the lack of straightforward success criteria [16]. We select the long-term open-ended tasks which usually need at least 5 minutes of suman-playing time to finish. The task prompts can be found in Appendix C.3 Following image generation and video generation tasks [19 411. we take the Fréchet Sequence Distance (FSD) metrics to evaluate the correlation between agent rollout video and creative instruction. Specifically, we first ask human experts to finish the creative task prompts under randomly generated worlds and record the game saae--llying videos Vnuman- Then, we provided the task prompts for different Minecraft agents, and obtained rollout video set V Vagent- Similar to FID 119,, we used MineCLIP [16] to calculate the embedding of video clips and computed FSD for the embedding distributions of human and agent rollout videos. The analysis of the metrics can be found in Appendix B
We further conduct open-ended embodied wwwsssiionnansseriie benchmarks to evaluate the ability of the agent to complete open-ended instructions and grasp world knowledge. The questions answering instructions set can be found in Appendix C.2 The evaluation results can be found in Table and Table 2] OmniJARV: is the agent that can simultaneously complete both types of tasks and has achieved the best performance in different task sets, surpassing strong baselines including Voyager 44 and DEPS [46]. Also, it maintains strong reasoning capability, especially embodied question answering compared to LLM basclines (with image captions as visual context).
4.5 Insights and Analysis
Interactive Dataset Format. We explore the crucial roles played by the different type of segments in interaction data, including the instruction, memory, thought, and caption tokens. The results can be found in Table 4 where we evaluate the loss on predicting the behavior tokens. It can be seen that instruction and thought can be more critical to the successful prediction of behavior tokens. This consistent with our hypothesis making informed decisions requires task instruction and reasoning.
7

---

Figure 4: Ablation experiments OmniJARVIS with different behav- ior tokenizers, vision tokenizers, and training different interactive datasets. The first line training on the unconditional interactive dataset, i... without instructions on the trajectories.
-072
O de
Grnusins TB
-1.00
OrraARvi 158
-.23
JARVIS with VQ- GROOT 42 [0] shows no results because of training collapse.
-1.5
Behavior
Vision
Datnset Format
Loss
-2.0
Tokenizer
Tokenizer
Instruction
Caption
Thought
Memory
Train
Eval
-2.5
x (unconditional)
0.33
-0.53
x
x
0.46
0.51
Traning Das too)
FSQ GROOT
LLaVA
x
x
0.44
0.48
/
x
0.32
0.33
Figure 5: Scaling potential
/
of OmniJARV S. Its evaluation loss continues drop with the growth of data and model parame- ters. The Pearson coefficients for the 2B, 7B, and 13B models are 0.9991, 0.9999, and 0.9989.
FSQ GROOT
Captioner+
5
/
FSQ GROOT
FUYU
x
x
0.12
0.11
GROOT
LLaVA
0.44
0.48
VQ GROOT
LLAVA
Table 5: Ablation experiments on behavior tokenizer with different code vocabulary size.
Codebook
FSQ Levels
Training Iterations
Train
Eval Loss
Reconstruction FSD 4
Sampling FSD 4
Average Rewards f
Codebook
size
Loss
Usage
e8
[8,6,5]
180k
2.746
3.161
46.57
68.90
0.63
93.75%
e10 [8,5,5,5|
180k
3.011
3.148
43.67
61.85
0.54-1.21
97.65%
e14 18,8,8,6,51
240k
3.092
3.116
42.72
57.37
2.27=2-45
92.36%
Vision Tokenization. We also evaluate training OmniJARVIS with different vision tokenization. including ImageCaptioner + LLaMA2-7B [12] 40) (basically converting the vision input into textual captions). fuyu-8b B3] and LLaVA-7B BI architecture. For the ImagcCaptionert, we fix the ImageCaptioner models and only fine-tune the language model, i.e., LLaMA2-7B. We use the prediction loss of behavior tokens as the evaluation criterion, namely eval loss. We found that the model trained with LLaVA-7B architecture has the lowest evaluation loss, so we chose this model as the default model.
Behavior Tokenizer. We explore Omn iJARVISS with different behavior tokenizers, including the default setting using FSQ codebook, a variant of using VQ-VAE instead of FSQ 42 and simply using sub-goal language annotation as behavior "tokens". The evaluation results on 4 programmatic tasks are listed in Table Using an FSQ tokenizer is generally better than a language goal. which confirms the advantages of using tokenized behavior over language descriptions of behavior. The use of VQ- VAE as a quantized behavior tokenizer collapsed during the training process, so there were no results in all test tasks.
Behavior Codebook. We conduct an in-depth investigation of behavior tokenizers with varying codebook sizes, utilizing recommended sets of FSQ levels to approximate specified codebook dimensions [34] delineated in Table 5 We evaluate performance across multiple metries for each codebook size. Codebook Usage is quantified as the proportion of codewords utilized at least once when encoding the validation datasets. Reconstruction FSD measured by the FSD scores derived from the MineCLIP encoder [16), processing 1,000 different demonstration videos through the FSQ-GROOT and subsequent rollout we measure Resampling FSD, which is the FSD score obtained when the environment rollout is conditioned on representations sampled from the codebook. Finally, we assess the average rewards for the task "collect wood" using Omn JARVIS S across varying codebook sizes. Our findings indicate that increases in codebook size correlate with enhanced average rewards and reduced FSD scores, suggesting a scalable performance in OmniJARVIS with larger codebooks.
randomly generated environment. Additionally,
Behavior Semantics. We provide some qualitative analysis on the learned FSQ-based behavior tokenizer. In Figure 6) we tokenize several reference videos, then feed the behavior tokens to the policy decoder and see if it can accomplish the same task as in reference videos. The results indicate that our behavior tokenizer is able to capture such behavior semantics and offers rich task information.
4.6 Generalization and Scaling Potential of Qmni JARVIS
We first explore adapting Omn i JARVIS to the Atari game Montezuma's Revenge. We created dataset from 500 episodes played by an agent trained with Random Network Distillation (8).
8

---

Provided
Generated
Figure 6: Examples behavior oakenization-deookeinztii.. Left: the reference video to be tokenized by our FSQ-based behavior tokenizer (encoder). Right: the behavior the policy decoder is conditioned on the bchavior tokens. The policy decoder can reproduce the task being accomplished in the reference vidco.
Affre
TOE
415
30
SC4E
337:
3581
0
3.8
0.00
0.00
0.00
Figure 7: Omni JARVIS plays Montezuma's Revenge and gets reward of 3600.
supplemented by random actions in early frames to enhance diversity. This dataset contains 1,823,699 transitions. We then trained the PSQ-GROOT tokenizer on this new dataset and subsequently trained Omn OIRVJS on the tokenized data. The finetuned Omn OmniJS achieved a score of 3600 in Montezuma's 's Revenge, indicating promising transferability. A rollout trajectory is Figure
We also investigate the scaling effect [25] 30 of data and model in OmniJARVIS by monitoring the instruction-following loss the validation tuning from the default LLaVA-7B, we include two additional scales: OmniJARVIS S-2B (fine-tuned from 2LB with Gemma-22 language models (20]) and OmniJARVIS- (fine-tuned from LLaVA-132 13B with LLaMA2-133 13B language models 31)).
the amount of data increases. In addition to fine-
The validation loss curves in Figure reveal the following insights: 1) When using Omni-Tokenizer,, instruction tuning aligns with the scaling law (25) All curves exhibit a log-linear decrease the data scale increases. 2) Scaling up VLM consistently enhances performance. Notably, S-AARISS-BB... However, while improvements are consistent, the difference between Omni JARVIS-7B and Omni OmniJARVIS: JARVIS- 13B seems less pronounced, hinting at potential saturation when further scaling up VLM. This underscores both the scalability of Omni JARVIS and the importance of increasing data volume to match the model.
OmniJARVIS-7B demonstrates significantly lower losses compared to
Related Works
Pretrained Language Models for Decision-making Several works have explored leveraging LLMs to generate action plans for high-level tasks in embodied environments [23] 27
5
52
To better
perform complex planning in the environment, existing methods usually utilize thain-of-though (49
or related methods [50]. To better cope with uncertainties in open worlds, some L.LM-based methods
generate plans interactively with human and environmental feedback (39 46 24 and retrieving
from memory 47 internet corpus [8]. However, those plans can only be executed in language environment or require an additional controller or code exccutor to interact in an open world.
Vision-Language-Actton Models. In order to better utilize the knowledge inside the language model for decision-making, some methods tend to use decision datasets fine-tune pretrained language models 15 14. Gato [38] was among the first to tokenize environment-providee actions to enable joint sequential modeling across modalities. PaLM-E [14] generates high-level instructions as texts and uses dedicated controllers to perform the task described by the output instructions. The RT series focuses more on robotics settings. Specifically, RT-1 pairs a VLM with language-conditioned controller; RT-2 extends the VL.M to directly include control tokens; RT-X generalizes to new robots and environments. recent VLA model LEO 22 expands the perception from 2D images to 3D world and enables rich scene-level reasoning and control tasks.

---

Instruction
Instruction
Instruction
Low Frequency
Low
Auto-segeessive
Auto-regrossive
Auto-regrossive
Frequency
Transformers
Vision-Languaae Models
-1
Vision-Language Models
Hz
Frequenry
Unijied
Language Plan
sonduagg-nndiioonedd
craft planiss...
mine loge,
Frequency
High
Language Tokent
Behavior &
Self-supervised Policy
chain of [Thoughi]
BEHWVIO 153
Frequency
High
Action
State
Policy(CodeTToo
-20 Hzz
Behavior De-Tokanizor
-2 Hz
Action
Action
State
State
Environment
Environment
Environment
(8) Native Tokenirer
(b) anguuge Tokkniter
(6) OmniJanvis
Figure 8: Comparative Framework of Vision-Language Action Models. (a) depicts a model where upon receiving a language instruction, actions are directly output based on the environmental state, facilitating immediate interaction with the environment
unified frequency. Smaller models with <IB parameters like VPT (2) maintain higher frequencies (>20Hz), though their capability for complex reasoning tasks is limited. Larger models with >7B parameters such as RT-2 [6. offer enhanced performance but operate at significantly reduced frequencies (2-3Hz). (b) illustrates a common approach utilizing large vision-language models for planning, subsequently outputting language goals 46 14 4. A ainguage-oondttioned policy then translates these language goals into actions a real-time interaction
of 20Hz, with high-level models re-planning less than HHz. This hierarchical structure balances interaction frequency and performance, while requires language as an intermediary and additional language labels. The training process of high-level rision-language models and language- conditoone policies are separate. thus performing poorly on tasks that can not be easily connected by language. (c) (ours) mirrors the hierarchieal structure of (b) but differentiates by employing self- supervieed encoder-decoder policy 10] and FSQ quantization (34] as behavior tokenizer. The apper-leval vision- vision-languaa language models produce sufersueer bchavior tokens, which are then conditioned by policy decoder to output actions, facilitating environment interaction. The behavior tokens are injected into the training corpus of ssiionlanguaag-actioo models, which enables end-to-end inference. This approach also eliminates the need for external language supervision and scales efficiently.
Open-world Agents in Minecraft. As LLMs have achieved remarkable reasoning results and understanding capabilities across various domains, the year 2023 has witnessed researchers adopting [55]
multiple LLM-based approaches to create open-world agents in Minecraft [56][55]]]] [47 44. Some
methods focus building policies for low-level skills 10 28, 2]. Building upon the low-level policies to interact with the Minecraft environment, Wang et al. [46]. Yuan et al. (51) and Wang et al. 47 focus on leveraging the pre-trained language models planners finish programmatic tasks with in-context learning. Wang et al. [44] adopts the life-long learning scheme and generates code as policies enable continual exploration. Some use expert trajectories and Minecraft corpus to fine-tune pre-trained vision language models for better embodied planning [36 54].
Conclusion
We'v ve presented Omn JARVIS, a novel VLA model that encompasses strong reasoning and ef- ficient decision-makken capabilities via unified tokenization of vision,, language, and actions in multimodal interaction data. The key ideas are learning behavior tokenizer (trajectory encoder) and de-tokenizer (IL policy decoder) using self-supervised learning on behavior trajectories and autoregressive modeling of tokenized multimodal interaction data using a pretrained multimodal lan- guage model (MLM). Evaluations the open-world Minecraft Universe demonstrate its impressive instruction- perruction-foloowing capabilities. Possible future directions include more in-depth investigation of behavior tokenization, language capabilities after VLA fine-tuning, and alignment concerns emerging from the unified interaction modeling and VLA capabilities.
Acknowledgments
This work is funded in part by the National Key R&D Program of China 00022ZD0160301 We thank grant from CCF-Tencent Rhino-Bing Bird Open Research Fund, One author is funded in part by NSF grants #IIS- 1944641, 1956411,,,, #CCF-1837129, an SRA from Meta and research gift from Amazon Alexa AI, and gift from RelationalAI.
10

---

References
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: visual language model for few-shot learning, arXiv preprint arXiv:2204. 14198, 2022.
[2] B. Baker, I. Akkaya, Zhokhov, Huizinga, J. Tang, A. Ecoffet, B. Houghton, Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv:2206. / 1795,, 2022.
[3] R. Bavishi, E. Elsen, C. Hawthorne, M. Nye, A. Odena, A. Somani, and S. Taşırlar. Introducing
our multimodal models, 2023. URL [tt.htts::// //wp://wwwwww/// www. www.:adeet.a.... wtt.aa///ll.... Booogg/uuu880 8b]
[4] A. Brohan, N. Brown, Carbajal, Y. Chebotar, Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv
preprint DXXiv2221200683
2022.
[5] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022.
[6] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess,
A. Dubey, C. Finn, et al. Rt-2: Vision-languagc-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 2023.
[7] T. Brown, B. Mann, N. Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1990 2020.
[8] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018.
[9] S. Cai, Z. Wang, X. Ma, A. Liu, and Y. Liang. Open-world multi-task control through goal- aware representation learning and adaptive horizon prediction, arXiv preprint arXiv:2301.1003: 2023.
[10] S. Cai, B. Zhang, Z. Wang, X. Ma, A. Liu, and Y. Liang. Groot: Learning to follow instructions by watching gameplay videos. arXiv preprint arXiv:23 08.3005.0....... 2023.
[11] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and
I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15099
2021.
[12] L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12799 2023.
[13] Y. Cheng, C. Zhang, Z. Zhang, X. Meng. S. Hong, W. Li, Z. Wang, Z. Wang, F. Yin, J. Zhao, and X. He. Exploring large language model based intelligent agents: Definitions, methods, and prospects. arXiv preprint arXiv: 2401.03428, 2024.
[14] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, al. Palm-e: An embodied multimodal language model. arXiv preprint M3Ki::3033.0376 2023.
[15] Z. Durante, B. Sarkar, R. Gong, R. Taori, Y. Noda, P. Tang, E. Adeli, S. K. Lakshmikanth, K. Schulman, A. Milstein, D. Terzopoulos, A. Famoti, N. Kuno, A. Llorens, H. Vo, K. Ikeuchi, L. Fei-Fei, J. Gao, N. Wake, and Q. Huang. An interactive agent foundation model. arXiv
preprint arXiv: 042.00992,, 2024.
[16] L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems Datasets and Benchmarks, 2022.
[17] R. Gong, Q. Huang. X. Ma, H. Vo, Z. Durante, Y. Noda, Z. Zheng, S.-C. Zhu, D. Terzopoulos, I. Fei-Fei, et al. Mindagent: Emergent gaming interaction. arXiv preprint arXiv:2309.0099//, 2023.
[18] W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov. Minerl: large-scale dataset of minecraft demonstrations, arXiv preprint arXiv:1901 13440, 2019.
11

---

[19] M. Heusel, H. Ramsauer. T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[20] M. Hinck, M. L. Olson, D. Cobbley, S.-Y. Tseng, and V. Lal. Llava-gemma: Accelerating multi- modal foundation models with a compact language model. arXiv preprint arXiv:2404.01331, 2024.
[21] S. Hu and J. Clune. Thought cloning: Learning to think while acting by imitating human thinking. Advances in Neural Information Processing Systems, 36, 2024.
[22] J. Huang, X. Ma, S. Yong, X. Linghu, et al. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12877 2023.
[23] W. Huang, Abbeel, D. Pathak, and Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. ICML, 2022.
[24] W. Huang, F. Xia, T. Xiao, H. Chan, Liang. P. Florence, A. Zeng, J. Tompson, Mordatch,
Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05660 2022.
[25] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0888
2020.
[26] P. Lewis. E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih. T. Rocktäschel, al.
generation for knowledge-intee intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-944 9474, 2020.
[27] Liang, W. Huang, Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint 27X1v:2009.07553, 2022.
[28] S. Lifshitz, K. Paster, H. Chan, J. Ba, and S. Mcllraith. Steve-1: A generative model for text-to-behavior in minecraft. arXiv preprint arXiv:2306 00937, 2023.
[29] H. Lin, Z. Wang, Ma, and Y. Liang. Mcu: A task-centric framework for open-ended agent evaluation in minecraft. arXiv preprini arXiv:2310.0883: 2023.
[30] H. Lin, B. Huang, H. Ye, Q. Chen, Z. Wang, S. J. Ma, X. Wan, Zou, and Y. Liang. Selecting large language model fine-tune via rectified scaling law. arXiv preprint arXiv:2402.02314, 2024.
[31] H. Liu, C. Li, Q. Wu, and Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[32] X. Ma, S. Yong, Z. Zheng, Q. Li, Y. Liang, S.-C. Zhu, and S. Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:22/0.07474, 2022.
[33] Y. Man, L.-Y. Gui, and Y.-X. Wang. Situational awareness matters in 3d vision language reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13678-13688, 2024.
[34] Mentzer, Minnen, F. Agustsson, and M. Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15506 2023.
[35] OpenAl. Gpt-4 technical report, 2023.
[36] Y. Qin, E. Zhou, Q. Liu, Z. Yin. L. Sheng, R. Zhang, Y. Qiao, and J. Shao. Mp5: A multi-modal open-ended embodied system in minecraft via active perception. arXiv preprint arXiv:2312.07472. 2023.
[37] C. Cafffll N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[38] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, al. A generalist agent. arXiv preprint wxXi:2205.00175. 2022.
[39] N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366 2023.
12

---

[40] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov. S. Batra, Bhargava, S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.00988,, 2023.
[41] T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprini
arXiv:1812.01171. 2018.
[42] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[43] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156-3164, 2015.
[44] G. Wang. Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.
[45] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2022.
[46] 7. Wang, S. Cai, G. Chen, A. Liu, X. Ma, Y. Liang, and T. CraftJarvis. Describe, explain, plan and select: interactive planning with large language models enables open-world multi-task agents. In Proceedings of the 37th International Conference Neural Information Processing Systems, pages 34153-3418 189,18997778 2023.
[47] Z. Wang. S. Cai, A. Liu, Y. Jin, Hou, B. Zhang, Lin, Z. He, Z. Zheng, Y. Yang. X. Ma. and Y. Liang. Jarvis-1: Open-world multi-tass task agents with memory memory-angmented multimodal language models. arXiv preprint arXiv: 2311.05997, 2023.
[48] Z. Wang, A. Liu, H. Lin, J. Li, X. Ma, and Y. Liang. Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation. arXiv preprint arXiv:2403.0531 2024.
[49] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. 36th Conference on Neural Information Processing Systems (NeurIPS 2022), 2022.
[50] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[51] H. Yuan, C. Zhang, H. Wang. F. Xie, P. Cai, H. Dong, and Z. Lu. Plan4me: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303. 1..303..6663.. 16563, 2023.
[52] C. Zhang, K. Yang, S. Hu, Z. Wang, G. Li, Y. Sun, C. Zhang, Z. Zhang, A. Liu, S.-C. Zhu, et al. Proagent: Building proactive cooperative ai with large language models. arXiv preprins arXiv:2308. 11339, 2023.
[53] H. Zhao, Z. Cai, S. Si, X. Ma, K. An, L. Chen, Z. Liu, S. Wang, W. Han, and B. Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915 2023.
[54] S. Zheng, Y. Feng, Z. Lu, et al. Steve-eye: Equipping based embodied agents with visual per- ception in open worlds. In The Twelfth International Conference on Learning Representations, 2023.
[55] X. Zhu, Y. Chen, H. Tian, C. Tao, W. Su, C. Yang, G. Huang, B. Li, L. Lu, X. Wang, et al. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305. 144, 2023.
13

---

A Training Details
OmniJARVIS. We utilized the SFTTrainer class from the TRL library by Hugging Face to train the VLM model. The learning rate was set at 1.4e-5, and cosine learning rate scheduler was employed. The weight decay parameter was set to with a warm-up ratio of 0.03. Training took place on 8 A800 GPUs with FSDP, with a batch size of and gradient accumulation steps of 4 using 6 precision. The training lasted for one epoch on our generated dataset. The raw interaction dataset comes from the sections 6xx, 7xx, and 10xx of the contractor dataset provided by OpenAl [2] and the recording interactions of JARVIS-1 Agents
Behavior Tokenizer. Each frame in our experiments has resolution of 128x 128x128 pixels. We segmented each episode into multiple trunks, with each trunk consisting of 128 frames. The learning rate was set at 0,00004, with weight decay of 0,001. The batch size was configured to 2, and training was conducted sections 6xx, 7xx, 9xx, and 10xx of the contractor dataset provided by OpenAl [2] The precision for training was set bfloat16.
cluster of eight NVIDIA 3090 Ti graphics cards. The training dataset comprised
B FSD Computation
This section outlines the computation of FSD score between the generated videos and human gameplay recordings. First, we divide the videos into trunks 128 frames. For each segment, we sample 16 frames, with 8 frames in between each sampled frame. These sequences of 16 frames are then fed through the video encoder MineCLIP [6] obtain 512-dimensional video embeddings. Finally, the score is calculated according to [99 between the embeddings of the generated videos and the reference videos.
We compute FSD scores between and within sets of videos using three distinct tasks, as illustrated in Figure B.T A noticeable gap exists between the FSD scores calculated within the same set of videos and those calculated between different sets. Furthermore, the metric exhibits relative insensitivity to the number of videos used for computing the score, demonstrating the validity of our proposed metric.
10
10
FSD Scote
10"
10"
troll-2 & 18sk-1
FSD
PSD
book-2 & took-
10
lask-]
took-3
task-1 & task-7
16'
task-1 tnok-1
tnsk-1 & toak-1
task-3 & tiok-2
20
40
60
80
100
60
100
Number
Number of videos
Number vidlos
Figure B.1: FSD scores between and within sets of videos for two distinct tasks. The horizontal axis represents the number of videos used for computing the scores, and the vertical axis depicts the corresponding score.
C Benchmarks Details
C.1 Programmatic Tasks
Table 6: Description and setting of programmatic tasks.
Task
Maximum
Task
Groop
Item List
Steps
Prompt
Wooden
10
3000
Make (item] from empty inventory.
stick, crafting.ttii tuble, chest, ladder, howl, button, door, hoat,
Food
5
(item).
cooked chicken, cooked mettoe, cooked, parkehop, cooked beef, bread charcoal, smaker, stone swond, fumace, torch
Stone
5
3000
Craft fitem).
Iron
5
6000
Smelt iron ingois craft (Item).
iron. sword, ingot, bucket, sugget, shears
Diamond
5
12000
down and cralt (item).
diamond pickaxe, diamond shovel, diamond boc, diamond axc, dismond
14

---

Table 7: Embodied Question Answering Examples.
Category
Question
Answer
Planning
How to obtain bucket with emply inventory step-by step Minecraft?
mine log without tool...
Planning
How obtain cooked heef with empty inventory step step-hy-ste hy-step Minecraft?
kill cow to obtain beef...
Knowledge
How many materials do need collect to make 2 iroe ingots in one go?
To make I irue ingot, you need liron ore and...
Knowledge
materials What the materials make iron helmet in Minccraft?
diamond pickaxe Minectaft?
diamond,
Knowledge
5 iron ingots.
Knowledge
What are materials make! golden ane Minecraft?
3 gold ingol, stick.
Knowledge
What materials
wooden shovel in Minecraft?
planks, stick.
Knowlodge
What are the materials make! becad in Minccraft?
3 wheat.
Reasoning
mined stoee pickuxe Minecraft?
Diamond only mined iron...
Reasnning
Can coal he mined with an iron pickaxe in Minecraft?
Yes. Coal can he mined...
Reasoning
Can obsidian be mined with un irom pickaxe in Minecraft?
No. Diamond can only be mined with iron...
Reasoning
lapis lazuli he mined with diamond pickaxe in Minecraft?
Yes. Lapis lazuli cun be mined
Reasoning
Can emeralds be mined with stone pickase Minccraft?
No. Emeralds can only be mined with an iron...
Decision Docision-making
<image><inventory>
diamonds,, behavior should
pickaxe
C.2 Embodied Question Answering Benchmarks
The embodied question-ans answering benchmarks consist of questions and instructions for Minecraft benchmarks, consisting of over 100 questions knowledge question answering, embodied planning, and math reasoning.
To enhance the L.LM-bas b-ssedd agent's understanding of Minecraft, we gathered a large dataset called 300k (mc-eqa-300k). Initially, we extracted language corpus from the Minecraft Wik and DigMinecraft website to create prompts for the language model (ChatGPT). This resulted around 50k question-ansee pairs. Using these as seed questions, ChatGPT generated approximately 250k diverse questions various aspects of Minecraft. To ensure accurate responses and avoid hallucinations common with existing models like self-instruct (45), we employed a retrieval augmented generation (26) method known RAT [48] to generate accurate answers. The resulting question question-answerte answering dataset, along with interaction data, form our training dataset for Omn JARVIS.
Lemncrraf-EEbodiice Emmooiidd-uuestion--
Example of Embodied Question Answering Generation for Reasoning Group
question: Can stone pickaxe be used to mine diamonds?
answer: No, a stone pickaxe cannot be used to mine diamonds in Minecraft. An iron pickaxe or better
required to mine diamonds.
question: Can golden pickaxe be used to mine diamonds?
answer: In Minecraft, you cannot use golden pickaxe to mine diamonds. Diamonds are one of the hardest materials in the game and require an iron pickaxe or better to mine. Using a gold pickaxe on diamond ore blocks will cause them to hreak, but you won't 't receive any diamonds as a drop. You 'll just get the diamond ore block itself.
To successfully mine diamonds, you should use iron, diamond, netherite pickaxe. These pickaxes have the required hardness to break diamond ore blocks and yield diamonds when mined.
Phurps//miineeratt.iiii
"btpss/wwww.dgminecrat.cc
15

---

Example of Embodied Question Answering Generation for Planning Group
Give you nothing in the inventory, how to ohtain diamond pickaxe from empty inventory, generate a step-by-step plan, from punching trees, and descrihe every step in detail.
question:
Ohtaining a diamond pickaxe Minecraft involves several steps, starting from punching trees and eventually progressing to mining diamonds. Here's 's step-by-step plan to obtain a diamond pickaxe from an empty inventory:
answer:
Step I: Punching Trees**
**Gather Wood: Start by punching trees collect wood blocks. You'l 'll need at least logs to proceed.
*"Step 2: Crafting Basic Tools**
2. **Craft Wooden Planks:* Open your inventory and turn the logs into wooden planks by placing them in
the crafting grid (4 wooden logs 16 wooden planks).
3. **Craft Crafting Table:* * Use the woaden planks craft crafting table. Place 4 wooden planks in the
crafting grid to create
4. **Craft Wooden Pickaxe:** With the crafting table, craft a wooden pickaxe using wooden planks and sticks. Place 3 wooden planks in the top row and 2 sticks in the middle row.
Step 3: Finding Stone
**Mine Stone:*s Use your wooden pickaxe to mine stone blocks. You'll need at least stone blocks to proceed.
**Seep 4: Upgrading Tools**
6. **Craft Furnace:* ** Use the crafting table to create furnace using 8 pieces of cobblestone. Place the cobblestone in all squares of the crafting grid except the center square.
**Smelt :obblestone: Use the furnace smell the cobblestone smooth stone. Place cobblestone in the top slot of the furnace and fuel (wooden planks, sticks, etc.) in the bottom slot. Collect the smooth stone when it's ready.
** Step 5: Crafting Stone Pickaxe**
**Craft Stone Pickaxe:** With the smooth stone, craft a stone pickaxe using cobblestone sticks. Place 3 cobblestones the top row and sticks the middle row.
Step 6: Upgrade to Iron Pickaxe**
9. **Find Iron Ore:*# Find iron ore blocks underground (usually lower depths) and mine them using your stone pickaxe. You'll '1l need at least iron ore blocks.
10. **Smelt Iron Ore:** Place your iron ore the top slot the furnace and use wooden planks as fuel the bottom slot. This will smelt the iron ore into iron ingots.
11. **Craft Iron Pickaxe:** With the iron ingots, craft iron pickaxe using iron ingots and sticks. Place 3 iron ingots in the top and 2 sticks in the middle column.
**Step 7: Mining D
12. **Find Diamonds:** Use your iron pickaxe mine for diamonds. Diamonds are found deep underground, usually between levels 5 to 12. They appear as blue, sparkly ore blocks.
**etpp 8: Crafting a Diamond *
13. **Cratt Diamond Pickaxe:*** Once you have collected at least diamonds, craft diamond pickaxe using top row sticks in the middle row.
diamonds and sticks. Place diamonds
**Seep 9: Using the Diamond **ckaxe**
14. Mine Ores:** The diamond pickaxe is more durable and efficient than lower-tier pickaxes. Use it to mine variety of ores, including iron, gold, redstone, and more.
And that's it! You've ve successfully obtained diamond pickaxe from an empty inventory by following these steps. Remember that Minecraft's world generation and mechanics might vary slightly depending on version you'te re playing, so adapt the plan as needed.
16

---

Example of Embodied Question Answering Generation for Knowledge Group
question: How to craft diamond pickaxe step-hy-step. describe every step in details. To craft diamond pickaxe in Minecraft, you will need follow these steps:
answer:
**Gather Resources:**
Start by gathering the necessary resources to craft a diamond pickaxe. You'l 'll need three diamonds and two sticks.
Diamonds can be found deep underground, typically between levels to 15 in the Overworld. You'l 'll need an iron pickaxe or beller mine diamonds. Use this iron pickaxe to mine the diamond ore blocks.
**Crafting Sticks:**
If you haven't already, craft sticks from wooden planks. Open your crafting table and place two wooden planks vertically create four sticks.
**Crafting the Diamond Pickaxe:**
Opee your crafting table, which consists a crafting grid.
Place the three diamonds in the top row.
Place two sticks in the middle row, one in the center slot and one in the slot just below it.
Your crafting grid should now look like this:
[D][D[[
|[SI
[[SI[]
[D] represents diamond.
[S] represents stick.
Once you't placed items in correct pattern, you'll see diamond pickaxe appear in result box the crafting table.
4. **Collect the Diamond Pickaxe:
Click or drag the diamond pickaxe from the crafting table's result box into your inventory.
Congratulations! You now have a diamond pickaxe in your inventory. Diamond pickaxes are extremely valuable Minecraft because they can mine most blocks faster than other pickaxes and are required to mine certain valuable resources like obsidian. Be sure to use it wisely and take care of it, as it's a valuable tool in
the game.
17

---

C.3
Open-ended Instruction-Folloiin Tasks
Prompts for Open-ended Instruction-followinng Tasks
Cook the beef with furnace and recycle the furnace. If you meet night, place and use the bed for sleeping.
Explore caves, mine resources, and craft items Minecraft to progress and survive.
3. Gather resources, craft tools, and cook food in Minecraft.
Place a torch on the wall light the environment. Collect and picking it up when you leave.
5. Craft oak boat and use it travel
Obtain resources for building and survival by gathering materials and farming resources.
7. Consistently interact with chests manage inventory contents.
Explore and gather resources in Minecraft.
9. Collect and mine azure bluets, deal damage mobs. and deffat mobs in the game.
0. Do the following tasks sequentially: 1. Gather oak logs and oak leaves from trees. Use oak logs to create oak planks and then crafting tablc. 3. Create sticks from oak planks using the crafting table. 4. Craft wooden and wooden pickaxe using sticks and oak planks, 5. Collect materials like mushrooms and Mine grass, tall grass, and stone using the
brown mushrooms by mining blocks with the wooden wooden tools for resources.
1. Harvest sugar to obtain multiple sugar cane pieces.
2. Plant and consume wither roses repeatedly.
3. Harvest wheat seeds, plant them, and use the harvested wheat seeds to feed animals or craft items such as bread.
14. Trade with a villager by giving emeralds and books to receive enchanted books as well as new emeralds and books.
5. Mine using an iron pickaxe and pick up ice block oblained.
6. Open chest in the game to access manage inventory items.
D Prompt for Instruction Generation
Prompt D. Proomp for Instruction Generation
"IIstructioon***
This is a paragraph of description of the player's gameplay in Minecraft. The caption summarizes the current environmental state and agent behavior, with the timestamp indicating which frame of the video this caption from. Please summarize what tasks the agent completed throughout the entire video. Please guess what instruction task the player received to exhibit such behaviors. This should clear and in details. *MMPORTANT**
DIRECTLY output the task. DO NOT repeat user input. DO NOT add additional explanations or introducement in the answer unless you asked to.
* *Observation
Stats cimnerrff.custommmmneeerr inttrinnttrrrraiitttttt_tccc _iitrarfrfcucceeeeeeeeeee furnace happens. Gui is open. New stats
minecraft.craf item:iinecraft.ookee beef happens. Get new item: cooked beef*9. Get pickaxe happens. Stats minecraft.min mine block:minecrat.ffumaa furnace happens. Stats minecraft, Siinecraft.iccuppmeee pickup:minecraf..fen furrace hap- pens. new item: 1 Stats minecraft.ss item:minecraft. white bed happens. Stats minecraft. mine wwt.tmmioiiiiiineeeeeeeeeeeee www.k white bed happens. Stats minccraft pickupmminecraft. white bed happens.
new item: stone 1 Stats minecraft.use htw.iuussitemmnnnernfffttttto wtw.ssttooeeeeeeeeeeeeeeeeee
Get new item: white_bed"1 New stats minecraft. use wtt.:uiceitmmmicccraff...m. woo.eeecraa....ooedddeeeo.e beef happens. Consume cooked 1 **Tak***
1. Interact with furnace smelt cooked beef and eat the cooked_ Place white bed and sleep survive the night.
*#Ossrvvation*. [observation)
18

---

Example of Instruction Generation
Example:
** **Obeervatio**:
Consume 1. Stats use item:ehest happens. Consume chest* Stats use item:chest happens. Consume 1. Stats use item:chest happens. Consume chest* Stats use item:chest happens. Consume item: 1.est*1. Stats use item:chest happens. Stats custom:open chest happens. Open Game 2D GUL Consume chepsteee_eeetteeeeeppeeee
oak 2lanks*24. 24. Consume item: birch, planks Stats custom:open chest happens. Open Game 2D GUL. Consume lapis_lazuli222
Consume item: 18n__inot*18. 18. Consume item: potato 30. Consume item: 9. Consume item: wheat 4. Stats custom:open_ccee _ceceet__esstteeeee happens. Consume oak vak_planks**6 Consume item: sak__laank**4. 44. Stats custom:open cheet happens, Open Game 2D GUL. Consume item: granite*20. Stats custom:open_chett chest happens. Open Game 2D GUT. Consume item: oak_ lokksapling*2. Consume item: birch serc__sapinn*4. Consume item: wheat seeds $ 12. Consume item:
Consume item:
sestoom:open_chestt cheest happens. Get new item: 4. Get new item: carrot*9. Get new item: potato*30. Stats custom:open cheest happens. Open Game 2D GUI. Consume item: wheat"4. Consume item: carrot*9. Consume item: potato Get item: potato 15. Stats custom:open chest happens. Open Game 2D GUL. Consume item: laiis_laulii64. Stats custom:interact_ soicctttttthhhheeeeee ffrraccurcceeee happens. Open Game 2D GUL Consume item: potato 15. Stats custom:open_chhss __teestttttteeeeeeee happens. Open Game 2D GUT. Get new item: iron_ rron_ingot* 18, Stats custom.interact_witeee wwwt__carfttng_tttte happens. Open Game 2D GUI. Consume item: stick*28. Get new item:
bread*1. Stats custom:open peceoopppepeesssttteeeeeee happens. Open Game 2D GUT. Stats custom:open
22ick*22. Consume item: iron iron_inggon pon__ngot*18.
**Generated #;
Task: Organize and manage inventory by storing items in chests, crafting various items, and using furnace to smelt resources efficiently. Also, focus on gathering resources like wood, ores, food items, and plant materials for crafting and survival purposes.
19

---

E Prompt for Thought Generation
Prompt E. I.Propppt for Thought Generation
A player is playing Minecraft. need you to give thought about what behavior it should take next given current situation. Here are some demonstrations:
Task: "Obtain a diamond"
State: "The image captures scene from the popular video game, There is grass block in front of the agent."
Inventory: !'dirt': 10)
Memory:
Thought: "The player has nothing in inventory. should first go to harvest some oak logs. However there is grass block in front of the agent, it should mine first to find a way. So hehavior should be saann__hoo:ggrrss1]] )."
Behavior:
('mine_blene block:grass':
Task: "Obtain iron pickaxe"
State: "In the image, player in the video game is standing in a dark cave." Inventory: { d'dirt': 20, 'stick'; ; 10, 'iron_ore"; 5, 'furnace"; 1, crone_picaaxe:: 5, ftruftittem iron ores in its inventory, it should smelt them to get iron ingots. However, it does not have enough coal to smelt the iron ores. The player should mine some coal ores first. And using the stone pickaxe in the inventory can help to mine the coal ores. So the behavior should be steemsstoe_pickaat mine block:coal _ore: Behavior: ['use_ item:stone piinsstteeepppcaxx''. mmine_block:cool ore':
Memory: { ('mine_blee balin__boocciroooofff
Thought: "The player
Task: "Harvest logs"
State:
"The image captures moment in the video game The player's character, standing in the center of the frame, is holding crafting table in their hands. The crafting table, which is the main focus of the image, gray and has crafting grid top of it."
Inventory: { ['oak_log': 20, 'stick': 8) Memory: ('use item:wood axe": 'break item:wood. _xe': 1)
40, 'craft item:stick':
8, "mine block:oak log': 20,
Thought: "The player needs harvest more logs. However the last memory shows that the player has player should craft new wood axe first. The player has enough logs in the inventory and the crafting table in the inventory can help to craft the wood axe. So the behavior should be (use ses_ittm.rratiigg table:1 ].."
broken the wood axe.
Behavior: ("use_ jtt__jtemmm crafting_table:.. 1)
demonstrations, "Task" is the goal of player. "State" descrihes image the player facing, "Inventory" is its current inventory and "Memory" contains past behaviors taken by the player. "Memory" is sorted by time, with the most recent behavior at the end. You should pay attention to recent behaviors. According these information, players first generate thoughts about what to do next(" "Thought"") and then (aarrdinnly""Beaviiiii
take behaviors second demo, the thought considers needs of the task and current inventory; the third demo considers the memory of the player and identify the need of crafting a new tool to replace the broken tool. Now think about the following situation:
the first demo, the behavior is not relevant the task; in the
Task: [task]
State: (state)
Inventory: |inventory]
Memory: [memory]
Thought: ()
Behavior: (hchavior)
Given current situation and the behavior the player will take, output simple thought that will di- rectly lead this behavior. Please carefully revise the need of the task. current inventory and recent memory of player. sure to explain every part of behavior. The output format should be "Thought: So the behavior should be (bchavior)".
20

---

Example of Thought Generation
Example:
Task:
"The player was instructed mine various resources and craft tools in Minecraft: Start by mining coal ore and crafting cooked beef from 2. Smelt iron ore and cook food in furnace. Mine stone to collect cobblestone. 4. Craft stone pickaxe and use it to mine various ores like coal, iron, and diorite. 5. Create torches from coal and sticks. 6. Craft stone pickaxe and an iron pickaxe. 7. Use the iron pickaxe to mine granite and gather resources. 8. Interact with crafting table craft items like iron pickaxe, torches, and iron ingots. 9. Utilize tools like pickaxes mine stones different ores efficiently. 10. Gather various resources like coal, iron, cobblestone, diorite, and granite. 11. Keep crafting and mining to progress in the game. These actions showcase a cycle of resource gathering, processing, and crafting to advance the player's capabilities and inventory in the game." State: "The image captures center the frame, is holding a crafting table their hands. The crafting table, which the main focus of the image, is gray and has crafting grid on top of it. In the crafting grid, there are several items arranged in rows and columns. Starting from the top left, there's book. followed by loom in the middle, and furnace the bottom. The crafting table is set against black background. which contrasts with the gray color of the table and the items on At the bottom of the image, there's 's red banner with the "Crafting" written on This banner adds a pop color to otherwise monochrome image. The overall composition of the image suggests that the player is in the process of crafting something, possibly book or a loom, using the items in the crafting grid."
in the video game Minecraft. The player's character, standing in the
Inventory: ( l'wooden shovel': 1, 'wooden axe': 1, "cobblestone': 51, 'dirt': 14, 'andesite': 23, 'iron ore': 'coal': 31, 'stick': 54, 'birch log": 5, prrck__lannk'' 47, 'furnace': 1, "crafting table": 1, "granite":
'diorite": 7, "wooden__pp
Memory: wooden_pickaxe"" pickaxe": 1)
tlem:stone_pickaseeee
63, "mine bww.bbbooocccoorre..
9. "pickup:coal": 9.
'mine wlo.k: torch': 1. "use item:torch'; 'pickup:torch': 1. 'mine_ block:granite" 4, 'pickup-granite.
4,
38, "pickup.cbbllstonee: 37, "mine blook::dioooiiie 8, "pickup:diorite:: heemsstone_pcckxxe' 1)
"use itte::.ohblestomee 3, 'mine_ block:iron_ore 3, pickuppiron ore": : 3, 'break item:stone pickaxe':
Updated Inventory: "wooden_ssie :aalen__hhvel::: 1, 'wooden_axx axe': 1, 'cobblestone': 51, 'dirt': 14, 'andesite": 23,
'iron ore': 7. 'coal': 31, 'stick': 54, "birch_ log': 5, "birch planks': 47, 'uurnace:: 1, "granite": 4. 'diorite': 7,
'wooden piccaeeppp.eeeeep 1)
Updated Memory: (use item:stone pickaxe': 63, "mine blockrcoal ore':
"pickuprcoal":
"mine_ wwtpe//oooowwlaooo.eeeoommo _torch' 1, 'use uis_iteemttocc"' : 3, 'pickupstorch': 1, 'mine_ mmine_boock-granie 4, 'pickup-granite::
4,
38, "pickup.cobblestone:: 37, blonc__boociiiorite"" 8, 'pickup:diorite:: 7,
'usc_ joo_item:cobblestone:: 3, mnin__bocck:iro_.og
pickuppiroo_ooe'' 3, 'break foaikk_ttmttoe__ickeeeee
'use iitem:craating.tte........ [able')
Generated Thought:
Thought: The player's current inventory includes various resources like cobblestone, coal, iron ore, and granite, along with essential tools like wooden pickaxe, shovel, and axe. Recent memory indicates the player has been actively mining various blocks and collecting resources, including coal, granite, and iron However, memory
shows the player's stone pickaxe has broken. progress efficiently and craft necessary items, such as torches and better pickaxes, player should utilize the crafting table. So behavior should be mnss_itemcrratting... :aanle" 1}.
21

---

Prompt for Memory Summarization
Prompt P.I:Prompt for Memory Summarization
A player is playing Minecraft. The situation of player contains 4 parts: task, state, inventory and memory. Under this situation, the player will take a behavior. And after this behavior, the player's 's memory will be updated to "Updated Memory". need you to give subpart of the player's updated memory that is most relevant its task.
Task is the goal of the player. State describes the image the player is facing, Inventory is its current inventory items. Memory contains its past behaviors, each item in memory is its past behavior and the number of this behavior. The memory is sorted by time, with the most recent behavior at the end. There are mainly 9 types of behavior:
craft item:x' means craft item 'dropx'' means drop an item x;
"use iis_iitem.x means to use an item x;
+ pickkp:x' means to pickup an item x; 'custom' means custom its playing status;
"mine block:x' means mine block
'kill entity:x' means kill an entity x;
+ entity killed
_illed_by:x' by:x' means the player is killed by an entity x;
+ "break_itens means an item got broken.
Here is the player's 's current situation:
Task: [task]
State: (state)
Inventory: (inventory)
Behavior: [behavior]
Updated Memory: (updated memory)
I need you to summarize what the player has done to complcte the task according to the updated memory. Please make sure every part your summary is relevant to the task. The output format should be: "The player first... ... then ... and finally Then in new line, try summarize which stage the task the player is in according to the memory.
22

---

Example of Memory Summarization
Example:
Task: "Gather various resources including andesite, granite, diorite, coal, iron ore, and cobblestone using stone pickaxe. Craft and use torches illumination. Upgrade from a wooden to stone pickaxe craft a stone sword for defense. Explore and mine in a systematic way, ensuring to light up the environment with torches and replacing tools they wear out."
State: "The image captures moment in video game, specifically Minecraft. The scene is set in dimly cave, with wooden pillar standing prominently the foreground. The player's inventory and score are displayed the top left corner of the sereen, providing glimpse into the player's 's progress in the game. In the bottom right corner, the player's s health and hunger bars are visible, indicating the player's 's current status in the game. The rest of the screen filled with series of lines of text, each line representing command instruction from the game. These commands seem be related the player's movement and interaction with the environment, guiding the player through their adventure in Minecraft. The image a snapshot complex digital world, where every command and action carefully calculated and executed. It's testament the immersive and engaging nature of video games like Minecraftt"
Inventory: ('stone pickaxe': 1, 'wooden axe': "oak log': 8, "stone _wondd' 1, 'andesite": 8, 'coal': 13, "oak_phan oak__lankss:: 2, "charcoal": 'torch': : 27, 'dirt': 1, 'furnace:: "cobblestone': 108, 'stick': 21, mrodden_pickaxe 1) Behavior: uuse_item:woode__ggg
'iron_ore": 2, granite"; ; 15, 'diorite'; shinn_block:oak_BP
1,
1)
Updated Memory: ( I'pickup:dirt': 1, '''mine block:oak leaves': 4, 'mine blocksandesit':: 8, 'pickup:fumace': "use item:stone pickaxe': 169, "mine_ block:stone': 125, 'craft itemzcharcoal': 4, 'use ittemttrch': 14, "mine block:oak log': 21, "mine block:furnace:: 1, 'craft item:stone siccllepppeee.eeeee 2, "pickuproak planks''
2. "craft_itee jttd:wwooden axc': 1, "caft jittt:sttettkk' 40, "use itom:woodee__piiiioo pickaxe": 13, "craft_ itemaaite::
0, "pickupaandestee: 8, "use
1,
1, 'mine_blen baane_bbooccrratng_tttaa table":
"craft_ite : 1, "pickup.crafting table': 2, 'pickup:coal": 21. 'mine_ block: mbossk'
'craft sanffiitem:torch:: 40, 'pickupzoak log': 23, pickup:cobbleettone: 124, "craft_ latpiiitt__plaamllaaa
36, "craft item:erafting table': 1, "use item:crafting table': 3, 'use 'use_item wood:woodenn axxe: 18, moaf_ittemwooden pickaxe": 1, "mine lime__boockccal ore": : 21, "mine_mlock:_ppeeemmmmmmm block:granite": 15, 'pickup-granite:: :
15, 'drop:stone_ pickaxe":
= 1, pickupp:tone pickaxe': 1, "mine block:diorite': 9, "pickuprdiorite:: 9.
"break item:stone pickaxe': I, craft item:stone sword': 'mine block:iron ore': 2, "pickup:iron ore': 2. "mine block:wall torch': "rickup:torch': I, "min block:oak fence": Generated Summary:
The player first gathered various resources including andesite, granite, diorite, coal, iron orc, and cobblestone using stone pickaxe. Then, they crafted and used torches for illumination, upgraded from wooden to stone pickaxe, and crafted stone sword for defense. Finally, they explored and mined systematically, ensuring to light up environment with torches and replacing tools as they wore out. The player is in the stage of gathering various resources, crafting torches, upgrading tools, and exploring systematically.
G Examples of OmniJARVIS Interaction Process
The tokens with blue color are prompted from the environment observation and user, and the tokens with red color are generated by OmniJARVIS.
23

---

Example of OmniJARVIS Interaction
TASK:
Mine stone with an iron pickaxe, gather cobhlestone, then switch to gravel, mining it with a wooden shovel collect
with the iron pickaxe, place torches for visibility, then proceed to mine iron ore as well. The agent collects items along the way, including gravel and iron ore while constantly gathering cobblestone throughout the process.
Return mining
MEMORY: I have mine 13 stone blocks and I gravel block. INVENTORY: iron pickaxe: 1, wooden shovel: wooden
cobblestone: 156, crafting table: 1, andesite: 23, iron_ore: coal: birch_log: 5, birch_ppankss planks: 47, furnace: 1, gravel: granite: 12, diorite: 28, wooden_ pickaxe:: 1, stone_ pinne_e 1, dirt: 14, stick: 43
OBSERVATION:
CAPTION:
The image captures a moment the video game Minecraft. The player's 's character is standing at the entrance of a cave, which is characterized by its dark gray walls and ceiling. The floor of the cave is contrasting light gray, with few small rocks scattered about. In the bottom left corner of the sereen, the player's health and hunger bars are visible, indicating screen displays the player's inventory, showcasing variety items including pickaxe, a sword, food. The most striking feature the image the text Tron Pickaxe that appears in the bottom right corner of the screen. This suggests that the player might be using iron pickaxe, common tool in the game for mining. The precise location of the objects and their relative positions contribute the immersive gaming experience.
status of the character. The top right comer of the
some
THOUGHT: The player needs to mine iron ore with an iron pickaxe to progress towards the task. Recent memory indicates a focus on using the iron pickaxe for various mining activities. Since iron ore a key component in the task's 's objectives and the player already possesses the required tool, the next step should be mine the iron So the behavior should iis_iitemiiro_ saantiron_pickxce,, 'mine_ chine_blockciron_com CONTROL: TOKENIBEHAVION 12372)....
24

---

Example of OnniJARVIS Interaction
TASK:
Task: 1. Smelt iron ingots in furnace. 2. Craft iron pickaxe from iron ingots and sticks, 3. Mine diamond blocks using the iron pickaxe, collecting diamonds. Mine redstone ore, collecting redstone. 5. Mine diorite blocks. Craft a diamond pickaxe using diamonds and sticks. Craft additional diamond pickaxes using cobblestone and the previous diamond pickaxe.
MEMORY: I have smelt 3 iron ingols.
INVENTORY: oak log: 9, andesite: 3, dirt: oak planks: 8, iron ingot: cobblestone: 124, stone 1,
stone_ pickkee: granite: 15, wheat_seeds: 1, diorite: 9, stick: 7
OBSERVATION:
The image captures scene from the video game Minecraft. the center the frame. a player's hand is seen holding a torch, which is ablaze with orange and yellow flames. The torch is being used to ignite pile of wooden planks that are stacked haphazardly on top of cach other. The planks are brown and appear be made of wood. The player's 's health and hunger bars visible in the bottom left corner of the sereen, indicating the player's status the game. In the bottom right corner, the player's inventory displayed, showcasing the tools their disposal. characterized by stalactites hanging from the ceiling and stalagmites rising from the floor. The walls of the cave are made of gray stone, providing stark contrast to the wooden planks. Overall, the image provides snapshot of moment in the game, capturing the player's interaction with the environment and the tools their disposal.
CAPTION:
background of the image reveals cave-like structure,
THOUGHT: The player holds enough iron ingots in their inventory, fulfilling the task requirement to craft iron ingots in furnace. However, to proceed with the subsequent tasks which involve mining diamond ore and redstone acquiring iron pickaxe is crucial. Considering the current inventory, the player possesses the necessary materials craft iron pickaxe iron ingots and sticks. Crafting the iron pickaxe now will enable the player to move efficiently towards their goal mining valuable resources like diamonds and redstone. Therefore, the immediate action should be to craft an iron pickaxe. So the behavior should be craft item:iron pickaxe.
CONTROL: TOKENIBEHAVIOR-117
25

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by review- ers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: [NA]
26

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might specific model and empirical evaluation, may be necessary
suffice, or if the contribution to either make it possible for others to replicate the model with the same dataset, or provide access to the model. general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (e.g., with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
The data and code will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
27

---

Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ public/guides/Coonee Submi wwt.miissioiii ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not possible, so "No" is acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (nttps:
http:///ppp..ipiiiecceeeeeeeeeee httpe////geeeee///eeuuuiooooo Cttp///ooeSummmmeecee ssi on olccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
28

---

8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:://///oooooooohtccccaaaeaa.
Question:
Answer: [Yes]
Justification:
[NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
[NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
29

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficieccy and accessibility MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [No]
Justification:
[NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperr saperswithcode com/ datasets has cu- rated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
30

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "EJUcVnxEyo", "reviewer": "Reviewer_LGfx", "summary": "This paper presents a Vision-Language-Action (VLA) model, OmniJARVIS, for open-world instruction-following agents in Minecraft. OmniJARVIS leverages unified tokenization of multimodal interaction data to enable strong reasoning and efficient decision-making capabilities. This work introduces a behavior tokenizer to encode behavior trajectories into compact representations that could be effectively modeled with other modality tokens via autoregressive transformers. The experiments are conducted in a variety of tasks in the Minecraft environment, along with some analysis of design choices and scaling properties.", "questions": "Since the authors leverage GPT3.5 to augment previous datasets with more language labels, what is the price/cost for constructing such a large dataset with GPT APIs?", "limitations": "There’s no separate \"Limitations\" section in this paper. The authors are encouraged to point out that the proposed approach is only validated in the Minecraft environment, and the generalization and transferability to other scenarios are not explored in this paper.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow. \n\n2. Illustrations of the pipeline are informative and clear. The authors demonstrate their proficiency in pipeline illustrations in Fig.1 and Fig.3, which contribute to improving readability.\n\n3. The proposed approach is scalable with respect to model sizes. As depicted in Fig. 4, a larger model results in lower evaluation loss, showcasing the scalability of this approach.\n\n4. The authors greatly utilized established LLMs to construct a large-scale multi-modal interaction dataset. The proposed strategy of data collection could provide inspiration for future research in multi-modal embodied understanding.", "weaknesses": "1. Incremental design with limited novelty. To my knowledge, the key idea of this work is tokenizing observations into compact behavior representations. However, the detail of the tokenizer works similarly to the previous work GROOT. The most notable difference between the two methods is that GROOT utilized continuous latent while this work uses quantized one (from lines 88-89). In my view, the idea of quantizing the action space seems incremental to the original GROOT and cannot adequately support the innovative quality of this paper. The authors are encouraged to add a separate section/subsection to detailedly discuss the difference between these two works.\n\n2. Lack of training/inference efficiency analysis and comparison. The authors claimed the efficiency of this approach multiple times throughout the paper, such as in lines 102-104. However, there is no direct comparison of the actual training/inference speed/memory cost between different approaches. Some quantitative supporting evidence could be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: Novelty concerns.\n\nThank you for your comments. As detailed in the General response, our core contribution is to propose a novel Vision-Language-Action (VLA) model architecture termed OmniJARVIS, as shown in **Figure 1 of the supplementary PDF**, aiming at resolving the issues including inference efficiency and language and annotation bottleneck. Please let us know if you still have further concerns about the novelty and we're more than happy to assist!\n\n> W2: Training/Inference efficiency analysis and comparison. \n\nThank you for your valuable feedback. We’ve added comparisons of inference speed, memory usage, and model parameters to **Table 4 in the supplementary PDF**:\n\n**Inference Efficiency Comparison**:\nThe inference efficiency experiments are conducted on the same workstation with one RTX3090 GPU.\n1. Inference Speed: Based on Minecraft’s rendering speed of approximately 25 fps, we found that the inference speed of models using a native action tokenizer (direct action output), such as GROOT and VPT, inversely correlates with model size. The same architecture’s VPT model shows decreasing inference speeds with increasing parameters: 1x, 2x, and 3x.\n2. Memory Usage: Models based on language tokenizers, which require in-context learning facilitated by ChatGPT, exhibited the lowest inference speeds due to their complex computational requirements.\n3. OmniJARVIS Performance: Thanks to its self-supervised behavior tokenizer and hierarchical architecture, OmniJARVIS achieves faster inference speeds (16.62 fps) even with a larger parameter count (7.2 billion) compared to native tokenizer models like VPT-3x (0.5 billion parameters at 12.34 fps).\n\n**Training Efficiency**:\nOmniJARVIS enhances training efficiency by encoding N=128 frames of trajectory data into k=5 behavior tokens. This compression allows the model to handle data more effectively compared to models like RT-2 and OpenVLA, which predict actions directly. This results in a significant reduction in training data volume and corresponding training time.\n\nThese quantitative results have been included in the revised paper to substantiate our claims regarding the efficiency of OmniJARVIS compared to other approaches.\n\n> Q: cost for synergized dataset.\n\nDue to the relatively certain generation of Thought, Memory, and Instruction, we used `gpt-3.5-turbo` to synthesize this data, and processing the Contractor dataset cost around $400.\n\n> L: Add a separate \"Limitations\" section and Generalization of OmniJARVIS. \n\nThank you for your suggestion. We have added a dedicated \"Limitations\" section in the **General Response** to address the scope of validation for OmniJARVIS. Initially, the model was validated solely within the Minecraft environment. Recognizing the need for broader applicability, we have also conducted preliminary generalization experiments in Atari Montezuma’s Revenge environment. These results, discussed in the **General Response**, demonstrate OmniJARVIS’s potential for adaptation to different scenarios. We will provide further details on these aspects in the updated manuscript to ensure a comprehensive understanding of the model’s limitations and capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tNFBuawyqc", "reviewer": "Reviewer_qkWa", "summary": "This work introduces OmniJARVIS, which jointly reasons over visual observations, instructions, self-generated text, and actions. OmniJARVIS models actions via behavior tokens, which are discrete embeddings that are separately learned on a behavior dataset. A policy decoder converts these behavior tokens to a sequence of low-level actions. OmniJARVIS also has a pipeline for synthesizing instructions, memory to track what happened in a long trajectory, and chain-of-thought text from offline observation action data. OmniJARVIS outperforms baselines in Minecraft on atomic, programmatic, and open-ended tasks.", "questions": "1. In regards to weakness (6), how can the OmniJARVIS be applied to non-Minecraft domains? \n\n1. What are the policy architectures for the encoder and policy decoder? How many parameters are there? How long are they trained for? \n\n1. For the behavior tokens, why introduce new tokens into the LLM vocabulary as opposed to reusing infrequently used tokens in the vocabulary as in RT-2?\n\n1. How does the paper arrive at 1T tokens on L187, given the previous token counts add up to 1B tokens? \n\n1. In Table 5, how can OmniJARVIS work without the instruction? \n\n1. What are the failure modes of OmniJARVIS? I am specifically interested in where OmniJARVIS fails in the programmatic tasks in Table 2. For the challenging task of Diamond, qualitatively, what behaviors does OmniJARVIS exhibit to succeed at this task?", "limitations": "No, the paper does not clearly state its limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. To the best of my knowledge, the behavior tokenization is a novel way to connect VLA models to actions. This approach has the advantage that the large LLM model does not need to be run for every action generation. Instead, the lighter weight policy decoder can generate a sequence of actions. \n\n1. The pipeline of using interleaved self-generated memory with a VLA is also novel. The paper shows the value of this additional data in Table 5 and presents a scalable way to generate it for Minecraft. \n\n1. OmniJARVIS presents a way to scale end-to-end policies for complex long-horizon tasks like Minecraft. Prior approaches like Voyager, while able to operate on long-horizon tasks, assume primitives that can be called via language. Other methods, like STEVE-1, directly output keyboard actions but struggle with long-horizon tasks (as shown in Table 2). OmniJARVIS operates directly from pixel inputs and outputs keyboard and mouse actions yet can complete long-horizon tasks with a high success rate.\n\n1. The paper shows extensive results in Minecraft with multiple tasks in 3 task setups of atomic, programmic, and open-ended tasks. In each setting, OmniJARVIS mostly outperforms the relevant baselines. \n\n1. OmniJARVIS can scale to larger models and datasets, as demonstrated in Fig 4.", "weaknesses": "1. The paper lacks many important details. Little detail is given about the encoder and policy decoder architectures (see (2) under the questions section).\n\n1. Important behavior token ablation experiments are missing. The context length for the behavior tokens is never analyzed and only 128 is used throughout the paper. However, this could be an important setting for the behavior tokens. Additionally, the paper does not ablate the FSQ settings to determine the necessary codebook size for the behavior tokens. The behavior tokens are also only conditioned on observation sequences without actions (L92). However, this decision is never justified. \n\n1. Experimental result details are unclear. On L215, the paper states it uses \"30 programmatic tasks to evaluate the performance of different agents\". But as far as I can tell, these 30 tasks are never described.\n\n1. The value of including the question answering dataset for instruction following is not verified. It is possible including this additional training source is the primary cause of the OmniJARVIS outperforming baselines considering it constitutes a third of the examples. \n\n1. OmniJARVIS performs worse than the GROOT baseline in collecting harder resources like wood and cobblestone (Table 1). However, I do not see this as a large weakness because OmniJARVIS is capable of programmatic tasks unlike GROOT.\n\n1. It's unclear how the OmniJARVIS can be used beyond Minecraft. OmniJARVIS exploits that the OpenAI Minecraft data includes oracle meta information for synthesizing the instruction, memory, and thought. \n\n1. The paper does not clearly discuss limitations.\n\n1. Agent behavior and failure modes are not analyzed. See point (6) of my questions.\n\nMinor:\n\n1. Table 2 should clarify what the numbers in parentheses next to the task type mean. I assume they are the number of programmatic tasks per category.\n\n1. A checkmark representing the training setting is _removed_ is confusing. I suggest changing to an \"x\" mark instead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1 and Q2: Details on encoder and decoder of behavior tokenizer.\n\nSorry for the confusion.\nAs shown in Fig. 2, the Behavior Tokenizer consists of three parts: Encoder, Decoder, and FSQ quantizer. We perform quantization based on the original GROOT, and the Encoder and Decoder use a design consistent with GROOT.\nSpecifically, the encoder includes a Convolutional Neural Network (CNN) to extract spatial information from image states $s_{1:T}$ and a non-causal transformer to capture temporal information from videos. The CNN is EfficientNet-B0 networks and the non-causal transformer is built on the code from minGPT-2 removing the causal mask. \nThe decoder consists of 4 identical blocks, where each block contains a Transformer-XL block and a Flamingo gated-attention dense layer to condition the decoder on FSQ code embeddings. More details can be found in GROOT paper Section 4.1 and Section 4.2. \nThe fsq quantizer consists of 5 levels of [8,8,8,6,5] with a codebook size of 15360. \nThe training hyperparameters of FSQ-GROOT are as follows:\nParallel Strategy: ddp, Accumulate Gradient Batches: 8, Batch size: 2, Precision: bf16, Image size 224*224, Encoder blocks: 8, Decoder blocks: 8, Hidden dimension: 1024, Chunk size: 128, Attention memory size: 256, Optimizer: AdamW, Weight decay: 0.001, Learning rate: 1.81e-05, Warmup steps: 2k. \n\n> W2: Ablation experiments on behavior tokenizer (codebook size and context length). \n\nThanks for your comments. \nWe conduct an in-depth investigation of behavior tokenizers with varying **codebook sizes**, utilizing recommended sets of FSQ levels to approximate specified codebook dimensions in the supplementary PDF Table-1. *Codebook Usage* is quantified as the proportion of codewords utilized at least once when encoding the validation datasets. *Reconstruction FSD* is measured by the FSD scores derived from the MineCLIP encoder, processing 1,000 different demonstration videos through the FSQ-GROOT and subsequent rollout in a randomly generated environment. Additionally, we measure *Resampling FSD*, which is the FSD score obtained when the environment rollout is conditioned on representations sampled from the codebook. Finally, we assess the *average rewards* for the task “collect wood” using OmniJARVIS across varying codebook sizes. Our findings indicate that increases in codebook size correlate with enhanced average rewards and reduced FSD scores, suggesting a scalable performance in OmniJARVIS with larger codebooks.\n\nWe also conduct an ablation of experiment on different context length of behavior token in PDF Table 1. We select the codebook size e14 as the default setting and explore different context lengths of 128, and 64.\nThe performances under context lengths of 64 and 128 are similar. Our insight is, that although shorter context lengths may offer better behavior granularity, they also require more frequent behavior code switching during inference, resulting in a slower speed of OmniJARVIS.\n\n> W2: Behavior tokens are only conditioned on observation sequence.\n\nThe reason we use observation sequence as tokenizer input is as follows: we inherit the self-supervised learning scheme from GROOT to train the behavior tokenizer, and the learning objective is to estimate missing actions from the observation-only sequence. Including actions in the input will break this learning scheme.\n\n> W3 and Minor1: Evaluation details of programmatic tasks. \n\nSorry for the confusion. The number behind the task category is the number of tasks in this group. The evaluated programmatic tasks are taken from DEPS, and the detailed task description and task settings are shown in the supplementary PDF Table 5. All tasks are evaluated over 20 times from an empty inventory under the open-ended generated environment in Minecraft. We will add this section to the updated manuscript. \n\n> W4: Ablation experiments on the training dataset.\n\nThank you for your comments. \nTo validate the effectiveness of including the QA dataset, we conducted ablation studies (detailed in **Supplementary PDF Table 2**). These studies revealed that while OmniJARVIS performs similarly on Atom Tasks with or without the QA dataset, there is a significant performance drop in Programmatic Tasks when the QA dataset is omitted. This drop underscores the necessity of the QA dataset for enhancing reasoning and planning capabilities essential for Programmatic Tasks.\n\nWe will further clarify this aspect in the updated manuscript, ensuring a comprehensive understanding of the QA dataset’s contribution to OmniJARVIS’s enhanced performance compared to baselines.\n\n> W5: Performance drop of OmniJARVIS in Table 1.\n\nThank you for your comments. We apologize for the confusion caused by an error in Table 1; the success rate for OmniJARVIS on the stone task should be 15.8, not 5.8 as mistakenly listed. This typo has skewed the comparison.\n\nFurthermore, it’s important to note that GROOT relies on manually selected reference videos as instructions, which can significantly influence its performance. In contrast, OmniJARVIS autonomously generates behavior tokens for the policy decoder to execute, providing a more consistent and scalable approach. \nWe further tested more atom tasks to compare GROOT and OmniJARVIS in the following table. Our tests show that OmniJARVIS performs comparably to GROOT across most Atom tasks, demonstrating its robustness not just in programmatic tasks but also in short-horizon atom tasks.\n\n| | log \t| dirt \t| stone \t| seeds \t| wheats \t| wool \t| llava \t|\n|-----|-----|-----|-----|-----|-----|-----|-----|\n| GROOT \t| **14.3±4.7** \t| 19.7±8.7 \t| **19.0±11.3** \t| 7.3±0.6 \t| 8.7±2.2 \t| 1.9±1.2 \t| **1.0±0.5** \t|\n| OmniJARVIS \t| 10.8±5.2 \t| **20.3±9.2** \t| 15.8±2.9 \t| **8.2±3.6** \t| **10.3±1.2** \t| **2.1±1.8** \t| 1.0±0.7 \t|\n\nDue to word limit restrictions, we have added more responses to the official comments, and we hope you can see them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0lO06ACXlD", "reviewer": "Reviewer_NWLR", "summary": "This work presents OmniJARVIS, a instruction following agent for open-world Minecraft. The agent works by learning a behavior encoder that generates behavior tokens conditioned on textual, visual and action inputs via self-supervised learning at first stage, then a multimodal interaction sequence can be packed with the learned tokenizer and a policy decoder is trained with this tokenizer autoregressively with the objective of predicting action sequence directly. The proposed method is achieve good performance on atomic action tasks, significantly better performance on programmatic tasks comparing with baselines, and better performance on open-ended tasks where instruction is given creatively. Comprehensive ablation experiments are conducted on behavior tokenizers, input modalities, and vision tokenizer.", "questions": "1. Is OmniJARVIS trained considering programatic task? If yes, in Table 2, do other baselines also trained with programatic task data? if not,  If not, how do you handle this difference when creating those baselines?\n2. Just to confirm, in Table 2, the baselines are retrain / finetuned with the data constructed in this work?\n3. GROOT and OmniJARVIS have close evaluation result on atom tasks as in Table 1, but far apart results for programatic tasks, could you provide more discussion on the reason? Clarifying first two questions would be helpful for this question.\n4. Figure 5. Left and right seems similar visually, more explanation and description on what those behaviors are would be helpful for understanding.\n\nI will consider raising score if above questions are addressed.", "limitations": "Authors provided a discussion of scalability of the agent: Scaling law is applicable for OmniJARVIS's instruction tuning process, the eval loss exhibit a log-linear decrease as data scale for tuning increases. Scaling up VLM improves performance but saturation is observed at 7B parameters.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The agent has a similar structure with GROOT, but replaced VAE-based approach with Finite Scalar Quantization (FSQ).\n2. The agent demonstrated significant performance gain on proposed experiments over all baselines.", "weaknesses": "1. Since this work get insights from GROOT, a slightly more comprehensive compare and contrast is preferred, especially, it appears the difference between these two works is not limited to how to learn trajectory representation, but also about input data format.\n2. Writing could be improved:\n    1. The description for 2nd stage of training seems incomplete (line 175)\n    2. Typos in general\n    3. The description of how to handle longer trajectories (> 128) is not clear to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: difference between GROOT and OmniJARVIS\n\nAs detailed in the **General Response**, our core contribution is to propose a novel Vision-Language-Action (VLA) model architecture termed OmniJARVIS, as shown in **Figure 1 of the supplementary PDF**, aiming at resolving the issues including inference efficiency and language and annotation bottleneck. \n\nGROOT is limited to short-horizon tasks (atom tasks) and **cannot accept language instructions**, as demonstrated in Tables 1 and 2 in the original paper. OmniJARVIS, on the other hand, uses the FSQ-GROOT decoder as a de-tokenizer, enabling it to generate environment-acceptable actions from discrete FSQ codes that are produced by the VLM. This allows OmniJARVIS to handle complex, long-horizon tasks (programmatic tasks) with advanced reasoning and planning abilities.\nPlease let us know if you still have further concerns about the novelty and we're more than happy to assist!\n\n> W2: Descriptions for OmniJARVIS training.\n\nSorry for the confusion. We utilized the SFTTrainer class from the TRL library by Hugging Face to train the VLM model. The learning rate was set at 1.4e-5, and a cosine learning rate scheduler was employed. The weight decay parameter was set to 0 with a warm-up ratio of 0.03. Training took place on 8 A800 GPUs with FSDP, with a batch size of 2 and gradient accumulation steps of 4 using bf16 precision. The training lasted for one epoch on our generated dataset.\n\n> W3: typos\n\nSorry for the typos. We've revised the manuscript accordingly.\n\n> W4: How to handle longer trajectories (>128).\n\nThank you for your question. OmniJARVIS is specifically designed to model trajectories that exceed this length, which is a limitation in models like GROOT which can only handle fixed 128-frame segments for instruction-following.\n\nGiven a long trajectory(>128), we first slice it into multiple segments of 128 frames each. These segments are then encoded into action tokens, $D^{beh}$, and integrated into an interaction dataset formulated as sequences of ${D^{instruct}, D^{mem}, D^{obs}, D^{th}, D^{beh}, D^{obs}, D^{th}, D^{beh}, ...}$. OmniJARVIS, as an autoregressive transformer model, supports a maximum token length of 8k, allowing it to model sequences that contain multiple $D^{behavior}$ tokens, effectively handling long sequences beyond 128 frames.\n\nAdditionally, during *training*, we employ a sliding window technique to manage sequences that exceed 8k tokens in VLM context length. In *inference*, OmniJARVIS outputs a new behavior token every 128 frames to address long-horizon tasks such as programmatic and creative tasks.\n\n> Q1 and Q2: Programmatic tasks are included in OmniJARVIS training datasets?\n\nOmniJARVIS leverages two datasets for training: a Question-answering dataset enriched with Minecraft knowledge and the Contractor dataset of human gameplay trajectories. While the instructions in the Contractor dataset don’t exclusively pertain to programmatic tasks, they support OmniJARVIS in generalizing from learned knowledge to generate new plans and behaviors in varied instructional contexts. For the comparisons in Table 2, all models, including the baselines, were trained using the same Minecraft dataset to ensure fairness in evaluating generalization capabilities across different tasks.\n\nAdditionally, the baseline models in Table 2 were not retrained or finetuned with the data constructed specifically for OmniJARVIS. However, we conducted further tests, shown in Table 3, finetuning open-source models on a Minecraft-specific Question-Answering dataset derived via ChatGPT self-instruct. This helped evaluate the impact of injecting specialized Minecraft knowledge on model performance, where the finetuned models showed improved understanding, aligning their capabilities closer to those of ChatGPT.\n\n> Q3: Discussion on GROOT performance on Atom and Programmatic tasks.\n\nAtom tasks are short-horizon tasks that can usually be completed within 600 timestamps (30s). These tasks primarily assess the agent’s mastery of basic skills in Minecraft, such as “mine oak log” and “kill sheep.” In contrast, programmatic tasks are long-horizon tasks, often requiring more than 6000 timestamps (5 min) to complete. These tasks demand complex reasoning and planning. For example, the programmatic task “obtain diamond” involves multiple intermediate steps, such as mining oak logs, crafting a wooden pickaxe, and acquiring stone and iron ore.\n\nGROOT and STEVE-I models are trained on the Minecraft Contractor data for instruction-following tasks of fewer than 128 frames. As a result, they possess certain atom skill proficiency but lack complex planning and reasoning capabilities as opposed to VLA models like OmniJARVIS. Consequently, they perform well on simple atom tasks but struggle with programmatic tasks that require sophisticated planning and reasoning.\n\n> Q4: Explanation of Figure 5.\n\nFigure 5 aims to visualize the behavior produced by our FSQ-GROOT decoder (the low-level policy adopted by OmniJARVIS) when conditioned on certain behavior codes. On the left is a screenshot of the input reference video to be encoded into behavior codes, and on the right is a screenshot of the FSQ-GROOT decoder policy's rollout in a new environment. \nThe behavioral similarity between left and right screenshots verifies that the code produced by the behavior tokenizer can be consistently decoded into the same behavior as the input by decoder policy, even in novel environments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EO8czHGz3m", "reviewer": "Reviewer_YroM", "summary": "The paper introduces a novel approach for behavior tokenization in agent domains, utilizing a unified token transformer. The key contributions include the development of a self-supervised behavior encoder that learns a vocabulary of actions. The generated discrete tokens for actions, augments vocabularies into MLMs for autoregressive modeling. The authors curate a large-scale Minecraft and multimodal QA dataset, including synthetic annotation generation, to train their model. The experiments demonstrate that the model surpasses baselines which go through a text bottleneck or directly map from pixels to low level actions.", "questions": "1. The paper proposes a new dataset and architectural methods. Can you clarify the individual contributions of each and their combined impact?  \n2. More comprehensive ablations to demonstrate the utility of the proposed architecture and the new dataset would strengthen the paper. Additionally, comparing against multimodal LLM training methods like QFormer or Perceiver to understand the advantages of the encoder approach and training stages. \n2. The ablations in Table 6 need more detail. Does the language goal include memory and caption text?\n3. How many episodes per task are used in Table 6? Why not ablate the full subtasks to provide a larger sample size? Additionally, why are ablations of the training phases and behavior tokenizer architecture not included? \n4. Why are Jarvis-1 and Voyager not included in the comparisons in Table 2?\n5. Can the codebook be used to derive interpretable skills?", "limitations": "The authors address some limitations, but additional suggestions for improvement include:\n- More comprehensive ablations to demonstrate the utility of the proposed architecture and the new dataset would strengthen the paper. Comparing against multimodal LLM training methods like QFormer or Perceiver could highlight the advantages of the proposed approach.\n- Ensuring consistent training data across baselines would provide a clearer comparison of model performance, or providing more experiments on the contributions of each component to back up paper claims.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The introduction of a quantized action codebook and the use of FSQ for behavior tokenization represent a novel approach to representing sub-goals, as opposed to text subgoals which may be limiting or require a predefined set. \n- The authors beat strong baselines DEPS and GROOT on long-horizon tasks in Minecraft, and show strong performance in open-ended instruction following and question answering. \n- The paper is generally clear and well-organized.\n- The proposed approach has potential applications beyond Minecraft, offering insights into behavior tokenization and hierarchical learning.", "weaknesses": "- The impact of the new dataset compared to the proposed architecture is unclear. Further analysis is needed to isolate the effects of the dataset from the architectural innovations.\n- Given the focus throughout the paper on the architecture contribution, the paper lacks comprehensive ablations to validate the contributions of individual components, such as FSQ and curated dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1, Q1, and L2: Impact of interaction dataset and model architecture. \n\nThank you for your insightful comments. To clarify the individual contributions and combined impact of the new dataset and architectural methods, we conducted specific ablation studies detailed in our paper.\n\n**Dataset Contributions**: OmniJARVIS utilizes two key datasets: a text QA dataset based on Minecraft knowledge and the Contractor dataset of human gameplay in Minecraft. The text QA dataset enhances OmniJARVIS’s understanding of Minecraft knowledge and high-level planning for complex tasks. The Contractor dataset, on the other hand, improves mastery of basic Minecraft skills such as “mine oak log.” We explored the impact of these datasets on different task types through new ablation experiments presented in **Table 2 of the supplementary PDF**. Results indicate that the QA dataset significantly influences OmniJARVIS’s performance on long-horizon programmatic tasks, while the Contractor dataset facilitates success in short-horizon atom tasks.\n\n**Architectural Innovations**: To assess the contributions of architectural innovations, we also conducted experiments on the effects of the vision encoder (Fuyu, LLaVA, isolated Captioner), behavior tokenizer (VQ-VAE, FSQ, Language), and various scales and architectures of the language model (Gemma-2B, LLaMA-7B, LLaMA-13B). These are detailed in Tables 4 and 6, and Figure 4 of the paper.\n\nOur findings demonstrate that both the datasets and architectural enhancements significantly contribute to OmniJARVIS’s ability to perform a wide range of tasks, from simple atom tasks to complex programmatic challenges. The synergy between the diverse datasets and innovative architectural elements is key to the robust performance of OmniJARVIS across different task domains.\n\n> W2: Comprehensive ablations on FSQ and dataset.\n\nThank you for your comments. We’ve conducted several ablation studies to validate the individual contributions of our architecture:\n\n1.\tFSQ Ablations: We’ve added new experiments on FSQ codebook size and code context length, detailed in the PDF-Table-1 and **General Response**.\n\n2.\tDataset Ablations: We add new dataset ablation in **Table 2 of the supplementary PDF** to investigate the effectiveness of the QA and interaction dataset. Detailed in Table 5 of the original paper, these experiments evaluate the impact of different dataset segments on OmniJARVIS’s performance.\n\n3.\tBehavior Tokenizer Ablations: Experiments on different tokenizer types (language, VQ-VAE, FSQ) are added to **Table 3 of supplementary PDF**, highlighting their effects on behavior generation.\n\nThese studies rigorously assess the influence of each component on the model’s effectiveness.\n\n> Q2 and L1: Ablations on multimodal LLM. \n\nThank you for the suggestion to include more comprehensive ablations and comparisons with multimodal LLMs. We have conducted experiments using various visual encoders, including FUYU (patch encoder), LLAVA (ViT), and an independent vision captioning model (ShareCaptioner+). The results of these experiments are detailed in Table 4 of the main paper.\n\nAdditionally, we are currently finetuning the OmniJARVIS model using the QFormer-based BLIP2-OPT-2.7B architecture. This variant of OmniJARVIS has not yet been finished; we will update the manuscript with the results once they are available.\n\n> Q3: Details on behavior tokenizer ablation in Table 6. \n\nThanks for your comments. This table showcases the success rates of OmniJARVIS in completing programmatic tasks using different behavior tokenizers.\n\nRegarding your specific question about language goals, they indeed include memory and caption text derived from the meta-information in the contractor data. These are converted into language goals that serve as decoder policy prompts for the language-conditioned STEVE-I model. For example, a language goal could be \"chop down the tree to mine oak logs.\"\n\nA note on the VQ-VAE tokenizer: during its training, we encountered a posterior collapse where only one code from the VQ codebook was utilized, leading to the failure of the VQ-GROOT training. The FSQ-GROOT, in contrast, represents the final setting used for OmniJARVIS.\n\nAll models in this comparison, including LLaVA-7B as the base model, used the same synthetic memory, thought, and caption data to ensure fairness in our evaluations. We continue to expand the scale of the ablation experiments (more tasks and test repetitions) and have placed the latest results in **supplementary PDF Table 3**.\n\n> Q4: More evaluation times in Table 6. \n\nIn Table 6, each task was evaluated 10 times. Due to the time-intensive nature of Programmatic Tasks, which require 6000-12000 timestamps to complete, this number of evaluations was deemed a practical balance between comprehensiveness and feasibility. To expand the scale of our experiments, we selected 1-2 representative tasks from each group of Programmatic Tasks and tested each 20 times. The results of this expanded testing are presented in Table 3 of the supplementary PDF, offering a larger sample size and further validation of our findings. \n\nDue to word limit restrictions, we have added more responses to the official comments, and we hope you can see them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank all. We would like to first explain some sharing concerns.

* Comparative Framework of existing Vision-Language-Action (VLA) Models.

According to the structure of the model and the training methods, the current mainstream VLA models can be roughly divided into three categories (**Figure 1 of the supplementary PDF**):
**(a)** depicts a model where upon receiving a language instruction, motor commands(actions) are directly produced based on the environmental state, facilitating immediate interaction with the environment at a unified frequency. Smaller models with <1B parameters like *VPT* maintain higher interaction speed (>20Hz), though their capability for complex reasoning tasks is limited. Larger models with >7B parameters such as *RT-2*, offer enhanced performance but operate at significantly reduced speed (2-3Hz). 
**(b)** illustrates a common approach utilizing large vision-language models (VLMs) for planning, subsequently outputting language goals, e.g., *DEPS* and *PaLM-E*. A language-conditioned policy (controller) then translates these language goals into actions at a real-time interaction rate of 20Hz, with high-level models re-planning at less than 1Hz. This hierarchical structure balances interaction frequency and performance, while it requires language as an intermediary and additional language annotations. Also, the training process of high-level VLMs and language-conditioned policies are separate and thus could struggle on tasks that can not be easily depicted by language (ex. building tasks, the language descriptions can be tedious). 
**(c )** OmniJARVIS(ours) mirrors the hierarchical structure of (b) but differentiates by employing a self-supervised encoder-decoder policy (GROOT) and FSQ quantization as a behavior tokenizer to connect the planner and the controller. The upper-level VLMs produce behavior tokens, which are effectively **compact** representations of tasks, and drive a policy decoder to output actions. The behavior tokens are automatically produced by tokenizing interaction data using the behavior tokenizer (see Fig.1 in the main paper), eliminating the need for language annotations on tasks/behaviors as used by the previous VLA scheme, therefore scaling more easily. 

* Differences between GROOT and OmniJARVIS

Our core contribution is the construction of a brand new Vision-Language-Action structure, as shown in **Figure 1 of the supplementary PDF**. GROOT can be classified as a (a)-class model. In addition to the aforementioned characteristics, GROOT is unable to accept language instructions and can only complete short-horizon tasks (atom tasks), as shown in Tables 1 and 2 in the main paper. In OmniJARVIS, the decoder of FSQ-GROOT is used as the de-tokenizer for OmniJARVIS, which conditions on the discrete fsq code generated by OmniJARVIS to output environment-acceptable actions $a_t$ based on observation $o_t$.


* Limitations of OmniJARVIS

1. More Evaluation Environments: While OmniJARVIS’s training method does not have specific requirements for the environment or data, allowing it to potentially generalize to other environments given the appropriate data, it has so far only been tested in the Minecraft environment. Collecting data from other environments to further validate its generalizability is an ongoing work. We have finetuned OmniJARVIS in the Atari Montezuma environment and show the primary result in **Supplementary PDF Figure 2**.
2. Large-Scale Data Requirements: Training large-scale Vision-Language-Action (VLA) models like OmniJARVIS requires a significant amount of interaction data. This can be resource-intensive and may limit the feasibility of deploying such models in environments where extensive data collection is difficult.
3. Model Hallucinations: Large models like OmniJARVIS may experience hallucinations, where the model generates incorrect or nonsensical outputs. This can lead to failures in decision-making processes for certain tasks, impacting the overall reliability of the agent.


* Generalization to Other Environments

Thank you for your suggestion. The primary validation of OmniJARVIS has been within the Minecraft environment as detailed in our paper. However, extending our approach to other environments to test generalization capabilities is an ongoing work. For instance, we have recently adapted OmniJARVIS to the Atari game Montezuma’s Revenge. Here, we created a dataset from 500 episodes played by an agent trained with Random Network Distillation, supplemented by random actions in early frames to enhance diversity. This dataset contains 1,823,699 transitions.

We then trained the FSQ-GROOT tokenizer on this new dataset and subsequently trained OmniJARVIS on the tokenized data. In initial tests, the finetuned OmniJARVIS achieved a score of 3600 in Montezuma’s Revenge, indicating promising transferability. We visualize a rollout trajectory in the **supplementary PDF Figure 2**.

The training components of OmniJARVIS include a self-supervised Behavior Tokenizer for short-horizon tasks and a Vision Language Transformer that leverages a long-horizon multimodal interaction dataset. This setup aligns with formats used in other embodied datasets, suggesting potential for broader applicability.

Due to time constraints, comprehensive experiments in additional environments are planned but not yet complete. Future work will focus on demonstrating OmniJARVIS’s generalization and robustness across diverse settings.

Meta-review: The paper presents OmniJARVIS, a vision-language-action (VLA) model for open-world instruction following. Integral to the model is a self-supervised behavior tokenizer that is conditioned upon text, vision, and action inputs. Behavior tokens are then decoded by a policy decoder and converted to a sequence of low-level actions. The encoder has the advantage that long-horizon sequences of instructions, observations, self-generated text, and behaviors can be combined into unified token sequences and processed via an autoregressive transformer. OmniJARVIS additionally includes a pipeline for synthesizing instructions, memory to track longer histories, and chain-of-thought text from offline observation-action data. Experiments demonstrate that OmniJARVIS outperforms contemporary baselines on a variety of tasks in Minecraft.

The paper was reviewed by four referees who agree on many of the strengths and weaknesses of the initial submission. Notably, at least two reviewers appreciate the novelty of the proposed behavior transformer as an effective means of connecting VLAs to low-level actions, with the advantage that it avoids needing to perform inference using LLMs for every action generation. Reviewers emphasized the fact that the method scales with regard to model sizes, which is not necessarily the case with existing methods. As several reviewers note, the experimental results demonstrate strong gains in performance over state-of-the-art baselines in Minecraft. Other reviewers found the paper to be well written and easy to follow, and the illustrations to be clear. 

The reviewers similarly agreed regarding the weaknesses of the paper as initially submitted. These include the lack of important algorithmic and experimental details (e.g., with regard to the encoder and policy decoder architectures). Reviewers found the experimental evaluation to be lacking, particularly with regard to the need for token and context length ablations, and the need to experimentally support claims about training/inference efficiency. Additionally, reviewers questioned the differences between OmniJARVIS and the existing GROOT method as well as the nature of dataset on which the model was trained. The authors made a concerted effort to address these and other issues during the rebuttal feedback, which helped to clarify the core contributions of the work as indicated by the fact that all four reviewers increased their overall scores. As they claim to do in their response, the authors are strongly encouraged to incorporate the results of this discussion, including the additional experiments and the nature and impact of the dataset, into the paper (and minimizing the need to refer readers to the GROOT paper for details as pointed out by Reviewer qkWa).
Decision: Accept (poster)