Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds
Authors: Sipeng Zheng
jiazheng liu
Yicheng Feng
Zongqing Lu
Venue: ICLR 2024 poster (2024)
Keywords: large multimodal pre-training
open-world embodied agent
large language model
Primary area: generative models

Abstract: Recent studies have presented compelling evidence that large language models (LLMs) can equip embodied agents with the self-driven capability to interact with the world, which marks an initial step toward versatile robotics. However, these efforts tend to overlook the visual richness of open worlds, rendering the entire interactive process akin to ``a blindfolded text-based game.'' Consequently, LLM-based agents frequently encounter challenges in intuitively comprehending their surroundings and producing responses that are easy to understand. In this paper, we propose Steve-Eye, an end-to-end trained large multimodal model to address this limitation. Steve-Eye integrates the LLM with a visual encoder to process visual-text inputs and generate multimodal feedback. We adopt a semi-automatic strategy to collect an extensive dataset comprising 850K open-world instruction pairs, enabling our model to encompass three essential functions for an agent: multimodal perception, foundational knowledge base, and skill prediction and planning. Lastly, we develop three open-world evaluation benchmarks and carry out experiments from a wide range of perspectives to validate our model's capability to strategically act and plan. The project’s website and code can be found at https://sites.google.com/view/steve-eye.

Published as a conference paper at ICLR 2024
STEVE-EYE: EQUIPPING LLM-BASED EMBODIED AGENTS WITH VISUAL PERCEPTION IN OPEN WORLDS
Sipeng Zheng', Jiazheng Liu", Yicheng Feng", Zongqing Lu Beijing Academy of Artificial Intelligence
2 School of Computer Science, Peking University s spzheng@baai. ac. on Fyc8138pku. edu. cn zongging. lu@pku. wdp. edu. on
ABSTRACT
Recent studies have presented compelling evidence that large language models (LLMs) can equip embodied agents with the self-driven capability interact with the world, which marks an initial step toward versatile robotics. However, these efforts tend to overlook the visual richness of open worlds, rendering the entire interactive process akin to "a blindfolded text-based game." Consequently, LLM- based agents frequently encounter challenges intuitively comprehending their surroundings and producing responses that are easy to understand. In this paper, we propose Steve-Eye, an end-to-end trained large multimodal model to address this limitation. Steve-Eye integrates the LLM with visual encoder to process visual-text inputs and generate multimodal feedback. We adopt a semi-automatic strategy collect extensive dataset comprising 850K open-world instruction pairs, enabling our model to encompass three essential functions for an agent: multimodal perception, foundational knowledge base, and skill prediction and planning. Lastly, we develop three open-world evaluation benchmarks and carry out experiments from wide range of perspectives to validate our model's capa- bility to strategically act and plan. The project's 's website and code can be found at https://sites, https///siee.goggll..ooo wtt.ge...co//ve//ttteee//e
INTRODUCTION
Developing embodied agents that can adapt the open world has long been sub- stantial challenge (Kolve et al., 2017; Savva al., 2019). Recently, the rapid progress of large language models (LLMs) (OpenAI, 2022; Touvron et al., 2023a) has shown their potential to serve as a general- purpose assis- tant. Driven by these pre-trained LLMs, re- cently proposed agents (Yuan et al., 2023; Wang et al., 2023a;b; Zhu et al., 2023) have managed to extract world knowledge and rea- soning capabilities from LLMs, allowing them to become self-driven. Thereby these agents are capable of generating executable policies or plans for a wide range of skills and tasks in an open-ended world.
(a)
Stove, you are phan for
NOOSEY FOR In Sirecutt
envirceent ment and yor
venbery stasa
craft slock
fann bedeotrrrodin aver
Sem the whet you
Stor. ciat eonden Mineroo??
in cochi oie dirs ley rexity costeick
Figure (a) LLM agent's feedback is un- controllable due to the uncertainty of input textual prompt, while visual cues can benefit the agent to generate feedbacks; (b) text-only driven agent of- ten finds difficult to produce intuitive feedback that humans can easily understand.
While current attempts integrate LLMs show promise in developing generic embod- ied agent, these efforts primarily translate the entire world into text, which overlooks the multifaceted richness of diverse visual reality
1 Corresponding author.

---

Published as a conference paper at ICLR 2024
and turns interacting with the environment into something akin to blindfolded text-based game." Consequently, such text-only agents often face difficulties when it comes to effectively and intu- itively representing the world. Imagine a situation where you request your agent to shop for pair of shoes. Would you prefer to send the agent a picture of the shoes or provide a lengthy description of the shoes to convey their appearance? Undoubtedly, you would opt for the former choice.
In fact, the agent's 's reliance on text input/output (I/O) imposes significant limitations on its ability to interact with the world. To illustrate this point, we consider Minecraft (Guss et al., 2019; Fan al., 2022) as an ideal example. Minecraft, being an expansive sandbox game, offers a vast realm for embodied agents to explore. which requires the acquisition of various basic skills (e.g.. crafting logs) and the ability to plan and execute diverse tasks. First, as shown in Figure (a), the LLLM-based agent produces uncontrollable outputs. The success of the agent's responses hinges heavily on careful prompt engineering (Huang et al., 2022b), ensuring that the L.L comprehends the environment and task objectives. Moreover, a universally applicable prompt that suits every L.LM and task is an unattainable goal. Therefore, this prompting process is labor-intensive and contradicts our aim of enabling agents to act in a self-driven manner. Second, when compared to visual feedback, language often encounters difficulties in intuitively conveying specific world concepts (e.g., recipes) to users, as illustrated in Figure 1 (b), thereby unavoidably creating obstacles for robust human- computerrm interaction (Preece et al., 1994; Fallman, 2003).
Unlike LLMs, humans possess an innate ability to process and generate information through both visual and text channels. This inherent gift significantly enhances our capability interact with the world. However, the coupling of LLM-based agents with multimodal I/O has been relatively underexplored in an open-ended environment. To fill this gap, we introduce Steve-Eye ah large multimodal model that enables LL-M-based embodied agents to engage with the open world via visual-text interfaces. Steve-Eye excels at producing responses that demonstrate comprehensive grasp of the environment, common-sense reasoning, and executable skill plans. To achieve this, Steve-Eye is equipped with three indispensable functions: (1) multimodal perception; (2) founda- tional knowledge base; and (3) skill prediction and planning. In this paper, we choose Minecraft as our validation platform considering its vast sandbox world and the high degree of freedom. More environments can also be considered, e.g., Virtual Home (Puig et al., 2018), AI2THOR (Kolve al., 2017). Due the space limit, we discuss the exploration of more generic environments in Appendix E and leave it as our future work. Our contributions can be summarized as follows:
Open-World Instruction Dataset. We construct an extensive instruction dataset to train Steve- Eye for the acquisition of three mentioned functions. The instruction data contains not only the agent's ' per-step status and environmental features but also the essential knowledge for agents act and plan. However, collecting such a dataset in an open world can be a costly endeavor, especially when aiming to gather fine-g fine-grained and diverse labels. As result, previous studies (Fan et al., 2022) have often relied on readily available unsupervised data (e.g., video-subtitle pairs) for pre- training. In these approaches, the agent's 's comprehension of its status and environment is implicitly learned through sulf-supe techniques, while its foundational knowledge is directly derived from general-purpose I.LMs. In contrast, our work involves curating multimodal instructional data specifically designed for open-ended embodied agents, by utilizing ChatGPT (OpenAI, 2022).
Large Multimodal Model and Training. Steve-Eye Eye combines a visual encoder which converts visual inputs into sequence of embeddings, along with a pre-trained LLM which empowers em- bodied agents to engage in skill or task reasoning in an open world. During the training process, we employ two-stage strategy similar to Liu et al. (2023). This strategy commences with the align- ment of multimodal elements between the visual encoder and the large language model, followed by the instruction tuning through our constructed dataset.
Open- World Benchmarks. We carry out extensive experiments to demonstrate that our pro- posed Steve-Eye outperforms LLM- -aseed agents in open-world setups. Specifically, we develop the following benchmarks to evaluate agent performance from a broad range of perspectives: (1) envi- ronmental visual captioning (ENV-VC), which assesses an agent's capacity to perceive and describe its surroundings effectively; (2) foundational knowledge question answering (FK-QA), which eval- uates the proficiency in mastering basic knowledge crucial for an agent's decision decision-making: (3) skill prediction and planning (SPP), which quantifies agent's 's capability to act and plan strategically.
2

---

Published as a conference paper at ICLR 2024
2
RELATED WORK
2.1 OPEN-WORLD EMBODIED AGENTS WITH LLMS
The rapid progress of large language models (Brown et al., 2020; Raffel et al., 2020; Zhang et al., 2022: Chowdhery et al., 2022) has significantly boosted their capacity to encode wide range of human behaviors within training data (Bommasani et al., 2021). When equipped with narrowly de- signed prompts, L.LM-based agents exhibit the capability to generate executable plans for tasks such as indoor robot manipulation. For instance. SayCan (Ahn et al., 2022) integrates skill affordances with LLMs to yield actionable plans, while Palm-E (Driess et al., 2023) takes a step further by constructing hierarchical agents capable of handling multimodal prompts. This approach has also proven its efficacy in open-world environments (Huang et al., 2022a; Li et al., 2022). In contrast to robot manipulation, agents in the wild require heightened level of real-time situational aware- ness and foundational knowledge to execute intricate skill plans across a diverse array of tasks. To simulate human behaviors in such open worlds, Generative Agents (Park et al., 2023) store agents' experiences and retrieve these memories to generate plans in a text-based sandbox game.
In recent years, the 3D sandbox Minecraft has received considerable attention owing to its remark- ably flexible game mechanics to serve as a prominent open-world benchmark (e.g., MineRL (Guss al., 2019) and Minedojo (Fan et al., 2022)). DEPS (Wang et al., 2023b) introduces the descrip- tor, explainer, and selector for plan generation with the help of LLM. Plan4MC (Yuan et al., 2023) constructs a skill graph and proposes a skill search algorithm to minimize planning errors. Voy- ager (Wang et al., 2023a) proposes an LLM-ppwered lifelong learning agent that continually ex- plores the Minecraft world. Similar to (Park et al., 2023), GITM (Zhu et al., 2023) integrates LLMs with text-based memory and knowledge to create generic agents in Minecraft. Among these studies, Voyager (Wang et al., 2023a) and GITM (Zhu et al., 2023) lean entirely on text descriptions of the environment to act and plan, while Plan4MC (Yuan al., 2023) and DEPS (Wang et al., 2023b) have visual-input skills but still rely on merely text for planning. None of them try to understand the rich visual observation provided natively by Minecraft. In contrast to these works, our work trains large multimodal model to fill this gap.
2.2 LARGE MULTIMODAL MODELS (LMMs)
In comparison to LLMs, large multimodal models (LMMs) (Awadalla al., 2023) encompass a broad range of information beyond text modality, which can be categorized into two primary streams. The first category (Gupta & Kembhavi, 2023; Huang al., 2023a; Patil et al., 2023; Surís et al., 2023) involves hinging on ChatGPT (OpenAI, 2022) or GPT-4 (OpenAI, 2023) to generate in-context responses without parameter tuning. However, these approaches heavily rely on the avail- ability of an LLM's API and the quality of the designed prompts. The second category comprises end-to-end pre-trained models. Within this category, models such as Huang et al. (2023b); Peng et al. (2023) are trained entirely from scratch. Conversely, some research explores efficient fine-tuning us- ing pre-trained LLMs by incorporating lightweight modality encoders, such as Qformer (Li et al., 2023) or Perceiver (Alayrac et al., 2022). Recently, Liu et al. (2023) propose to explicitly instruction- tune LLM using vision- language instruction data.
In this work, we propose Steve-Eye by building upon pre-trained LLMs, aiming develop an open- world agent powered by a large-scale model with versatile multimodal I/O capabilities.
METHODOLOGY
In this section, we first provide our instruction-followinn dataset to develop three key functions for the agent's open-world interaction in Section 3.1. We then propose large multimodal agent Steve-Eye in Section 3.2, and clarify details of the training procedure in Section 3.3. We adopt Minecraft
open-ended platform in this paper to collect data and validate the model, anticipat- ing to explore a broader range of environments for Steve-Eye in future studies.
To empower agent with the self-driven capacity to act and plan in open world, posit that the following embodied functions are indispensable: (1) multimodal perception function which offers detailed description of the agent status and environmental features; (2) foundational knowledge
3

---

Published as a conference paper at ICLR 2024
base which imparts understanding of how the world works and conveys crucial basic knowledge related to skills and tasks; (3) skill prediction and planning which responsible for generating skill execution feedback (e.g., success failure) and crafting high-level skill plans for handling more complex and long-horizon tasks. We develop these functions by building the corresponding instruction dataset to pre-train Steve-Eye follows.
3.1 OPEN- WORLD INTRUCTTION-OOLL.CO
DATASET
Multimodal Perception Instructions. Human players can perform actions in Minecraft mainly relying on their visual perception, without any prior hints or imposed game judgments. In order to endow Steve-Eye with the same ability, it is required to provide it with comprehensive visual descrip- tions of the environment. To achieve this, we use Minedojo (Fan al., 2022) obtain Minecraft snapshots which contain wide array of details within the agent's surroundings, including environ- mental features, the agent's life and food status, inventory items, and equipment, as illustrated in Figure 2. In addition, we leverage MaskCLIP (Zhou al., 2022) to identify the in-sight objects of these snapshots without supervised annotations. During our data collection process, for each snap- shot I and its corresponding description &c, we initiate three-step approach. Firstly, prompt ChatGPT to curate a list of 40 instructions as shown in Figure in Appendix A. Then we enrich snapshot details dense caption to describe its content, with the assistance of ChatGPT. Finally, we select an instruction NQ randomly from the list and combine it with the snapshot's caption to create single-round multimodal description pair (e.g., ### Human: Xg I'I n ### Embodied Agent: XC (.). By doing so, we collect 200K instructional pairs for multimodal perception learning.
Foundational Knowledge Instructions. Embodied agents require a foundation of essential knowledge to facilitate action- Minecraft, such knowledge
taking and skill planning. should contain item recipes, details of item attributes, their as- sociated numerical value, etc. We access this vital information from Minecraft- Wiki (Fandom, 2023), which comprises tensive collection of over 9,000 HTML pages. To be specific, first obtain all item icons from Minecraft-We Wiiii and gener- ate 200K icon inventory images, as illustrated in Figure (a). Each icon image corresponds ciated caption adhering to a standardized template: "There is Minecraft inventory with 4 rows. From left right, they
and food object in sight
inventory equipment
[Description] Stave walking the forest and sky.
front him. He stil has 20.0 and 20.0 faed. He is now equipped with the iran pickaxe and the inventory contains pickaxes,
4-row table with asso-
Figure 2. Multimodal perception
are ...". As shown in Figuire 7 in Appendix A, we curate set of 20 distinct prompts designed to challenge the model's s ability to recognize items. Subsequently, we further collect all recipe-related information from the Wiki as illustrated in Figure 3 (b), and design similar prompt templates to for- mulate 10,000 cmipe-image instructional pairs. Lastly, we process the Wiki and utilize this corpus to produce 40,000 single-round question-answer pairs. In total, we collect high-quality dataset with 250K foundational knowledge instructions.
Skill-related Interaction Instructions.
The environmen- tal description and foundational knowledge serve as prerequi- sites for agent's interaction within the open world. How- ever, a successful interaction requires more than these ele- ments alone. It relies upon the mastery of basic skills, such as log, harvesting, and food preparation, as well as high-level skill planning abilities tackle complex, hong-horizon tasks, such as crafting an iron pickaxe. To facilitate this, we gather corresponding training data for skill prediction and planning, which enables our model to provide correct feedback on both basic skills and horizon tasks across a spectrum of agent or environmental conditions. Specifically, the data collection process involves two steps. First, we sample skill trajectories based on the pre-trained basic skill policies and collect 200K snapshot pairs with corresponding statuses from these trajec- tories. Each snapshot pair (Iv, I.} denotes the 0-th and t-th timestamp of the skill trajectory. Next, we employ ChatGPT generate question-answe pairs about diverse aspects of skill execution status. These questions
JL
(a) Item icons
(b) recipes
Figure 3. Icons and recipes
4

---

Published as a conference paper at ICLR 2024
Heupl pencratio
How to c'aft wooden axe?
Answer me via recipe image.
4mage>
multimodal
not ingredients required
craft detector ral??
Ingot. Stone
capt capleterg
Large Language Model
Provide a detailed description the given Minecraft image <images,
00
right, sheers in the inventory, which moans he equippod with
<image>
projector
shears. S leve still has 20.0 life and 20.0 food.
stat takonizer
sti prrningg
Make pian lst fnish tack <mage> Minecraft.
executable can
multimodal
log neartyy: (2) crafting log. crafting planks; craiting
<mzge>
stick.
Figure 4. Illustration of Steve-Eye: large multimodal model designed to seamlessly process both visual and language inputs. Steve-Eye excels in acquiring fundamental knowledge of the world lives in, understanding the nuances of its surroundings, and generating executable plans complete a wide array of open-ended tasks. Furthermore, Steve-Fye responds to user instructions through either visual text-hased cues, enhancing the convenience and flexibility of human-Al interaction.
delve into whether the agent completes the skill, encounters unexpected failures, or seeks explana- tions for such failures. More details can be found in Appendix A. Second, we sample 40K task tra- jectories using the planner in Yuan et al. (2023), each of which can be denoted as T = {s1. $11)) representing the task is finished via T-round planning procedure, where 8; is the skill plan for i-th round. At each round i, we feed our model with its start snapshot and task initialization, and curate instructional questions to inquire about &; with reasonable explanation. In this manner, we obtain 200K instructional pairs from task trajectories.
3.2 MODEL ARCHITECTRRE
Figure 4 illustrates the overall architecture of our proposed model. Steve-Eye, functioning as a generative model, connects an image-oriented tokenizer fo with the pre-trained L.LM backbone Q. We adopt the image tokenizer, e.g., VQ-GAN (Esser et al., 2021), to encode the raw images into token embeddings V {v1, 12 ... 2n E Rndd where denotes the number of visual tokens and d is the dimensionality of each token. We further utilize a lightweight projection module fi with a trainable projection matrix W. This module maps the visual tokens to the same space with text emheddings, yielding & {Ū1, 82 in E Rxdd
= WV; where V = fo(T).
(1)
To effectively process visual-language inputs and generate corresponding outputs, our model inte- grates the visual codebook C, into the pre-existing language vocabulary C, This integration leads the formation of unified multimodal codebook, denoted as C.. C, U,. Additionally, in order to mark the starting and ending points of visual elements in I/O sequences, we introduce two special tokens, namely <vis>> and </vis>.. The LLM backbone Ô of our Steve-Eye is built upon decoder-only architecture with casual transformers. Our model employs auto-regressive predic- tion mechanism, generating responses based on the provided multimodal input tokens. The resulting response is mixed sequence of visual and textual tokens, represented as ) {21, 32...., Fm ). For each embedding yı, we pass it through a linear layer fp followed by softmax operation, mapping it into probability distribution of the multimodal vocabulary. The final prediction for the i-th token 2; is determined by selecting the token from the multimodal codebook with the highest score:
arg max
3.3 TRAINING
Each inslruction-folloiingg instance can be formulated as a multi-round conversation (x} C X: XNN, where each (2%,2i represents question-answer interaction between human and
5

---

Published as a conference paper at ICLR 2024
the embodied agent and N indicates the total number of rounds in the conversation. The entire instructional dataset follows this unified template, as demonstrated in Figure 11 in Appendix A. To efficiently train model, employ the negative log-likelihood objective over the prediction tokens with instruction tuning:
[(0) = Σ log (o D1:j-)),
(3)
1=1
where y and ğ respectively denote the input and target token sequences, with & representing the model parameters, and L representing the length of the target sequence. The input visual content I may represent an empty image depending on the input instruction. It is worth noting that we constrain the loss computation to only consider the answer tokens Nc. This constraint prevents training from becoming excessively straightforward and ensures that the model's primary focus on learning to precisely generate coherent responses. Similar to Liu et al. (2023), we adopt a two- stage Instruction-tuning strategy to train our model:
Two-Stage Instruction-Tuninn -uniin.. (1) Multimodal feature alignment: In the first stage, our pri- mary objective is to align visual features with the language token space. In order to strike a bal- ance between efficient tuning and comprehensive coverage of the world's concepts, we curate our open-ended instruction dataset to 600K snapshot-text pairs. These pairs are then transformed into cestructiionfollowweei data as described in Section 3.1. During the feature alignment stage, main- tain the visual encoder and the LLM parameters in a frozen state, exclusively training the projection module. Additionally, this training phase involves fine-tuning token embeddings to accommodate the newly introduced visual codebook and two special tokens <vis>> and </vis>>. (2) End-to-end instruction tuning: In the second stage, we continue keep the visual encoder frozen while concur- rently training the projection module and LLM. This second stage leverages the entire open-ended instructions and contributes significantly enhancing the model's capability of comprehending and effectively responding to complex multimodal instructions.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Implementation Details. In this paper, we use the LLaMA-2 model (Touvron et al., 2023b) as the LLM backbone. Additionally, we use CLIP (Radford et al., 2021) as our visual encoder to achieve the best performance for non-visual generative tasks, and use VQ-GAN (Esser et al., 2021) as the default visual tokenizer for visual generation. The size of visual codebook C, and language vocabulary is 8192 and 32000, respectively. In addition, we add <vis>> and </vis> to the final unified codebook, indicating the starting and ending points of visual content. Similar to Liu et al. (2023), we construct 850K instruction-anss answer pairs for model training. Note that the model is trained to predict the agent's 's answer, and thus only sequence/tokens of answer will be used to compute the loss in the auto-regressive model. We also adopt LoRA (Hu al., 2021) to reduce the computational cost for efficient tuning. We choose MineDojo (Fan et al., 2022) as the Minecraft platform to collect our instruction data and conduct experiments. Following Yuan al. (2023), we use the environments of programmatic tasks to train basic policies with RL. These policies are trained execute corresponding skills and keep fixed all testing tasks.
Evaluation Benchmarks. We conduct experiments on three benchmarks to evaluate an agent's interaction ability in an open world. (1) Environmental visual captioning (ENV-VC): given a snapshot, the model is asked to describe the agent's current status and environmental features from diverse aspects (e.g., life, food...). We evaluate the prediction's ss accuracy of each aspect by ex- tracting corresponding answers from the output description to compare with the groundtruth. (2) Foundational knowledge question answering (FK-QA): to assess the model's grasp of essential knowledge, we collect set of 10,000 Minecrafi-related questions from different sources, including the Wiki pages, Wiki tables, and Minecraft recipes. The performance is measured by the model's ability to provide correct answers to these questions. (3) Skill prediction and planning (SPP): utilize our proposed Steve-Eye to predict whether skill has been successfully completed and assert its capability to generate executable high-level skill plans for long-horizon tasks.

---

Published as a conference paper at ICLR 2024
Table I. Comparisons of different model settings on the environmental visual caption benchmark. The experiments are conducted on 20K ENV-VC test set.
Model
visual encoder
inventory
equip A
object in sight ž
life
food
sky
BLIP-2
CLIP
41.6
58.5
64.7
88.5
87.9
57.6
Llama-2-7h
Steve-Eye-7b
VQ-GAN
89.9
78.3
87.4
92.1
90.2
68.5
13b 13b
MineCLIP
44.5
61.8
72.2
89.2
88.6
68.2
Stove-Eye-13b
VQ-GAN
91.1
79.6
89.8
92.7
90.8
72.7
Steve-Lyye-13b
CLIP
92.5
82.8
92.1
93.1
91.5
73.8
Table 2. Comparisons of different data configurations on the environmental visual captioning bench-
mark, where "snapshot desc." " denotes the 200K multimodal perception instruction dataset.
inventory
cquip
object in sight
life
food
sky
no instruction tuning
22.7
24.3
39.8
81.2
80.4
61.1
w/o snapshot desc.
46.2 (-23.5)
40,9 (116.))
41.2 (1L.))
83.0 (:13)
82.4 (22.)
63.3 (12.)
w/o icon images
52.3 (-29.6)
48.1 (425,0)
91 e451
92.5 (41.3)
90.9 (-105)
73.5 (-124)
full data
92.5 (-69.8)
82.8 (458.5)
92.1 (482.)
93.1 (+11.)
91.5 (411.))
73.8 (4127)
4.2 ENVIRONMENTAL VISUAL CAPTIONING (ENV-VC)
We introduce this evaluation protocol for asserting Steve-Eye's multimodal perception function, which serves as an initial stride toward comprehensive evaluation of large multimodal models. Specifically, we collect 20,000 Minecraft snapshots (named ENV-VC test set) using Minedojo and apply the proposed data generation pipeline to create six questions for each snapshot, resulting in total of 120K questions. These six questions pertain the prediction of various aspects, including inventory items I, equipment 8, objects in sight ž life food 9, and the visibility of sky
During the inference phase, Steve-Eye predicts answers based these questions and the input snapshot. Experimental results are presented in Table I and Table 2. As shown in Table 1, our visual encoder, when combined with multimodal instruction tuning, significantly enables the ability of the text-only language model LLM (Llama-2-765 to comprehend the contents of the snapshots (Steve-Eye-7b). Notably, Steve-Eye outperforms BLIP-2 by substantial margin due the im- proved reasoning ability enabled by the larger LLM. Furthermore, the visual encoder plays crucial role in facilitating multimodal understanding. Surprisingly, the model equipped with CLIP (Radford et al., 2)21) surpasses the performance of the model using MineCLIP (Fan et al., 2022), achieving over +48.9%, +21.0% and +19.9% improvements in inventory, equipment, and object-in-sight pre- dictions, respectively. We attribute this performance difference to the fact that MineCLIP does not prioritize fine-grani ine-grained alignment during pre-trane vre-training, despite being exposed to a diverse range of Minecraft videos. In summary, Steve- Eye's 's ability to comprehend visual cues from its surroundings lays the foundation for subsequent interactions with the world.
To investigate the effictiveness of various types of instructional data for multimodal perception, carry out experimental comparisons with diverse data configurations in Table 2. First, our results showcase significant improvement in the model's capacity to respond instructional questions through instruction tuning, which leads to impressive gains of over +50% for inventory, equipment, and object-in-sigh prediction. Furthermore, the inclusion of the multimodal perception dataset and icon images in the training data both contribute to substantial improvement in the model's 's overall performance. Ultimately, the best results are achieved when combining all available data sources.
4.3 FOUDATIONAL KNOWLEDGE QUESTION ANSWERING (FK-QA)
Following Team (2022), we establish question database specialized to assess our model's pro- ficiency in generating responses pertaining to fundamental Minecraft knowledge. This evaluation is carried out through validation dataset known the FK-QA test set, which is further divided into two distinct subsets: TEXT and IMG. In the FK-QA TEXT subset, we generate a collection of 10,000 question-answer pairs curated from various sources, including the Minecraft-Win pages, Minecraft- Wiki tables, and Minecraft recipes. Each category comprises 2,000, 5,000, and 3,000 pairs, respectively. Upon receiving response from Steve-Fye, feed both the generated response
7

---

Published as a conference paper at ICLR 2024
Table 3. Comparisons FK-QA test of the foundational knowledge question answering bench-
mark. The evaluation metrics consider both the scoring and accuracy dimensions simultaneously.
Scoring
Accuracy
Wiki Page
Wiki Table
Recipe
TEXT All
TEXT
IMG
Llana-2-7b
6.90
6.21
7.10
6.62
37.01% 7.01%
Llama-2-138
6.31 (40.9)
6.16 (4.05)
6.31 6-079)
6.24
37.96%
Llama-2-700
6.91 (e0.1)
6.97 (40.70
7.23 (+0.13)
7.04 (+0.-2)
38.27%
gpt-turbo-3.5
7.26 (40
7.15 (40.9)
7.97 (40,87)
7.42 (+0.60)
41.78%
Steve-Eye-7b
7.21 (60.31)
7.28 (41.7)
7.82 (40.22)
7.54 (40.92)
43.25%
62.83%
Steve-Eye-133
7.38 (+0.45)
7.44 (+1.35)
7.93 (+083)
7.68 (+1.05)
44.36%
65.13%
and the corresponding groundtruth answer to ChatGPT. ChatGPT will first examine the accuracy of the response as measure of answer correctness. To minimize variability in error, ChatGPT con- ducts a further evaluation, considering the response's 's accuracy, relevance, and level of detail. This comprehensive evaluation yields overall score on scale ranging from 0 to 10, where higher score signifies superior overall performance. In the FK-QA IMG subset, we shift our focus to visual generation by employing 3,000 recipe images as groundtruth data. Here, our model tasked with generating visual outputs for each item within the recipe inventory, following specific order. The visual output is considered correct only if every element of the recipe is accurately generated. We adopt this metric to assert our model's ability to produce multimodal feedback.
Table 3 presents both scoring and accuracy results. It's worthy note that Llama-2 exhibits consis- tent performance regardless of the model's scale, with Llama-2-70b only marginally outperforming the 7b-version by +1.26% accuracy, meanwhile ver-son performs even worse than 7b- version on the scoring results. We hypothesize that this phenomenon can be attributed to distinct variations in difficulty levels encountered within our FK-QA test set. Llama-2 fails to answer correctly for challenging part regardless of its size due to essential knowledge missing. In contrast, Steve-Fye outperforms both Llama-2 and gpt-turbo-3.5, despite its considerably smaller scale. Furthermore, model exhibits more substantial improvement in responding to Recipe and Wiki Table ques- tions as compared to Wiki Page questions. This disparity can likely be attributed to the fact that Wiki Page contains large proportion of invalid questions (e.g.. version, history), whereas Recipe and Wiki Table predominantly feature knowledge-related inquiries. Such result further validates the effectiveness of our approach in acquiring foundational knowledge. Unlike text-only LLMs, our model exhibits considerable ability to output visual contents, which achieves 65. 63%1%% accuracy on FK- IK-QA IMG using the 33b-versoon. The multimodal generation ability enables Steve-Eye to better serve as an assistant for potential needed people such as beginners of this game. We show more details and cases in Appendix D.
4.4 SKILL PREDICTION AND PLANNING (SPP)
Skill Prediction. Similar to Section 3.1, collect another 20K snapshot pairs in the form of [Io from skill trajectories (referred to as Skill-Pred test). These pairs are input into our model query the current execution status of the skill. The execution status fall into of three cate- gories: success, failure, and running, with "running" signifying that the skill is currently in progress. As shown in Table our model exhibits com- mendable performance in skill status predic- still far
Table Recall/Accuracy results on Skill-Pred test set for the skill prediction benchmark.
tion. However, the performance from enough to completely replace the rule- based game judgment adopted by the existing RL-based skill agents. These experiments indi- cate that, despite the excellent multimodal un- derstanding capabilities of our model in open- world environments in previous experiments, it still falls short in fine-grained reasoning tasks that involve consecutive frames some extent.
running (%)
success (%)
fail (%)
BLIP-2 Steve-Eye-7b Steve-Eye- 13b
65.2/58.8
49.8/54.3
42.1/51
89.8/82.5
77.6/81.4
74.2/79.9
92.1/84.2
80.5/83.1
76.8/81.5
Skill Planning. Following Yuan et al. (2023), we carry out evaluation on 24 difficult tasks in Minecraft. These tasks be categorized into three types: cutting trees to craft primary items (7), mining cobblestones to craft advanced items (7), and interacting with mobs to harvest food and ma- terials (10). Each task is tested for 30 episodes, where an episode refers a multi-round interaction
8

---

Published as a conference paper at ICLR 2024
Table 5. Comparisons on the skill planning benchmark. We test the mean success rates of all tasks,
where each task is executed for 30 episodes using the same seeds for initialization.
Model
7
MineAgent
0.03
0.00
0.00
0.00
0.00
gpt assistant
0.30
0.17
0.07
0.00
0.03
0.00
0.20
0.00
0.20
0.03
0.13
0.00
0.10
0.00
Sieve-Fye-auto
0.30
0.27
0.37
0.23
0.20
0.17
0.26
0.07
0.13
0.17
0.20
0.33
0.00
0.13
Steve-Eye
0.40
0.30
0.43
0.53
0.33
0.37
0.43
0.30
0.43
0.47
0.47
0.40
0.13
0.23
Model
7
MineAgent
0.46
0.50
0.33
0.35
0.0
0.0
0.06
0.0
0.0
0.0
gpi assistant
0.57
0.76
0.43
0.30
0.00
0.00
0.37
0.00
0.03
0.00
Sieve-Eye-aute
0.70
0.63
0.40
0.30
0.17
0.37
0.03
0.07
0.00
Steve-Eye
0.73
0.67
0.47
0.33
0.23
0.07
0.43
0.10
0.17
0.07
process. At each round, the model receives the environmental feedback from the last round, plans skill list based on the current status, and then picks up the top skill execute. For each task episode, we set a maximum step between [3000, 10000]. our evaluation, we compare Steve-Eye against two baseline approaches: (1) MineAgent (Fan al., 2022). which completes tasks without decomposing them into basic skills, and uses PPO and self-imitation learning with CLIP reward, and (2) GPT Assistant, which employs ChatGPT high-level planner to generate skill plans by prompting itself with information from the environment and the agent's status. The results in Ta- ble 5 demonstrate that Steve-Eye significantly outperforms both baseline methods. Additionally, conduct experiments in which Steve-Eye takes over the skill prediction function from the rule-based game judgment Minecraft. This self-driven variant is referred as 'Steve-Eye-auee model's 's skill prediction is not always 100% accurate, fieve-Eye-aute does experience some perfor- mance degradation when compared to Steve-Eye. This degradation is more pronounced in longer, complex tasks (e.g., 70.
Since the
Nevertheless, Beeve-Eye-auto still demonstrates significant performance improvements most tasks, compared to the baselines. For additional details about this benchmark, please refer to Appendix C.3.
as opposed to short-term tasks (e.g..
For better visualization, we provide a qualitative example of Steve-Eye completing the task "crafting stone with wooden pickaxe" shown in Figure 5.
bithplace
find cobolestone
harvest cobblestone
find trees
PIRIN 1
stone
craft place table
planks
harvest
Figure 5. Snapshots of qualitative example, illustrating how Steve-Eye completes the task of "crafting stone axe with wooden pickaxe." Our model generates a skill plan each interaction round and selects the top skill from the plan list for execution.
5 CONCLUSION
In this paper, we explore enabling a large multimodal model to serve as generative embodied agent open worlds. We achieve this goal by proposing Steve-Eye, which combines the text-only language model with visual encoder, allowing for a multimodal I/O interface to interact with the environment. With the help of ChatGPT, we curate questions to generate 850K followiinn data to facilitate the agent's multimodal perception function, foundational knowledge mastery, as well as the capability of skill prediction and planning. Experiments on three open-world benchmarks verify the advantages of Steve-Fye over wide range of perspectives.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by NSFC under grant 62250068.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.010
2022.
Man-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint wwXiv:2308.01390, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.0775 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.03111 2022.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.03377 2023.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference computer vision and pattern recogni- tion, pp. 12873-12888 2021.
Daniel Fallman. Design-oriented human-computse interaction. In Proceedings of the SIGCHI con- ference on Human factors in computing systems, pp. 225-232, 2003.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:
18343-18362, 2022.
Fandom. Minecraft wiki. http://miieccctt.fffdmmooommomm 2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953-14966 2023.
William Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint 1rX.v 2019.
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 2Xiv:2/06009666 2021.
10

---

Published as a conference paper at ICLR 2024
Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong. Jiawei Huang, Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. arXiv preprint arXiv:2304.12299, 2023a.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv: 14045, 2023b.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference Machine Learning, pp. 9118-9143 PMLR, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022b.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:/712.05444 2017.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision- making. Advances in Neural Information Processing Systems, 35:31 199-319993,,,, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0848, 2023.
OpenAl. Chatgpt. https://ppena.cco//boggccccce 2022.
OpenAl. Gpt-4 technical report. arXiv:2303.0877: 2023.
Joon Sung Park, Joseph C O'Bren, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442. 2023.
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint 2.Xiv:23005.5334. 2023.
Zhiliang Peng. Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023.
Jenny Preece, Yvonne Rogers, Helen Sharp, David Benyon, Simon Holland, and Tom Carey. Human-computer interaction. Addison-Wesles Longman Ltd., 1994.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8494-8502. 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pp- 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 218555585--55,, 2020.
11

---

Published as a conference paper at ICLR 2024
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pp-
9339-9347, 2019.
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08125, 2023.
Vicuna Team. Vicuna: An open-source chatbot impressing
gpt-4 with 90quality.
https://vicma..msss.orr// 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 33X11:2302.13971,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykow, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.094 09.08888, 2023b.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305 16291, 2023a.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents.
arXiv preprint arXiv:2302.01560 2023b.
Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303. 16563, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01065 2022.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pp. 696-712. Springer, 2022.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. arXiv preprint 1rX14:2305.17144. 2023.
12

---

Published as a conference paper at ICLR 2024
Appendices
We offer a detailed description of the construction of our open- wpendworld instruction dataset, as outlined in Appendix A, including (1) multimodal perception instructions, (2) foundational knowledge in- structions, (3) skill-related interaction instructions, and (4) template of instructional training data. Furthermore, provide additional implementation details of proposed benchmarks and duct more analysis in Appendix B and Appendix C. In Appendix D, we present qualitative cases that illustrate our model's ability to provide intuitive visual feedback and serve as intelligent chatbot with multimodal input-output interface. Finally, in Appendix E, we offer further discussion of our work, such as the exploration of our model's potential applications in diverse environments.
A DATASET
Minedojo, we set our agent's birthplace using a seed value 500 during both data sampling and evaluation phases. This large seed value leads to spawning in wide variety of locations within the map, including forests, plains, and lakes.
Multimodal Perception Instruction. This dataset contains 200K instructional pairs. Figure 6 illustrates a partial listing of instructional questions employed for describing the content of the Minecraft snapshots. These instructions convey similar meanings, albeit with slight variations in natural language. Our generated responses to these instructions can be divided into two segments: (I) Responses directly derived from processed dense captions. To foster data diversity, we instruct ChatGPT to prompt variety of description templates. Each caption is then rephrased by randomly selecting one of these templates, ensuring a broad spectrum of expressions. (2) Responses to the following questions about the caption. The model is required to accurately respond based on the selected question and the visual content of the caption.
"Describe the following Minecraft image in detail",
rovide a detalled description of the given Minecraft image",
"G ivee an elaborate explanation of the Minecraft game image you see",
"S hare comprehensive rundown of the presented Minecraft image",
"O ffer thorough analysis of the Minecraft frame".
"E xplain the various aspects of the Minecraft image before you',
"E "axamin the Minecraft image closely and share its details",
"Write an exhaustive depiction of the given Minecraft image"
"Clarify the contents of the displayed Minecraft image with great detail",
'Narrate the contents of the Minecraft image with precision'
Figure Ten instruction examples for multimodal perception instructions.
Foundational knowledge Instructions. The dataset comprises 250K training instances, which organized into three distinct subsets: 200K icon image instructions, 10K recipe image instructions, and 40K Minecraft-Wike corpus instructions. For the icon images, we generate questions aimed at prompting the model to recognize and describe item icons within the inventory, depicted in Figure 7. Similarly, we curate instructional questions for recipe images as shown in Figure with the objective of extracting information on completing specific recipes. In addition, we preprocess the raw Minecraft- Wiki HTML pages by removing irrelevant information (e.g., reference links) and unresolved data, transforming the raw corpus into formatted, clean Markdown version. Leveraging the capabilities of ChatGPT, we employ this powerful language model to generate 10 questions, each with its corresponding answer, for every page of the cleaned Wiki corpus. This process yields a collection of 40K single-round question-answar answer pairs, which can be utilized for instruction tuning.
Skill-related Interaction Instructions. For skill prediction,
utilize the skill policies trained by Yuan et al. (2023) to create a dataset comprising 200K skill trajectories. each trajectory, we extract timestamps from the initial and /-th points to generate snapshot pair, denoted (To, I,)-
13

---

Published as a conference paper at ICLR 2024
"Clarify the contents of the displayed inventory image with great attention to detail."
"Characterize the inventory image with a meticuloualy detailed description."
"Break down the individual slot elements within the inventory image with precision."
"Take step-by-step journey through the important details of the Minecraft inventory image."
"Paint a vivid and descriptive narrative of the Minecraft iniertottinvennty image."
"Provide precise narration of the contents within the Minecraft inventory image."
"Thoroughly analyze the Minecraft inventory image in comprehensive and detailed manner."
"llustrate the Minecraft inventory image through a descriptive and informative explanation."
"Examine the Minecraft inventory image closely and share its intricate details."
"Compose an exhaustive depiction of the given Minecraft inventory Image."
Figure 7. Ten instruction examples of icon images for foundational knowledge instructions.
rovide brief descripton given recipe image."
succinct exp lanation of the recipe picture presented."
"S ummaarize the recipe content about kons of the image."
Giive short and clear explanation of the subsequent recipe image."
"Shan hare concise interpretation of the recipe image provided."
"rrssett a compact recipe description
photo's key features."
"Relay a brief, clear account the recipe picture shown."
'Render clear and concise recipe summary of the photo."
WWitte terse but informative recipe summary of the picture."
"Create narrative representing the recipe image presented."
Figure 8. Ten instruction examples of recipe image for foundational knowledge instructions.
"Steve demonstrating his proficiency KILL NAME) with the objective of achieving SKILL DE FEFITIION) We'll now both initial and current frames to determine he has successfully executed the skill and the reasons behind it"
"The skill Steve perferming (SKILL_ NAME }. and its intended outcome KKLL DEFINITION) To determine whether Steve has accompliheed this skill, we need to analyze both the starting and current frames:"
"Steve currently engaged executing KKILL NAME aiming achieve KKLLL DEFINITION) order to evaluate success in performing this skill, well examine both the Initial frame and current frame:"
"The task at hand Steve involves mastering (KILL NAME with the ultimate goal accompishing (SKILL_ DEFINITION) we'l analyze
ascertain whether he has successfully completed
starting current
"Steve the process of mastering the of S SKILL NAME with the specific objective accomolishing [SKILL_ DEFINITION; We
successfully
by companing
"The skill that tave is currently executing is KILLL NAME) and the intended outcome is KSILLL DE DEFINITION) To determine if Steve has effectively executed
"S teve is currently performing the KIILL NAME skill, with the aim achieving SKILL DE DEFINITION). Let's analyze both the start fiame determine
suceeeded
reasons behind
""S teve is demonstrating proficiency in the skill of (SKILL NAME which is designed to accomplish SKILL DEFINITION: Our comparison
evalustion
the initial current
"The task Steve undertaking is the mastery of (5 KILL NAME] with the end goal of achieving (6KILL, DEFINITION) To determine whether Steve successfully
this skill, wil analyze both the initial frame
frame:"
"S teve is in the grocess of executing the SKILL ssME) with the ultimate aim of accomplishing SKKLL DELL_LEIINTNO Our assessment will involve comparion between the starting frame the current frame determine has succeeded and why:"
Figure 9, Ten instruction examples for skill prediction instructions.
We then construct questions aimed at determining whether the agent successfully executed the skill or, in the case of failure, identifying the underlying reasons for the unsuccessful attempt. Illustrative examples of these skill prediction questions are provided in Figure 9, We also provide examples with such snapshot pairs in Figure 10.
14

---

Published as a conference paper at ICLR 2024
Tou dial Inllutiin Frone
300000 SOC
Q: Steve finding logs nearoy, complete this skill?
Q: finding sheep nearby, he fail? A: No. Steve found
Q: Steve plans to craft table, does finish this skill successfully? does.
A: No.
Q: Why does he fail?
Az Becouse Steve died.
Q: Steve collects wool Minecraft, what the consequence?
Q: Steve mining cooblestone, does he finish this skill?
Q: Seee is asked collectplanks, what the results? A: Steve collect planks successfully.
A: Steve fails finish the sill.
A: No.
Q: Please help analyze the reasons
you know
for the failure.
A: No additionalcabbblnnn cabblestone shown inventory
A: Sure, there is sheep sight
Figure 10. Illustrative examples of skill prediction instruction data with snapshot pairs.
Template of Instructional Training Data. Similar to Liu al. (2023), formulate each in- structional instance as a multi-round conversation as shown in Figure 11, where X message is a sentence describe this assistant (e-g.. "You are in chat between curious human and artificial intelligence assistant. You should serve as an assistant to give helpful, detailed, and polite answers the human's questions."), The number of rounds relies on the input instruction content. And the input images (denoted as < image >) will only be fed in the first round, while A' may contain visual outputs with two additional tokens vis and < /vvs
#### Xeed message In
### Human:
X3 <image> In ### Embodied Agent: X In
### Human: 2 In ### Embodied Agent: X
### Human: 2 In #### Embodied Agent: 2 In
Figure 11. The unified template to generate input sequence for instructional tuning.
B IMPLEMENTATION DETAILS OF BENCHMARK
Foundational Knowledge Question Answering. The 1000 evaluation QA pairs are generated by ChatGPT based on Minecraft knowledge from Wiki. It's important to note that these pairs are distinct from our instruction dataset and are not utilized in the pre-training phase to avoid data leakage. We will release them ensure reproducibility. The evaluation process cost-efficient, as the usage of gpt-turbo-3.5 charges only 0.0015 per 1,000 tokens. Consequently, the entire evaluation costs less than dollar.
Skill Planning. To clarify this benchmark, we begin by offering comprehensive task setup details in Table and Table 7. During evaluation, we relocate the agent to a random location at the initiation of every episode, with distances of up 500 units, ensuring that the agent spawns in unfamiliar environment. Furthermore, for tasks that involve interacting with mobs, we enforce a maximum spawning distance of 30 units for cows and sheep.
15

---

Published as a conference paper at ICLR 2024
Table 6. The setups of 24 tasks used in our skill planning evaluation, where "Initial Items" refers to the tools provided in the agent's s inventory at the beginning of each episode, and "Max Steps" represents the maximum episode duration. Any episode exceeding this limit is classified as task failure. The tasks are originally developed by Yuan et al. (2023).
(a) 7 tasks involving the process "cutting trees to craft primary items".
Task Icon
>
Task Name
stick
crafting. lable nearby
buwl
chest
trap_door
sign
wooden wooden.pickuxe pickuxe
Initial Ttems
Max Steps
3000
3000
3000
3000
3000
3000
3000
(b) 7 tasks involving the process of "mining cobblestones to craft advanced items".
Task Icon
Task Name
furnarne_nearty neerry
stone staesstairs
stone_slab
cobblestone
lever
torch
stone koonepickaxe
Initial Trems
$10
10
>
*10
>>
Max Steps
5000
5000
3000
5000
5000
5000
10000
(c) 10 tasks involving the process of "interacting with mobs to harvest food and materials".
Task Icon
Tusk Name
milk. bucket
wool
beef'
mutton
bed
painting
carpet
item. frame
cooked. beef
cooked.
button
Initial Items
Steps
3000
3000
3000
3000
10000
10000
3000
10000
10000
10000
Table 7. The setups of 10 long-horizon iron-based tasks, where "Initial Items" are provided in the agent's inventory task beginning, and "Max Steps" refers to maximum environmental steps.
Task icon
Task description
Initial tools
Max steps
craft iron ingot
7
#64
8000
craft shears craft bucket craft iron pickaxe craft iron axe craft iron sword craft iron shovel craft tripwire hook craft heavy weighted pressure plate craft iron trapdoor
715.
#64
10000
75
*64
12000
705,
12000
7
64
12000
715.
64
10000
64
8000
64
8000
7
64
10000
705.
$64
12000
For "Steve-Eye" is rooted a hierarchical framework complete tasks. To be more specific, our model does not engage in predicting low-level moves. Instead, we implement low-level moves by additionally training an RL--hased policy for each involved skill as introduced by Yuan et (2023). Then, our model exclusively generates high-level skill plans, delegating the actual skill execution to these pre-trained basic skill policies. Notably, we introduce self-driven variant named "Steve- Eye-auto", which serves not only planner but also replaces the Minecraft rules verify the successful execution of skills.
For the "gpt-assistant" baseline, develop interactive planning framework using ChatGPT. In this setup, ChatGPT serves as the planner, formulating skill plans in response to prompts that in- clude task descriptions and environmental observations. This approach involves thhin-of-
16

---

Published as a conference paper at ICLR 2024
prompting, where we initially provide the planner with few-shot demonstration with explanations to guide the initial planning step. To address frequent errors observed during testing, we incorporate specific planning rules into the prompts. As the planning process progresses, the planner encounters various scenarios: invalid skill names, redundant skills (already executed), successful skill execu- tion, failed skill execution. For each of these cases, carefully craft language feedback and ask the planner to re-plan based on inventory changes.
For "Plan4MC", this approach utilizes strategy similar ours break down a given Minecraft task into several skills and execute these skills one by one. By employing a specialized skill graph, Plan4MC guarantees the generation of executable plans with 100% certainty, thereby serving as an upper bound for our approach. As shown in the following, our Steve-Eye presents competitive performance when compared to Plan4MC, This noteworthy considering Steve-Fye uses a signifi- cantly less optimal LLM (LLaMA-2) compared to ChatGPT used by Plan4MC.
C ADDITIONAL BENCHMARK ANALYSIS
C.1 FOUNDATIONAL. KNOWLEDGE QUESTION ANSWERING
It's important to acknowledge that the performance boost of this benchmark is limited by comparing Eyre--Eye13bb and LlaMA-13b. We attribute this to the data noises within our Wiki data, partic- ularly those extracted from Wiki Tables. As you can in https:/ /mi hecraft.. fandom. com/wikee Wiki data is highly structured. which inevitably causes data noises even with careful processing. To address this issue, we are dedicating more resources to refine the collected Wiki data for a better-quality version. Given that such an endeavor has not been previously undertaken, we believe that offering this knowledge- resource will significantly benefit the community.
C.2 SKILL PREDICTION
In Minecraft, the rule-based analysis is beneficial since simply combining two adjacent frames may not provide enough information for skill assessment. Nevertheless, we do not directly employ in our skill prediction evaluation. Instead, it plays an important role during the pre-training of the ENV-VC task, which is tailored to enable our model to interpret the current status primarily via visual cues. We believe relying solely on visual cues for predictions without explicit rule-based information is more in line with human's 's doccisionmmaking process, compared to a dependency on rule-based analysis, since there won't be explicit rules in our real life.
C.3 SKILL PLANNING
We take closer look skill planning and identify several key factors contributing the standout performance of Steve-Eye compared to other game agents like MineAgent:
Adaptive Skill Planning Ability: MineAgent employs multi-task policy learned through RL, which can be seen as relying on static. predefined plans to accomplish each task. In contrast, Steve- Eye adopts different approach by decomposing task into individual executable skills, forming new skill plans based on current, real-time conditions. This method endows Steve-Eye with ex- ceptional adaptability in various gaming environments, enabling it to devise more effective and executable skill plans. To validate this advantage, we conduct experiments (referred to as "ours w/o decomp"") in Table 8, and 10, which reveal a significant decline in performance without task decomposition. The results indicate that integrating task decomposition is beneficial for solving complex tasks with basic skills trained via RL.
Additional Visual Understanding Ability. Our Steve-Eye directly leverages visual cues to sess its current status and respond accordingly. As shown in Table in our main paper, visual cues empower Steve-Eye to outperform text-only models like "gpt-assistant" even based much leggtt LLM. Our work also suggests a universal application of visual cues in gaming agents, moving beyond the traditional reliance rule- rule-based indicators for status assessment.
More Effective In-Context Reasoning Ability. Steve-Fye outperforms the vanilla llma-2--7b by 6.4% the foundational knowledge QA bench-
As depicted in Table 3 in our main paper,
17

---

Published as a conference paper at ICLR 2024
Table 8. Comparison of tasks involving the process of "cutting trees craft primary items."
Model
stick
table
bowl
chest
trapdour
sign
wooden pickuxe
MineAgent
0.00
0.00
0.00
0.21
0.0
0.05
0.0
Plan4MC:
0.30
0.30
0.47
0.23
0.37
0.43
0.53
w/o decomp
0.03
0.03
0.23
0.13
0.03
0.07
0.00
w/o ENV-VC
0.27
0.27
0.33
0.43
0.27
w/o FK-QA
0.37
0.23
0.40
0.47
0.27
0.33
0.37
ours
0.40
0.3
0.43
0.53
0.33
0.37
0.43
Table 9. Comparison of tasks involving the process of "mining cobblestones to craft advanced items."
Model
furnace
stairs
stoneslab
stonewall
lever
torch
pickaxe
MineAgent
0.00
0.03
0.00
0.00
0.00
0.00
0.00
Plan4MC
0.37
0.47
0.53
0.57
0.10
0.37
0.17
w/o decomp
0.13
0.23
0.17
0.13
0.03
0.07
0.00
w/o ENV.VC
0.23
0.30
0.33
0.37
0.27
0.23
w/o FK-QA
0.33
0.37
0.43
0.37
0.23
0.10
0.17
ours
0.30
0.43
0.47
0.47
0.40
0.13
0.23
Table 10. Comparison of 10 tasks involving the process of "interacting with mobs to harvest food and materials."
Model
bucket
wool
boef
mutton
bed
paint
carpet
itemframe
bcef
mutton
MineAgent
0.46
0.50
0.33
0.35
0.0
0.0
0.06
0.0
0.0
0.0
Plan4MC
0.83
0.53
0.43
0.33
0.17
0.13
0.37
0.07
0.20
0.13
w/o decomp
0.50
0.43
0.23
0.13
0.03
0.03
0.17
0.03
0.00
0.03
w/o ENV-VC
0.67
0.43
0.33
0.27
0.13
0.07
0.27
0.07
0.23
0.03
w/o FK-QA
0.63
0.53
0.43
0.37
0.17
0.37
0.13
0.10
0.00
0.73
0.67
0.47
0.33
0.23
0.07
0.43
0.10
0.17
0.07
Table 11. Comparison of 10 long-horizon iron-based tasks.
Model
iron ingot
shears
iron bucket
iron pickaxe
iron
iron sword
iron shovel
tripwire hook
pressure plate
iron trapdoor
MineAgent
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
gpt-assistant
0.20
0.00
0.00
0.03
0.00
0.00
0.03
0.00
0.03
0.00
ours zero-shot
0.17
0.10
0.13
0.07
0.03
0.03
0.27
0.03
0.00
0.00
ours
0.23
0.13
0.10
0.13
0.07
0.03
0.23
0.10
0.03
0.00
mark. Remarkably, it even exceeds GPT-Turbo-3.5 highlighting its acquisition of world knowledge. Therefore, Steve-Hye has enhanced adaptability to environmental changes when planning skills.
Long-Horizon Planning Ability. Long-horizon objectives in the skill planning benehmark (e.g., tasks of "mining cobblestones to craft stone pickaxe", "crafting bed" and "harvest cooked beef"") typically can be broken down into dozens of skills and may require ten thousand steps to complete. Here, we conduct experiments 10 long-horizon iron-based tasks in Table 11 and introduce their setup in Table Steve-Eye consistently exhibits promising results across these long- morizoo objec- tives, showing its robustness in handling complex game tasks. To further demonstrate Steve- learn came-playing strategies beyond the confines of the intructiio-folloor dataset, we demonstrate the results without the skill-related interaction instruction for training ("ours zero-shot"), Since the
18

---

Published as a conference paper at ICLR 2024
furmace
gold ingot
green candle
oak boat
orange bed
painting
Figure 12. Qualitative examples of recipe image generation.
tipped-arrow
acacia-sign
stone-axe
Figure 13. Illustrative examples of recipes that our model struggles to predict accurately. We attribute this failure the complexities arising from fine-graine tne-grained semantically overlapping image information.
skill plans these unseen tasks are not included in the training dataset, we believe these experiments effectively demonstrate Steve-Eye's ability beyond the scope of the collected instructions.
D QUALITATIVE RESULTS OF MULTIMODAL. GENERATION
Recipe Image Generation. Figure 12 showcases qualitative examples of our evaluation on the FK-QA IMG dataset. Utilizing visual tokenizer like VG-GAN, our model demonstrates the ability to engage visual generation, enabling to provide visual feedback based on its comprehension of textual input. However, shown in Figure 13, model encounters difficulties when generating image content characterized by fine-grained or semantically overlapping elements. These challenges warrant further exploration in future work.
Multimodal ChatBot. In Figure 14, we present an overview of Steve-Eye functioning as chatbot to receive task commands and execute them.
E FURTHER DISCUSSION
Additional Open-World Exploration. In this paper, have selected Minecraft as our open- world platform. Steve-Eye is not directly transferrable to other games due to each game's 's unique interface, especially for Minecraft known for distinct pixel-style visuals. Nevertheless, Steve- Eye can be applied to other open-world environments, such Virtual Home (Puig et al., 2018) and AI2THOR (Kolve et al., 2017), with minimal manual effort using the same methodology. These alternative benchmarks, when compared to Minecraft, exhibit closer alignment with the real world. To some extent, this choice holds greater significance since our ultimate objective is to deploy the agent in the real world. To achieve this goal, expand the Virtual Home benchmark by introducing a more extensive range of environments (50+ rooms), human-interaan interaction tasks (200+ for each room), well as diverse categories of actions (20+) and objects (100+), illustrated in Figure 15. As
19

---

Published as a conference paper at ICLR 2024
your bitt. place, yoursee!
Minecraft Environment Feedback. Feedbatk.
Envirommnnt eedback.
can see the sky, trees front of me. I have 20 lives and 10 food. There is a bucket my Invent. tory
There a cow fron: of mo, finding cows s- uccesstully!
mikk bucket firished! The task is success- fully completed
I want you harvest milk
Make a plan again.
Thank you, Steve
plan
The plan 1st to harves: mile (1) mile bucket
harvest find
cow nearbyy milk bucket.
Round 2 milk bucket
Round cow nearby
Figure 14. An overview of the process through which the chatbot receives and fulfills a task com- mand.
simulated world, Virtual-Home demonstrates the possibility of seamlessly adapting Steve-Eye to range of real-world simulation games. The corresponding validation and further exploration of open-ended embodied agents in a real-world context will be the focus of our future work.
Association between VN and FK-QA for the Agent's Decision Making. We verify this association can improve the agent's ' decision-making processes in Table 8, 9 and 10. When the ENV- VC task is removed, and we directly utilize visual cues without further instruction tuning, there's a notable drop in performance. This clearly demonstrates the critical role of the ENV-VC task in enhancing the model's ability to interpret and utilize visual cues effectively. Similarly, the exclusion of the FK-QA task yields performance decrease, emphasizing the essential contribution of these two pre- pre-training tasks enhancing our model's 's decision-making capabilities.
Comparison with more advanced methods like Voyager. We choose not to compare Steve- Eye with methods like Voyager, as there are significant differences in their underlying mechanisms. Voyager controls the agent by generating code for the Mineflayer API using GPT-4, process that inherently different from Steve-Eye, which is driven by a considerably lighter-weight LLM and employs pre-defined RL policies for each skill control. Additionally, Voyager heavily relies curated prompting for its planning and skill selection process, a feature not as prominently utilized in our approach. Given these fundamental differences, direct comparison between Steve- Eye and Voyager would not be equitable. However, it's crucial to recognize that Steve-Eye is compatible with Voyager and other similar methods. Voyager uses LLMS to translate agent's observations into text, while Steve-Eye excels at interpreting multimodal cues. This capability positions Steve- Eye potential multimodal foundation model replace the LLMs in methods like Voyager.
Implementation of Fully Embodied Agent. A fully embodied agent trains the model to predict the next move from videos and instructions, which could perform better compared with our existing hierarchical framework. However, our initial attempt to predict moves from videos and instructions has proven to be a substantial challenge and did not yield satisfactory results before. As result, currently, our Steve-Fye assesses only the success or failure of skill execution at each step, so that Steve-Fye can decide whether to re-plan. Nevertheless, we agree that directly predicting low-level moves based on visual cues could be more advantageous for creating realistic embodied agents. Thus, we are committed to exploring this direction in our future work.
Clarification Image Output Utilization. Firstly, integrating visual generation into our pre- training introduces a novel task that significantly broadens the scale and diversity of our instruction- following dataset. This expansion helps prevent oversinplification of the training process and en- sures that the model consistently produces accurate and coherent responses, especially when dealing with complex environments. Secondly, discussed in introduction, we argue that multimodal
20

---

Published as a conference paper at ICLR 2024
Task Name: relax on sofa
Task Description: go the living and sit in the sofa
Task Plan: ("1": walk living room, "2": walk couch, "3": find couch, "4" walk couch, '5" sit couch}
Task Name: browse the Internet
Task Description: to the office and sit in chair, turn the computer and grab the mouse. type the keyboard and starting working the computer.
Task Plan: ("1": walk living-room, "2": walk desk, "3": find desk, *4" find chair, "5"; sit chair, "6": find computer, "7": switch-on computer, "8"; find mouse, "9": grab mouse, find keyboard, "10": type
keyboard, "11": turn-to computer, "12": look-at computer)
Task Name: put milk in the freezer
Task Description: walk into kitchen, look for the milk, walk to milk, look for refrigerator, walk to refrigerator, open door, put the milk in the refrigerator
Task Plan: ("1": walk dining-room, "2": walk milk, "3": find milk, "4": tum-to milk, "5": grab milk, "6": look-at freezer, "7": walk freezer, "8": open freezer, "9": put milk
Task examples from the extended Virtual-Home benchmark, where elements in green, cyan and red represent action, room, and object categories, respectively. Our benchmark includes diverse range of tasks that simulate interactions between individuals and their room environments. It contains over 50 distinct room setups, involving 20 unique actions, and 100 objects. Each room presents selection of more than 200 distinet tasks.
Figure 15.
outputs offer intuitive way represent the world, leading effective interface for inter- action with human users. Therefore, although the multimodal output in this work is not generated for specific tasks, believe that this crucial for developing a general-purpose game assistant.
21


Reviews:
[{"review_id": "Y4PQZ4t41f", "reviewer": "Reviewer_m8nW", "summary": "The proposed method seeks to overcome \"text-only\" problems by combining an LLM with a visual encoder, which processes both visual and textual inputs and produces multimodal responses. This model is trained on a new dataset proposed by the authors, which contains multimodal perception, a foundational knowledge base, and skill prediction & planning. The trained model can perform multimodal I/O, benefitting from the two-stage training and fine-tuning.\n\nThe effectiveness of Steve-Eye is demonstrated through three proposed new benchmarks: environmental visual captioning (ENV-VC), foundational knowledge question answering (FK-QA), and skill prediction and planning (SPP).  The results show that finetuned Steve-Eye surpasses existing LLM in Minecraft scenarios, especially in multimodal Q&A.", "questions": "I'm still confused about the Multimodal Perception Instructions section. Are the responses for the instructions drawn from those processed dense captions? How exactly is the extraction done? Also, it would be clearer for readers if example instructions, responses, and snapshots were provided together, as the current format requires flipping back and forth between the text and the appendix to understand these benchmarks.\n\nFor foundational knowledge question answering, evaluation relies on ChatGPT, which may be hard to reproduce and could incur high costs. Moreover, I doubt about the evaluation performance. Given that Steve-Eye-13b is fine-tuned on a vast amount of domain data, it should theoretically perform much better than Llama. Could the limited performance boost be due to the evaluation method or insufficient training on Wikipedia data?\n\nRegarding skill prediction, it seems that rule-based analysis is necessary since simply combining two random frames likely won't give enough information for skill assessment, even with human expertise. Yet, this kind of pretraining appears to somewhat improve performance, according to results from the authors. However, I believe a different approach, like training the model to predict the next move from videos and instructions, might be more beneficial, which requires a stronger capability to process sequences of frames or videos.\n\nIn summary, the proposed Steve model shows promise in multimodal Q&A for Minecraft, but it's not yet a fully embodied agent. Future work could improve its in-context Q&A and control capabilities, which would allow for understanding temporal context and environmental feedback-based control.\n\nThough there's potential to refine the evaluations and finetuning tasks, the authors do contribute massive datasets and pretrained multimodal large models to the community, which they plan to make publicly available. Hence, I am giving a positive review.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Included in the summary and questions section.", "weaknesses": "Included in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NAXEaZubQZ", "reviewer": "Reviewer_ijGQ", "summary": "The \"Steve-Eye\" paper introduces embodied game-playing agents with visual perception capabilities and the potential to interact intelligently in open-world environments. By integrating large language models with a visual encoder, the paper enables agents to represent their surroundings and comprehend critical knowledge in the environment in a more intuitive manner. Utilizing Minecraft as a primary evaluation platform, the model demonstrates its effectiveness in a complex and dynamic setting, showcasing its versatility and potential as a general-purpose assistant.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**1. Strong Motivation:** The paper identifies and addresses a critical challenge in the field of embodied agents, highlighting the limitations of text-only interactions and demonstrating the necessity of incorporating visual perception for a more intuitive and effective user experience.\n\n**2. Extensive Task Definitions:** The authors have meticulously defined a variety of tasks to evaluate the agent’s performance, spanning across environmental visual captioning, foundational knowledge question answering, and skill prediction and planning. This comprehensive approach ensures a thorough assessment of the agent’s capabilities, showcasing its versatility and adeptness in handling different aspects of open-world interaction.\n\n**3. Large and Rich Dataset:** The paper introduces an extensive open-world instruction dataset, specifically curated to train the Steve-Eye model. This dataset can be a good contribution to the field for future instruction tuning of large multimodal models.", "weaknesses": "**1. In-Depth Analysis on Skill Planning Required:** The paper presents Steve-Eye, an innovative integration of visual perception with Large Language Models (LLMs) for embodied agents. However, there’s a discernible lack of comprehensive discussion and analytical depth in the aspect of skill planning, particularly when compared to other game-playing agents such as “MineAgent.” While Steve-Eye showcases commendable performance, a more exhaustive exploration of its strategic capabilities, especially in comparison to MineAgent, is crucial. This detailed analysis could unveil the underlying reasons behind Steve-Eye’s superior performance, providing readers with substantial insights and a clearer understanding of its capabilities in the embodied task of planning.\n\nOn a related note, the paper discusses two other tasks: Environmental Visual Captioning (ENV-VC) and Foundational Knowledge Question Answering (FK-QA). While these tasks are undoubtedly interesting and add value to the paper, their direct connection to the agent’s in-game execution and decision-making processes is not explicitly clear. Strengthening this connection and elaborating on how these tasks intricately weave into the agent’s planning and action sequences would significantly enhance the paper’s overall contribution to the field.\n\n**2. Baseline Selection Could Be Improved:** The inclusion of “gpt-assistant” as a baseline in the performance evaluation, particularly noted in Table 5, brings about certain ambiguities. The choice of baselines is critical, and in this context, one might wonder if “Voyager” or other models specifically tailored for the Minecraft environment would serve as more apt comparisons. By opting for baselines that are more closely aligned with the Minecraft gaming milieu, the paper could present a more robust and convincing argument for Steve-Eye’s effectiveness.\n\n**3. Clarification on Image Output Utilization Needed:** As depicted in Figure 4 and discussed in various sections of the paper, Steve-Eye has the capability to generate image outputs. However, the practical application and utility of these image outputs in the context of the tasks at hand appear somewhat nebulous. Providing readers with a clearer exposition on how these image outputs can be leveraged for specific tasks would enhance the paper’s clarity and comprehensibility.\n\nMinor:\n**4. Figure 1 Requires Better Visual Clarity:** My initial interaction with Figure 1 was marked by confusion, necessitating several revisits to fully grasp its intended message. A visual element such as a horizontal separator line could significantly improve the figure’s readability, distinctly delineating the unfavorable responses from the preferable ones. Currently, the figure could be misinterpreted as a continuous chat, leading to potential misunderstanding. Implementing this small yet impactful change would streamline the reader’s experience, fostering a quicker and clearer comprehension of the depicted scenarios.\n\nMinor: \n5. Figure 7/8/12 in the appendix are not in the correct order.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MOEW0IjEJ5", "reviewer": "Reviewer_k9Af", "summary": "This paper proposed Steve-eye, a multi-modal LLM for visual-based embodied game playing in the simulated environment. Especially, \nas previous game agent mainly obtain environment informations directly from game engine, Steve-eye want to intergrate perception system into game playing agent. To achieve this objective, this paper construct a multi-modal instruction dataset for instruction-tuning. And, the paper construct some benchmark to quantitatively evaluate the different capability of the Steve-eye.", "questions": "No anymore questions.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper dedicate to a interesting and valuable problem that make game agent more human-like (use visual to sense the world).\n2. Steve-eye divided the visual perception for several ends: generation, QA, caption, skill planning, which suit the minecraft game well.\n3. Generate a large multi-modal instruction dataset for this task.", "weaknesses": "1. The most important question for Steve-eye is the evaluation. Steve-eye is not evaluated on some long-horizon objectives like Voyager and GITM. With generated multi-modal instruction dataset, it is so intuitive that the model can perform well on the ENV-VC, FK-QA, SPP benchmarks as the instructions are generated for these tasks. The skill plan, visual caption or QA are actually studied on real-world images by some VLLM such as LLaVA. There is no reason to go back and evaluate VLLM on simulation environment just for such tasks.\nI think the main reason for the community to carry on research on game engine is to evaluate much more complex tasks that are not hard to attempt in real world. In this regard, I think the Steve-eye should have some evaluation manners as Voyager and GITM, to show what happens when we replace the environment information of game engine to visual perception.\n\n2. Is the model trained on such instruction dataset generalizable to new game? Such as GTAV?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a5dk2oQNEP", "reviewer": "Reviewer_mQD9", "summary": "This paper proposes a new end-to-end architecture for multimodal agent interacting with a simulated environment. Specifically, it proposes to fuse a visual encoder with a pre-trained large language model for visual and textual generation. The model is trained on a curated dataset of Minecraft data, and is able to achieve convincing results. Overall, the work attempts to enhance interaction the capability of text-only LLM interaction for embodied agents and the results are promising for future work.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed architecture is intuitively sound. It seems to be the natural way to merge a visual encoder, an LLM, and an image generator in an end-to-end pipeline.\n\nThe paper contains some smart ways to curate a large dataset.\n\nThe author suggests to publish codes and projects for reproducibility, which is nice.", "weaknesses": "The tasks are still fairly constraint in that everything is image-text or text-text pairs for evaluation. The task does not really test on the agent’s ability to navigate or perform tasks in Minecraft environment directly.\n\n(FK-QA) Evaluation is still focusing on in-domain knowledge about Minecraft-related questions. There seems to have no procedures or details specifying the methods to ensure data leakage. Especially, given that the LLM is selected as a LLAMA2, and directly evaluate on data procured from the internet. There could be an additional benchmark on using LLAMA to fine-tune on the text-only datasets and to evaluate on the same sets of text-only evaluation set.\n\nTable-3 has Llama 2-7b model outperforming Llama 2-13b model. What would be a good explanation for this behaviour?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)