EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
Authors: Yao Mu
Qinglong Zhang
Mengkang Hu
Wenhai Wang
Mingyu Ding
Jun Jin
Bin Wang
Jifeng Dai
Yu Qiao
Ping Luo
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Embodied AI
Multi-modal Foundation Model
Embodied Control
TL;DR: We introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI.

Abstract: Embodied AI is a crucial frontier in robotics, capable of planning and executing action sequences for robots to accomplish long-horizon tasks in physical environments.
In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi-modal understanding and execution capabilities. To achieve this, we have made the following efforts: (i) We craft a large-scale embodied planning dataset, termed EgoCOT. The dataset consists of carefully selected videos from the Ego4D dataset, along with corresponding high-quality language instructions. Specifically, we generate a sequence of sub-goals with the "Chain of Thoughts" mode for effective embodied planning.
(ii) We introduce an efficient training approach to EmbodiedGPT for high-quality plan generation, by adapting a 7B large language model (LLM) to the EgoCOT dataset via prefix tuning. (iii) We introduce a paradigm for extracting task-related features from LLM-generated planning queries to form a closed loop between high-level planning and low-level control.
Extensive experiments show the effectiveness of EmbodiedGPT on embodied tasks, including embodied planning, embodied control, visual captioning, and visual question answering.
Notably, EmbodiedGPT significantly enhances the success rate of the embodied control task by extracting more effective features. It has achieved a remarkable 1.6 times increase in success rate on the Franka Kitchen benchmark and a 1.3 times increase on the Meta-World benchmark, compared to the BLIP-2 baseline fine-tuned with the Ego4D dataset.

EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
Yao Mu', Qinglong Zhang", Mengkang Hu', Wenhai Wang', Mingyu Ding" .1, Jun Jin",
Bin Wang', Jifeng Dai2.5, Yu Qiao", Ping Luo The University of Hong Kong, 'Shanghai Al Laboratory,
The Chinese University of Hong Kong, "Noah's Ark Laboratory Tsinghua University
Abstract
Embodied Al crucial frontier robotics, capable of planning and executing action sequences for robots to accomplish long-horizon tasks in physical envi- ronments. In this work, we introduce EmbodiedGPT, an end-to-end multi modal foundation model for embodied AI, empowering embodied agents with multi- modal understanding and execution capabilities. To achieve this, we have made the follow- ing efforts: (i) We craft large-scale embodied planning dataset, termed EgoCOT. The dataset consists of carefully selected videos from the Ego4D dataset, along with corresponding high-quality language instructions. Specifically, we generate sequence of sub-goals with the "Chain of Thoughts" mode for effective embodied planning. (ii) We introduce efficient training approach to EmbodiedGPT for high-quality plan generation, by adapting a 7B large language model (LLM) to the EgoCOT dataset via prefix tuning. (iii) We introduce a paradigm for extracting task-related features from LLM-generated planning queries to form closed loop between high-level planning and low-level control. Extensive experiments show the effectiveness of EmbodiedGPT on embodied tasks, including embodied plan- ning, embodied control, visual captioning, and visual question answering. Notably, EmbodiedGPT significantly enhances the success rate of the embodied control task by extracting more effective features. It has achieved a remarkable 1.6 times increase in success rate on the Franka Kitchen benchmark and 1.3 times increase on the Meta-World benchmark, compared to BLIP-2 baseline fine-tuned with the Ego4D dataset. More demos, code, and dataset information can be found at our homepage.
Introduction
Embodied AI tasks, e.g., embodied planning, embodied VQA, and embodied control, aim to imbue robots with the ability to perceive, reason, and act within their environment, enabling them to perform long-horizon plans and execute actions autonomously based on real-time observations. Recently, large language models (LLMs) such as GPT4 [1] and PaLM-E [2], have shown promising language understanding, reasoning, and "chain-of -fthoughtth capabilities. Such advances may open new possibilities for developing robots capable of processing natural language instructions, performing multi-modal of-though"", and planning actions in physical environments.
Large-scale datasets play important roles in training large language models. For example, OpenCLIP trains its ViT-G/14 model on the LAION-2B dataset [3], which contains 2B image-language pairs. Unlike seneral-purposs visual language tasks that can get huge amount of weakly labeled image- caption pairs from the Internet, embodied AI tasks require egocentric data in robotics domains. Also,
Corresponding authors: Mingyu Ding and Ping Luo ([dingmyu, pluo.Ihi| @ gmmil.com)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Human: Deseribe the video in detail.
Assistant: In video, robot engaged in manipulation in a kitchen environment. The scene features a white robotic black platform with four gray-white circles representing the heating elements of induction cooktop. white kettle with handle can he seen in the bottom left corner of the platform, and black microwave with handle for opening sits on the left side of the platform. Two blue-gra cabinets are positioned above the platform, with the left cabinet having two doors with handles and the right cabinet featuring sliding door with handle, The video caplures the process opening right cabinet's sliding door by first grabhing
robotic
and pulling outwards.
Embodied Control Performance
Human: Can you write detailed plan for the task the robot performing?
(LIP-2(EgoDD
Success Rate (%G)
R3M
Assistant: task opening the sliding door. Step I: Move the left side of the platform
60
EubsdiiolPP
Step 2: Grip the handle on the right side of the door.
40
Step 3: Pull to the right.
Step 4: Hold the handle still while the door slides open.
Step 5: Release the handle when thedoor is open.
Kitchen
Metaworld
Figure I: EmbodiedGPT'ss capabilities for video captioning, multi-turn question answering, embodied planning, and low-level control. The plans given by EmbodiedGPT are highly executable and incorporate task-specific features, leading to a significant improvement in the success rate of embodied control tasks, outperforming both R3M [12] (a video-language contrastive learned model) and BLIP- 2 [13] (a multi-moda foundation model) on Franka Kitchen [14] and Meta- World [15] environments.
structured language instructions needed for precise planning, which usually requires huge manual efforts and costs. This poses a challenging problem collecting high-quality embodied multi-modal data. Some researchers [4, 5, 6, 7] explore creating large-scale embodied datasets with simulators, but a significant gap remains between simulation and the real world. Recent works [8, 9, 10] also explore adapting the pre-trained LLMs
domain by efficient tuning strategies like LoRA [1]]. However, several open questions still remain: how to apply LLMs to the field of robotics which may face large domain gaps; how to leverage the Shain-offtthoullee capability for structured planning: and how to use the output language plan for downstream manipulation tasks in an end-to-end manner.
To solve the above challenges, in this work, we first build large-scale embodied planning dataset, termed EgoCOT, which features chain-of-thought planning instructions. It contains carefully selected egocentric videos from the Ego4D dataset [16] and corresponding high-quality step-by-step language instructions, which are machine- gennraaed, then semantics-bal based filtered, and finally human-verified. Additionally, we also create the EgoVQA dataset egocentric human-object object interaction video question answering tasks, which aims to offer a wider range of egocentric multi-modal data.
extension of the Ego4D dataset, focusing on
Based on our EgoCOT and Ego we present an end-to-end multi-mode embodied foundation model called EmbodiedGPT, which can interact with the physical world in more natural and intuitive manner, and perform many embodied tasks, as shown in Figure such as embodied planning. embodied VQA, and embodied control. EmbodiedGPT comprises four integrated modules that work together, including i) a frozen vision model for encoding visual features of current observations, ii) a frozen language model used to execute natural language for question answering. captioning, and embodied planning tasks, iii) an baabodied-formee. with a language mapping layer for aligning the visual and embodied instructions and extracting task-relevant mesence-level features with the generated planning for low-level control, and iv) a policy network, which is responsible for producing low-level actions based on the task-relevant features, enabling the agent to effectively interact with the environment. To further enhance EmbodiedGPT 's performance in generating reliable planning containing sub-gat goal sequences, implement prefix tuning the frozen language model to encourage the generation of more executable planning.
Our method possesses the following core advantages: i) the generated planning exhibits strong executability and granularity at the object part level, such as the gripper of robotic arm or the handle of a door, manifested in sub-goal sequences. ii) the proposed EgoCOT dataset is built based

---

on an open-souree large-scale dataset, which offers greater scalability compared to the PaLM-E [2] model trained on proprictary robot data. And both the EgoCOT dataset, and the EmbodiedGPT model will be open-sourced. iii) EmbodiedGPP forms a closed-loop from high-level planning to low-level control, which enables seamless integration of high-level planning and low-level control, providing efficient task performance and adaptability to wide range of tasks. To achieve this, we utilize the embodied-forme former to query task-relevant instance-level features through cross-attention between visual observations and generated embodied planning. This enables the policy network to complete low-level control tasks with fewer than 25 demonstrations.
The contributions can be summarized as follows: (i) We build an end-to-ens multi-modee foundation model EmbodiedGPT for embodied AL, which is featured with "chain-of Chain-of-thoughtt capability, em- powering embodied agents to interact with the physical world in more natural and intuitive manner. (ii) We develop two datasets, EgoCOT and EgoVQA, consisting of 200M annotated videos from the Ego4D dataset with corresponding detailed planning instructions and VQA data. The datasets are first machine- generraeed then semantics- saaaattiss-bseed filtered, and finally human-verified for quality control. (iii) We introduce EmbodiedGPT cost-effective training approach and paradigm for extracting task-relevant features from LLM-ggeerated planning queries, thereby forming closed loop between high-level planning and low-level control. We demonstrate our approach's 's effectiveness by achieving state-of of-theeart or comparable performance on multiple embodied tasks, including embodied control, embodied planning, video captioning, and video QA. Notably, in comparison to BLIP-2 [17] fine-tuned on the Ego4D dataset and R3M [12] specifically designed for manipulation tasks, EmbodiedGPT outperforms both models on the Franka Kitchen [14] benchmark with a margin of 22.1% % and 5.5%, respectively. Similarly, on the World [14] benchmark, EmbodiedGPT surpasses both models with margins of 22.5% and 4.2%, respectively.
Related Work
2.1 Vision Language Pre-training with large scale foundation model
Vision-Languanl Pre-training focuses strengthening the link between visual observation and natural language. The goal is to develop models that can better understand and process visual content, such recognizing objects and actions, and generating descriptive text. As models become larger, the computational expense for end-to-end pre-training rises, leading to the need for modular vision-language pre- training methods. These methods smartly pre-trained models, keeping them 'frozen' during vision language pre-training to save on computational costs. For example, models like Uniter [18], Oscar [19], VinVL [20], and LiT [21] freeze the image encoder, while Frozen [22] and VGPT [23] freeze the language model. Furthermore, Flamingo [24] and BI.IP-2 [17] use both frozen image encoders and language models, providing a balance between performance and computational efficiency. Due to the lack of open-source data for multi-modal embodied planning, previous works struggled to perform detailed task decomposition and lacked the ability to generate precise and executable plans. To tackle this issue, we create the EgoCOT dataset and develop an embodied chain-of of-thought vision language pre-training framework to enhance the capacity of multi-moda. models for embodied reasoning and planning.
2.2 Egocentric Video Datasets.
Egocentric videos, which are captured using wearable cameras, provide a natural perspective of daily activities and pose several challenging research questions [25, 26, 27]. Several egocentric video datasets have been created over the years, including [28, 29, 30]. However, the collection of egocentric videos is expensive, and previous datasets tend be small-scale and domain-specific. Recently, a massive egocentric video dataset, Ego4D [16], has been released and has been used for embodied representation learning. The dataset comprises 3,670 hours of videos collected by 931 people from 74 locations across 9 countries, with videos accompanied by narrations. For embodied AI tasks, learning from large and diverse egocentric human videos has emerged as a promising approach to acquiring generally useful visual representation for controlling such tasks. For example, R3M [12] developed a sparse and compact visual representation using the Ego4D human video dataset through combination of time-contrastive learning and video-language alignment. VIP [3 I]. learns general-purpose reward functions for goal-conditioned robotic manipulation using the Ego4D) dataset.
3

---

Asslatart Hlas Svep shop
Embodied Queries
Text Querries
ovan Srepi: pull nandle oct
Vision Transformer
Embodied-
Language Mapping
LLaMA
Full axtcurdte)
Embodied Planning
Instance information
glen caption
Resistant: ress, 14 aateeniiee been
Text promat
Questions
Video Caption
Global information
Task description
concat
Examples Dialogue Memory
oRJICE
rabet
Recietant::
Coat
CNN w/ global
Policy
Video Q&AA
average pooling
Mapping
seccce apd) speratang!
Rasistant: The
Viet: the coler of the
Physical Manipulation
Multi-turn Dialogue
Figure 2: Overall framework of EmbodiedGPT. The black arrow shows the vision- laiiun-anguage planning process, while the red arrow represents that we leverage the queried language plans for better policy learning in low-level control tasks.
2.3 Large Foundation Model Assistant System
Recent advancements in large-scale multi-modal language models (LLMs), such as GPT-3 [32] and GPT-4 [1], have resulted in the creation of various models that can understand multiple modes of information. Two main approaches are used in this field: systematic collaboration and end-to- trained models. Systematic collaboration approaches involve coordinating multiple vision models or tools with language models to combine visual information with textual descriptions. Examples include models like Visual ChatGPT [33], MM-REACT [34], and HuggingGPT [35]. However, this approach limited by the accuracy and capacity of fixed modular models, which can lead to an accumulation of errors. On the other hand, end-to-end models aim to provide unified models for multi-modal tasks. For example, Flamingo [24] combines vision and language by freezing pre-trained vision encoders and language models. BLIP-2 [13] introduces Q- Q-Former to align visual features from frozen visual encoders with large language models. Recently, models such as MiniGPT-4 [36] and LLaVA [37] align instruction-tune -unde language models with visual features from frozen visual backbones. VideoChat(38], mPLUG-OWl [39] and X-LLM [40]. further expand support for video input. PaLM-E [41] is the first large embodied multi-modal model, which directly incorporates features from sensor modalities to improve real-world performance and is trained with their large- scale everyday robot data [42]. Compared to PaLM-E, EmbodiedGPT is more compact, with size of only 1OB and offers additional support for video captioning, video QA and making planning according demonstration video. Furthermore, we form a closed-loop system that spans from high-level planning to low-level control.
3 Method
The goal of the embodied foundation model to imitate human-like perception and interaction with the environment by accurately perceiving the environment, identifying relevant objects. analyzing their spatial relationships, and formulating a detailed task plan. To achieve this, the EmbodiedGPT employs a pre-trained vision transformer the visual encoder and a pre-trained LLaMA [43] model as the language model. As shown in Figure 2, the embodied- the visual and language domains, it first extracts compact visual features from the output of the vision model through attention based interaction involving visual tokens, text queries, and learnable embodied queries and then maps to the language modality through language mapping layer. These embeddings are sent to the frozen LLaMA [43] language model for visual caption, visual QA, and embodied planning. The generated planning then used to query highly relevant features from the general visual tokens encoded by the visual model via the embodied- formerr. These features
acts as a bridge between

---

are utilized to generate low-level control commands for task execution through the downstream policy network. To enhance performance across range of embodied tasks, we introduce novel video-language pre-training paradigm that leverages a cognitive chain of thought to produce embodied planning from egocentric video inputs. We formulate this task as standard VQA (Visual Question Answering) task, using "how to do the task that + original caption" as the question and embodied planning as the answer. This framework enriches the data of embodied planning and standard visual QA tasks, encouraging the embodied-formet to capture task-specific features that are more suitable for embodied control tasks.
3.1 Framework
The training process consists of three stages, each designed to incrementally develop reasoning and planning capabilities. The first two stages focus on pre-training in basic cognitive and responsive skills, while the third stage involves training the embodied AI task with egocentric video-text data on EgoCOT. In the first stage, we focus on image-text conversation alignment pre-training, which involves using three datasets: COCO Caption [44]. 595 thousand finely filtered image-text pairs from CC3M [45], and 491 thousand filtered image-text pairs obtained by re-captioning LAION-400M using 2LIP-2 [17]. The primary goal of this stage is to pre-train the cebbodidd-formor and language projection while keeping the vision and language model parameters frozen to save computational resources. In the second stage, our goal is to enhance the model's 's ability to comprehend and generate more complex sentences and improve its reasoning skills. We achieve this by updating the language projection and prefix language adapter and utilizing the "Complex_ Beaeeeee_eaennnnn_eeeeaeee 77k" and multi-turn conversation datasets provided by "LLaVA_Instruct__a Instruct_150K [46].
Embodied Sohain-of-thoughhe training with EgoCOT: During the third stage, we employ Conv3D [47] to adapt the pre-trained vision model from stage 2 for video encoding, using a total of eight evenly distributed keyframes from each video. Each keyframe is partitioned into three- dimensional (3D) patches, which can be visualized as spatio-temporal cubes, adeptly capturing both the visual content and the sequence of events within the video. These 3D patches are subsequently encoded into visual tokens via the Conv3D module with time offset of 2 and are then integrated into the internal vision transformer. Then, we introduce the pre-training paradigm where the model takes 8 keyframes of the video input, along with the task description, embodied planning, and structured verb-noun pairs summary to reason with a prompt, such Listing 1. To avoid overfitting, we provide prompt set that has different instructions with the same meaning. In this stage, we fine-tune the patch embedding, the language projection layer, and the prefix language adapter to better capture temporal information.
vision language
Watch this video, identify the actions and devise a plan using chain thought. Extract datailed actions using this schana:
Task: ('task description")
Plan: ('plan with oh -thhoughtt" Actions: (('aunber"): ('verb (((Couu )))),
Listing I: Prompt we used for chain-of-thought pre-training.
3.2 Model Architecture
The Embodied-formect denoted E(), serves bridge between visual input zvis and the frozen language model, acting as an information bottleneck that delivers the most relevant visual data to the language model. The Embodiee-former consists of two sub-modules: one for extracting features from the image input, denoted as Evis Evis + Ivis. and another for extracting features from the text input, denoted Eixt : Fixt Vixt. We employ N learnable embodied query embeddings Squery as the input of E to interact with Ivis through cross-attention layers and with Ztxt. through self-attention layers. We denote the output query representation as 2 E NND. where D is the dimensionality of the embeddings. The dimension of is significantly smaller than that of the visual
features. The output query embeddings are then transformed to 2 E NxD', which have the same dimensionality D' as the LLM's text embedding in the language modality. This transformation is performed by mapping function denoted as M + which is accomplished by linear projection via fully-connected (FC) layer. The projected embeddings, z', serve as "soft visual prompts for the language model," decoupling the whole interaction into visual-query interaction and
5

---

query-text interaction. The final embodied planning is inferred by the language model with a' and text prompt(shown as Listing 1) as imput. For low-level control which aims generate actions to interact with the environment, the embodied plan pplan is used as input text for embodied-former to query the task-relevant instance level features Einstance = E (Irvis "plan, Vquery). Subsequently, the agent is capable of generating control commands, such as the turning angle of the servo, represented as a = g (Finstance: Fglobat). This function combines both the instance-specifie information Zinstance and the global context Zglobal- The global context is inferred using a ResNet50 model [48] that has been pre-trained on ImageNet [49], employing global average pooling. Here, g(-) represents the policy network, which is Multi-Layer Perceptron (MLP) [50] mapping function. The output of the policy network consists of specific executable actions, such as positions and velocities in the Cartesian coordinate system. More implementation details can be found in Appendix A.
3.3 Training Settings
We employ the same pre-trained image encoder as BLIP-2[17]. Specifically, we utilize the ViT-G/14 model from EVA-CLIP [51] and remove its last layer, using the output features of the second last layer instead. For the frozen language model, we adopt a pre-trained LLaMA-7B [43] model and fine-tune it using the ShareGPT dataset and a GPT-4 generated 52K English mnstruction-followiii dataset [52]. We then utilize the well-finettundd language model as the frozen language model for vision-language pre-training. Additionally, we convert the data type of parameters of the frozen VIT [53] and language model to FP16 during pre-training to increase efficiency.
3.4 Creating EgoCOT and VQo Dataset
For our EgoCOT dataset, we obtain basic data from the Ego4D dataset [16], which includes 9, 645 untrimmed videos of various durations ranging from 5 seconds to 7 hours. To prepare the data for our purposes, we conducted two stages of data cleaning to prepare our data. In the first stage, we filtered out videos with missing or very short narrations (which made up 7.4% and 0.9% of the text, respectively), as well as those with unsure tags (which accounted for 4.0% of the text). We also excluded videos without human-object interaction, such as watching TV or walking. After this stage, we were left with 2.9 thousand hours of video, containing 3.85 million narrations, from 129 different scenarios covering 2927 hours of video.
To generate pairs of captions, embodied plannings, and corresponding video segments with time intervals, we utilized the ELP framework [54] to segment the video. The narrations are organized as a sequence of sentences To, .., Tn with precise timestamps lo ... bn that indicate when a described event occurred. For each narration T, with timestamp t,, we paired with a clip V; by determining its start and end time points:
[t (and] = [t, - 3,/2a, to + B,/2a),
(I)
where 3; =E (4)+1 /n is an adjustable parameter equal to the average temporal distance between consecutive narrations in a given video. Conversely,
a scale factor computed as the average of all B; across all videos in the EgoCOT dataset (a = 4.9 seconds). For each video segment, we provide prompts and corresponding captions for ChatGPT [55] to generate a reasonable and detailed embodied planning. The caption is typically a brief introduction such as "C opens a drawer." We use the ChatGPT to generate a chain of thought according to the caption and organize it into a list of verb-noun pairs, such as "plans: grasp the handle with the gripper and pull the handle; actions: grasp(handle, gripper) 2. pull(handle). " "The prompt we used to generate EgoCOT dataset is shown in Listing 2. To enhance the diversity of generated chain of thoughts, we employ a temperature parameter of 0.9 and a top-p parameter of 0.95. For each prompt, we perform five sampling iterations.
fost-procedure. To ensure the quality of the generated planning instructions, we perform the second stage of data cleaning. We used the CLIP model [56] to assess the similarities between the video and text pairs. For each video, we compared it against five potential embodied plans and selected the one with the highest similarity as the corresponding label for the embodied plan. We then took our data-cleaning process a step further by filtering out any video-caption caption-planning pairs with similarities lower than the threshold. We eliminated both data with the low similarity between the video and caption and between the video and planning to ensure the highest quality data for our EgoCOT dataset. For each keyframe of the video segment, we use the CLIP model to encode both the text data T' and the image data I into shared embedding space. The similarity is calculated using the

---

You need to generate plens with chain of thought for each task, and then extract detailed actiona (collocation of nouna and verba) from the plan.
folloving form:
saction_nane] eg., turn left;
[action_name] argumenti, eg., pick up(apple); [action_name] argunenti argunent2,
(uplaapl,, table) plans: graep the handle of the cup with the gripper and lift up
Task: pick up cup on the table
Actions:
graep
cup, gripper)
2. lift ucup))
Listing 2: Prompt we used for creating EgoCOT dataset.
cosine similarity function as Syrr, vr) = I wrll where Syr yr) denotes the similarity between the text and image, and yr and are the respective embeddings. Given that each video contains multiple keyframes, ensemble of similarity scores is obtained for each video. This ensemble strategy helps to alleviate the problem of variability among individual frames and ensures a more robust and representative measure of overall similarity. The ensemble similarity score between video V with n keyframes and text data T is given by:
yr-yy
n
E(V, T)
Σ S(yra 72
where E(V, T) is the ensemble similarity score, S(yr 91;) is the similarity score for the i-th keyframe, and is the total number of keyframes. We also created the EgoVQA dataset specifically for egocentric human-object interaction video question answering tasks to enrich the training data. For each caption in the Ego4D dataset,
used ChatGPT generate five QA pairs. To ensure relevance, we guided ChatGPT to focus on core key verbs and nouns by designing prompts as shown in Listing 3. The sampling schema when crafting EgoVQA the same to that EgoCOT.
sone questions accroding For example, in this sentence "a man is picking cup", the verb is picking and the questions
and noune
noun is cup, "shat operation is performed the cup?". Then You need to give the anever.
"shat object man is picking up?"
input: a man is picking a object ansver: The cup
question:
man is picking up
Listing 3: Prompt used for creating EgoVQA dataset.
Experiments
In this section, present comprehensive evaluation of multi-modal foundation models and EmbodiedGPT, across various tasks including visual captioning, embodied planning, and control.
Evaluation on image input tasks. In order to evaluate the quality of generated captions and planning with the given image, we conducted user study with 30 participants. The study included 10 cases of image caption tasks from MS-COCO dataset [44], 5 embodied planning scenarios in different embodied AT simulators, and real-world scenes with accompanying embodied planning tasks. Participants were asked to rate the generated captions from different end-to-end models on five dimensions using a scoring system ranging from to 10: object recognition accuracy, spatial
Model
Object(t)
Spatial(t)
Redundancy(L)
Plan Reasonable(t)
Plan Executable(t)
Minigpt4
5.6
4,8
4.4
4.5
4.8
LLaVA-7B
7.3
7.4
3.9
7.5
6.6
LLaVA-13B
8.5
8.6
3.4
8.4
7.6
EmbodiedGPT
8.4
8.8
2.6
8.8
8.4
Table Generate Quality Evaluation on image input tasks.
7

---

Humaas detailed explonation the anage, deseritle scene and the observed objects, spuzial relanonshipe. Where should go It') wish find cont lisager?
EmbodiedGGPT:
The image depicts doorway leading into bedroom, with clock hanging on the wall beside it and of
further right.
There a closet suitable storing clothes on right side bedroom. 3. if fooe needs to hang a cost, the white closet on the left seem to be the lngical choice.
Visual CharGiF:
The shows bataroom with clock on wall and toilet the comerr The clock white and has black face with
owalls tiled The image shows batihroom with clock and toilet. There is no cost hanger visible in the imauge
window un roem.
Figure 3: Comparison between EmbodiedGPT and VisualGPT in the question-answering task.
Video:
Human: describe this video in detail
Assistant:
The video displays a red robotic the ring positioned next to it, with their centers aligned. The robotic arm is then used to pick up the ring. Carefully, the center of the ring is aligned with the center of the cylinder, and the ring is slowly lowered onto the cylinder, joining the two together. The final step is to inspect the join using the robotic arm, ensuring that it is secure and without gaps.
red cylinder, and a ring. The cylinder is positioned on flat surface, and
Human: can you write a detailed plan for the task the robot is performing?
Assistant:
1. Position the first object, cylinder, on a flat surface.
2. Position the second object, a ring, next to the red cylinder, with their centers aligned.
Use the robatic arm to pick up the ring.
Caretully align the center the ring with the center of the cylinder.
Slowly lower the ring onto the cylinder, joining the two objects together.
Figure 4: Example of video input embodied Al tasks on Meta- BeradWorld benchmark. EmbodiedGPT accurately analyzes embodied control tasks in demonstration videos and provides precise planning.
relationship understanding. level of redundancy in the answer, and reasonability of the planning and the executability of the planning. The average scores among all the participants for different models are shown in Table 1. The results demonstrate that EmbodiedGPT achieves a comparable level of object recognition and spatial relationship understanding as the LLaVA-11 model, despite having only 7B parameters in the language model. Furthermore, EmbodiedGPT generates less redundant content in relation to the given embodied AI task, and produces the most reasonable and executable planning outputs. We also compared the performance of EmbodiedGPT with Visual ChatGPT [33], which adopts a hierarchical approach by combining several pre-trained vision models and language models to answer questions. In the Virtual-Homa [57] benchmark, Visual ChatGPT uses a visual caption model to generate dense captions that are subsequently passed into ChatGPT for deriving a solution. As shown in Figure 3, Visual ChatGPT failed to find coat hanger due to its limitations of relying solely on the caption model for extracting visual information, resulting in poor performance when compared to the end-to-end model like EmbodiedGPT. These findings highlight the advantages of adopting a unified, end-to-end model over hierarchical approaches that rely on multiple stages.
Evaluation on video input embodied AI tasks. We evaluate the recognition ability of videos and planning abilities of our model for embodied control tasks on standard embodied AI benchmarks, Franka Kitchen [14] and WerldWorld [15]. Meta-World provides challenging set of tasks that require complex object manipulation skills, including assembling a ring on a peg. picking and placing a block between bins, pushing a button, opening drawer, and hammering a nail. Franka Kitchen benchmark focuses on tasks like sliding open the right door, opening the cabinet, turning on the light, turning the stovetop knob, and opening the microwave. As shown in Figure 4, given demonstration video, EmbodiedGPT can accurately interpret the embodied control task and provide step-by-step planning. The output planning is fed into the babbodidd-forme module of EmbodiedGPT to query
8

---

Suices Ralt
60
RBM
Embooliddgpt
40
oabinee Jeft
webirei_nght
micro Jeit
mirr right
knote, Jeit
knobs right
let
ighh_right
adoor juft
Adbor Light
Performance comparison in Franka Kitchen with only 10 demos.
BLIP 21Ege40)
SHOKSS Rate
60
RAM
EmoodilesrPP
20
bhrufggt
buttonJief
buttinn Aght assemby_JoR y_
Srawer
hammerjef hammerrigg
(b) Performance comparison in Meta-World with only 10 demos.
Figure 5: Performance of EmbodiedGPT in low-level control tasks with 10 demonstration demos.
SUP-21tg:4D)
1.3
Emood
cabrnet.ngtt
micr
fgtht
right
light
Fight right
500o7 right
(a) Performance comparison in Franka Kitchen with 25 demos.
Success Patt
EmboskeaorP
kinjen
Nutton, Aight
wovembly Jef
y.9gt
drower jett
dmawe digh
lammerJeh
hammer foght
(b) Performance comparison in Meta- Meta-World with 25 demos.
Figure 6: Performance of EmbodiedGPT in low-level control tasks with 25 demonstration demos.
highly relevant features for inputs in the policy network and the low-level actions generated by the policy network to interact with the environment (see more visualizations Appendix B).
Evaluation on embodied control tasks. For embodied control tasks, we compare our model with R3M[12], which is the state-of-the BLIP-2[Ego4D]] which has the same structure and same amount of parameters as EmbodiedGPT,, and only fine-tuned on the video caption task using the Ego4D dataset without incorporating EgoCOT. In all experiments, the policy network is learned using few-shot learning small amount of demonstration data. There are two settings, one which utilizes demonstrations, and the other utilizes 25 demonstrations. We report the success rate in 100 random evaluations with only visual observations in 5 tasks per benchmark over 5 seeds and 2 different camera views for each setting, respectively, As shown in Figure and Figure 6, EmbodiedGPT outperforms the bascline methods, demonstrating the effectiveness of learning with EgoCOT.
method in these two benchmarks, and an ablation version called
Model
Franka(10 demos) Franka(25 demos)
Meta- World(10 demos)
Meta- Pera-Worldd255aoo demos)
EmbodiedGPT
50.8% +:2.8
58.5% +2.7
76.4%
+.22
81.2%
Close-loop
38.6% +.2.9
47.3% +.2.5
62.7% 4.2.2
64.9% 4.2.0
COT
26.2% +3.2
36.4% +2.7
55.2% +2.4
58.7% +2.0
Table 2: Ablation on the closed-loop spans from planning to low-level control, and "chain-of- tf-thughhth (COT) training with 25 and 10 demonstrations("" symbol indicates "removing"). We report the average success rate over tasks and camera views per henchmark.

---

You are the controller of the robot shown in this image, now your task is put all the objects into the brown bag next to the robot, please write step by step plan:
Human:
EmbodiedGPT:
Step1. Move the robotic arm to the top of the gray bottle.
Step 2. Close the gripper and grasp gray bottle.
Step 3. Move the arm to the brown bag.
Step 4. Release the bottle.
Step Move the
Step Close the gripper top the cereal bag.
grasp cereal bag. Step Move the the brown bag.
Step Finally, release the cereal bag.
Franka Emika Panda arm with frant facing camera
Figure 7: Generated plan by EmbodiedGPT for rearrangement task.
Figure 8: Real-world experiment on the Franka Emika robot arm.
Ablation study. We perform ablation studies to analyze the effectiveness of the CChain-of-Thoughhe training mode and the importance of closed-loop design for embodied control. The results, shown in Table 2, demonstrate a significant improvement success rate when using the EgoCOT approach compared training solely with the EGO4D caption task. Moreover, the closed-loop design is necessary as the generated plans contained specific and relevant sub-goal information, which proved crucial for control tasks. In summary, EmbodiedGPT exhibits strong ability to generate reasonable planning, accurately extract task-relevant features from visual inputs, as well as execute low-level actions interact with the environment. The ablation experiments demonstrate that both the training paradigm based on EgoCOT and the closed-loop design from embodied planning to low-level control significantly contribute the performance improvement of EmbodiedGPT.
Real robot experiment. We also conducted a real-world experiment using the Franka Emika robot arm, emphasizing rearrangement task where the objective was to pack scattered bottles from table into a box. As depicted in Figures and 8, model is capable of generating intricate plans and executing of low-level actions based 50 demonstration.
5 Conclusion
In this paper, we present EmbodiedGPT, an end-to-end multi-mode foundational model for embodied Al that enables agents to perform step-by-step planning and execute low-level commands. To achieve this, we create large-scale scale embodied planning dataset called EgoCOT and develop an efficient training approach that utilizes prefix tuning to generate high-quality plans with "chain-of-thought" Furthermore, our embodied control paradigm seamlessly coordinates high-level planning and low- level control. Extensive experiments demonstrate the effectiveness of EmbodiedGPT on various embodied tasks, achieving statc-of-the-an or comparable performance. We believe that EmbodicdGPT represents significant step towards developing more intelligent embodied AI agents.
Future works and limitations: EmbodiedGPT freezes the parameters of the vision and language model due to limited computational resources. Joint training with all modules and exploring other modalities, such as speech, could be future works. We do not foresee obvious undesirable ethical or social impacts at this moment.
Acknowledgement. This paper is partially supported by National Key R&D Program China No. N0.2022ZD011 61 61000 and the General Research Fund of Hong Kong No. 1720062.. We would like to express our appreciation to Dr. René Zurbrügg from ETH AT Center for his support on robot experimental platform.
10

---

References
[1] OpenAL. Gpt-4 technical report, 2023.
[2] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng. Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint ar3ivv23333.03333 2023.
[3] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.
[4] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv: Arxiv-2210003000 2022.
[5] Yizhou Zhao, Qiaozi Gao, Liang Qiu, Govind Thattai, and Gaurav S Sukhatme. Opend: A benchmark for language-driven door and drawer opening. arXiv preprint arXiv:2212.05211, 2022.
[6] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference Robot Learning, pages 894 906. PMLR, 2022.
[7] Kaizhi Zheng. Xiaotong Chen, Odest Chadwicke Jenkins. and Xin Wang. VImbench: A compositional benchmark for siiiion-nndllanuuaaa manipulation. Advances in Neural Information Processing Systems, 35:665-678 6688, 2022.
[8] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu. Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023.
[9] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficeent visual instruction model, arXiv preprint arXiv:2304.155 15010, 2023.
[10] Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. Lim-adapters: An adapter family for parameter-efficien fine-tuning of large language models. arXiv preprint arXiv:2304.0199 2023.
[11] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Ahu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 2021.
[12] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12661 2022.
[13] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. CoRR, abs/2301. 12597, 2023.
[14] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving honghoorzoon tasks via imitation and reinforcement learning. arXiv preprint arXiv: 1910. 1956, 2019.
[15] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1 100. PMLR, 2020.
[16] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1-995-19013 2022.
[17] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping Linguage-inage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12555 2023.
[18] Yen
Yen-Chun Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In ECCV, volume 12375, pages 104-120, 2020.
11

---

[19] Xiujun Li, Xi Yin. Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei. Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, pages 121-137, 2020.
[20] Pengehuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang. Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Making visual representations matter in vision- Sainnnlanguage models. arXiv preprint arXiv:2101.0053 2021.
[21] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers. Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, pages 18102-18112, 2022.
[22] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, NeurlPS, pages 200-212, 2021.
[23] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning, In CVPR, pages 18009-18019, 2022.
[24] Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong. Sina Samangooei, Marianne Monteiro, Jacoh Menick, Sehastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning, arXiv preprint wxXiv:2204.441999 2022.
[25] Fabian Caba Heilbron. Victor Escorcia, Bemard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961-970, 2015.
[26] Yazan Abu Farha, Alexander Richard, and Juergen Gall. When will you do what?-anticipating temporal occurrences of activities. In CVPR, pages
2018.
[27] Benita Wong, Joya Chen, You Wu, Stan Wcixian Lei, Dongxing Mao, Difci Gao, and Mike Zheng Shou. Assistq: Affordance-centric question-driven task completion for egocentric assistant. In ECCV, 2022.
[28] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egncentric vision: Collection, pipeline and challenges for epic-kitchens- -00e IJV, 130(1):33 3)3-555, 2022.
[29] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-cgo: A large-scale dataset of paired third and first person vidcos. arXiv preprint arXiv: 00466666 2018.
[30] Yin Li, Zhefan Ye, and James M Rehg. Delving into egocentric actions. In CVPR, pages 287-295, 2015.
[31] Yecheng Jason Ma. Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.000:: 2022.
[32] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz I Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
I Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances
[33] Chenfei Wu, Shengming Yin, Weizhen Qi. Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. CoRR, abs/2303.046) 2023.
[34] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint 2333: 1381, 2023.
[35] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving Al tasks with chatgpt and its friends in huggingface. CoRR, abs/2303. 17580, 2023.
[36] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models, 2023.
12

---

[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. CoRR. abs/2304.08485, 2023.
[38]
KunChang Li. Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang. Limin Wang. and Yu Qiao. Videnchat: Chat-centric video understanding. arXiv preprint arXiv: 2305.06355, 2023.
[39] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[40] Feilong Chen, Minglun Han. Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-IIm: Bootstrapping advanced large language models by treating multi-modalitie as forcign languages. arXiv preprint arXiv:2305.00.000 040000 2023.
[41] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chehotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine. Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff. Audy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. CoRR, abs/2303.03378, 2023.
[42] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint
2022.
[43] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2302 2023.
[44] Tsung- Yi Lin, Michael Maire, Serge J. Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, ECCV, volume 8693, pages 740-755, 2014.
[45] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Pupers), pages 2556-2565, 2018.
[46] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0845. 2023.
[47] Rahul Dev Singh, Ajay Mittal, and Rajesh K Bhatia. 3d convolutional neural network for object recognition: a review. Multimedia Tools and Applications, 78:15951 15995,, 2019.
[48] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[49] Jin Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEFE conference on computer vision and pattern recognition, pages 248-255
Ieee, 2009.
[50] Martin Riedmiller and A Lemeen Multi layer perceptron. Machine Learning Lab Special Lecture, University of Freiburg, pages 7-24, 2014.
[51] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang. and Yue Cao. Eva: Exploring the limits of masked visual representation learning scale. arXiv preprint arXiv:2211.0766 2022.
[52] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023.
[53] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. In ICIR, 2020.
[54] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z. XU, Difei Gao, Rong- Cheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, WANG HongFa, Dima Damen, Bemard Ghanem, Wei Liu, and Mike Zheng Shou. Egocentric video-language pretraining. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors.Avvaneee Advances in Neural Information Processing Systems. volume 35, pages 7575-7586. Curran Associates, Inc., 2022.
13

---

[55] OpenAL. Chatgpt (mar 14 version) [large language model], 2023.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint XxXi:2103.00020 2021.
[57] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torrala. Virtualhome: Simulating household activities via programs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494 8502, 2018.
14


Reviews:
[{"review_id": "mboGfFhfAb", "reviewer": "Reviewer_BD8Q", "summary": "This paper introduces EmbodiedGPT, a new multi-modal foundation model designed for application to embodied (robotic) tasks. EmbodiedGPT combines existing large pretrained (and frozen) vision and language models (namely the ViT-G/14 EVA-CLIP vision model and LLaMA-7B) through the use of an \"Embodied-Former\" which converts input text tokens and vision features into prefix tokens that can be fed directly to the LLaMA model. To pretrain this Embodied-Former model, this work introduces two new datasets (EgoCOT and EgoVQA), which contain video clips taken from the Ego4D dataset annotated with text captions generated by ChatGPT that, for EgoCOT, describe chain-of-thought-style plans corresponding to the actions taken within the clip, and, for EgoVQ, include questions and answers related to the clip. Using these new datasets and a 3-stage training pipeline, this paper trains the EmbodiedGPT model and, via human evaluations, show that humans generally prefer the captions and embodied plans generated by EmbodiedGPT even when compared to a LLaVA-13B baseline which has many more parameters. Finally, the paper shows that, by attaching an MLP policy head to the output of the Embodied-Former, it is possible to use imitation learning from a small set of demonstrations to train performant physical-manipulation agents in simulation. Experiments show that EmbodiedGPT outperforms ablations of itself and a R3M-based SoTA model for these tasks.", "questions": "I have listed some of my major questions and weaknesses above. If some of these weaknesses can be addressed (especially the point about statistical significance) I expect to increase my score. In the below I include, more minor, line-by-line comments and questions.\n\nLine 60\n- \"EmbodiedGPT ’s\" should be \"EmbodiedGPT’s\". I suspect this is a problem due to latex macros, try adding {} after your macro inline, i.e. \\mymodel{}.\n\nLine 77\n- This \"200M annotated videos\" statement seems like it must be wrong. Later on, you note that there are 2927 hours of video in total, this is approximately 10M seconds of video which would correspond to ~200M frames depending on the framerate, is that what you mean?\n\nFigure 2\n- This figure could use some work, for instance, it's very unclear how videos are being input into the model.\n\nLines 146-147\n- What is the value of the \"learnable embodied queries\"? What makes them \"embodied\"?\n\nLines 172-174\n- I found this description of the video encoding procedure to be too vague, please add more details.\n\nLines 181-204\n- This entire discussion could be simplified and better presented, the idea here is quite simple and much of the formalism is extraneous. For instance, you introduce a \"mapping function (...) M: z -> z'\" but this mapping function is just a linear projection and the \"M\" notation seems to be never used again. You could just say \"z is then mapped to z' by via a linear projection.\"\n\nLine 241\n- Extra space after the comma in 9,645.\n\nLines 225-228\n- How is \\beta_i \"adjustable\" if it's defined in this way? It seems to be a fixed function of other fixed values.\n- Why not just set [t^{start}_{i}, t^{end}_{i}] to be [(t_{i-1}+t_{i})/2, (t_{i+1} + t_{i})/2]?\n\nFigure 3\n- This seems like a surprising qualitative example to choose as the EmbodiedGPT model says many wrong things. E.g. (1) the clock is not actually above the door, (2) the coat hanger is not on the floor, and (3) the coat hangers can be seen towards the top left, not the \"near the right side of the doorway\".\n\nFigure 4\n- Again, this is a surprising example as the video shows it lowering the ring over the cylinder twice whereas EmbodiedGPT seems to only suggest this happens once.", "limitations": "I would have liked to see some discussion of the limitations I discussed in the _Really \"foundational\"?_ section of the weakness. Otherwise, limitations are reasonably well addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper studies a very popular, important, and challenging area of work: building foundational models for robotic control and embodied world understanding. The proposed EmbodiedGPT model represents a modest but non-trivial step towards this grand goal. In my view, the major contributions of this are the substantial efforts taken to create the EgoCOT/EgoVQA datasets and execute the complex and computationally expensive 3-stage strategy required for training the EmbodiedGPT model. Generating these datasets and training the model clearly required significant effort and the experimental results appear quite strong.", "weaknesses": "While I lean positive for this paper, I will highlight some weaknesses below that temper my excitement.\n\n## Statistical significance of the human results\n\nThe only evaluation of EmbodiedGPT against other existing large vision-and-language models comes in a 30-participant human study using 20 tasks (10 captioning, 5 simulated planning, 5 real planning). As this sample size is quite small, and the performance difference sometimes relatively minor, it's challenging to draw meaningful conclusions from the results in Table 1. Moreover, it is hard to know how representative the 20 tasks are of general captioning/planning tasks, how were these chosen?\n\nThere are ways to provide a statistical analysis of this data but they are not trivial: you'd need to account for the correlated nature of the data (e.g. that individual participants rated multiple examples and that performance may be correlated with the task type and within each question). While not a perfect treatment, at the very least a paired t-test could be used to compare EmbodiedGPT with the other models.\n\n## Really \"foundational\"?\n\nWhile EmbodiedGPT is touted as a \"foundation model for embodied AI\", the results do not quite support this in my view. In particular, I feel that the EmbodiedGPT misses the critical capacity to handle partial observability and I was surprised not to see any real-world examples of robotic control, more details below.\n\n### Lack of tasks involving partial observability\n\nThe only embodied tasks evaluated in this paper involve static robotic arms acting in fully observed (or almost fully observed) environments. There are many embodied tasks (e.g. visual navigation) for which partial observability plays a critical role. From my understanding, the policy component of the EmbodiedGPT model makes decisions from a single visual observation in isolation from all previously seen states and thus would be expected to fail in all such tasks.\n\n### Lack of application to real-world embodied tasks\n\nWhile it is great to see that EmbodiedGPT outperforms the R3M model in the simulated environments, one of the strengths of the R3M paper was demonstrating that it also functioned well when applied to real robots. While not necessary for acceptance, it's difficult to call EmbodiedGPT a robotic foundation model without at least some real-world results.\n\n## Architectural novelty\n\nThis is a very minor point as I consider the training of large models to be a significant contribution in itself, but the proposed EmbodiedGPT architecture is not especially novel: it combines popular pretrained vision/language models with well-known prefix tuning ideas. While you cite Tsimpoukelli et al. (2021), I was a bit surprised to see more discussion of how their work relates to the proposed approach. Moreover, simply searching for \"embodied\" in the citations of the above paper (i.e. https://scholar.google.com/scholar?hl=en&as_sdt=5%2C48&sciodt=0%2C48&cites=16154696122208258147&scipsc=1&q=embodied) appears to surface several additional citations that should be included in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your careful review and meticulous line-by-line suggestions!\n\n**Q1. [It is hard to know how representative the 20 tasks are of general captioning/planning tasks, how were these chosen?]**\n\nTo ensure comprehensive representation, the tasks are selected across real-world and simulated settings (Virtual Home, Habitat, and AI2THOR) that span various indoor areas like kitchens, bedrooms, and offices. These tasks address key interaction and cognitive elements, from basic actions like \"how to wash a vegetable\" to complex reasoning questions indicative of decision-making and planning.\n\n**Q2. [User study's sample size and lack of statistical significance testing in human study.]** \n\nWe've conducted a Welch t-test comparing EmbodiedGPT to the baselines. The table below presents the T-statistic and P-value derived from the responses of 30 individuals. A p-value below 0.05 signifies a statistically significant difference between the groups. EmbodiedGPT demonstrates a statistically significant improvement over the baselines, underscoring its effectiveness.\n| Scoring Standard | Groups Compared | T-statistic | P-value  | Significant |\n|------------------|-----------------|-------------|----------|-------------|\n| Plan Reasonable  | Minigp4 vs EmbodiedGPT | -23.39 | 6.26e-30 | Yes |\n| Plan Reasonable  | LLaVA-7B vs EmbodiedGPT | -8.15  | 1.73e-11 | Yes |\n| Plan Reasonable  | LLaVA-13B vs EmbodiedGPT | -2.35  | 0.022    | Yes |\n| Plan Executable  | Minigp4 vs EmbodiedGPT | -24.74 | 3.90e-31 | Yes |\n| Plan Executable  | LLaVA-7B vs EmbodiedGPT | -9.19  | 2.61e-13 | Yes |\n| Plan Executable  | LLaVA-13B vs EmbodiedGPT | -2.12  | 0.038    | Yes |\n\n**Q3. [Lack of evaluation on tasks requiring partial observability.]** \n\nGood Question! We address your concerns in the following aspects.\n- **EmbodiedGPT's Design with Respect to Partial Observability**: EmbodiedGPT has been carefully designed to consider the problem of partial observability. It supports multi-frame input and is capable of understanding the changes and motions across different frames thereby handling occlusion. Specifically, through the use of EgoCOT training, it can easily capture the variations and partial observations in object states/motions.\n- **Additional Experiments**: We add experiments on the **visual navigation task** which aims to find the flush toilet with partial observation. The results show that EmbodiedGPT provides robust priors and efficiently consolidates historical observations into textual records for future decisions. See detailed results **Figure 1 of rebuttal PDF**  in General Response. \n\n**Q4. [Lack of application to real-world embodied tasks.]** \n\nWe add a real-world experiment on the Franka Emika robot arm, focusing on a rearrangement task to pack bottles cluttering the table into a box. See the demo in the **Figure 2 of rebuttal PDF**. \n\n**Q5. [Architectural novelty.]** \n\nThank you for recognizing that our training large models is a significant contribution in itself. We will conduct a broader literature review.\n\n1. We'd like to clarify that the primary novelty of our work does not reside in the architecture design itself, but rather in the unique approach we have taken in the \"embodied chain of thought\" training paradigm and the data handling. The core emphasis of our work is on guiding the model to understand the interactivity of objects through intuitive knowledge, such as \"if a door has a handle, one must grasp the handle to open it.\" This stands in contrast to conventional multimodal pre-trained models that mainly focus on grounding in common semantics.\n2. The contributions we have highlighted concerning architecture design are not meant to stress the key novelty but rather to offer an efficient and feasible training method within these constraints. \n\n**Q6. [Video Input Mechanism into the Model]**\n\nVideos are processed by:\n1. Keyframe Extraction: Extracting eight uniformly spaced keyframes.\n2. Segmentation to 3D patches: Keyframes are fed into the Conv3D module, segmented into 3D patches each defined by `patch_size*patch_size*frame_stride`.\n3. Patch Representation: These patches act as spatio-temporal cubes in the video, capturing both visual content and temporal sequence.\n4. Token Conversion: Patches are transformed into visual tokens via Conv3D  with \" kernel_size=(frame_stride, patch_size, patch_size), stride=(frame_stride, patch_size, patch_size) \"and then fed into the vision transformer.\n\n**Q7.[What is the value of the \"learnable embodied queries\"? What makes them \"embodied\"?]**\n\nThe learnable embodied queries are vector embeddings that interact with the visual features through cross-attention layers in the Embodied-Former module (Section 3.2). They are considered \"embodied\" because their goal is to extract only the most relevant visual information for embodied tasks like planning and control. These queries offer a compact representation with task-specific semantics and also reduce the input tokens to the language model(an image is encoded into far more visual tokens than text tokens).\n\n**Q8.[How is $\\beta_i$ \"adjustable\".]**\n\nSorry for the confusion. We would like to clarify that the discrepancies in the length of video segments corresponding to a single caption are inherent to the dataset itself. The calculation method we used is fixed, and we apologize for any confusion this may have caused.\n\n**Q9.[Minor issues and typos.]**\n\nWe will revise our paper carefully according to your suggestions.\nAs for Figure 3, spatial misalignment is prevalent in image captioning, even in state-of-the-art models. Scaling the model can enhance detail alignment and capacity. As for Figure 4, we apologize for the mistake in the order of the plotted frames. We appreciate your keen eye and patience, and we have revised it with the correct order in **Figure 3 of rebuttal PDF**.\n\n\\\nThanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window.\n"}, {"review_id": "7QDqjTMhmU", "reviewer": "Reviewer_P2YP", "summary": "In this work, the authors propose an end-to-end multi-modal model for Embodied AI field. Based on the Ego4D dataset, the authors build a large-scale Embodied planning dataset, i.e., EgoCOT, which contains machine-generated language instruction with chain-of-thoughts mode. Besides, an EmbodiedGPT based on LLM is proposed to achieve the plan generation task, where the corresponding training approach is designed for prefix tuning on EgoCOT. Experimental results demonstrate that EmbodiedGPT can achieve the promising ability of plan generation.", "questions": "See Weaknesses.", "limitations": "I do not foresee obvious undesirable ethical or social impacts at this moment.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This work provides the EgoCOT dataset with EgoVQA as an extension, which can promote the development of LLM in the Embodied AI field.\n\n- This work makes the pioneering attempt to build a foundation model with the chain-of-thought capability. Building the framework based on the Vision Transformer and LLaMA, and designing the Embodied-Former as the bridge between the visual and language modality is reasonable.\n\n- Extensive quantitative and qualitative results are provided for examining the model's effectiveness. The results show that EmbodiedGPT can produce reasonable answers to describe what happened in the input video and generate the step-by-step action sequence.\n\n- The paper is well written and easy to follow with detailed instructions about how to construct and train the model.", "weaknesses": "- In the field of Embodied AI, an important task is \\*agent navigation*, but this work does not cover this part. I understand that the authors focus on the multi-modal foundation model, and there is a vision-language navigation (VLN) task [1][2] that requires the agent to conduct navigation according to the given language instruction. Thus I believe the VLN is a promising task for authors to test the model's navigation ability, e.g., verifying if the model can generate step-by-step navigation instructions with chain-of-thoughts.\n\n- I am confused about the setting of EmbodiedVQA in this paper. In my view, EmbodiedVQA [3] is actually the task that embodied agents navigate in rich 3D environments in order to answer the given question.\n\n- Currently, the model can generate detailed descriptions of a given video and convert them into planning steps with chain-of-thoughts. However, can the model generate executable steps for completing a specific task based solely on a static image of an object, rather than a demonstration video?\n\n- The authors should provide more details on how to use the model for Embodied control, specifically how to convert the text output into step-by-step control signals, as this is a crucial aspect for implementing embodied agents.\n\n- Minor: the colors of Figure 2 are a bit chaotic, and if there is no special meaning, I suggest it can be simplified.\n\n[1] Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR 2018.\n\n[2] Reverie: Remote embodied visual referring expression in real indoor environments. In CVPR 2020.\n\n[3] Embodied Question Answering. In CVPR 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you have invested in reviewing our paper. Your meticulous evaluation and insightful recommendations have significantly contributed to improving the quality of our research.\n\n**Q1.[Evaluating on vision-language navigation (VLN) tasks to test the navigation ability.]**\n\nThank you for your insightful suggestion regarding the exploration of vision-language navigation (VLN) tasks with EmbodiedGPT. We concur that VLN is a fascinating direction for future investigation, particularly as it involves the generation of navigation instructions.\n\nTo that end, we have conducted an experiment on the vision-language navigation task using EmbodiedGPT. The task chosen was an object-finding exercise, guided by the human instruction, \"please find a flush toilet.\" EmbodiedGPT processed both historical and current observations to infer the direction needed to locate the target object. The performance details can be found in **Figure 1 of rebuttal PDF**.\n\nEmbodiedGPT demonstrates the ability to generate step-by-step navigation instructions, employing a coherent chain of thought when provided with language instructions and visual observations of new environments. We appreciate this recommendation and plan to extend EmbodiedGPT's navigation capabilities, benchmarking it on VLN tasks in future research.\n\n**Q2.[Confused about our embodied VQA setting compared to prior work.]**\n\nSorry for the confusion. We would like to clarify that the Embodied VQA tasks presented in our paper are specifically focused on egocentric vision question answering for both embodied tasks and general recognition tasks, such as video/image recognition and task-oriented planning. This approach distinguishes itself from navigation-based EmbodiedVQA tasks as defined in prior work, which necessitates navigating complex 3D environments. We will revise the paper to make our setting clearer. Once again, we thank you for pointing out this crucial aspect.\n\n\n**Q3.[If the model can generate plans from static images without demonstration videos.]**\n\nAs illustrated in the appendix figures (Figure 3 and Figure 4), EmbodiedGPT can generate plans based solely on static images of environments or objects, without the necessity of demonstration videos. While videos may offer supplementary guidance, the model is fully capable of formulating reasonable plans utilizing static images alone. To further substantiate this capability, we have included additional experiments on an embodied rearrangement task, as demonstrated in **Figure 2 of rebuttal pdf**, EmbodiedGPT can also generate clear plans with static images.\n\n**Q4.[More implementation details on converting text plans to control signals.]**\n\nSpecifically, we leverage cross-attention modules in the Embodied-Former to extract features relevant to the plan from the encoded visual observations. These plan-relevant features are then passed to the policy network, which maps them to parameterized low-level actions and control signals, such as gripper poses and joint velocities, that can be executed to perform the task.\n\n\n\\\nThanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window."}, {"review_id": "mcROno1v4k", "reviewer": "Reviewer_5cDX", "summary": "This paper proposes an end-to-end multi-modal foundation model for embodied AI. Specifically, the authors create a dataset termed EgoCOT from the Ego4D dataset and introduce a paradigm for both the vision and embodied tasks using large language models. However, I have some concerns about this paper. My detailed comments are as follows.", "questions": "See the weaknesses.", "limitations": "See the weaknesses.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1、The paper well combines the large language models with embodied planning.\n2、The proposed EgoCOT dataset and EgoVQA dataset are helpful for downstream tasks such as embodied planning and VQA.", "weaknesses": "To me, this paper is close to being interesting but I, unfortunately, do feel there are some weaknesses that, currently, prevent me from recommending acceptance.\n1、The main idea has limited technical contribution:\nThe entire paper reads like a technical report and many parts of it are not coherent. The network architecture level in this regard can be said to have no innovative contribution. The frozen modules in the paradigm have been proposed and used in many papers related to large language models. Essentially, embodied former is also actually a cross-attention network used for visual and language modality interaction. As for the policy network, it simply encodes vision and combines it with multi-modal interaction output as input. I understand this paper has the purpose to use large language models for embodied tasks. However, at the method level, I see more about how to interact and align the image or video modality with the text modality, and in the experimental part, it also focuses on reflecting the ability of the method in image caption and object grounding, not even in some important experiment on embedded tasks (such as embedded navigation, embedded rearrangement, Instruction Following and Dialogue, etc.) \n\n2、As a side note, I would like to let the authors know about some flaws in the presentation of the paper:\n（1） In Section 3.1, in the second stage of the training, how are the language projection and prefix language adapter updating? I suggest that the authors explain this in detail.\n（2） In Section 3.1, in the third stage of the training, does the Conv3D module need to be learned? And how to decide the time offset to be 2? I didn’t see any explanation or ablation studies of these hyperparameters.\n（3） In Section 3.4, how to decide the start and end time points of a clip? Please explain the relationship of  \\beta and \\alpha and why the \\beta/2*\\alpha becomes the bound.\n（4）I suggest adding more descriptions of the tasks conducted in the experiments. For example, I am confused about that why are the indicators Plan Reasonable and Plan Executable related to image caption tasks from MS-COCO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts to improve the quality of our paper. We appreciate the flaws you identified in our paper's presentation and we believe they have been thoroughly clarified and addressed as follows.\n\n**Q1.[Limited novelty on architecture.]**\n\nWe would like to clarify that our approach's novelty lies principally in the embodied chain of thought training methodology and the EgoCOT dataset, not the architectural design. We provide a  generic multimodal pretraining with the chain-of-thought strategy applicable to most existing model architectures. The core focus is exploring how chain-of-thought training can impart understanding of object interactability - for example, grasping a handle to open a door. Such knowledge further enables generating executable, high-quality plans. In manipulation tasks, while other multimodal models tend to concentrate on the detection of objects present on a table, our focus shifts toward a more nuanced understanding. \n\nSpecifically, through the pre-training method guided by the embodied chain of thought, we endeavor to allow the multimodal system to comprehend how each object can be manipulated, as well as the intricate steps involved in a particular task. This distinct emphasis enables a more profound grasp of object interactions, fostering a comprehensive view of the task at hand, rather than simply identifying the presence of objects. It represents an advancement in modeling object manipulation and provides a more enriched understanding of complex task execution.\n\nAs Review BD8Q commented, architecture is a minor point as training large models is a significant contribution in itself. We present these architectures not to emphasize their novelty, but rather their utility in making large language model pretraining viable for embodied goals with modest resources.\n\n**Q2.[Add more experiment on embodied tasks such as embodied navigation, embodied rearrangement, Instruction Following and Dialogue, etc.]**\n\nThank you for your suggestion. We have added experiments on embodied navigation and embodied rearrangement. For navigation, we are focusing on the object navigation task of locating a flush toilet under visual inputs. See details in **Figure 1 of rebuttal PDF** at General response. Our experiments demonstrate that EmbodiedGPT offers robust priors for navigation tasks and can infer the most probable direction based on its knowledge. In the rearrangement task, we are using a Franka robotic arm to clean up cluttered items on a table and place them into a box.  See details in **Figure 2 of rebuttal PDF**.\nAs for the Instruction Following and Dialogue tasks, our experiments in the original paper already utilize human language instructions as input to accomplish specific tasks, and they support multi-turn dialogue. See Fig. 4 in our main paper and Fig. 2 in the Appendix for examples.\n\n**Q3.[ In Section 3.1, in the second stage of the training, how are the language projection and prefix language adapter updating? I suggest that the authors explain this in detail. ]**\n\nIn the second stage of training, the language projection layer and prefix language adapter are updated through backpropagation during pre-training. This allows the model to better capture complex language and reasoning skills. \n\n**Q4.[ In Section 3.1, in the third stage of the training, does the Conv3D module need to be learned? And how to decide the time offset to be 2?]**\n\n1) Yes, the Conv3D module is learned during the pre-training process to extract relevant temporal information. \n2) Regarding the choice of time offset being 2, it's mainly due to the constraints on computational resources and considerations for online inference speed. Our video input is limited to only 8 frames, and the time offset generally has two possible values: 2 and 4. We found that a time offset of 4 led to an inadequate understanding of the video content due to the larger span. Therefore, we selected 2 as the time offset. We acknowledge that with increased computational resources, the number of frames in the video input could be increased, which might lead to better performance, and there would be more options available for the time offset.\n\n**Q5.[In Section 3.4, how to decide the start and end time points of a clip? Please explain the relationship of $\\beta$ and $\\alpha$ and why the $\\beta/2\\*\\alpha$ becomes the bound.]**\n\nSorry for the confusion. $\\beta_i$ is defined as the average temporal distance between consecutive narrations in a video. $\\alpha$ is the average of all $\\beta_i$ across the dataset. So $\\beta_i/2\\*\\alpha$ determines the start and end time of a clip based on the timestamp $t_i$ of narration $T_i$. This creates a clip centered around the narration's timestamp with some padding. As shown in our paper, we use the same framework in the EgoVLP paper [A] to clip the video segment, and they have provided the ablation on the choice of clip strategy in their paper's Table 2.  \n\n[A] Qinghong Lin, Kevin, et al. \"Egocentric Video-Language Pretraining.\" NeurIPS 2022.\n\n**Q6.[Confused about that why are the indicators Plan Reasonable and Plan Executable related to image caption tasks from MS-COCO]**\n\nThe utilization of the MS-COCO image captioning task in our experiments is aimed at testing the model's general visual language understanding ability and to provide comparisons with other models. It is not specifically designed to evaluate the indicators \"Plan Reasonable\" and \"Plan Executable.\" These two indicators are not related to the image captioning tasks from MS-COCO, and we appreciate your feedback in highlighting this confusion. Further clarification will be added to the manuscript to ensure the distinction between the different tasks and evaluation metrics is clearer.\n\n\\\nWe wish that our response has addressed your concerns, and turns your assessment to the positive side. If you have any questions, please feel free to let us know during the rebuttal window. We appreciate your suggestions and comments! Thank you!"}, {"review_id": "m3nD5sMBXo", "reviewer": "Reviewer_eM5s", "summary": "The given text discusses the field of embodied AI and its applications in robotics. It highlights the potential of large language models (LLMs) in enabling robots to process natural language instructions, perform multi-modal tasks, and plan actions in physical environments. However, the collection of high-quality embodied multi-modal data poses challenges due to the need for egocentric data and structured language instructions.\n\nTo address these challenges, the authors propose the creation of two datasets: EgoCOT and EgoVQA. EgoCOT is a large-scale embodied planning dataset that contains egocentric videos from the Ego4D dataset, along with carefully generated, filtered, and human-verified step-by-step language instructions. EgoVQA is an extension of the Ego4D dataset, focusing on egocentric human-object interaction video question answering tasks.\nBased on these datasets, the authors present an end-to-end multi-modal embodied foundation model called EmbodiedGPT. \n\nIt consists of four integrated modules: a frozen vision model for encoding visual features, a frozen language model for natural language processing, an embodied-former for aligning visual and embodied instructions, and a policy network for low-level action generation. The model incorporates \"chain-of-thought\" planning capabilities and aims to enhance task performance and adaptability.\n\nThe contributions of this work include the development of the EmbodiedGPT model, the creation of the EgoCOT and EgoVQA datasets, and the demonstration of the model's effectiveness in various embodied tasks such as embodied control, embodied planning, video captioning, and video question-answering. EmbodiedGPT outperforms other models on benchmark tasks, showcasing its strong performance and advantages in generating reliable planning sequences and integrating high-level planning with low-level control.", "questions": "What is the computational cost of training such a large model?\n\nHow it the long-term behavior of EMbodiedGPT?\n\nHow can embodiedGPT benefit real-world robotic applications?", "limitations": " The authors adequately addressed the limitations", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Addressing a key challenge in embodied AI by creating new datasets: One major strength of the paper lies in recognizing a critical issue in embodied AI - the lack of good quality embodied multimodal data, especially those accompanied by structured language instructions. To tackle this issue, the authors created two new datasets - EgoCOT and EgoVQA which have the potential to significantly improve research in this area.\n\nProposing an effective solution with state-of-the-art results: Building upon these newly developed datasets, the authors introduced the EmbodiedGPT model. This model achieved impressive performance across multiple embodied tasks including embodied control, embodied planning, video captioning, and video question answering. \n\nFocus on improving the interpretability of LLMs: Another notable aspect of the paper is its focus on improving the transparency and explainability of LLMs used in embodied AI. The authors designed the EmbodiedGPT architecture so that each module can be interpreted separately or jointly analyzed together with attention weights. \n\nEncouraging open source collaboration through code release: Finally, the authors released their code publicly, fostering further research collaborations within the community. ", "weaknesses": "Potential overfitting to specific domains: Despite being trained on diverse tasks, EmbodedGPT might have become too specialized in certain areas where the training datasets were biased towards, leading to suboptimal generalization abilities when facing unseen situations or domains. \n\nBesides, the interpretability and explainability of the EmbodiedGPT may remain problematic, griven transformer-based training on language, vision, and action policy, which lacks explicit memory components to ensure accurate responses based on past experiences. \n\n Training EmbodedGPT requires substantial computational resources, making it less accessible to smaller organizations or individual researchers who cannot afford expensive hardware. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are immensely grateful for your invaluable efforts and insightful suggestions, which have been of great significance and played a crucial role in the improvement of our paper.\n\n**Q1.[Potential overfitting to specific domains]**\n\nGreat question! Your insight highlights the very reason we constructed the EgoCOT dataset, following the same format as the general Visual Question Answering (VQA) task. By aligning the structure in this way, it allows for the joint training of both the EgoCOT data and other general vision-language datasets, such as MS-COCO captions. This approach ensures that EmbodiedGPT incorporates a more diverse range of data, enhancing its ability to generalize across various tasks.\n\n\n**Q2.[interpretability and explainability]**\n\nwe recognize that it is an inherent challenge in dealing with large multi-modal models. We are actively investigating methods and chain-of-thoughts to enhance understanding, such as incorporating more explicit memory and reasoning, either from multi-frame/video or from symbolic language. This includes generating textual forms from symbolic graphs, which can provide more insights about the reason for decision-making, using language models and updating them continuously with new observations.\n\nThere is an example:\n\nsymbolic graph:\n```python\n{\n  \"Central node\": {\n    \"name\": \"Front view\",\n    \"features\": {\n      \"sink\": \"white\",\n      \"mirror\": \"above sink\",\n      \"atmosphere\": \"well-lit\",\n      \"trash can\": \"next to sink\",\n      \"floor\": \"blue tiled\"\n    },\n    \"connected_nodes\": [\"Left view\", \"Right view\"]\n  },\n  \"Left view\": {\n    \"name\": \"Left view\",\n    \"features\": {\n      \"urinals\": \"3 in a row\",\n      \"stalls\": {\n        \"left\": 2,\n        \"right\": 1\n      },\n      \"walls\": \"lined with white tiles\",\n      \"window\": \"above urinals\",\n      \"atmosphere\": \"spacious, well-lit\"\n    },\n    \"connected_nodes\": [\"Central node\"]\n  },\n  \"Right view\": {\n    \"name\": \"Right view\",\n    \"features\": {\n      \"sinks\": \"2 white sinks next to mirror\",\n      \"bottles\": \"2 (possibly containing hand sanitizer or other cleaning agents)\",\n      \"hand dryer\": \"mounted on the wall above the sinks\"\n    },\n    \"connected_nodes\": [\"Central node\"]\n  }\n}\n```\nQuestion: To find a flush toilet, you need to \"move forward\" \"turn left\" or \"turn right\", tell me your decision and tell me why?\n\nEmbodiedGPT: Based on the observations, it appears that the toilet stalls are likely to contain flush toilets. In the \"left view\" description, it was mentioned that there are stalls in the back of the bathroom. As such, to find a flush toilet, you should \"move forward\" and then \"turn left\".\n\nSincerely thanks for your insightful suggestion again.  Improving transparency remains a vital direction that we are committed to pursuing in our future work.\n\n**Q3.[What is the computational cost of training such a large model?]**\n\nThe computational cost of EmbodiedGPT for jointly training the model using the EgoCOT dataset, web video dataset, and COCO caption dataset was effectively managed on 8 A100 GPUs within a span of 16 hours. We have implemented a feasible and efficient method to train large models via a technique that combines a frozen module and prefix tuning. This approach significantly reduces the need for computational resources compared to traditional methods.\n\n**Q4.[What is the long-term behavior of EmbodiedGPT?]**\n\nThank you for your insightful suggestion! We have added experiments on rearrangement tasks that require manipulation and interaction with multiple objects to evaluate the long-term performance of EmbodiedGPT. Specifically, the robot is tasked to organize scattered items on a table into a box using the Franka Emika Robot Arm. The performance and demo are demonstrated in **Figure 2 of rebuttal PDF** in the General Response section. Furthermore, as we show in Q2, EmbodiedGPT can also maintain a text-based symbolic graph, continuously updating this text graph based on the latest observations. This feature can further enhance its success rate in tasks that require long-sequence memory dependence, such as navigation tasks across multiple rooms. \n\n**Q5.[How can EmbodiedGPT benefit real-world robotic applications?]**\n\nGood question! EmbodiedGPT provides a strong foundation tool that robots can leverage to efficiently adapt to new instances of these tasks.  And can benefit real robot applications in the following points：\n\n1. Improved Object Recognition and Interaction: The combined vision-language understanding of EmbodiedGPT allows robots to not just visually recognize objects, but also interpret their meaning and function based on language context. This enables more accurate object identification and more sensible interactions aligned with human intents.\n2. Enhanced Human-Robot Communication: By processing both visual and natural language inputs, EmbodiedGPT can enable more intuitive communication between humans and robots. Robots can interpret complex commands involving speech and gestures and provide feedback in a more natural, human-like way.\n3. Advanced Task Execution: EmbodiedGPT's joint understanding of visual scenes and language instructions empowers robots to perform tasks requiring both, such as \"pick up the red apple on the left\". This demands recognizing the apple visually while comprehending the language-based directions.\n4. Adaptive Learning: Robots can leverage EmbodiedGPT to learn new skills by observing human demonstrations and explanations. Watching a person perform a task while listening to narrations allows adaptable learning to replicate the task.\n\nWe are actively exploring these application directions and aim to demonstrate concrete real-world results in our future work. EmbodiedGPT provides a strong foundation for advancing robots' abilities to perform diverse skills in human-centric environments.\n\n\\\nThanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window. "}]

Author rebuttal:
# General Response
We sincerely appreciate all reviewers’ time and efforts in reviewing our paper. We are glad to find that reviewers generally recognized our contributions:
* **Model Development.** Taking a pioneering step in developing a foundational model for embodied AI [BD8Q, P2YP], integrating large multi-modal models with embodied planning [5cDX].
* **Dataset Creation.** Introducing the EgoCOT dataset, along with the EgoVQA extension, to enhance the multi-modal chain-of-thought capabilities in the Embodied AI domain [BD8Q, P2YP, 5cDX]. This addresses a significant challenge in embodied AI through the generation of new datasets [eM5s]. 
* **Experiments.** Showing remarkable and promising results, demonstrating the effectiveness of generating high-quality step-by-step action sequences [BD8Q,P2YP], and achieving impressive performance across multiple embodied tasks [eM5s]. 
* **Writing.** Well-written and easy to follow [P2YP]. 

And we also thank all reviewers for their insightful and constructive suggestions, which help a lot in further improving our paper. In addition to the pointwise responses below, we summarize supporting experiments added in the rebuttal according to reviewers’ suggestions.

**New experiments**

- Additional **Visual Navigation** Experiment [BD8Q,5cDX]. We conducted an experiment to test EmbodiedGPT's capabilities in a **partially observed** setting with a visual object navigation task. Specifically, the task required the model to locate a target object (a flush toilet) using only limited visual inputs and a language command. Our results indicate that EmbodiedGPT provides valuable priors for visual navigation tasks. Even in instances where the observed images don't contain the target object, the model can deduce the most likely direction based on its knowledge. This suggests that, when paired with cutting-edge navigation algorithms, EmbodiedGPT could emerge as a formidable tool in this domain. Please see more details in Figure 1 of the following rebuttal PDF.

- **Real robot** experiment on **embodied rearrangement** task with Franka Emika robot arm [BD8Q,5cDX].  The video is provided in a separate link using "official comments" according to the strict regulation.  In the embodied rearrangement task, we are using a Franka robotic arm to clean up cluttered items on a table and place them into a box. Please see more details in Figure 2 of the following rebuttal PDF.

**New Data Analysis**
- Statistical significance testing of the human evaluation results, confirming the superior performance of EmbodiedGPT over baselines [BD8Q]. 

We believe these new additions help address reviewer concerns around partial observability, real-world evaluation, novelty, and presentation issues. We thank the reviewers' time and feedback in improving the quality of our work and we hope the revisions further highlight the contributions made. Please let us know if any clarification or additional experiments would further strengthen the paper. We would be happy to incorporate all these suggestions in the final version.

Meta-review: The authors introduce both an approach and data for converting a video (e.g. Ego4D) to a task plan for robot execution.  This CoT approach helps the model build a representation of low-level skill sequences that improves performance.  The same underlying model can then also be used for VQA and related multimodal tasks.  The authors expanded their set of experiments in the rebuttal period -- all of which once incorporated will make the paper much stronger.  The reviewers are generally in agreement that the paper is in fact a useful and interesting contribution, there are a few rhetorical points (e.g. generally by BD8Q or where terms might be confused -- P2YP) that the authors should consider to strengthen the paper, reduce possible confusion, and scope the contributions.
Decision: Accept (spotlight)