Title: EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
Venue: NeurIPS 2023
Accepted: yes
Keywords: Embodied AI

Multi-modal Foundation Model

Embodied Control
Abstract: Embodied AI is a crucial frontier in robotics, capable of planning and executing action sequences for robots to accomplish long-horizon tasks in physical environments.
In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi-modal understanding and execution capabilities. To achieve this, we have made the following efforts: (i) We craft a large-scale embodied planning dataset, termed EgoCOT. The dataset consists of carefully selected videos from the Ego4D dataset, along with corresponding high-quality language instructions. Specifically, we generate a sequence of sub-goals with the "Chain of Thoughts" mode for effective embodied planning.
(ii) We introduce an efficient training approach to EmbodiedGPT for high-quality plan generation, by adapting a 7B large language model (LLM) to the EgoCOT dataset via prefix tuning. (iii) We introduce a paradigm for extracting task-related features from LLM-generated planning queries to form a closed loop between high-level planning and low-level control.
Extensive experiments show the effectiveness of EmbodiedGPT on embodied tasks, including embodied planning, embodied control, visual captioning, and visual question answering.
Notably, EmbodiedGPT significantly enhances the success rate of the embodied control task by extracting more effective features. It has achieved a remarkable 1.6 times increase in success rate on the Franka Kitchen benchmark and a 1.3 times increase on the Meta-World benchmark, compared to the BLIP-2 baseline fine-tuned with the Ego4D dataset.

EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought

Abstract

Embodied AI is a crucial frontier in robotics, capable of planning and executing action sequences for robots to accomplish long-horizon tasks in physical environments. In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi-modal understanding and execution capabilities. To achieve this, we have made the following efforts: (i) We craft a large-scale embodied planning dataset, termed EgoCOT. The dataset consists of carefully selected videos from the Ego4D dataset, along with corresponding high-quality language instructions. Specifically, we generate a sequence of sub-goals with the "Chain of Thoughts" mode for effective embodied planning. (ii) We introduce an efficient training approach to EmbodiedGPT for high-quality plan generation, by adapting a 7B large language model (LLM) to the EgoCOT dataset via prefix tuning. (iii) We introduce a paradigm for extracting task-related features from LLM-generated planning queries to form a closed loop between high-level planning and low-level control. Extensive experiments show the effectiveness of EmbodiedGPT on embodied tasks, including embodied planning, embodied control, visual captioning, and visual question answering. Notably, EmbodiedGPT significantly enhances the success rate of the embodied control task by extracting more effective features. It has achieved a remarkable 1.6 times increase in success rate on the Franka Kitchen benchmark and a 1.3 times increase on the Meta-World benchmark, compared to the BLIP-2 baseline fine-tuned with the Ego4D dataset. More demos, code, and dataset information can be found at our homepage.

1 Introduction

Embodied AI tasks, e.g., embodied planning, embodied VQA, and embodied control, aim to imbue robots with the ability to perceive, reason, and act within their environment, enabling them to perform long-horizon plans and execute actions autonomously based on real-time observations. Recently, large language models (LLMs) such as GPT4 [1] and PaLM-E [2], have shown promising language understanding, reasoning, and "chain-of-thought" capabilities. Such advances may open new possibilities for developing robots capable of processing natural language instructions, performing multi-modal "chain-of-thought", and planning actions in physical environments.

Large-scale datasets play important roles in training large language models. For example, OpenCLIP trains its ViT-G/14 model on the LAION-2B dataset [3], which contains 2B image-language pairs. Unlike general-purpose visual language tasks that can get a huge amount of weakly labeled imagecaption pairs from the Internet, embodied AI tasks require egocentric data in robotics domains. Also,

Assistant:

In this video, a robot is engaged in a manipulation task in a kitchen environment. The scene features a white

robotic arm, a black platform with four gray-white circles representing the heating elements of an induction cooktop. A

white kettle with a handle can be seen in the bottom left corner of the platform, and a black microwave with a handle for

opening sits on the left side of the platform. Two blue-gray cabinets are positioned above the platform, with the left

cabinet having two doors with handles and the right cabinet featuring a sliding door with a handle. The video captures the

robotic arm in the process of opening the right cabinet's sliding door by first grabbing the handle and pulling it outwards.

Human:

Can you write a detailed plan for the

task the robot is performing?

Assistant:

The task is opening the sliding door.

Step 1: Move to the left side of the platform.

Step 2: Grip the handle on the right side of the door.

Step 3: Pull it to the right.

Step 4: Hold the handle still while the door slides open.

Step 5: Release the handle when thedoor is open.

Human:

Describe the video in detail.

®

EmbodiedControlPerformance

BLIP-2[Eg04D]

(%)

80

R3M

Rate

60

Success

40

20

0

FrankaKitchen

Metaworld

®

structured language instructions are needed for precise planning, which usually requires huge manual efforts and costs. This poses a challenging problem in collecting high-quality embodied multi-modal data. Some researchers [4, 5, 6, 7] explore creating large-scale embodied datasets with simulators, but a significant gap remains between simulation and the real world. Recent works [8, 9, 10] also explore adapting the pre-trained LLMs to a new domain by efficient tuning strategies like LoRA [11]. However, several open questions still remain: how to apply LLMs to the field of robotics which may face large domain gaps; how to leverage the "chain-of-thought" capability for structured planning; and how to use the output language plan for downstream manipulation tasks in an end-to-end manner.

To solve the above challenges, in this work, we first build a large-scale embodied planning dataset, termed EgoCOT, which features chain-of-thought planning instructions. It contains carefully selected egocentric videos from the Ego4D dataset [16] and corresponding high-quality step-by-step language instructions, which are machine-generated, then semantics-based filtered, and finally human-verified. Additionally, we also create the EgoVQA dataset as an extension of the Ego4D dataset, focusing on egocentric human-object interaction video question answering tasks, which aims to offer a wider range of egocentric multi-modal data.

Based on our EgoCOT and EgoVQA, we present an end-to-end multi-modal embodied foundation model called EmbodiedGPT, which can interact with the physical world in a more natural and intuitive manner, and perform many embodied tasks, as shown in Figure 1, such as embodied planning, embodied VQA, and embodied control. EmbodiedGPT comprises four integrated modules that work together, including i) a frozen vision model for encoding visual features of current observations, ii) a frozen language model used to execute natural language for question answering, captioning, and embodied planning tasks, iii) an embodied-former with a language mapping layer for aligning the visual and embodied instructions and extracting task-relevant instance-level features with the generated planning for low-level control, and iv) a policy network, which is responsible for producing low-level actions based on the task-relevant features, enabling the agent to effectively interact with the environment. To further enhance EmbodiedGPT 's performance in generating reliable planning containing sub-goal sequences, we implement prefix tuning on the frozen language model to encourage the generation of more executable planning.

Our method possesses the following core advantages: i) the generated planning exhibits strong executability and granularity at the object part level, such as the gripper of a robotic arm or the handle of a door, manifested in sub-goal sequences. ii) the proposed EgoCOT dataset is built based

on an open-source large-scale dataset, which offers greater scalability compared to the PaLM-E [2] model trained on proprietary robot data. And both the EgoCOT dataset, and the EmbodiedGPT model will be open-sourced. iii) EmbodiedGPT forms a closed-loop from high-level planning to low-level control, which enables seamless integration of high-level planning and low-level control, providing efficient task performance and adaptability to a wide range of tasks. To achieve this, we utilize the embodied-former to query task-relevant instance-level features through cross-attention between visual observations and generated embodied planning. This enables the policy network to complete low-level control tasks with fewer than 25 demonstrations.

The contributions can be summarized as follows: (i) We build an end-to-end multi-modal foundation model EmbodiedGPT for embodied AI, which is featured with "chain-of-thought" capability, empowering embodied agents to interact with the physical world in a more natural and intuitive manner. (ii) We develop two datasets, EgoCOT and EgoVQA, consisting of 200M annotated videos from the Ego4D dataset with corresponding detailed planning instructions and VQA data. The datasets are first machine-generated, then semantics-based filtered, and finally human-verified for quality control. (iii) We introduce EmbodiedGPT a cost-effective training approach and a paradigm for extracting task-relevant features from LLM-generated planning queries, thereby forming a closed loop between high-level planning and low-level control. We demonstrate our approach's effectiveness by achieving state-of-the-art or comparable performance on multiple embodied tasks, including embodied control, embodied planning, video captioning, and video QA. Notably, in comparison to BLIP-2 [17] fine-tuned on the Ego4D dataset and R3M [12] specifically designed for manipulation tasks, EmbodiedGPT outperforms both models on the Franka Kitchen [14] benchmark with a margin of 22.1% and 5.5%, respectively. Similarly, on the Meta-World [14] benchmark, EmbodiedGPT surpasses both models with margins of 22.5% and 4.2%, respectively.

2 Related Work

2.1 Vision Language Pre-training with large scale foundation model

Vision-Language Pre-training focuses on strengthening the link between visual observation and natural language. The goal is to develop models that can better understand and process visual content, such as recognizing objects and actions, and generating descriptive text. As models become larger, the computational expense for end-to-end pre-training rises, leading to the need for modular vision-language pre-training methods. These methods smartly use pre-trained models, keeping them 'frozen' during vision language pre-training to save on computational costs. For example, models like Uniter [18], Oscar [19], VinVL [20], and LiT [21] freeze the image encoder, while Frozen [22] and VGPT [23] freeze the language model. Furthermore, Flamingo [24] and BLIP-2 [17] use both frozen image encoders and language models, providing a balance between performance and computational efficiency. Due to the lack of open-source data for multi-modal embodied planning, previous works struggled to perform detailed task decomposition and lacked the ability to generate precise and executable plans. To tackle this issue, we create the EgoCOT dataset and develop an embodied chain-of-thought vision language pre-training framework to enhance the capacity of multi-modal models for embodied reasoning and planning.

2.2 Egocentric Video Datasets.

Egocentric videos, which are captured using wearable cameras, provide a natural perspective of daily activities and pose several challenging research questions [25, 26, 27]. Several egocentric video datasets have been created over the years, including [28, 29, 30]. However, the collection of egocentric videos is expensive, and previous datasets tend to be small-scale and domain-specific. Recently, a massive egocentric video dataset, Ego4D [16], has been released and has been used for embodied representation learning. The dataset comprises 3,670 hours of videos collected by 931 people from 74 locations across 9 countries, with videos accompanied by narrations. For embodied AI tasks, learning from large and diverse egocentric human videos has emerged as a promising approach to acquiring a generally useful visual representation for controlling such tasks. For example, R3M [12] developed a sparse and compact visual representation using the Ego4D human video dataset through a combination of time-contrastive learning and video-language alignment. VIP [31], learns general-purpose reward functions for goal-conditioned robotic manipulation using the Ego4D dataset.

Physical Manipulation

Vision

Transformer

Embodied-Former

Language

Mapping

Embodied Queries

Text Queries

CNN w/ global

average pooling

Instance information

Global information

Policy

Mapping

LLaMA

Questions

Task description

Examples

Dialogue Memory

Text prompt

<imgs>[Image/8de837d8.mp4]

How

to

do this

task?

Assistant : Plans

are

Step 1:

grasp the

handle

of

the microwave oven

door

pull the

handle

out

Actions:

Grasp(handle)

Pull out(handle)

Embodied Planning

Human:

Please

give

caption

of Video/8de837d8.mp4

Assistant: A

robot

is

opening

the door

of

the

microwave Oven

Video Caption

<imgs>[Video/8de837d8.mp4]

What object

is

the

robot

operating?

Assistant:

The microwave

Oven.

Video Q&A

Multi-turn Dialogue

<imgs>[Video/8de837d8.mp4]

What

object

is

the robot

operating?

What

is the

color

of

the

button?

Assistant:

The button

Assistant:

Red

concat

+

2.3 Large Foundation Model Assistant System

Recent advancements in large-scale multi-modal language models (LLMs), such as GPT-3 [32] and GPT-4 [1], have resulted in the creation of various models that can understand multiple modes of information. Two main approaches are used in this field: systematic collaboration and end-to-end trained models. Systematic collaboration approaches involve coordinating multiple vision models or tools with language models to combine visual information with textual descriptions. Examples include models like Visual ChatGPT [33], MM-REACT [34], and HuggingGPT [35]. However, this approach is limited by the accuracy and capacity of fixed modular models, which can lead to an accumulation of errors. On the other hand, end-to-end models aim to provide unified models for multi-modal tasks. For example, Flamingo [24] combines vision and language by freezing pre-trained vision encoders and language models. BLIP-2 [13] introduces Q-Former to align visual features from frozen visual encoders with large language models. Recently, models such as MiniGPT-4 [36] and LLaVA [37] align instruction-tuned language models with visual features from frozen visual backbones. VideoChat[38], mPLUG-Owl [39] and X-LLM [40], further expand support for video input. PaLM-E [41] is the first large embodied multi-modal model, which directly incorporates features from sensor modalities to improve real-world performance and is trained with their largescale everyday robot data [42]. Compared to PaLM-E, EmbodiedGPT is more compact, with a size of only 10B and offers additional support for video captioning, video QA and making planning according to a demonstration video. Furthermore, we form a closed-loop system that spans from high-level planning to low-level control.

3 Method

The goal of the embodied foundation model is to imitate human-like perception and interaction with the environment by accurately perceiving the environment, identifying relevant objects, analyzing their spatial relationships, and formulating a detailed task plan. To achieve this, the EmbodiedGPT employs a pre-trained vision transformer as the visual encoder and a pre-trained LLaMA [43] model as the language model. As shown in Figure 2, the embodied-former acts as a bridge between the visual and language domains, it first extracts compact visual features from the output of the vision model through attention-based interaction involving visual tokens, text queries, and learnable embodied queries and then maps it to the language modality through a language mapping layer. These embeddings are sent to the frozen LLaMA [43] language model for visual caption, visual QA, and embodied planning. The generated planning is then used to query highly relevant features from the general visual tokens encoded by the visual model via the embodied-former. These features

are utilized to generate low-level control commands for task execution through the downstream policy network. To enhance performance across a range of embodied tasks, we introduce a novel video-language pre-training paradigm that leverages a cognitive chain of thought to produce embodied planning from egocentric video inputs. We formulate this task as a standard VQA (Visual Question Answering) task, using "how to do the task that + original caption" as the question and embodied planning as the answer. This framework enriches the data of embodied planning and standard visual QA tasks, encouraging the embodied-former to capture task-specific features that are more suitable for embodied control tasks.

3.1 Framework

The training process consists of three stages, each designed to incrementally develop reasoning and planning capabilities. The first two stages focus on pre-training in basic cognitive and responsive skills, while the third stage involves training the embodied AI task with egocentric video-text data on EgoCOT. In the first stage, we focus on image-text conversation alignment pre-training, which involves using three datasets: COCO Caption [44], 595 thousand finely filtered image-text pairs from CC3M [45], and 491 thousand filtered image-text pairs obtained by re-captioning LAION-400M using BLIP-2 [17]. The primary goal of this stage is to pre-train the Embodied-former and language projection while keeping the vision and language model parameters frozen to save computational resources. In the second stage, our goal is to enhance the model's ability to comprehend and generate more complex sentences and improve its reasoning skills. We achieve this by updating the language projection and prefix language adapter and utilizing the "Complex_Reasoning_77k" and multi-turn conversation datasets provided by "LLaVA_Instruct_150K" [46].

Embodied "chain-of-thought" training with EgoCOT : During the third stage, we employ Conv3D [47] to adapt the pre-trained vision model from stage 2 for video encoding, using a total of eight evenly distributed keyframes from each video. Each keyframe is partitioned into threedimensional (3D) patches, which can be visualized as spatio-temporal cubes, adeptly capturing both the visual content and the sequence of events within the video. These 3D patches are subsequently encoded into visual tokens via the Conv3D module with a time offset of 2 and are then integrated into the internal vision transformer. Then, we introduce the 'chain-of-thought' vision language pre-training paradigm where the model takes 8 keyframes of the video as input, along with the task description, embodied planning, and structured verb-noun pairs summary to reason with a prompt, such as Listing 1. To avoid overfitting, we provide a prompt set that has different instructions with the same meaning. In this stage, we fine-tune the patch embedding, the language projection layer, and the prefix language adapter to better capture temporal information.

3.2 Model Architecture

The Embodied-former, denoted as E ( · ) , serves as a bridge between visual input x vis and the frozen language model, acting as an information bottleneck that delivers the most relevant visual data to the language model. The Embodied-former consists of two sub-modules: one for extracting features from the image input, denoted as E vis : x vis → y vis , and another for extracting features from the text input, denoted as E txt : x txt → y txt . We employ N learnable embodied query embeddings y query as the input of E to interact with x vis through cross-attention layers and with x txt through self-attention layers. We denote the output query representation as z ∈ R N × D , where D is the dimensionality of the embeddings. The dimension of z is significantly smaller than that of the visual ′ N × D ′

features. The output query embeddings are then transformed to z ∈ R , which have the same dimensionality D ′ as the LLM's text embedding in the language modality. This transformation is performed by a mapping function denoted as M : z → z ′ , which is accomplished by a linear projection via a fully-connected (FC) layer. The projected embeddings, z ′ , serve as "soft visual prompts for the language model," decoupling the whole interaction into visual-query interaction and

☎

query-text interaction. The final embodied planning is inferred by the language model with z ′ and text prompt(shown as Listing 1) as input. For low-level control which aims to generate actions to interact with the environment, the embodied plan x plan is used as input text for embodied-former to query the task-relevant instance level features z instance = E ( x vis , x plan , y query ) . Subsequently, the agent is capable of generating control commands, such as the turning angle of the servo, represented as a = g ( z instance , z global ) . This function combines both the instance-specific information z instance and the global context z global . The global context is inferred using a ResNet50 model [48] that has been pre-trained on ImageNet [49], employing global average pooling. Here, g ( · ) represents the policy network, which is a Multi-Layer Perceptron (MLP) [50] mapping function. The output of the policy network consists of specific executable actions, such as positions and velocities in the Cartesian coordinate system. More implementation details can be found in Appendix A.

3.3 Training Settings

We employ the same pre-trained image encoder as BLIP-2[17]. Specifically, we utilize the ViT-G/14 model from EVA-CLIP [51] and remove its last layer, using the output features of the second last layer instead. For the frozen language model, we adopt a pre-trained LLaMA-7B [43] model and fine-tune it using the ShareGPT dataset and a GPT-4 generated 52K English instruction-following dataset [52]. We then utilize the well-fine-tuned language model as the frozen language model for vision-language pre-training. Additionally, we convert the data type of parameters of the frozen ViT [53] and language model to FP16 during pre-training to increase efficiency.

3.4 Creating EgoCOT and EgoVQA Dataset

For our EgoCOT dataset, we obtain basic data from the Ego4D dataset [16], which includes 9 , 645 untrimmed videos of various durations ranging from 5 seconds to 7 hours. To prepare the data for our purposes, we conducted two stages of data cleaning to prepare our data. In the first stage, we filtered out videos with missing or very short narrations (which made up 7.4% and 0.9% of the text, respectively), as well as those with unsure tags (which accounted for 4.0% of the text). We also excluded videos without human-object interaction, such as watching TV or walking. After this stage, we were left with 2.9 thousand hours of video, containing 3.85 million narrations, from 129 different scenarios covering 2927 hours of video.

To generate pairs of captions, embodied plannings, and corresponding video segments with time intervals, we utilized the EgoVLP framework [54] to segment the video. The narrations are organized as a sequence of sentences T 0 , · · · , T n with precise timestamps t 0 , · · · , t n that indicate when a described event occurred. For each narration T i with timestamp t i , we paired it with a clip V i by determining its start and end time points:

where β i = ∑ n -1 j =0 ( t j +1 -t j ) /n is an adjustable parameter equal to the average temporal distance between consecutive narrations in a given video. Conversely, α is a scale factor computed as the average of all β i across all videos in the EgoCOT dataset ( α = 4 . 9 seconds). For each video segment, we provide prompts and corresponding captions for ChatGPT [55] to generate a reasonable and detailed embodied planning. The caption is typically a brief introduction such as "C opens a drawer." We use the ChatGPT to generate a chain of thought according to the caption and organize it into a list of verb-noun pairs, such as "plans: grasp the handle with the gripper and pull the handle; actions: 1. grasp(handle, gripper) 2. pull(handle)." The prompt we used to generate EgoCOT dataset is shown in Listing 2. To enhance the diversity of generated chain of thoughts, we employ a temperature parameter of 0.9 and a top-p parameter of 0.95. For each prompt, we perform five sampling iterations.

Post-procedure. To ensure the quality of the generated planning instructions, we perform the second stage of data cleaning. We used the CLIP model [56] to assess the similarities between the video and text pairs. For each video, we compared it against five potential embodied plans and selected the one with the highest similarity as the corresponding label for the embodied plan. We then took our data-cleaning process a step further by filtering out any video-caption-planning pairs with similarities lower than the threshold. We eliminated both data with the low similarity between the video and caption and between the video and planning to ensure the highest quality data for our EgoCOT dataset. For each keyframe of the video segment, we use the CLIP model to encode both the text data T and the image data I into a shared embedding space. The similarity is calculated using the

cosine similarity function as S ( y T , y I ) = y T · y I ∥ y T ∥∥ y I ∥ , where S ( y T , y I ) denotes the similarity between the text and image, and y T and y I are the respective embeddings. Given that each video contains multiple keyframes, an ensemble of similarity scores is obtained for each video. This ensemble strategy helps to alleviate the problem of variability among individual frames and ensures a more robust and representative measure of overall similarity. The ensemble similarity score between a video V with n keyframes and text data T is given by:

where E ( V, T ) is the ensemble similarity score, S ( y T i , y I i ) is the similarity score for the i -th keyframe, and n is the total number of keyframes. We also created the EgoVQA dataset specifically for egocentric human-object interaction video question answering tasks to enrich the training data. For each caption in the Ego4D dataset, we used ChatGPT to generate five QA pairs. To ensure relevance, we guided ChatGPT to focus on core key verbs and nouns by designing prompts as shown in Listing 3. The sampling schema when crafting EgoVQA is the same to that as EgoCOT.

4 Experiments

In this section, we present a comprehensive evaluation of multi-modal foundation models and EmbodiedGPT, across various tasks including visual captioning, embodied planning, and control.

Evaluation on image input tasks. In order to evaluate the quality of generated captions and planning with the given image, we conducted a user study with 30 participants. The study included 10 cases of image caption tasks from MS-COCO dataset [44], 5 embodied planning scenarios in different embodied AI simulators, and 5 real-world scenes with accompanying embodied planning tasks. Participants were asked to rate the generated captions from different end-to-end models on five dimensions using a scoring system ranging from 1 to 10: object recognition accuracy, spatial

☎

☎

Human

: Give a detailed explanation of the image, describe the scene and the observed objects, and

spatial relationships. Where should I go if I wish to find a coat hanger?

:

1. The image depicts a doorway leading into a bedroom, with a clock hanging on the wall beside it and a

picture of a woman further to the right.

2. There is a white closet suitable for storing clothes on the right side of the bedroom.

3. If one needs to hang a coat, the white closet on the left seem to be the logical choice.

Visual ChatGPT

:

The image shows a bathroom with a clock on the wall and a toilet in the comer. The clock is white and has

a black face with two hands owalls are tiled and there is a window in the room.

The image shows a bathroom with a clock and a toilet. There is no coat hanger visible in the image.

Human: describe this video in detail

Assistant:

The video displays a red robotic arm, a red cylinder, and a ring. The cylinder is positioned on a flat surface, and

the ring is positioned next to it, with their centers aligned. The robotic arm is then used to pick up the ring.

Carefully, the center of the ring is aligned with the center of the cylinder, and the ring is slowly lowered onto the

cylinder, joining the two together. The final step is to inspect the join using the robotic arm, ensuring that it is

secure and without gaps.

Human: can you write a detailed plan

for the task the robot is performing?

Assistant:

1. Position the first object, a cylinder, on a flat surface.

2. Position the second object, a  ring, next to the red cylinder, with their centers aligned.

3. Use the robotic arm to pick up the ring.

4. Carefully align the center of the  ring with the center of the cylinder.

5. Slowly lower the  ring onto the cylinder, joining the two objects together.

Video:

CIIIID

relationship understanding, level of redundancy in the answer, and reasonability of the planning and the executability of the planning. The average scores among all the participants for different models are shown in Table 1. The results demonstrate that EmbodiedGPT achieves a comparable level of object recognition and spatial relationship understanding as the LLaV A-13B model, despite having only 7B parameters in the language model. Furthermore, EmbodiedGPT generates less redundant content in relation to the given embodied AI task, and produces the most reasonable and executable planning outputs. We also compared the performance of EmbodiedGPT with Visual ChatGPT [33], which adopts a hierarchical approach by combining several pre-trained vision models and language models to answer questions. In the Virtual-Home [57] benchmark, Visual ChatGPT uses a visual caption model to generate dense captions that are subsequently passed into ChatGPT for deriving a solution. As shown in Figure 3, Visual ChatGPT failed to find a coat hanger due to its limitations of relying solely on the caption model for extracting visual information, resulting in poor performance when compared to the end-to-end model like EmbodiedGPT. These findings highlight the advantages of adopting a unified, end-to-end model over hierarchical approaches that rely on multiple stages.

Evaluation on video input embodied AI tasks. We evaluate the recognition ability of videos and planning abilities of our model for embodied control tasks on standard embodied AI benchmarks, Franka Kitchen [14] and Meta-World [15]. Meta-World provides a challenging set of tasks that require complex object manipulation skills, including assembling a ring on a peg, picking and placing a block between bins, pushing a button, opening a drawer, and hammering a nail. Franka Kitchen benchmark focuses on tasks like sliding open the right door, opening the cabinet, turning on the light, turning the stovetop knob, and opening the microwave. As shown in Figure 4, given a demonstration video, EmbodiedGPT can accurately interpret the embodied control task and provide step-by-step planning. The output planning is fed into the Embodied-former module of EmbodiedGPT to query

cabinet_left

cabinet_right

micro_left

micro_right

knobs_left

knobs_right

light_left

light_right

sdoor_left

sdoor_right

Tasks

0

20

40

60

80

Success Rate

BLIP-2(Ego4D)

R3M

Embodied-gpt

(a) Performance comparison in

Franka Kitchen

with only 10 demos.

bin_left

bin_right

button_left

button_right

assembly_left assembly_right

drawer_left

drawer_right

hammer_left

hammer_right

Tasks in Metaworld

0

20

40

60

80

100

Success Rate

BLIP-2(Ego4D)

R3M

(b) Performance comparison in

Meta-World

with only 10 demos.

cabinet_left

cabinet_right

micro_left

micro_right

knobs_left

knobs_right

light_left

light_right

sdoor_left

sdoor_right

Tasks

0

20

40

60

80

Success Rate

BLIP-2(Ego4D)

R3M

Embodied-gpt

(a) Performance comparison in

Franka Kitchen

with 25 demos.

bin_left

bin_right

button_left

button_right

assembly_left

assembly_right

drawer_left

drawer_right

hammer_left

hammer_right

Tasks in Metaworld

0

20

40

60

80

100

Success Rate

BLIP-2(Ego4D)

R3M

(b) Performance comparison in

Meta-World with 25 demos.

highly relevant features for use as inputs in the policy network and the low-level actions are generated by the policy network to interact with the environment (see more visualizations in Appendix B).

Evaluation on embodied control tasks. For embodied control tasks, we compare our model with R3M[12], which is the state-of-the-art method in these two benchmarks, and an ablation version called 'BLIP-2[Ego4D]', which has the same structure and same amount of parameters as EmbodiedGPT, and is only fine-tuned on the video caption task using the Ego4D dataset without incorporating EgoCOT. In all experiments, the policy network is learned using few-shot learning on a small amount of demonstration data. There are two settings, one of which utilizes 10 demonstrations, and the other utilizes 25 demonstrations. We report the success rate in 100 random evaluations with only visual observations in 5 tasks per benchmark over 5 seeds and 2 different camera views for each setting, respectively. As shown in Figure 5 and Figure 6, EmbodiedGPT outperforms the baseline methods, demonstrating the effectiveness of learning with EgoCOT.

Human : You are the controller of the robot shown in this image, now your task is put all the objects into the brown bag next to the robot, please write a step by step plan:

EmbodiedGPT :

Step1. Move the robotic arm to the top of the gray bottle.

Step 2. Close the gripper and grasp the gray bottle.

Step 3. Move the arm to the brown bag.

Step 4. Release the bottle.

Step 5. Move the arm to the top of the cereal bag.

Step 6. Close the gripper and grasp the cereal bag.

Step 7. Move the arm to the brown bag.

Step 8. Finally, release the cereal bag.

Franka Emika Panda arm with front facing camera

大

公

Ablation study. We perform ablation studies to analyze the effectiveness of the "Chain-of-Thought" training mode and the importance of a closed-loop design for embodied control. The results, as shown in Table 2, demonstrate a significant improvement in success rate when using the EgoCOT approach compared to training solely with the EGO4D caption task. Moreover, the closed-loop design is necessary as the generated plans contained specific and relevant sub-goal information, which proved crucial for control tasks. In summary, EmbodiedGPT exhibits a strong ability to generate reasonable planning, accurately extract task-relevant features from visual inputs, as well as execute low-level actions to interact with the environment. The ablation experiments demonstrate that both the training paradigm based on EgoCOT and the closed-loop design from embodied planning to low-level control significantly contribute to the performance improvement of EmbodiedGPT.

Real robot experiment. We also conducted a real-world experiment using the Franka Emika robot arm, emphasizing a rearrangement task where the objective was to pack scattered bottles from a table into a box. As depicted in Figures 7 and 8, model is capable of generating intricate plans and executing of low-level actions based on 50 demonstration.

5 Conclusion

In this paper, we present EmbodiedGPT, an end-to-end multi-modal foundational model for embodied AI that enables agents to perform step-by-step planning and execute low-level commands. To achieve this, we create a large-scale embodied planning dataset called EgoCOT and develop an efficient training approach that utilizes prefix tuning to generate high-quality plans with a "chain-of-thought". Furthermore, our embodied control paradigm seamlessly coordinates high-level planning and lowlevel control. Extensive experiments demonstrate the effectiveness of EmbodiedGPT on various embodied tasks, achieving state-of-the-art or comparable performance. We believe that EmbodiedGPT represents a significant step towards developing more intelligent embodied AI agents.

Future works and limitations: EmbodiedGPT freezes the parameters of the vision and language model due to limited computational resources. Joint training with all modules and exploring other modalities, such as speech, could be future works. We do not foresee obvious undesirable ethical or social impacts at this moment.

Acknowledgement. This paper is partially supported by the National Key R&D Program of China No.2022ZD0161000 and the General Research Fund of Hong Kong No.17200622. We would like to express our appreciation to Dr. René Zurbrügg from ETH AI Center for his support on robot experimental platform.

References

OpenAI. Gpt-4 technical report, 2023.

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03378 , 2023.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv: Arxiv-2210.03094 , 2022.

Yizhou Zhao, Qiaozi Gao, Liang Qiu, Govind Thattai, and Gaurav S Sukhatme. Opend: A benchmark for language-driven door and drawer opening. arXiv preprint arXiv:2212.05211 , 2022.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning , pages 894-906. PMLR, 2022.

Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Wang. Vlmbench: A compositional benchmark for vision-and-language manipulation. Advances in Neural Information Processing Systems , 35:665-678, 2022.

Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 , 2023.

Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010 , 2023.

Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933 , 2023.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 , 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. CoRR , abs/2301.12597, 2023.

Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956 , 2019.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning , pages 1094-1100. PMLR, 2020.

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18995-19012, 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In ECCV , volume 12375, pages 104-120, 2020.

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV , pages 121-137, 2020.

Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Making visual representations matter in vision-language models. arXiv preprint arXiv:2101.00529 , 2021.

Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR , pages 18102-18112, 2022.

Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, NeurIPS , pages 200-212, 2021.

Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In CVPR , pages 18009-18019, 2022.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 , 2022.

Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR , pages 961-970, 2015.

Yazan Abu Farha, Alexander Richard, and Juergen Gall. When will you do what?-anticipating temporal occurrences of activities. In CVPR , pages 5343-5352, 2018.

Benita Wong, Joya Chen, You Wu, Stan Weixian Lei, Dongxing Mao, Difei Gao, and Mike Zheng Shou. Assistq: Affordance-centric question-driven task completion for egocentric assistant. In ECCV , 2022.

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. IJCV , 130(1):33-55, 2022.

Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626 , 2018.

Yin Li, Zhefan Ye, and James M Rehg. Delving into egocentric actions. In CVPR , pages 287-295, 2015.

Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030 , 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020.

Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. CoRR , abs/2303.04671, 2023.

Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381 , 2023.

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving AI tasks with chatgpt and its friends in huggingface. CoRR , abs/2303.17580, 2023.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. CoRR , abs/2304.08485, 2023.

KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 , 2023.

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178 , 2023.

Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.04160 , 2023.

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. CoRR , abs/2303.03378, 2023.

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, ECCV , volume 8693, pages 740-755, 2014.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, 2018.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485 , 2023.

Rahul Dev Singh, Ajay Mittal, and Rajesh K Bhatia. 3d convolutional neural network for object recognition: a review. Multimedia Tools and Applications , 78:15951-15995, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Martin Riedmiller and A Lernen. Multi layer perceptron. Machine Learning Lab Special Lecture, University of Freiburg , pages 7-24, 2014.

Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636 , 2022.

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2020.

Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z. XU, Difei Gao, RongCheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, WANG HongFa, Dima Damen, Bernard Ghanem, Wei Liu, and Mike Zheng Shou. Egocentric video-language pretraining. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 7575-7586. Curran Associates, Inc., 2022.

OpenAI. Chatgpt (mar 14 version) [large language model], 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 , 2021.

Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8494-8502, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe given text discusses the field of embodied AI and its applications in robotics. It highlights the potential of large language models (LLMs) in enabling robots to process natural language instructions, perform multi-modal tasks, and plan actions in physical environments. However, the collection of high-quality embodied multi-modal data poses challenges due to the need for egocentric data and structured language instructions.\n\nTo address these challenges, the authors propose the creation of two datasets: EgoCOT and EgoVQA. EgoCOT is a large-scale embodied planning dataset that contains egocentric videos from the Ego4D dataset, along with carefully generated, filtered, and human-verified step-by-step language instructions. EgoVQA is an extension of the Ego4D dataset, focusing on egocentric human-object interaction video question answering tasks.\nBased on these datasets, the authors present an end-to-end multi-modal embodied foundation model called EmbodiedGPT. \n\nIt consists of four integrated modules: a frozen vision model for encoding visual features, a frozen language model for natural language processing, an embodied-former for aligning visual and embodied instructions, and a policy network for low-level action generation. The model incorporates \"chain-of-thought\" planning capabilities and aims to enhance task performance and adaptability.\n\nThe contributions of this work include the development of the EmbodiedGPT model, the creation of the EgoCOT and EgoVQA datasets, and the demonstration of the model's effectiveness in various embodied tasks such as embodied control, embodied planning, video captioning, and video question-answering. EmbodiedGPT outperforms other models on benchmark tasks, showcasing its strong performance and advantages in generating reliable planning sequences and integrating high-level planning with low-level control.\n\nSTRENGTHS:\nAddressing a key challenge in embodied AI by creating new datasets: One major strength of the paper lies in recognizing a critical issue in embodied AI - the lack of good quality embodied multimodal data, especially those accompanied by structured language instructions. To tackle this issue, the authors created two new datasets - EgoCOT and EgoVQA which have the potential to significantly improve research in this area.\n\nProposing an effective solution with state-of-the-art results: Building upon these newly developed datasets, the authors introduced the EmbodiedGPT model. This model achieved impressive performance across multiple embodied tasks including embodied control, embodied planning, video captioning, and video question answering. \n\nFocus on improving the interpretability of LLMs: Another notable aspect of the paper is its focus on improving the transparency and explainability of LLMs used in embodied AI. The authors designed the EmbodiedGPT architecture so that each module can be interpreted separately or jointly analyzed together with attention weights. \n\nEncouraging open source collaboration through code release: Finally, the authors released their code publicly, fostering further research collaborations within the community.\n\nWEAKNESSES:\nPotential overfitting to specific domains: Despite being trained on diverse tasks, EmbodedGPT might have become too specialized in certain areas where the training datasets were biased towards, leading to suboptimal generalization abilities when facing unseen situations or domains. \n\nBesides, the interpretability and explainability of the EmbodiedGPT may remain problematic, griven transformer-based training on language, vision, and action policy, which lacks explicit memory components to ensure accurate responses based on past experiences. \n\n Training EmbodedGPT requires substantial computational resources, making it less accessible to smaller organizations or individual researchers who cannot afford expensive hardware.\n\nQUESTIONS:\nWhat is the computational cost of training such a large model?\n\nHow it the long-term behavior of EMbodiedGPT?\n\nHow can embodiedGPT benefit real-world robotic applications?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an end-to-end multi-modal foundation model for embodied AI. Specifically, the authors create a dataset termed EgoCOT from the Ego4D dataset and introduce a paradigm for both the vision and embodied tasks using large language models. However, I have some concerns about this paper. My detailed comments are as follows.\n\nSTRENGTHS:\n1、The paper well combines the large language models with embodied planning.\n2、The proposed EgoCOT dataset and EgoVQA dataset are helpful for downstream tasks such as embodied planning and VQA.\n\nWEAKNESSES:\nTo me, this paper is close to being interesting but I, unfortunately, do feel there are some weaknesses that, currently, prevent me from recommending acceptance.\n1、The main idea has limited technical contribution:\nThe entire paper reads like a technical report and many parts of it are not coherent. The network architecture level in this regard can be said to have no innovative contribution. The frozen modules in the paradigm have been proposed and used in many papers related to large language models. Essentially, embodied former is also actually a cross-attention network used for visual and language modality interaction. As for the policy network, it simply encodes vision and combines it with multi-modal interaction output as input. I understand this paper has the purpose to use large language models for embodied tasks. However, at the method level, I see more about how to interact and align the image or video modality with the text modality, and in the experimental part, it also focuses on reflecting the ability of the method in image caption and object grounding, not even in some important experiment on embedded tasks (such as embedded navigation, embedded rearrangement, Instruction Following and Dialogue, etc.) \n\n2、As a side note, I would like to let the authors know about some flaws in the presentation of the paper:\n（1） In Section 3.1, in the second stage of the training, how are the language projection and prefix language adapter updating? I suggest that the authors explain this in detail.\n（2） In Section 3.1, in the third stage of the training, does the Conv3D module need to be learned? And how to decide the time offset to be 2? I didn’t see any explanation or ablation studies of these hyperparameters.\n（3） In Section 3.4, how to decide the start and end time points of a clip? Please explain the relationship of  \\beta and \\alpha and why the \\beta/2*\\alpha becomes the bound.\n（4）I suggest adding more descriptions of the tasks conducted in the experiments. For example, I am confused about that why are the indicators Plan Reasonable and Plan Executable related to image caption tasks from MS-COCO.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nSee the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose an end-to-end multi-modal model for Embodied AI field. Based on the Ego4D dataset, the authors build a large-scale Embodied planning dataset, i.e., EgoCOT, which contains machine-generated language instruction with chain-of-thoughts mode. Besides, an EmbodiedGPT based on LLM is proposed to achieve the plan generation task, where the corresponding training approach is designed for prefix tuning on EgoCOT. Experimental results demonstrate that EmbodiedGPT can achieve the promising ability of plan generation.\n\nSTRENGTHS:\n- This work provides the EgoCOT dataset with EgoVQA as an extension, which can promote the development of LLM in the Embodied AI field.\n\n- This work makes the pioneering attempt to build a foundation model with the chain-of-thought capability. Building the framework based on the Vision Transformer and LLaMA, and designing the Embodied-Former as the bridge between the visual and language modality is reasonable.\n\n- Extensive quantitative and qualitative results are provided for examining the model's effectiveness. The results show that EmbodiedGPT can produce reasonable answers to describe what happened in the input video and generate the step-by-step action sequence.\n\n- The paper is well written and easy to follow with detailed instructions about how to construct and train the model.\n\nWEAKNESSES:\n- In the field of Embodied AI, an important task is \\*agent navigation*, but this work does not cover this part. I understand that the authors focus on the multi-modal foundation model, and there is a vision-language navigation (VLN) task [1][2] that requires the agent to conduct navigation according to the given language instruction. Thus I believe the VLN is a promising task for authors to test the model's navigation ability, e.g., verifying if the model can generate step-by-step navigation instructions with chain-of-thoughts.\n\n- I am confused about the setting of EmbodiedVQA in this paper. In my view, EmbodiedVQA [3] is actually the task that embodied agents navigate in rich 3D environments in order to answer the given question.\n\n- Currently, the model can generate detailed descriptions of a given video and convert them into planning steps with chain-of-thoughts. However, can the model generate executable steps for completing a specific task based solely on a static image of an object, rather than a demonstration video?\n\n- The authors should provide more details on how to use the model for Embodied control, specifically how to convert the text output into step-by-step control signals, as this is a crucial aspect for implementing embodied agents.\n\n- Minor: the colors of Figure 2 are a bit chaotic, and if there is no special meaning, I suggest it can be simplified.\n\n[1] Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR 2018.\n\n[2] Reverie: Remote embodied visual referring expression in real indoor environments. In CVPR 2020.\n\n[3] Embodied Question Answering. In CVPR 2018.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nI do not foresee obvious undesirable ethical or social impacts at this moment.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces EmbodiedGPT, a new multi-modal foundation model designed for application to embodied (robotic) tasks. EmbodiedGPT combines existing large pretrained (and frozen) vision and language models (namely the ViT-G/14 EVA-CLIP vision model and LLaMA-7B) through the use of an \"Embodied-Former\" which converts input text tokens and vision features into prefix tokens that can be fed directly to the LLaMA model. To pretrain this Embodied-Former model, this work introduces two new datasets (EgoCOT and EgoVQA), which contain video clips taken from the Ego4D dataset annotated with text captions generated by ChatGPT that, for EgoCOT, describe chain-of-thought-style plans corresponding to the actions taken within the clip, and, for EgoVQ, include questions and answers related to the clip. Using these new datasets and a 3-stage training pipeline, this paper trains the EmbodiedGPT model and, via human evaluations, show that humans generally prefer the captions and embodied plans generated by EmbodiedGPT even when compared to a LLaVA-13B baseline which has many more parameters. Finally, the paper shows that, by attaching an MLP policy head to the output of the Embodied-Former, it is possible to use imitation learning from a small set of demonstrations to train performant physical-manipulation agents in simulation. Experiments show that EmbodiedGPT outperforms ablations of itself and a R3M-based SoTA model for these tasks.\n\nSTRENGTHS:\nThis paper studies a very popular, important, and challenging area of work: building foundational models for robotic control and embodied world understanding. The proposed EmbodiedGPT model represents a modest but non-trivial step towards this grand goal. In my view, the major contributions of this are the substantial efforts taken to create the EgoCOT/EgoVQA datasets and execute the complex and computationally expensive 3-stage strategy required for training the EmbodiedGPT model. Generating these datasets and training the model clearly required significant effort and the experimental results appear quite strong.\n\nWEAKNESSES:\nWhile I lean positive for this paper, I will highlight some weaknesses below that temper my excitement.\n\n## Statistical significance of the human results\n\nThe only evaluation of EmbodiedGPT against other existing large vision-and-language models comes in a 30-participant human study using 20 tasks (10 captioning, 5 simulated planning, 5 real planning). As this sample size is quite small, and the performance difference sometimes relatively minor, it's challenging to draw meaningful conclusions from the results in Table 1. Moreover, it is hard to know how representative the 20 tasks are of general captioning/planning tasks, how were these chosen?\n\nThere are ways to provide a statistical analysis of this data but they are not trivial: you'd need to account for the correlated nature of the data (e.g. that individual participants rated multiple examples and that performance may be correlated with the task type and within each question). While not a perfect treatment, at the very least a paired t-test could be used to compare EmbodiedGPT with the other models.\n\n## Really \"foundational\"?\n\nWhile EmbodiedGPT is touted as a \"foundation model for embodied AI\", the results do not quite support this in my view. In particular, I feel that the EmbodiedGPT misses the critical capacity to handle partial observability and I was surprised not to see any real-world examples of robotic control, more details below.\n\n### Lack of tasks involving partial observability\n\nThe only embodied tasks evaluated in this paper involve static robotic arms acting in fully observed (or almost fully observed) environments. There are many embodied tasks (e.g. visual navigation) for which partial observability plays a critical role. From my understanding, the policy component of the EmbodiedGPT model makes decisions from a single visual observation in isolation from all previously seen states and thus would be expected to fail in all such tasks.\n\n### Lack of application to real-world embodied tasks\n\nWhile it is great to see that EmbodiedGPT outperforms the R3M model in the simulated environments, one of the strengths of the R3M paper was demonstrating that it also functioned well when applied to real robots. While not necessary for acceptance, it's difficult to call EmbodiedGPT a robotic foundation model without at least some real-world results.\n\n## Architectural novelty\n\nThis is a very minor point as I consider the training of large models to be a significant contribution in itself, but the proposed EmbodiedGPT architecture is not especially novel: it combines popular pretrained vision/language models with well-known prefix tuning ideas. While you cite Tsimpoukelli et al. (2021), I was a bit surprised to see more discussion of how their work relates to the proposed approach. Moreover, simply searching for \"embodied\" in the citations of the above paper (i.e. https://scholar.google.com/scholar?hl=en&as_sdt=5%2C48&sciodt=0%2C48&cites=16154696122208258147&scipsc=1&q=embodied) appears to surface several additional citations that should be included in this work.\n\nQUESTIONS:\nI have listed some of my major questions and weaknesses above. If some of these weaknesses can be addressed (especially the point about statistical significance) I expect to increase my score. In the below I include, more minor, line-by-line comments and questions.\n\nLine 60\n- \"EmbodiedGPT ’s\" should be \"EmbodiedGPT’s\". I suspect this is a problem due to latex macros, try adding {} after your macro inline, i.e. \\mymodel{}.\n\nLine 77\n- This \"200M annotated videos\" statement seems like it must be wrong. Later on, you note that there are 2927 hours of video in total, this is approximately 10M seconds of video which would correspond to ~200M frames depending on the framerate, is that what you mean?\n\nFigure 2\n- This figure could use some work, for instance, it's very unclear how videos are being input into the model.\n\nLines 146-147\n- What is the value of the \"learnable embodied queries\"? What makes them \"embodied\"?\n\nLines 172-174\n- I found this description of the video encoding procedure to be too vague, please add more details.\n\nLines 181-204\n- This entire discussion could be simplified and better presented, the idea here is quite simple and much of the formalism is extraneous. For instance, you introduce a \"mapping function (...) M: z -> z'\" but this mapping function is just a linear projection and the \"M\" notation seems to be never used again. You could just say \"z is then mapped to z' by via a linear projection.\"\n\nLine 241\n- Extra space after the comma in 9,645.\n\nLines 225-228\n- How is \\beta_i \"adjustable\" if it's defined in this way? It seems to be a fixed function of other fixed values.\n- Why not just set [t^{start}_{i}, t^{end}_{i}] to be [(t_{i-1}+t_{i})/2, (t_{i+1} + t_{i})/2]?\n\nFigure 3\n- This seems like a surprising qualitative example to choose as the EmbodiedGPT model says many wrong things. E.g. (1) the clock is not actually above the door, (2) the coat hanger is not on the floor, and (3) the coat hangers can be seen towards the top left, not the \"near the right side of the doorway\".\n\nFigure 4\n- Again, this is a surprising example as the video shows it lowering the ring over the cylinder twice whereas EmbodiedGPT seems to only suggest this happens once.\n\nLIMITATIONS:\nI would have liked to see some discussion of the limitations I discussed in the _Really \"foundational\"?_ section of the weakness. Otherwise, limitations are reasonably well addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}