SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Post-procedure. To ensure the quality of the generated planning instructions, we perform the second stage of data cleaning. We used the CLIP model [56] to assess the similarities between the video and text pairs. For each video, we compared it against five potential embodied plans and selected the one with the highest similarity as the corresponding label for the embodied plan. We then took our data-cleaning process a step further by filtering out any video-caption-planning pairs with similarities lower than the threshold. We eliminated both data with the low similarity between the video and caption and between the video and planning to ensure the highest quality data for our EgoCOT dataset. For each keyframe of the video segment, we use the CLIP model to encode both the text data  $T$  and the image data  $I$  into a shared embedding space. The similarity is calculated using the
Listing 2: Prompt we used for creating EgoCOT dataset.
You need to generate plans with chain of thought for each task, and then extract detailed actions (collocation of nouns and verbs) from the plan. The action can be of the following form: [action_name], eg., turn left; [action_name] argument1, eg., pick up/apple); [action_name] argument1 argument2, eg., put(apple, table) Task: pick up a cup on the table plans: grasp the handle of the cup with the gripper and lift it up Actions: 1.grasp(handle of the cup, gripper) 2.lift up(cup)
cosine similarity function as  $S(y_{T},y_{I}) = \frac{y_{T}\cdot y_{I}}{\|y_{T}\|\|y_{I}\|}$ , where  $S(y_{T},y_{I})$  denotes the similarity between the text and image, and  $y_{T}$  and  $y_{I}$  are the respective embeddings. Given that each video contains multiple keyframes, an ensemble of similarity scores is obtained for each video. This ensemble strategy helps to alleviate the problem of variability among individual frames and ensures a more robust and representative measure of overall similarity. The ensemble similarity score between a video  $V$  with  $n$  keyframes and text data  $T$  is given by:
$$
E (V, T) = \frac {1}{n} \sum_ {i = 1} ^ {n} S \left(y _ {T i}, y _ {I i}\right) \tag {2}
$$
where  $E(V, T)$  is the ensemble similarity score,  $S(y_{T_i}, y_{I_i})$  is the similarity score for the  $i$ -th keyframe, and  $n$  is the total number of keyframes. We also created the EgoVQA dataset specifically for egocentric human-object interaction video question answering tasks to enrich the training data. For each caption in the Ego4D dataset, we used ChatGPT to generate five QA pairs. To ensure relevance, we guided ChatGPT to focus on core key verbs and nouns by designing prompts as shown in Listing 3. The sampling schema when crafting EgoVQA is the same to that as EgoCOT.
Listing 3: Prompt used for creating EgoVQA dataset.
Please ask some questions accroding to the verbs and nouns in the sentence. For example, in this sentence "a man is picking up a cup", the verb is picking up and the noun is cup, therefor questions can be "what is the object the man is picking up?" or "what operation is performed on the cup?". Then You need to give the answer. input: a man is picking up a cup question: What is the object the man is picking up answer: The cup
In this section, we present a comprehensive evaluation of multi-modal foundation models and EmbodiedGPT, across various tasks including visual captioning, embodied planning, and control.
Evaluation on image input tasks. In order to evaluate the quality of generated captions and planning with the given image, we conducted a user study with 30 participants. The study included 10 cases of image caption tasks from MS-COCO dataset [44], 5 embodied planning scenarios in different embodied AI simulators, and 5 real-world scenes with accompanying embodied planning tasks. Participants were asked to rate the generated captions from different end-to-end models on five dimensions using a scoring system ranging from 1 to 10: object recognition accuracy, spatial
Model	Object(↑)	Spatial(↑)	Redundancy(↓)	Plan Reasonable(↑)	Plan Executable(↑)
Minigpt4	5.6	4.8	4.4	4.5	4.8
LLaVA-7B	7.3	7.4	3.9	7.5	6.6
LLaVA-13B	8.5	8.6	3.4	8.4	7.6
EmbodiedGPT	8.4	8.8	2.6	8.8	8.4
Table 1: Generate Quality Evaluation on image input tasks.
Figure 3: Comparison between EmbodiedGPT and VisualGPT in the question-answering task.
Human: Give a detailed explanation of the image, describe the scene and the observed objects, and spatial relationships. Where should I go if I wish to find a coat hanger?
EmbodiedGPT:
1. The image depicts a doorway leading into a bedroom, with a clock hanging on the wall beside it and a picture of a woman further to the right.
2. There is a white closet suitable for storing clothes on the right side of the bedroom.
3. If one needs to hang a coat, the white closet on the left seem to be the logical choice.
Visual ChatGPT:
The image shows a bathroom with a clock on the wall and a toilet in the corner. The clock is white and has a black face with two hands owalls are tiled and there is a window in the room.
The image shows a bathroom with a clock and a toilet. There is no coat hanger visible in the image.
Human: describe this video in detail
Assistant:
The video displays a red robotic arm, a red cylinder, and a ring. The cylinder is positioned on a flat surface, and the ring is positioned next to it, with their centers aligned. The robotic arm is then used to pick up the ring.
Carefully, the center of the ring is aligned with the center of the cylinder, and the ring is slowly lowered onto the cylinder, joining the two together. The final step is to inspect the join using the robotic arm, ensuring that it is secure and without gaps.
Human: can you write a detailed plan for the task the robot is performing?
Figure 4: Example of video input embodied AI tasks on Meta-World benchmark. EmbodiedGPT accurately analyzes embodied control tasks in demonstration videos and provides precise planning.
1. Position the first object, a cylinder, on a flat surface.
2. Position the second object, a ring, next to the red cylinder, with their centers aligned.
3. Use the robotic arm to pick up the ring.
4. Carefully align the center of the ring with the center of the cylinder.
5. Slowly lower the ring onto the cylinder, joining the two objects together.
relationship understanding, level of redundancy in the answer, and reasonability of the planning and the executability of the planning. The average scores among all the participants for different models are shown in Table 1. The results demonstrate that EmbodiedGPT achieves a comparable level of object recognition and spatial relationship understanding as the LLaVA-13B model, despite having only 7B parameters in the language model. Furthermore, EmbodiedGPT generates less redundant content in relation to the given embodied AI task, and produces the most reasonable and executable planning outputs. We also compared the performance of EmbodiedGPT with Visual ChatGPT [33], which adopts a hierarchical approach by combining several pre-trained vision models and language models to answer questions. In the Virtual-Home [57] benchmark, Visual ChatGPT uses a visual caption model to generate dense captions that are subsequently passed into ChatGPT for deriving a solution. As shown in Figure 3, Visual ChatGPT failed to find a coat hanger due to its limitations of relying solely on the caption model for extracting visual information, resulting in poor performance when compared to the end-to-end model like EmbodiedGPT. These findings highlight the advantages of adopting a unified, end-to-end model over hierarchical approaches that rely on multiple stages.
Evaluation on video input embodied AI tasks. We evaluate the recognition ability of videos and planning abilities of our model for embodied control tasks on standard embodied AI benchmarks, Franka Kitchen [14] and Meta-World [15]. Meta-World provides a challenging set of tasks that require complex object manipulation skills, including assembling a ring on a peg, picking and placing a block between bins, pushing a button, opening a drawer, and hammering a nail. Franka Kitchen benchmark focuses on tasks like sliding open the right door, opening the cabinet, turning on the light, turning the stovetop knob, and opening the microwave. As shown in Figure 4, given a demonstration video, EmbodiedGPT can accurately interpret the embodied control task and provide step-by-step planning. The output planning is fed into the Embodied-former module of EmbodiedGPT to query
(a) Performance comparison in Franka Kitchen with only 10 demos.
(b) Performance comparison in Meta-World with only 10 demos.
Figure 5: Performance of EmbodiedGPT in low-level control tasks with 10 demonstration demos.
(a) Performance comparison in Franka Kitchen with 25 demos.
(b) Performance comparison in Meta-World with 25 demos.
Figure 6: Performance of EmbodiedGPT in low-level control tasks with 25 demonstration demos.
highly relevant features for use as inputs in the policy network and the low-level actions are generated by the policy network to interact with the environment (see more visualizations in Appendix B).
Evaluation on embodied control tasks. For embodied control tasks, we compare our model with R3M[12], which is the state-of-the-art method in these two benchmarks, and an ablation version called 'BLIP-2[Ego4D]', which has the same structure and same amount of parameters as EmbodiedGPT, and is only fine-tuned on the video caption task using the Ego4D dataset without incorporating EgoCOT. In all experiments, the policy network is learned using few-shot learning on a small amount of demonstration data. There are two settings, one of which utilizes 10 demonstrations, and the other utilizes 25 demonstrations. We report the success rate in 100 random evaluations with only visual observations in 5 tasks per benchmark over 5 seeds and 2 different camera views for each setting, respectively. As shown in Figure 5 and Figure 6, EmbodiedGPT outperforms the baseline methods, demonstrating the effectiveness of learning with EgoCOT.
Model	Franka(10 demos)	Franka(25 demos)	Meta-World(10 demos)	Meta-World(25 demos)
EmbodiedGPT	50.8% ±2.8	58.5% ±2.7	76.4% ±2.2	81.2%±2.0
- Close-loop	38.6% ±2.9	47.3% ±2.5	62.7% ±2.2	64.9% ±2.0
- COT	26.2% ±3.2	36.4% ±2.7	55.2% ±2.4	58.7% ±2.0
Table 2: Ablation on the closed-loop spans from planning to low-level control, and "chain-of-thought" (COT) training with 25 and 10 demonstrations(- - symbol indicates "removing"). We report the average success rate over 5 tasks and 2 camera views per benchmark.
Human: You are the controller of the robot shown in this image, now your task is put all the objects into the brown bag next to the robot, please write a step by step plan:
Step1. Move the robotic arm to the top of the gray bottle.
Step 2. Close the gripper and grasp the gray bottle.
Step 3. Move the arm to the brown bag.
Step 4. Release the bottle.
Step 5. Move the arm to the top of the cereal bag.
Step 6. Close the gripper and grasp the cereal bag.
Step 7. Move the arm to the brown bag.
Step 8. Finally, release the cereal bag.
Franka Emika Panda arm with front facing camera
Figure 7: Generated plan by EmbodiedGPT for rearrangement task.
Figure 8: Real-world experiment on the Franka Emika robot arm.
Ablation study. We perform ablation studies to analyze the effectiveness of the "Chain-of-Thought" training mode and the importance of a closed-loop design for embodied control. The results, as shown in Table 2, demonstrate a significant improvement in success rate when using the EgoCOT approach compared to training solely with the EGO4D caption task. Moreover, the closed-loop design is necessary as the generated plans contained specific and relevant sub-goal information, which proved crucial for control tasks. In summary, EmbodiedGPT exhibits a strong ability to generate reasonable planning, accurately extract task-relevant features from visual inputs, as well as execute low-level actions to interact with the environment. The ablation experiments demonstrate that both the training paradigm based on EgoCOT and the closed-loop design from embodied planning to low-level control significantly contribute to the performance improvement of EmbodiedGPT.
Real robot experiment. We also conducted a real-world experiment using the Franka Emika robot arm, emphasizing a rearrangement task where the objective was to pack scattered bottles from a table into a box. As depicted in Figures 7 and 8, model is capable of generating intricate plans and executing of low-level actions based on 50 demonstration.
In this paper, we present EmbodiedGPT, an end-to-end multi-modal foundational model for embodied AI that enables agents to perform step-by-step planning and execute low-level commands. To achieve this, we create a large-scale embodied planning dataset called EgoCOT and develop an efficient training approach that utilizes prefix tuning to generate high-quality plans with a "chain-of-thought". Furthermore, our embodied control paradigm seamlessly coordinates high-level planning and low-level control. Extensive experiments demonstrate the effectiveness of EmbodiedGPT on various embodied tasks, achieving state-of-the-art or comparable performance. We believe that EmbodiedGPT represents a significant step towards developing more intelligent embodied AI agents.
Future works and limitations: EmbodiedGPT freezes the parameters of the vision and language model due to limited computational resources. Joint training with all modules and exploring other modalities, such as speech, could be future works. We do not foresee obvious undesirable ethical or social impacts at this moment.
Acknowledgement. This paper is partially supported by the National Key R&D Program of China No.2022ZD0161000 and the General Research Fund of Hong Kong No.17200622. We would like to express our appreciation to Dr. René Zurbrugg from ETH AI Center for his support on robot experimental platform.
[55] OpenAI. Chatgpt (mar 14 version) [large language model], 2023.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021.
[57] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494-8502, 2018.