EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Authors: Rui Yang
Hanyang Chen
Junyu Zhang
Mark Zhao
Cheng Qian
Kangrui Wang
Qineng Wang
Teja Venkat Koripella
Marziyeh Movahedi
Manling Li
Heng Ji
Huan Zhang
Tong Zhang
Venue: ICML 2025 oral (2025)
Keywords: Embodied Agent
Multi-modal Large Language Models
Primary area: deep_learning->large_language_models
TL;DR: We introduces EmbodiedBench, a benchmark designed to evaluate the finegrained capabilities of vision-driven embodied agents.

Abstract: Leveraging Multi-modal Large Language Models (MLLMs) to create embodied agents offers a promising avenue for tackling real-world tasks. While language-centric embodied agents have garnered substantial attention, MLLM-based embodied agents remain underexplored due to the lack of comprehensive evaluation frameworks. To bridge this gap, we introduce EmbodiedBench, an extensive benchmark designed to evaluate vision-driven embodied agents.
EmbodiedBench features: (1) a diverse set of 1,128 testing tasks across four environments, ranging from high-level semantic tasks (e.g., household) to low-level tasks involving atomic actions (e.g., navigation and manipulation); and (2) six meticulously curated subsets evaluating essential agent capabilities like commonsense reasoning, complex instruction understanding, spatial awareness, visual perception, and long-term planning.
Through extensive experiments, we evaluated 24 leading proprietary and open-source MLLMs within EmbodiedBench. Our findings reveal that: MLLMs excel at high-level tasks but struggle with low-level manipulation, with the best model, GPT-4o, scoring only $28.9\\%$ on average. EmbodiedBench provides a multifaceted standardized evaluation platform that not only highlights existing challenges but also offers valuable insights to advance MLLM-based embodied agents. Our code and dataset are available at [https://embodiedbench.github.io](https://embodiedbench.github.io).

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large
Language Models for Vision-Driven Embodied Agents
Rui Yang" 1* Hanyang Chen 1* Junyu Zhang" 1* Mark Zhao 32
Cheng Qian
Kangrui Wang? Qineng Wang?
Teja Venkat Koripella' Marziyeh Movahedi 41 Manling Li² Heng Ji Huan Zhang!? 1t Tong Zhang"
Abstract
1. Introduction
Developing embodied agents capable of solving complex tasks in real world remains a significant challenge (Du- rante et al., 2024). Recent advancements in foundation models, including Large Language Models (LLMs) (Brown et al., 2020: Achiam al., 2023; Touvron al., 2023; Yang al., 2024a) and Multimodal Large Language Mod- els (MLLMs) (OpenAI, 2024a; Reid et al., 2024; Liu et al., 2024a; Wang et al., 2024; Chen et al., 2023c; 2025), have un- locked unprecedented potential toward this goal. These mod- els, trained on extensive internet-scale datasets, demonstrate exceptional proficiency understanding human knowledge and performing human-like reasoning. Based on these ca- pabilities, researchers now design intelligent agents that use off-the-shee foundation models solve complex tasks through interaction with environments (Huang et al., 2022a;b; 2023c; Ahn et al., 2022; Song et al., 2023; Singh et al., 2023; Liang et al., 2023; Qian et al., 2024).
Leveraging Multi-modal Large Language Mod- els (MLLMs) create embodied agents offers promising avenue for tackling real-world tasks. While aenuugge-enttrc embodied agents have gar- nered substantial attention, ML.LM-hased embod- ied agents remain underexplored due to the lack of comprehensive evaluation frameworks. To bridge this gap, we introduce EMBoDIEDBENCH, an extensive benchmark designed to evaluate vision- driven embodied agents. EMBODIEDDENCII fea- tures: (1) a diverse set of 1,128 testing tasks across four environments, ranging from high-level se- mantic tasks (e.g., household) to low-level tasks involving atomic actions (e.g., navigation and ma- nipulation); and (2) six meticulously curated sub- sets evaluating essential agent capabilities like commonsense reasoning, complex instruction un- derstanding, spatial awareness, visual perception, and long-term planning. Through extensive exper- iments, we evaluated 24 leading proprietary and open-source MLLMs within EMBODIEDBENCH Our findings reveal that: MILMs excel at high- level tasks but struggle with low-level manipula- tion, with the best model, GPT-4o, scoring only 28.9% on average. EMBODIEDBENCH provides multifaceted standardized evaluation platform that not only highlights existing challenges but also offers valuable insights to advance MLLM- based embodied agents. Our code and dataset are available https///embodiedbeech hhtp://bbbbiiibbbbccccuubb......
Given the multitude of proposed algorithms, there a press- ing need for standardized and automated evaluation frame- works enable comprehensive assessment and comparison. To address this need, several initiatives have been exploring ILL--basd embodied agent evaluation (Liu et al., 2023b; Choi et al., 2024; Li et al., 2024b). While these efforts significantly contribute to understanding LLM-based agent design, the evaluation of MLLM embodied agents remains underexplored, posing a challenge for creating more versa- tile agents. Visual (Liu al., 2024e) represents the first benchmark for evaluating MLLM agents, covering embodied tasks such as household and Minecraft. However, its limited scope, focusing exclusively on high-level plan- ning, leaves critical questions unanswered, such as the role of vision in embodied tasks and the performance MLLM agents in low-level tasks like navigation and manipulation.
"qqual contribution Equual advising 'University of Illinois Urbana- "haana-Caappain "Northwestern University. University of Toronto. 'Toyota Technological Institute at Chicago. ' Woor done during internship UIUC. Correspondence to: Rui Yang <ry21 illinois.eduss Huan Zhang <huan heahhhuahhhh zhang.com>> Tong Zhang <tongzhang @ tongzhang-ml.co ml.org?.
To address these questions, we introduce EMBODIED- BENCH, comprehensive benchmark comprising 1,128 test- ing instances across four environments. EMBODIEDBENCH is designed with two key features that set it apart from exist- ing benchmarks: 1. Diverse tasks with hierarchical action levels. Among the four environments, EB-ALFRED and EB-
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Hierarchical Action Representation
Vision-Driven Embodied Agents
High-level:
[["ectaon'ss "Fand Handtowel"), ("actionns "ptck up sondfowe"") [
EmbodiedBench
Gemini
Qwen
Low-level:
(4, Y, 2, tol), Pitch, vou, Gripper]
Tasks with various action levels
Capobility-oriented fine fine-grained evaluation
High-level Household
Base Capability
Spatial Awareness
EB-ALFRED
EB-Habitat
Stack the right cylinder of the left moon.
Place plate with spoon
Common Sense
Visual Appearance
Instruction: Put the
Instruction: Find an
orange the TV stand and nove it to the sink.
I'm fecling thirsty and nend a small container water Please novigate to that object and stay near (cun)
Oétain round fruit and put it the counter.
Low-level Navigation EB-Navigation
Low-level Manipulation
EB-Manipulation
Complex Instruction
Horizon
While you're tisying up, find spot the back torlet rolls of toilet paper. it's great place
Place cold apple siice on toble.
Instruction: Navigaate to the laptop and stay close.
Instruction: Pick
silver
Figure 1. Overview of EMBODIEDBENCH. Two key features of our benchmark: various action levels and capability eupability-oriented evaluation.
Habitat focus on high-level task decomposition and planning (e.g.. "put a book on the desk"), while EB- Navigatiiin and EB-Manipulation demand planning with low-level actions (e.g., trannlltiionl//rotttoonni control) and require precise perception and spatial reasoning. 2. Capability-orient evaluation. Unlike previous benchmarks that primarily emphasize overall accuracy (Liu et al., 2023b; Choi et al., 2024; Liu et al., 2024c) or module-specific performance (Li al., 2024b). EMBODIEDBENCE introduces fine- ine-grained evaluation framework that assesses six critical capabilities of embodied agents, including basic task solving, common- sense reasoning, complex instruction understanding, spatial awareness, visual perception, and long-horizon planning.
40%-70% when removed, whereas its impact on high-level tasks is minimal. Additionally, our ablation studies provide practical insights into MILMM agent design, particularly re- garding image resolution, multi-step image input, and visual in-context learning.
Our contributions are threefold: (1) proposing a compre- hensive benchmark suite for evaluating MLLM- -bseed em- bodied agents with different action levels and fine-grained capability-oriente subsets, (2) the development of an ef- ficient MILM agent framework, (3) conducting extensive evaluations and ablation studies of leading MLLMs, provid- ing valuable insights for vision-driven agent design.
To facilitate the evaluation of MLLMs as embodied agents, we design unified agent framework that integrates ego- centrie visual perception, few-shot in-context examples, in- teraction history, and environment feedback for decision- making. This powerful framework can unlock the full po- tential of current fffthhe-sheff MI.LMs and tackle both high- level and low-level tasks effectively. Based on EMBOD- IEDBENCH and our agent pipeline, we evaluate 24 lead- ing closed-source MILMs (c.g., GPT-4o, Gemini, Claude- 3.7, and Qwen-VL--Me VL-Max) and 7B-90B open-source models (e.g., Llama-3.2 Vision (Meta, 2024), InternVL3 (Zhu al., 2025), Qwen2.5-VL (Bai et al., 2025), and Gemma-3 (Team et al., 2025)). Our evaluation yields three key findings: (1) While MLLMs excel at high-level tasks, they struggle with low-level manipulation. (2) Long-horizon planning emerges as the most challenging subset. (3) Vision input is crucial for low-level tasks, with performance degrading by
2. Related Work
In embodied agent research, LLMs are primarily used to support high-level planning (Ahn et al., 2022; Huang et al., 2022a;b; Yao et al., 2023; Huang et al., 2023d; Rana et al., 2023: Chen et al., 2023a; Gao al., 2024b). MLLMS are then integrated for perception-related tasks (Chen al., 2023b; Wang et al., 2023d; Gao et al., 2024b). Beyond perception, MI.LMs also contribute to decision-making. ei- ther by directly generating actions in an end-to-end manner (Shridhar et al., 2022; Driess et al., 2023; Du et al., 2023; Mu et al., 2024) or by producing code to develop policy or value functions (Liang et al., 2023; Huang et al., 2023c).
As this field rapidly evolves, variety of simulators (Kolve et al., 2017; Shridhar et al., 2020a; Xiang et al., 2020; Li et al., 2021; 2023) and evaluation benchmarks (Shridhar et al., 2020b:a; James et al., 2020; Zheng et al., 2022; Szot
2

---

At timestep t, the EMBODIEDBENCH, Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Table 1. Comparison with related benchmarks. EMBODIED is multi-domal benchmark including household, manipulation, and navigation tasks. "Fine-gra "Fine-grained" indicates a multi mulli-diimenional evaluation approach rather than an overall accuracy. 'AgentBench and VisualAgenlBench include domains such as household, games, and Weh. *VLABench is originally used for evaluating VLA models.
Benchmark
Category
Action Level #Env. #Test Tasks Multimodal Fine-grained LLM/VLM Support
ALFWorld (Shridhar et al., 2020b)
Household
High
1
274
Alfred (Shridhar et al.. 2020a)
Household
High
1
3062
VLMbench (Zheng et al., 2022)
Manipulation
Low
1
4760
V
Behavior-1K (Li et al., 2023)
Houschold
High
1000
V
Language Rearrangement (Szot al., 2023)
Houschold
High
1000
V
V
GOAT-bench (Khanna al., 2024)
Navigation
Low
3919
V
AgentBench (Liu et al., 2023b)
Multi-domain'
High
1091
X
X
Lota-bench (Choi et al., 2024)
Household
High
2
308
X
VisualAgentBench (Liu et al., 2024e)
Multi-domain'
High
5
746
X
Embudied Agent Interface (Li et al., 2024b)
Houschold
High
2
438
X
VLABench (Zhang et al., 2024u)
Manipulation
Low?
1
100
V
EMBODIEDBENCH (ours)
Multi-domain
High & Low 4
1128
et al., 2023: Liu al., 2023b; 2024e; Choi et al., 2024; Li et al., 2024b; Zhang et al., 2024a; Cheng et al., 2025) have emerged. Table provides a comprehensive comparison with existing works, highlighting how EMBODIEDBENCI sets itself apart from prior works in several aspects. More related works are listed in Appendix A.
agents; Ω is the visual perception space, where each ob- servation Ie E Ω corresponds to an image frame at time t; 'T is the transition dynamics; O relates the underly- ing states to the agent's visual observations; L is the lan- guage instruction that specifies the desired goal; R evalu- ates task completion given the language instruction L: Tr
1 if 8t } L. (instruction achieved)
3. Problem Formulation
0 otherwise
Definition of Action Levels. In embodied agent re- search, actions can be systematically classified into hi- erarchical levels based on their executability in robotic systems (Ma et al., 2024b; Belkhale al., 2024). Low- level actions correspond to atomic commands directly exe- cutable by robots, defined operations that specify transla- tional or rotational displacements. For instance, a robotic arm's 's action is often parameterized as a 7-dimensional vector: a [X, Y, Z, Roll, Pitch, Yaw, Gripper]. where (X,Y, Z) denote incremental translational displacements, (Roll, Pitch, Yaw) represent rotational deltas in Fuler an- gles, and Gripper encodes the binary open/closed state of the end-effector. Similarly, commands like "move for- ward 0.1 m" qualify as low-level actions, as they map ambiguously to kinematic transformations. In contrast, high-level actions can be decomposed into sequences of lowelevel primitives. Formally, a high-level action is defined as a = [a1, a2........ an), where each a; is a low-level executable primitive. For example, executing "find a HandTowel" might involve iterating through low- level behaviors: rotating certain degrees, scanning for the target, and moving towards it.
agent maintains a history he = (To, ao...., 1 at a1 Tt) and selects actions through a policy T(arLL,he) The objective is to maximize the probability of task success: max, E [ry]] where T is the terminal timestep-eithe when the task is successfully completed (s, }= L.) or when the maximum horizon is reached.
4. EmbodiedBench
To thoroughly assess MLLMs as embodied agents across various action levels and capabilities, we introduce EM- BODIEDBENCH, a benchmark comprising four environ- ments: EB-ALFRED, EB- HabHattt EB- EB-Navigation, and EB- Manipulation. To evaluate six core embodied agents' capa- bilities, we developed new datasets and enhanced cxisting simulators to support comprehensive assessments. Below is an overview of the four benchmark tasks, with further details available in Appendix C.
4.1. High-level and Low-level Tasks
EB-ALFRED. We develop EB-ALFRED based on the AL- FRED dataset (Shridhar et al., 2020a) and the 1H-T simulator (Kolve et al., 2017). Our simulator is based on Lota-Bench's implementation (Choi et al., 2024) for 8 high- level skill types: "pick up". "'pen",, "close", "turn on". "turn off". "slice", "put down", and "find", each customiz- able with specific objects, for example, "find apple". The simulator provides an egocentric view as observation, along with textual feedback on action validity and possible failure reasons. Despite its strengths, Lota-Bench's 's simulator has several limitations, which we outline in Appendix C. To
Vision-driven agents are au- tonomous systems that make sequential decisions based on visual perception and language instructions. This problem can be formally modeled as Partially Observable Markov Decision Process (POMDP) augmented with language in- structions, defined by the tuple (8, 2, L, R). Here, S is the complete state space unobservable to the agent: A is the space of high-level or low-level actions for the
Vision-driven Agents.
3

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Visual Perception
Human Instruction
In-context Demonstrations with Reasoning
General Information
"Stack noon and the gray noon in sequence.
Exanple 1: Huzan Instruction: Stack the red star and the olive star in sequence.
Input: [C'object 1"s (42, 24, 19), "object 2": [56, 55, 10]
Output: ["visual_
purple "reasoelig_aane and reflectionn: "I need to understand the instructton first.
description": "Fron left right, can [42, 24, 19], an
star at [54, 53, .*]...>>
Skill Sets
Optional Information
stack the objects in sequence, the red stan should be at the bottor and the olive star should be placed on top. The plan involves noving the gripper to...". "executable plan": "I[54, 51, 28, 3, 66, 37, 11, (54, 54, 28, 66, 37, G)...'"
Task- Specific Information
("Nove forward 0.25", "Nove lefftward 0.20", "Rotate to right by se degrees" "Rotate to leftt by 20 degrezs...""
Object Poxition Detectioon
Example
Environment Feedback
Interaction History
Output Action History
MLLM-baseed Task Planner
Visual State Description
Reflection
Reasoning
Language Plan
Executable Plan
From Jeft to right, can see 1ime
The previcus actions indicate
task ... The 1ime moon at [82, 17,
1. Move to the gray woon's position.
[["action": "[S7, 61, 38, 8, 68, 25, 8]"),
roon at [82, 17, 19), bl noon at [57, 61, 27], and green at [52, 19].
some inwalid paths, need to ensure gripper connectly..
19] should be at the
("action": *[82, 17,
botton, and the
Open gripper to release gray moon anto the lime moon.
0]"),
[57, 61, 27]
("action": *[82, 17,
should placed top.
26, 9, 50, 25, 13")]
EB-ALFRED
EB-Habitat
Embodied Environments
EB (B-Navigation
EB-Manipulation
Updated Visual Perception
Environment Feedback
Interaction History
Outout Action History
"Last action executed successfully."
("action": "[57, 9,
2. "At this noment, you have completed executing 2 stops. Last
1]"), ("actiions "[S7, 61, 27, 6, 60, 25, 9]"), ("action": ...b ...]
action is invalid. The current reward obtained is 8.0 ....
Figure 2. The vision-driven agent pipeline used in EMBODIEDBENCH. This pipeline serves as robust framework for processing multimodal imputs, reflection and reasoning, and generating executable plans. For detailed descriptions, refer to Section 4.3.
enhance the simulation, we introduced key improvements, such as support for multiple instances of the same object type, allowing us to cover all task types in ALFRED. Addi- tionally, we streamlined the action space by merging "put down" actions into single action, since only one object can be held at time. Due to the varying number of objects in ALFRED, the action space of EB-ALFRED is dynamic, ranging from 171 to 298 actions. Furthermore, we manually corrected simulator errors and refined instruction quality, ensuring more accurate action execution and improved task solvability. These enhancements make EB-ALFRED high- quality benchmark for evaluating embodied agents.
bodied agents' navigation abilities. Each unique navigation task is primarily defined by: (I) initial Robot Pose, (2) target object information, and (3) language instruction that speci- fies which target object locate, such "navigate the laptop". The robot can only rely on visual observations and textual feedback, without direct positioning data, to navigate to the target object. Success is defined as reaching within a specified distance of the target. The action space includes low-level actions: (1) Move forwardd/bcckwwr//eef/// by Az. (2) Rotate to the right/left by A degrees. (3) Tilt the camera upward/downward by Ayp degrees. The envi- ronment provides textual feedback on action validity, such as collision detection. Additionally, we offer script for automatic task generation, allowing users to create custom task datasets by specifying the configuration.
Habitat EB-Habitat is built upon the Language Re- arrangement benchmark (Szot et al., 2023), featuring 282 diverse language instruction templates. It leverages the Habi- tat 2.0 simulator (Szot et al., 2021) and focuses on planning and executing 70 high-level skills to achieve user-defined goals. These skills fall into five categories: "navigation", "pick", "place", "open", and "close", with each skill parame- terized by set of objects. Unlike ALFRED, which permits navigation to any object, EB-Habitat restricts navigation to receptacle-type objects, requiring robots to visit multi- ple locations to find desired items. With its wide variety of language instructions and unique navigation constraints, EB- Habitat serves as a valuable complement to EB-ALFRED.
EB--anipulation. EB-Manipulation extends VLMBBncc (Zheng et al., 2022) to evaluate MI.LM-based embod- ied agents in low-level object manipulation. The agent controls robotic arm using 7-dimensional action vec- tor, specifying movement parameters. Direct low-level manipulation is challenging for MLLMs. To overcome this challenge, we implemented enhancements, as illus- trated in Figure 2: (1) action space discretization (Yin et al., 2024), which divides the position components (a, y, into 100 bins and the orientation components (roll, pitch, yaw) into 120 bins, enabling valid actions to take forms like [x, y, z, roll, pitch, yaw.griiper
EB- -a--avigtnion. EB-Navigation is an evaluation suite based on AI--THOR (Kolve et al., 2017), designed to assess em-
4

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Step 1
Step 7
Step 11
Step 14
Stop 16
Step 17
Step 20
High-level Planning Trajectory
Planning
Fail
Replanning
Success
Instruction:
>>>>
witth spatulla the kitchen counter.
Step 1
Step
Step 3
Stop 4
Step 5
Low-level Planning Trajectory
Planning
Success
Testructian lan: stack the right acon on too of the left stan,
>>>>
>>>>
>>>>
Figure 3. Planning examples in EB-ALFRED and EBBManipuuatiion based on GPT-4o.
[57, 61, 20, 10, 60, 1]: and (2) additional information like YOLO (Redmon, 2016) detection boxes with index markers (Yang et al., 2023a) and object pose estimation for indexed objects, reducing the need for precise 3D location.
EB- Navigation consists of 300 test cases distributed across 5 subsets (60 instances each), while EB-Manipulation con- tains total of 228 instances, with 48 instances for each sub- set except visual appearance, which includes 36 instances. Detailed data collection is provided in Appendix C.
4.2. Capability-orienteed Data Collection
4.3. Vision-driven Agent Design
We aim to collect cupability-orientedd data for our four envi- ronments. To accomplish this, we have identified six capa- bility categories, as outlined in Table 5: (1) The Base subset evaluates basic task-solving skills necessary for planning action sequences across tasks of low to medium difficulty. (2) The Common Sense subset focuses on the use of com- mon sense knowledge to indirectly refer objects, such as describing& refrigerator as "a receptacle that can keep food fresh for several days." This subset evaluates the abil- ity of embodied agents to reason using common sense. (3) The Complex Instruction subset includes relatively longer contexts, which can be relevant or irrelevant, to obscure the instruction. This measures an agent's s ability to discern user intent from long context. (4) The Spatial Awareness subset refers to objects by their location relative to other objects. (5) The Visual Appearance subset involves refer- ring to objects based on their visual attributes, such as color or shape. (6) The Long Horizon subset comprises tasks requiring extended action sequences, typically more than 15 steps in EB.... These subsets cover a broad range of scenarios, enabling a fine-grained evaluation of embodied agents' capabilities.
To evaluate MIL.Ms as agents in EMBODIEDBENCH, we de- sign a unified embodied agent pipeline, illustrated in Figure 2. This pipeline provides a robust framework for processing multimodal inputs, reasoning through interactions, and gen- erating structured, executable plans composed of sequential actions. Two planning examples are provided in Figure 3, with additional examples available in Appendix J. Below, we outline the key components of our agent design.
The agent processes a variety of inputs, in- cluding language instructions, visual perceptions, in-context demonstrations, interaction history, and task-specific infor- mation. For visual perception, the agent can utilize either the current step image or a sequence of historical images within sliding window. However, we observe that current MLLMs struggle to understand multiple historical images effectively, so we primarily rely on the current step image for efficiency. An exception made for -aviaiiga,,,,, which is discussed in more detail in Appendix G. Task-specific information varies by task type. For high-level tasks and EB- -aviiiiga,,,,,, the agent requires valid skill sets, while EB-Manipulation includes descriptions of the action format. Additionally, Man-aaniullttion incorporates detection boxes with visual markers and object positions to help MLLMs accurately identify 3D locations. More examples of input prompts are provided in Appendix I.
Agent Input:
To construct diverse dataset, we employ different data col- lection strategies. For EB-ALFRED and EB-Manipulation, data was gathered through a combination of manual annota- tion and instruction augmentation using GPT-4o (OpenAI, 2024a). For EB-Habitat, we reorganized and adapted an existing dataset from (Szot et al., 2023), aligning it with our specific objectives. Differently, data for EB- EB-Navigation was generated entirely through automated Python programs. In summary, EB-ALFRED and EB- BB-Habit each include 300 test instances, with 50 instances for 6 subsets. Due to design challenges, EB-Navigation omits the spatial awareness sub- set and AB-Manipulation excludes the bong-horizon subset.
Task Planner:
At each planning step, the agent: (1) gen- erates a textual description of the current visual input; (2)
reflects on past actions and environmental feedback; (3)
reasons about how to achieve the goal using available in- formation; (4) formulates a language- plan; and (5)
converts it into an executable plan the required format. All outputs are structured in JSON. Unlike prior work plan- ning one action per timestep (Liu et al., 2024e), we support
5

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
multi-step planning, allowing the agent dynamically de- cide the number of actions needed. It offers two advantages: (I) better alignment with in-context examples for sequential decision-making and (2) reduced plan redundancy, espe- cially in low-level tasks where single action causes limited changes in images, thereby minimizing MLLM API calls. If a plan fails or triggers an invalid action, the agent restarts planning from the latest state.
size increases. Nevertheless, a substantial performance gap remains between the top proprietary and open-source mod- els, particularly on high-level tasks that demand advanced reasoning capabilities.
The Role of Vision in Embodied Agent. By comparing the performance of embodied agents with and without vi- sual information (marked as "Lang") in Tables 2 and 3, we observe a clear distinction between low-level and high-level tasks. Low-level tasks show a much stronger reliance vision compared to high-level tasks. For example, dis- abling vision causes GPT-4o's EB EB-Navigation performance to drop sharply from 57.7% to 17.4%, with long-horizon planning completely collapsing to 0%. This sharp decline highlights the critical importance of visual signals for low- level control tasks. Conversely, high-level tasks show much less dependence visual input. GPT-4o (Lang) and GPT- 4o-mini (Lang) perform on par with or even outperform their vision-enabled counterparts in EB-ALFRED and EB- Habitat, suggesting that these tasks may rely more heavily on textual information rather than visual input. We will further investigate the impact of language- Section 5.3. These findings emphasize two key insights: (I) when designing MLLM-based embodied Al benchmarks, it is essential to consider action-level taxonomy, with greater attention to low-level action tasks, and (2) more advanced methods are needed to effectively leverage visual input for high-level embodied tasks.
5. Experiments
In this section, conduct comprehensive experiments to evaluate the performance of various MLLMs in EMBODIED- BENCH, followed by ablation studies in Sections 5.3 and 5.4 and error analysis in Section 5.5.
5.1. Experimental Setups
We benchmark 24 models, including 8 leading proprietary models and 16 SOTA open-source models. The propri- etary models include GPT-4o and GPP-44-mini (OpenAI, 2024a;b), Claude-3.5-Sonne and Claude-3.7-Somee (An- thropic, 2024), Gemini Pro and Gemini Flash (Team al., 2024a; DeepMind, 2024), and VL-Max (Bai et al., 2023). The open-source models include Intern and V1. (8B/ / 38B/ / 78B) (Chen et al., 2025; Zhu et al., 2025), Qwen2-VL and Qwen2.5-VL (7B / 72B) (Wang et al.,
factors in
2024; Bai et al., 2025), Gemma-3 (12B / 27B) (Team et al.,
2025), Ovis2 (16B/ 34B) (Lu et al., 2024), and LLaMA3.2 Vision Instruct (11B/ / 90B) (Meta, 2024). For consistency, all models are set with temperature of 0 and maximum completion token length of 2048. All images are standard- ized to a resolution of 500x500 pixels. The maximum number of environment steps is 30 for high-level tasks, 20 for -aBiaaaiit,,,n, and 15 for EB-Manipulation. We use the task success rate as the primary metric in our experiments. More results and ablations are deferred to Appendix F.
Fine-grained Results across Subsets. We have the follow- ing findings based on our evaluation across 6 subsets.
(1) Performance Varies across Different Subsets. We ob- serve that models perform differently across various subsets. For instance, while EB-Habitat overall, GPT-4o surpasses it on long-horizon subsets (64% vs. 58%). This divergence is even more evident in low-level tasks. In EB- EB-Manipulation, for exam- plc, Claude-3.5-Somme scores 14.6 and 5.6 points higher than GPT-4o on the complex instruction and visual appear- ance subsets, respectively, but falls significantly behind on other capabilities. These results highlight the importance of fine-grained evaluations to uncover nuanced limitations in current models.
is the best model on
5.2. Benchmark Results
Overall Results. Tables 2 and 3 summarize the results for high-level and low-level tasks, respectively. Over- all, current MLLMs demonstrate strong performance on high-level tasks but struggle with low-level tasks, espe- cially EB--anipulation. Among proprietary models, we observe that different models excel at different task lev- els: Claude-3.5 3.5-onnet achieves the highest average ac- curacy on high-level tasks, with 64.0% on EB-ALFRED and 68.0% on 3BB-Habitat while GPT-4o leads in low-level tasks, scoring 57.7% on EB-Navigation and 28.9% on EB- Manipulation. For open-source models, VLtemVLL3-88 delivers the strongest overall performance. surpassing sev- eral proprietary models and closely matching GPT-4o low-level tasks with 53.7% on EB-Navigation and 26.3% on EB- EB-Manipulation. Additionally, open-source models exhibit a clear scaling trend, with performance improving as model
(2) Long-Horiae Planning Is the Most Challenging Task. The long-horizon subset consistently proves be the most difficult, showing the largest performance gap compared to base scores. For instance, in Habita,, Claude-3.3 Claude-3.5-Sonnet 3.5-Sonnet achieves 96% on the base subset but drops to 58% on the long-horizon subset. Similarly, GPT-4o falls from 86% to 64%. This trend holds true across both high-level and low- level tasks, suggesting that long-horizon planning remains a significant bottleneck for current MLLM-based agents.
6

---

EMBODIEDBENCH, Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Table Task subsets of EB-ALFRED and EB-Habitat, with the best proprictary model in bold and open-source model underlines per column. Success rates for subsets are integers since each subset consists of 50 test instances.
EB-ALFRED
EB EB-Habitat
Model
Avg
Common Complex
Visual
Spatial
Long
Avg
Common Complex
Visual
Spatial
Long
Proprictary MLLMs
GPT-4a
56.3
64
68
46
54
59.0
86
44
68
36
64
GPT-4o-min
24.0
34
36
24
32.7
74
22
32
22
32
14
Chande 3,7 Sonnet
67.7
68
70
68
70
58.7
90
58
58
62
38
46
Chaude335 Someet
64.0
72
76
60
68.0
96
68
78
38
58
Gemini-1.5 1.5-Pro
62.3
70
64
72
58
52
58
56.3
92
52
48
56
38
52
Gemini-2 2.0-Bash
52.3
62
54
46
46
58
42.3
82
38
38
36
34
26
Gemini- 1.5 Bass
39.3
44
56
42
28
39.3
76
32
48
36
32
12
Owen-VLL-Max
41.3
44
48
44
42
38
32
45.3
74
40
50
42
30
36
(Lang)
58.0
70
52
56.0
52
34
GPT-lo-mini (Lang)
31.3
30
Open Open-Source Source MLLMs
L12m--3.2 3.2 3.2-90B----- Vision-Ins
32.0
38
34
44
28
32
16
40.3
94
24
50
32
28
14
Llama-3.2 3.2- 3.2-11B---. Vision las
13.7
24
16
22
25.0
70
16
28
10
20
IntemV1.2_5.0 21,85-788
37.7
38
34
42
34
36
42
49.0
80
42
56
58
30
28
23.3
36
30
36
22
14
26
38,3
60
28
48
34
32
28
VL2.58L2..-8B
2.0
2
0
0
11.3
36
4
10
16
2
InteraVL33-5BB
39.0
38
34
46
42
38
36
55.0
84
58
6D
56
32
40
InteraV1.5 InternVL.3-388 3-388
38.0
42
34
48
30
30
44
43.3
80
26
52
40
30
32
InteraVI 3-88
10.3
20
14
14
12
2
24.3
60
14
24
18
20
10
Down2-VLL-28--
33.7
40
30
40
30
32
30
35.7
70
30
36
32
28
18
Boen2-NLL7BB-ass
1.7
2
D
2
18.3
48
6
16
20
18
2
Quen2.5-V1--22B.10 In-
39.7
50
42
42
36
34
34
37.7
74
28
42
40
24
18
Owen2.5-V1-77B.1k
4.7
10
6
2
2
14.3
32
2
26
10
14
2
Ovis2-34B
28.7
34
30
38
28
18
24
37.0
68
34
38
38
30
14
Ovis2-16B 16B
16.3
26
16
24
12
16
32.0
66
26
42
28
22
8
gemma-3-276-it
37.0
42
40
48
30
36
26
35.7
68
26
30
40
28
22
gemma-3 3- 12b-it
25.7
32
26
38
26
20
12
23.0
58
10
24
18
24
4
Impert Senditarks for EB FRFD
Import of Connaat Fesmole Mi mine for FR (RFA/FRDD
fredbbbk
Teadbeck
Curst
Difisrent camara reschutlons Ee-Mampplation
Impact Debection Bones ES-Man pulation
300-004
308d300
N000
72:2
72.5
37.5
32.
5400000 4000
27.2
292
22
Caude
Gaude Sernet
0P/-0
GPI-43
Figure 4. Language-centric ablations EB- EB-ALDRED.
(u) Different Camera Resolutions
(b) Detection Boxes
50
5.3. Language-centric Ablation
Faccoms tawe Ph
22.6
30.5
51.5
OU
We explore the role of the language-centric components, specifically focusing environment feedback and the number of in-context examples. Comparisons are con- ducted using the base subset of EB-ALFRED. Our find- ings in Figure reveal that removing environment feed- back-which provides critical information during interac- tion tion-causes 10% drop in success rate for GPT-4o and 8% drop for Slaude-3.5-Somee Furthermore, while our periments use 10 in-context examples by default, reducing this number significantly affects performance. In 0-shot setting, the success rate drops to around 40%. When com- pared with results in Table 2, where removing vision can even lead to performance gains, these findings highlight that high-level tasks rely more heavily on textual information
15
Eia 1-0000
(c) Multi-step Images
(d) Visual In-context Learning
Figure 5. Visual-centric ablations EB-Manipulation.
than on visual input.
5.4. Visual-centrie Ablation
Visual information is critical for the performance of low- level tasks. In this section, we thoroughly analyze the impact of four factors or potential enhancements: camera resolution, detection boxes, multi-step images, and visual in-context

---

EMBODIEDBENCH. Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Table
Task success rates subsets of EB- 2B-Navigation and AB-Manipulation, with the best proprietary model in bold and open-source model underlines per column.
Navigation
EB-Maniiuuationn
Model
Avg
Base
Common Complex
Visual
Long
Base
Common Complex
Visual
Spatial
Proprietary MILMss
GPT-4o
57.7
55.0
60.0
58.3
60.0
55.0
28.9
39.6
29.2
29.2
19.4
25.0
GPT-to-mini
32.8
31.7
33.3
35.0
28.3
33.3
4.8
4.2
6.3
2.1
0.0
10.4
Ch
61.7
31.3
20.8
43.8
25.0
20.8
Slaude11.--Somm
44.7
66.7
51.7
41.7
36.7
26.7
25.4
37.5
16.7
29.2
19.4
22.9
Gemini-1.5-Pro
24.3
23.3
25.0
25.0
28.3
20.0
21.1
14.6
14.6
22.9
16.7
35.4
Gomini-2.0- 1.
48.7
63.3
65.0
50.0
51.7
13.3
16.7
14.6
8.3
14.6
13.9
31.3
Gemäni- Bashh
41.7
56.7
50.0
46.7
50.0
9.6
14.6
10.4
4.2
8.3
10.4
Oween-VL-Max
39.7
50.0
46.7
41.7
35.0
25.0
18.0
25.0
10.4
18.8
2.8
29.2
GPT-40 (Lang)
17,4
21.7
21.7
26.7
16.7
0.0
16.2
16.7
16.7
14.6
19.4
14.6
GPT-4o-mini (Lang)
8.3
3.3
13.3
10.0
15.0
0.0
12.5
0.0
2.1
2.8
14.6
Open-Source MLLMS
Lluma 3.2 Vision-Ins
30.0
48.3
23.3
38.3
33.3
6.7
14.9
10.4
12.5
16.7
10.4
20.8
1lama-3.2-11 1-----1---iioonn....
21.4
23.3
21.7
26.7
18.3
17.0
0.9
0.0
0.0
2.1
0.0
2.1
VnternV1.2_5-788
30.7
36.7
38.3
33.3
21.7
23.3
18.0
16.7
16.7
14.6
22.2
20.8
VL2.5388
30.3
35.0
28.3
38.3
26.7
23.3
22.9
16.7
8.3
13.9
16.7
InlemVL2.5588 5-8B
21.3
35.0
23.3
21.7
26.7
0.0
7.0
8.3
2.1
6.3
8.3
10.4
VL3-780
53.7
66.7
63.3
31.7
26.3
29.2
22.9
22.9
25.0
31.3
VL3-31L
50.7
$5.0
61.7
55.0
56.7
25.0
22.6
20.8
14.6
20.8
19.4
37.5
InternVL3-88
29.3
38.3
30.0
10.4
10.4
12.5
13.9
10.4
Mwen2-VL-728-lle
21.2
26.7
30,0
28.3
16.0
5.0
13.6
18.8
20.8
4,2
8.3
14.6
Qwen2-VL-77
26.7
3.3
0.0
0.0
0.0
0.0
0.0
0.0
wwen2.5-VL-72B-Im
40.0
46.7
46.7
46.7
26.7
33.3
16.2
12.5
12.5
16.7
22.2
18.8
Qwen2.5-V1.-7--he
20.3
20.0
26.7
38.3
16.7
0.0
9.6
8.3
8.3
8.3
5.6
16.7
Ovis2-34B
45.7
63.3
50.0
56.7
46.7
11.7
26.8
31.3
25.0
18.8
27.8
31.3
Ovis2-16B
47.7
60.0
46.7
58.3
48.3
25.0
11.3
10.4
4,2
16.7
16.7
8.3
gomma-3-270-it
45.4
53.3
45.0
61.7
50.0
16.7
17.5
25.0
16.7
16.7
8.3
20.8
2omma--3-12b-it
34.0
38.3
36.7
48.3
40.0
6.7
20.6
20.8
22.9
20.8
19.4
18.8
learning. All comparisons based on the base subset of EB-Manipulation. Additional ablation results be found in Appendix F.
their important role in object localization for low-level tasks.
Multi-step Image Input. We also explore whether incor- porating multi-step historical observations can enhance per- formance in our agent framework, as they may help address partial observability. For EBB-Manipulation, we include ob- servations from the past two steps in addition to the current step. Two multi-step image examples are shown in Figure 10 and 11. Figure (c) presents the quantitative results. Our experiments reveal that current MILLMs struggle to effectively utilize multiple image inputs, often leading to confusion about their current state. Future work could focus on developing methods to better leverage multiple images for enhanced understanding and reasoning.
Camera Resolutions. We investigate the effect of three camera resolutions on task performance. Our results, shown in Figure (a), indicate that mid-range resolutions (500 500) achieve better results compared to both lower (300 x 300) and higher (700 700) resolutions. While low- resolution images may lack fine-grained details necessary for task execution, excessively high resolutions can intro- duce unnecessary complexity. making it harder for MLLMs to focus on relevant information for decision-making. These results highlight the importance of selecting an appropriate resolution when deploying MLLM-based embodied agents.
Visual In-context Learning (ICL), Previous work has pri- marily relied on text-based ICI. demonstrations. In this study, investigate the impact of visual ICL for embod- ied agents by including image observations part of the in-context examples for EB AB-Maanpulatiin. This approach helps the model better understand the relationship between successful low-level actions and the object positions the
Detection Boxes. In EB- EB-Manipulation, detection boxes and visual markers are used to align language instructions with visual information, helping to localize key objects in the scene. Figure (b) shows that removing detection boxes reduces success rates from 39.6% to 27.1 % for GPT-4o and
from 37.5% to 29.2% for Claude-33 3.5- Sonnet, emphasizing
8

---

EMBODIEDBENCH, Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
6. Conclusion
holleciaation
IrANid aniiidaction 20%
5%
insuffiiens auploiation 1295
modic notion 116
We introduce EMBODIEDBENCIL a comprehensive evalua- tion framework designed to assess MLLM- L--seeed embodied agents across tasks with varying action levels and capability- oriented subsets. Through extensive experiments, we identi- fied key challenges, including difficulties in low-level ma- nipulation and long-horizon planning. and the varying sig- nificance of vision input across tasks. By highlighting these areas for improvement, we hope EMBODIED BENCH will inspire and guide future research toward building more ca- pable and versatile vision-driven embodied agents.
WTOT& neEnn 2214
TRIMTIUNT desision 12%
In rain scdin 42%
porcEption emo 339
mismooon 1%
enorr 4oft
G planning BITER
FRancringg DITOr
spotial understoncing
assonind encr 28%
mi stop 23%
reflection engr
10%
18%
(8) EB-ALFRED
(b) EB-Manipulation
Figure 6. Error Analysis.
Limitations
image. Visual ICL examples are demonstrated in Figure 15. We limit the number of examples to two to avoid over- whelming the model with excessive visual input. This may slightly lower the baseline performance, as the main results use more than two text-based examples. As shown in Figure 5 (d), the results demonstrate that visual ICL significantly outperforms language-only ICL. For instance, Claude-3.5. Sonnet achieves a 16.7% performance boost. These findings underscore the potential of visual ICL as promising avenue for future research in embodied agents.
A key limitation of this work is that our evaluation is con- ducted solely in simulated environments, without real-world experiments. This reflects a common trade-off between reproducibility, cost, safety, and real-world applicability. While real-world testing is essential for practical deploy- ment, simulated benchmarks offer a standardized and re- producible setting, significantly reducing time, financial costs, and safety risks (Li et al., 2024c; Liu et al., 2024e). EMBODIEDBENT BNCH represents step forward in evaluating MLLM agents across diverse simulated embodied tasks. Fu- ture work could explore more realistic and complex simula- tions (Li et al., 2023) or develop standardized, cost-effective real-world test suites (Zhao et al., 2023; Fu et al., 2024) to bridge the gap toward practical deployment.
5.5. Error Analysis
We conducted error analysis on GPT-4o to identify po- tential failure modes in EB-ALFRED and EB- EB-Manipulation. For each environment, we sample 10 failure episodes from each subset, resulting in a total of 1 10 failed episodes to be analyzed. We found three main types of errors: perception errors, reasoning errors, and planning errors. Each error cat- egory corresponds to a specific stage in our agent pipeline, with definitions of sub-errors provided in Appendix H.
Impact Statement
This work aims to advance the development of -isien-driven embodied agents. There are many potential societal con- sequences of our work, none of which we feel must be specifically highlighted here. addition, a discussion of possible future directions is provided in Appendix B.
Overall, planning errors are the most common issue in both environments, while perception errors are more prevalent in low-level tasks. In EB-ALFRED, planning errors (55%) and reasoning errors (41%) %)) dominate, while only 4% of er- rors are perception errors. Among planning errors, missing steps (23%) and invalid actions (22%) are the most common issues, highlighting challenges in generating complete and valid plans. Reflection errors (17%) suggest the model often fails to recognize planning mistakes in its action history. Another common failure is wrong termination errors (13%), where the model prematurely assumes the task is complete and stops too early. For Man-aiipult planning errors remain the primary cause of failure (44%), due to inaccurate actions, indicating difficulties in estimating precise gripper poses. Perception errors make up 33% of failures, with wrong recognition errors (22%) being the most frequent. These errors show that even with detection boxes annotated in the visual input, the model still fails to recognize object attributes correctly. This highlights considerable room for improvement in the visual capabilities of GPT-4o.
A Acknowledgement
Tong Zhang acknowledges the support of NSF grant No. 2416897, Huan Zhang acknowledges the support of the University Research Program (URP) from Toyota Research Institute (TRI). This research reflects solely the opinions and conclusions of its authors, not those of TRI or any other Toy- ota entity. This research is also based upon work supported by U.S. DARPA ITM Program No. FA8650-23-C-7316 and DARPA ECOLE Program No. #HR001 12390060, The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily repre- senting the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for govern- mental purposes notwithstanding any copyright annotation therein.

---

EMBODIEDBENCH,
Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
References
transfer web knowledge to robotic control, arXiv preprint arXiv:2307 2023.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M.. Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Lan- guage models are few-shot learners. In Proceedings of the 34th International Conference on Neural Informa- tion Processing Systems, NIPS'20, 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829544
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E.,, Cai, T., Chattopadhyay, P., Chen, Y., Cui, Y., Ding, Y., et al. Cosmos world foundation model platform for physical ai. arXiv preprint WrXiv:2501.03575 2025.
Ahn, M.. Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C.,, Fu, C.,, Gopalakrishnan, K., Hausman, K., et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., and Dollar, A. M. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pp. 510-51 517. IEEE, 2015.
Ajay, A., Han, S., Du, Y., Li, S., Gupta, A., Jaakkola, T., Tenenbaum, J., Kaelbling, L.,, Srivastava, A., and Agrawal, P. Compositional foundation models for hi- erarchical planning. Advances in Neural Information Processing Systems, 2222304--23255 2023.
Chang. M., Chhablani, G., Clegg, A., Cote, M. D., Desai, R., Hlavac, M., Karashchuk, V., Krantz, J., Mottaghi, R., Parashar, P., et al. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. arXiv preprint arXiv:2411.0008) 2024.
Anthropic Claude 3.5 sonnet, 2024. URL
http:///wwwwathroiccccoo cwwp:////www/weeeeeeeeeeeee 3-5- sonnet.
Bai, J., Bai, S., Yang, S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou. C., and Zhou, Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Chattopadhyay, P., Hoffman, J., Mottaghi, R., and Kemb- havi, A. Robustnav: Towards benchmarking robustness in embodied navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15691- 15700, 2021.
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song. S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., and Lin, J. Qwen2.5-vl technical report, 2025.
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, I.., and Xia, F. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024a.
Belkhale, S., Ding, T., Xiao, T., Sermanet, P., Vuong, Q., Tompson, J., Chebotar, Y., Dwibedi, D., and Sadigh, D. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.0188 001.40...113,.... 2024.
Chen, Y., Cui. W., Chen, Y., Tan, M., Zhang, X., Zhao, D., and Wang. H. Robogpt: intelligent agent of making embodied long-term decisions for daily instruction tasks. arXiv preprint arXiv:2311 1. 15649, 2023a.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg. J., Bosse- lut, A., Brunskill, E., et al. On the opportumities and risks of foundation models. arXiv preprint 108iv:2108.0720 2021.
Chen, Y., Wang, X., Li, M., Hoiem, D., and Ji, H. Vistruct: Visual structural knowledge extraction via curriculum guided code-vision representation. In Proc. The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP2023) 2023b.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J.,
Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A.,
Chen, Y., Wang. X., Peng. H., and Ji, H. Solo: single transformer for scalable vision-language modeling. In Transactions on Machine Learning Research, 2024b.
Hsu, J., al. Rt-1: Roboties transformer for real-world control at scale. arXiv preprint arXiv:2212.06811 2022.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-action models
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., Li, B., Luo, P., Lu, T., Qiao, Y., and Dai, J. Internvl: Scaling up
10

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
vision foundation models and aligning for generic visual- linguistic tasks. arXiv preprint arXiv:23/2.142366 2023c.
Du, Y., Yang. M., Florence, P., Xia, E, Wahid, A., Ichter, B., Sermanet, P., Yu, T., Abbeel, P., Tenenbaum, J. B., et al. Video language planning. arXiv preprint arXiv:23100 10625, 2023.
Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z.., Gu, L.., Wang. X., Li, Q.. Ren, Y., Chen, Z., Luo, J., Wang. J., Jiang, T., Wang. B., He, C., Shi, B., Zhang, X., Lv, H., Wang, Y., Shao, W., Chu, P., Tu, Z., He, T., Wu, Z., Deng, H., Gc, J., Chen, K., Zhang, K., Wang, I.., Dou, M., Lu, L.., Zhu, X., Lu, T., Lin, D., Qiao, Y., Dai, J., and Wang, W. Expanding performance boundaries of open-soune source multimodal mod- els with model, data, and test-time scaling, 2025. URL https: http///axivv.oo/ abs wwwp////...0 2412.0521.
Durante, Z.,, Huang, Q. Wake, N., Gong, R., Park, J. S., Sarkar, B.. Taori, R., Noda, Y., Terzopoulos, D., Choi, Y., et al. Agent ai: Surveying the horizons of multimodal interaction. arXiv preprint arXiv:2401.03556 2024.
Fu, Z., Zhao, T. Z., and Finn, C. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117 2024.
Gao, C.,, Zhao, B., Zhang, W., Mao, J., Zhang. J., Zheng, Z., Man, F., Fang. J., 7hou, Z.,, Cui, J., et al. Embodiedcity: A benchmark platform for embodied agent in real-world city environment. arXiv preprint arXiv:2410.0990 04600494 2024a.
Cheng, A.-C Yin, H., Fu, Y., Guo, Q. Yang, R., Kautz, J., Wang, X., and Liu, S. Spatialrgpt: Grounded spa- tial reasoning in vision language model. arXiv preprint 5884 2024.
Gao, J., Sarkar, B., Xia, F., Xiao, T., Wu, J., Ichter, B., Ma- jumdar, A., and Sadigh, D. Physically grounded vision- language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12462-12469, IEEE, 2024b.
Cheng, Z., Tu, Y., Li, R., Dai, S., Hu, J., Hu, S., Li, J., Shi, Y., Yu, T., Chen, W., et al. Embodiedeval: Evaluate multimodal ilms as embodied agents. arXiv preprint arXiv:2501. 2025.
Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K. M.,
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song. S. Diffusion policy: Visuomo- tor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241273668, 2023.
Sen, B., Agarwal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5021-5028. IEEE, 2024.
Choi, J. Yoon, Y., Ong, H., Kim, J., and Jang, M. Lota- bench: Benchmarking ianguage-orienter task planners for embodied agents. arXiv preprint urXiv:2402.081 2024.
Gulino, C., Fu, J., Luo, W., Tucker, G., Bronstein, E., Lu, Y., Harb, J., Pan, X., Wang, Y., Chen, X., al. Waymax: An accelerated, data-driven simulator for large-scale au- tonomous driving research. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Contributors, L. Lmdeploy: A toolkit for compressing, deploying, and serving 1mm. https:://// / / thub.com/ InternLM/Imdeploo 2023.
Huang, H., Lin, F, Hu, Y., Wang, S., and Gao, Y. Copa: General robotic manipulation through spatial con- straints of parts with foundation models. arXiv preprint wwXh:22403.08248, 2024a.
DeepMind, G. Introducing gemini 2.0: our new ai model for the agentic era, 2024. https:::::: //b: /olgg//olllll technology / google Lecchooologg/oooogeeeeeeeeee///o deeppind/googlli.... gemini-ais -iii--ai--pdatee detrtteeedeembbbe-2222..
Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y., Li, Q., Zhu, S.-C., Jia, B., and Huang, S. An embodied gener- alist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023a.
Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q. Yu, T., et al. Palm-e: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, pp. 8469-8488, 2023.
Huang, S., Jiang, Z., Dong, H., Qiao, Y., Gao, P., and Li, H. Instruct2act: Mapping multi- modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305.1 /166 2023b.
Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., Tang, C., Wang, C., Zhang, D., Yuan, E., Lu, E., Tang, F., Sung, F., Wei, G., Lai, G., Guo, H., Zhu, H., et al. Kimi k1.5: Scaling reinforcement learning with lms.. arXiv preprint arXiv:2501. 2025.
Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Lan- guage models as zero-shot planners: Extracting ac- tionable knowledge for embodied agents. In Interna- tional Conference on Machine Learning, pp. 9118-9147. PMLR, 2022a.

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Huang, W., Xia, F,, Xiao, T., Chan, H., Liang, J., Florence, P. Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint 0rXiv:2207.05608, 2022b.
with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
Li, C.,, Xia, F., Martin-Martín, R., Lingelbach, M., Srivas- tava, S., Shen, B., Vainio, K., Gokmen, C., Dharan, G., Jain, T., et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint arXiv:2108.032. 2021.
Huang, W., Wang, C., Zhang, R., Li, Y., Wu, J., and Fei-Fei, L. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.059733 2023c.
Li, C., Zhang, R., Wong, J., Gokmen, C., Srivastava, S.. Martín- Marrin,,arttn,, R., Wang, C., Levine, G., Lingelbach, M., Sun, J., et Behavior- Ik: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pp. 80-93. PMLR, 2023,
Huang, W. Xia, F,, Shah, D., Driess, D., Zeng, A., Lu, Y., Florence, P., Mordatch, I., Levine, S., Hausman, K., et al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprint arXiv:2303.000 00855, 2023d.
Li, K., Yu, B., Zheng, Q. Zhan, Y., Zhang, Y., Zhang, T. Yang. Y., Chen, Y., Sun, L., Cao, Q.. Shen, L.,, Li, L., Tao, D., and He, X. Muep: A multimodal bench- mark for embodied planning with foundation models. In Larson, K. (ed.), Proceedings of the Thirty-Third In- ternational Joint Conference on Artificial Intelligence, IJCAI-24, pp. 129-138. International Joint Conferences on Artificial Intelligence Organization, 8 2024a. doi: 10. N0.24966/jccaai2222/15 URL /ttps::/ttee /doi.org/ 10.24963/1jcai.200
Huang. W. Wang, C., Li, Y., Zhang, R., and Fei-Fei, L. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint 1rX1v22409.01662, 2024b.
James, S., Ma, Z., Arrojo, D. R., and Davison, J. J. Rlbench: The robot learning benchmark learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
Main Track.
Li, M., Zhao, S., Wang, Q., Wang, K., Zhou, Y., Srivastava, S., Gokmen, C., Lee, T., Li, L. E., Zhang, R., et al. Em- bodied agent interface: Benchmarking 1lms for embod- ied decision making. arXiv preprint arXiv:440.00166. 2024b.
Jiang, H., Huang, B., Wu, R., Li, Z., Garg, S., Nayyeri, H., Wang, S., and Li, Y. Roboexp: Action-conditioned scene graph via interactive exploration for robotic manipulation. arXiv preprint arXhv:2402.15487 2024.
Khanna, M., Ramrakhya, R., Chhablani, G., Yenamandra, S., Gervet, T., Chang. M.. Kira, Z., Chaplot, D. S., Batra, D., and Mottaghi, R. Goat-bench: A benchmark for multi-modal lifelong navigation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16373-16383, 2024.
Li, X.,, Hsu, K., Gu, J., Pertsch, K., Mees, O., Walke, H. R., Fu, C., Lunawat, I, Sich, I., Kirmani, S., al. Evaluat- ing real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.0500
2024c.
Liang. J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493 9500. IEEE, 2023.
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., et al. Openvla: An open open-source linion-language- action model. arXiv preprint 2406 2024.
Koh, J. Y., Lo, R., Jang, L., Duvvur, V. Lim, M. C., Huang, P.-Y,, Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. Visualwebarena: Evaluating multimodal agents on re- alistic visual web tasks. arXiv preprint arXiv:2401.13649 2024.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. Llava-next: Improved reasoning. oer, and world knowledge, January 2024a. URL http://llaaaa..iittb......... 01 11-30-1lava-next]
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L.,, Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint
Liu, J., Li, S., Wang. Z., Li, M., and Ji, H. A language first approach for procedural planning. In Proc. The 61st Annual Meeting of the Association for Computational Linguistics (ACL2023) Findings, 2023a.
2017.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H.. Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving
Liu, S., Chen, J., Ruan, S., Su, H., and Yin, Z. Exploring the robustness of decision-level through adversarial attacks on llm-based embodied models. Proceedings of the
12

---

EMBODIED Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
32nd ACM International Conference on Multimedia, pp- 8120-8128, 2024b.
Mazzaglia, P., Verbelen, T., Dhoedt, B., Courville, A., and Rajeswar, S. Genrl: Multimodal-foundatie world mod- els for generalization in embodied agents. arXiv preprint arXiv:2406. 18043, 2024.
Liu, S., Wu, L.,, Li, B., Tan, H., Chen, H., Wang, Z., Xu, K., Su, H., and Zhu, J. Rdt-lb: a diffusion founda- tion model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024c.
Meta. Llama 3.2:
Revolutionizing edge ai and vision with open, customizable models, 2024. URL https: / /a wwt.em.acm/.bll..mm ht/ 2am---22-connece 2024 1122-vvist wiwiiiiiiieiiii wwwedddoieeeeeeeeeeeeeeeeeeeeeee
Liu, S., Ren, Z., Gupta, S., and Wang. Physgen: Rigid- body physics-grounded grounded image-to-video generation. In European Conference on Computer Vision, pp. 360-378. Springer, 2025.
Mu, Y. Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., and Luo, P. Embod- iedgpt: Vision- langone pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36, 2024.
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. Agentbench: Evalu- ating ilms as agents. arXiv preprint arXiv:2308.0.2: 03688, 2023b.
Nasiriany, S., Maddukuri, A., Zhang, L.,, Parikh, A., Lo, A., Joshi, A., Mandlekar, A., and Zhu, Y. Robocasa: Large- scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.0000 02523, 2024a.
Liu, X., Guo, D., Zhang, X., and Liu, H. Heterogeneous embodied multi-agent collaboration. IEEE Robotics and Automation Letters, 2024d.
Nasiriany, S., Xia, F., Yu, W., Xiao, T., Liang, J., Dasgupta, I., Xie, A., Driess, D., Wahid, A., Xu, Z., et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872 2024b.
Liu, X., Zhang. T., Gu, Y., long, L., Xu, Y., Song, X., Zhang, S., Lai, H., Liu, X., Zhao, H., et al. Visualagent- bench: Towards large multimodal models as visual foun- dation agents, arXiv preprint ar arXiv:2408.0633
2024e.
OpenAL. Hello gpt-4o, 2024a. ieeema..omminnex//iee
URL https://
Lu, S., Li, Y., Chen, Q-G., Xu, Z., Luo, W., Zhang, K., and Ye, H.JJ Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024.
OpenAl. Gpt-4o mini: costnccingoost-ffffiiii intelligenne, 2024b. URL https://openai.com index/
1et-44o-ini--o aello-mmiiadvaacee mwwiiaavaanningg-aaaeeaa cost -ost----ttttttfiieeeeeee---- efffeieffiiiiin intelligence/.
Luo, J., Xu, C., Liu, F., Tan, L.., Lin, Z., Wu, J., Abbeel, P., and Levine, Fmb: functional manipulation bench- mark for generalizable robotic learning. The International Journal of Robotics Research, pp. 027833492241260017, 2023.
Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., and Torralba, A. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 8494-8502, 2018
Ma, Y. Cui, C., Cao, X., Ye, W., Liu, P., Lu, J., Abdelraouf, A., Gupta, R., Han, K., Bera, A., et al. Lampilot: An open benchmark dataset for autonomous driving with lan- guage model programs. In Proceedings of the IEEF/CV Conference Computer Vision and Pattern Recognition, pp. 15141-15151, 2024a.
Qian, C., Han, P., Luo, Q. He. B., Chen, X., Zhang, Y., Du, H., Yao, J., Yang, X., Zhang, D., Li, Y., and Ji, H. Escapehench: Pushing language models to think outside the box. In arxiv, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763. PMLR, 2021.
Ma, Y., Song, Z., Zhuang, Y., Hao, J., and King, I. A survey on vision-language-action models for embodied ai. arXiv preprint arXiv:2405.14099 2024b.
Madan, N., Møgelmose, A., Modi, R., Rawat, Y. S., and Moeslund, T. B. Foundation models for video under- standing: A survey. arXiv preprint arXiv:2405.037700 2024.
Rana, K., Haviland, J., Garg, S., Abou-Chakra, J., Reid, I. D., and Suenderhauf, N. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. CoRR, 2023.
Mao, J.. Qian, Y., Zhao, H., and Wang. Y. Gpt-driver: Learn- ing to drive with gpt. arXiv preprint arXiv:2310.01415, 2023.
Redmon, J. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
13

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5; Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint ar arXiv:2403.055520 2024.
Song, C. H., Wu, J., Washington, C.,, Sadler, B. M., Chao, W.-L.,, and Su, Y. Llm- LIm-planner: Few-shot grounded plan- ning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2998-3009, 2023.
Rohmer, E., Singh. S. P., and Freese, M. V-rep: versa- tile and scalable robot simulation framework. In 2013 JEEE/RSJ international conference intelligent robots and systems, pp. 1321-1326. IEEE, 2013.
Song, X., Chen, W., Liu, Y., Chen, W. Li, G., and Lin, L. Towards long-horizon vision-language naviga- tion: Platform, benchmark and method. arXiv preprint arXiv:2412.09082. 2024.
Sarch, G., Somani, S., Kapoor, R., Tarr, M. J., and Fragki- adaki, K. Helper-x: A unified instructable embod- ied agent to tackle four interactive vision-language do- mains with memory-augmented language models. arXiv preprint ar arXiv:2404. 19065, 2024a.
Stone, A., Xiao, T., Lu, Y., Gopalakrishnan, K., Lee, K.- H., Vuong, Q. Wohlhart, P., Kirmani, S., Zitkovich, B., Xia, F., et al. Open-world object manipulation us- ing pre-trained vision- assion-language models. arXiv preprint arXiv:2303.00m 303000905 2023.
Sarch, G. H., Jang, L., Tarr, M. J., Cohen, W. W., Marino, K., and Fragkiadaki, K. Vlm agents generate their own memories: Distilling experience into embodied programs of thought. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Sun, H. Reinforcement learning in the era of llms: What is essential? what is needed? an rl perspec- tive on rlhf, prompting, and beyond. arXiv preprint arXiv:2310.06 2023.
Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao, Y., Turner, J., Maestre, N., Mukadam, M., Chaplot, D. S., Maksymets, O., et al. Habitat 2.0: Training home as- sistants to rearrange their habitat. Advances in neural information processing systems, 34:251-266, 2021.
Sharma, S., Huang, H., Shivakumar, K., Chen, L. Y., Hoque, R., Ichter, B., and Goldberg, K. Semantic mechanical search with large vision and language models. arXiv preprint ar :rxvv:2302 2202.22922...... 129/5, 2023.
Shen, B., Xia, F. Li, C., Martin, R., Fan, L., Wang, G., Pérez-D' Arpino, C.,, Buch, S., Srivastava, S., Tchapmi, L., et al. igibson 1.0: simulation environ- ment for interactive tasks in large realistic scenes. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7520-7521 IEEE, 2021.
Szot, A., Schwarzer, M., Agrawal, H., Mazoure, B., Metcalf, R., Talbott, W., Mackraz, N., Hjelm, R. D., and Toshev, A. T. Large language models generalizable policies for embodied tasks. In The Twelfth International Conference on Learning Representations, 2023.
Szot, A., Mazoure, B., Attia, O., Timofeev, A., Agrawal, H., Hjelm, D., Gan, Z., Kira, Z., and Toshev, A. From multimodal I1ms to generalist embodied agents: Methods and lessons. arXiv preprint arXiv:2412.088: 2024.
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 10740-10749, 2020a.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.0553 2024a.
Shridhar, M., Yuan, X., Cote, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. Alfworld: Aligning text and embod- ied environments for interactive learning. In International Conference on Learning Representations, 2020b.
Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19788 2025.
Shridhar, M.. Manuelli, L.. and Fox, D. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning, pp. 894-906. PMLR, 2022.
Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J.. Kreiman, T., Xu, C., et al. Octo: An open-source generalist robot policy, arXiv preprint 1xXx:22405.12213. 2024b.
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. Prog- prompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1,523-11530 IEFE, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.,
14

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Bhosale, S., al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Xiao, T., Chan, H., Sermanet, P., Wahid, A., Brohan, A., Hausman, K., Levine, S., and Tompson, J. Robotic skill acquisition via instruction augmentation with vision- language models. arXiv preprint arXiv:221..11336, 2022.
Wang, G., Xie. Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open- ended embodied agent with large language models. arXiv preprint ar ar0vv:2305. 1629/, 2023a.
Xie, J., Chen, Z. Zhang, R., Wan, X., and Li, G. Large multimodal agents: survey, arXiv preprint arXiv:2402.151 2024.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z.,, Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Xu, J., Yang, R., Luo, F., Fang, M., Wang, B., and Han, I. Robust decision transformer: Tackling data corrup- tion in offline rl via sequence modeling, arXiv preprint SKlv:2407.042885 2024.
Wang, Q., Li, M., Chan, H. P., Huang. L.,, Hockenmaier, J., Girish, C., and Ji, H. Multimedia generative script learning for task planning. In Proc. The 6 Annual Meeting of the Association,fos Computational Linguistics (ACL2023) Findings, 2023b.
Yang. A., Yang. B., Zhang. B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2. technical report. arXiv preprint arXiv:2412.
2024a.
Yang, J., Zhang, H., Li, F. Zou, X., Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4y, arXiv preprint arXiv:23 10. 1441, 2023a.
Wang. Y. Xian, Z., Chen, F., Wang, T.-H., Wang, Y., Fragki- adaki, K., Erickson, Z.. Held, D., and Gan, C. Robo- gen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.014555 2023c.
Yang, R., Zhong, H., Xu, J., Zhang, A., Zhang, C., Han, L., and Zhang, T. Towards robust offline reinforcement learning under diverse data corruption. arXiv preprint arXiv:2310.1299: 2023b.
Wang. Z., Blume, A., Li, S., Liu, G., Cho, J., Tang, Z., Bansal, M., and Ji, H. Paxion: Patching video-language foundation models with action knowledge. In Proc. 2023 Conference on Neural Information Processing Systems (NeurIPS2023) [Spotlight Paper], 2023d.
Yang, R., Ding, R., Lin, Y., Zhang, H., and Zhang, T. Reg- ularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406. 102 0216, 2024b.
Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X., and Liang, Y. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi- task agents. arXiv preprint arXiv:2302.01560, 2023e.
Yang. R., Pan, X., Luo, F., Qiu, S., Zhong, H., Yu, D., and Chen, J. newards-in-conters Multi-objective alignment of foundation models with dynamic preference adjust- ment. arXiv preprint 24x2 2024c.
Wu, C. H., Shah, R. R., Koh, J. Y., Salakhutdinov, R., Fried, D., and Raghunathan, A. Dissecting adversarial robust- ness of multimodal Im agents. In NeurlPS 2024 Workshop on Open- World Agents, 2024a.
Yang, Y., Zhou, T., Li, K., Tao, D., Li, L., Shen, L., He, X., Jiang, J., and Shi, Y. Embodied multi-modal agent trained by I1m from parallel textworld. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26275-26288 2024d.
Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., al. Deepseek-vi2: Mixture- of-experts vision-language models for advanced multi- modal understanding. arXiv preprint arXiv:2412.10300, 2024b.
Yao, S., Zhao, J., Yu, D., Du, N.. Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
Xiang, F., Qin, Y., Mo. K., Xia, Y., Zhu, H. Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 1 1199--11007 2020.
Yin, Y., Wang, Z., Sharma, Y.. Niu, D., Darrell, T., and Herzig, R. In-context learning enables robot action pre- diction in llms. arXiv preprint 110.:::2110..2788. 2024.
Xiang. J., Liu, G., Gu, Y., Gao, Q., Ning, Y., Zha, Y., Feng, Z., Tao, T., Hao, S., Shi, Y., et al. Pandora: Towards general world model with natural language actions and video states, arXiv preprint 240::2406.00455, 2024.
Zawalski, M., Chen, W., Pertsch, K., Mees, O., Finn, C., and Levine, S. Robotic control via embodied thain-of-thougeee reasoning. arXiv preprint arXiv:2407.0862 2024.
15

---

EMBODIEDBENCH. Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Zhai, S., Bai, H., Lin, Z., Pan, J., Tong, P., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., al. Fine-tuning large vision-language models decision-making agents via reinforcement learning. Advances in Neural Information Processing Systems, 37:1 10931-9991000 110971, 2025.
Zhang. S., Xu, 7., Liu, P., Yu, X., Li, Y., Gao. Q., Fei, Z., Yin, Z., Wu, Z., Jiang, Y.G., et al. Vlabench: A robotics
sarge-scale benchmark for Iinuuae--conitionn manipulation with long-horizon reasoning tasks. arXiv preprint drXiv:2412.18194 2024a.
Zhang, X., Li, J., Chu, W., Hai, J., Xu, R., Yang, Y., Guan, S., Xu, J., and Cui, P. On the out-of-dee futt-f--dittrbutiinn gener- alization of multimodal large language models. arXiv preprint arXiv:2402.06599 06099, 2024b.
Zhao, T. Z.. Kumar, V., Levine, S., and Finn, C. Learn- ing fine-grained bimanual manipulation with low-cost hardware. arXiv preprint Sitiv:2304.17003 2023.
Zheng, K., Chen, X., Jenkins, O. C., and Wang, X.
VImbench: A compositional benchmark for vision-and- language manipulation. Advances in Neural Information Processing Systems, 35:665-678, 2022.
Zhou, G., Hong, Y., and Wu, Q. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 38, pp. 7641-7649, 2024a.
Zhou, Y., Li, X., Wang. Q., and Shen, Visual in-context learning for large vision-language models. arXiv preprint arXiv:2402. 11574, 2024b.
Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L.,, Duan, Y., Tian, H., Su, W. Shao, J., et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10473 2025.
Zou, C., Guo, X., Yang, R., Zhang, Hu, B., and Zhang, H. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:2411.0088. 2024.
16

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
A. Additional Related Works
Foundation models (Bommasani et al., 2021), particularly Large Language Models (LLMs) (Brown et al., 2020; Achiam et al.. 2023; Touvron al., 2023; Yang et al., 2024a;c) and Multi-Modal Large Language Models (MLLMs) (Radford al., 2021; Team et al., 2024a; Wang et al., 2024; Wu et al., 2024b; Du et al., 2025; Chen al., 2024b; Xie et al., 2024), fundamentally transform how embodied agents perceive, make decisions, and act in physical and simulated environments.
The integration of these models into embodied agents evolves through several key approaches. Initially, Large Language Models (LLMs) are introduced to assist with high-level planning (Ahn et al., 2022; Huang et al., 2022a;b; Rana et al., 2023; Gao et al., 2024b; Huang et al., 2023d; Wang et al., 2023a; Huang et al., 2023b; Liu et al., 2023a; Wang et al., 2023b; Chen et al., 2023a; Huang et al., 2023a; Zhou et al., 2024a). They are also adopted for low-level controls (Mao et al., 2023; Yin et al., 2024). MILMS are then incorporated for perception tasks such as object attribute identification, visual relation extraction, and action recognition (Xiao et al., 2022; Chen et al., 2023b; Wang et al., 2023d:e; Gao al., 2024b; Gu et al., 2024). Subsequently, the role of MLLMs extends into policy-making through various approaches. Some works implement MI..Ms in an end-to-end manner for direct action generation (Shridhar et al., 2022; Driess et al., 2023; Du et al., 2023; Yang et al., 2024d; Mu et al., 2024). Others enhance policy generation by using MLLMs to create visual markers or generate constraints or guidance with visual masks (Sharma et al., 2023; Stone et al., 2023; Nasiriany et al., 2024b; Huang et al., 2024a; Jiang et al., 2024). A different approach involves prompting MLLMs to generate code for creating policy or value functions (Liang et al., 2023; Huang et al., 2023e; 2024b).
Most recently, Vision Language Action Models (VLAs) (Brohan et al., 2022; 2023; Chi al., 2023; Belkhale et al., 2024; Team et al., 2024b; Liu et al., 2024c; Kim et al., 2024) have emerged as promising direction. These models typically language-conditioned conditioned diffusion models as their foundation and are trained on low-level robotics action data. Another promising direction leverages world models as action simulators (Xiang et al., 2024; Agarwal et al., 2025; Liu et al., 2025). These approaches employ diffusion models conditioned on language inputs to predict future states given actions or task descriptions.
utilize MLLMs or language-
In response to the rapid advancements in this field, various simulators (Kolve al., 2017: Puig et al., 2018; Shridhar et al., 2020a; Xiang al., 2020; Shen et al., 2021; Li et al., 2021; 2023; Nasiriany al., 2024a) and evaluation benchmarks (Shridhar et al., 2020b;a; Zheng et al., 2022; Li et al., 2023; Szot et al., 2023; Luo et al., 2023; Li et al., 2024a; Koh et al., 2024; Choi et al., 2024; Khanna et al., 2024; Liu et al., 2024e; Li et al., 2024b; Zhang et al., 2024a; Song et al., 2024) have been developed. However, existing benchmarks exhibit notable limitations. For instance, ALFWorld (Shridhar et al., 2020b), AgentBench (Liu et al., 2023b), Lota-bench (Choi et al., 2024), and Embodied Agent Interface (Li et al., 2024b) lack support for multimodal input evaluation. Furthermore, most benchmarks are narrowly focused on specific domains, particularly high-level household tasks (Shridhar al., 2020a: Li et al., 2023; Szot et al., 2023), while others, such as VLMbench (Zheng et al., 2022) and GOAT-bench (Khanna et al., 2024), concentrate on low-level control for manipulation and navigation, respectively. Although VisualAgentBench (Liu et al., 2024e) pioneers the evaluation of MLLMs across multiple domains, it is limited to high-level tasks like household and Minecraft, and does not support fine-grained capability assessment. Embodied Agent Interface (Li et al., 2024b) and VLABench (Zhang et al., 2024a) introduce fine-grained evaluation metrics with language model support, but their focus remains primarily on LLMs and VLAS rather than MLLMs. Concurrently, EmaodiedEvall (Cheng et al., 2025) proposes a multi-domain benchmark for evaluating MILMSs across navigation, object interaction, social interaction, attribute question answering, and spatial question answering. While it overlaps with our work in navigation and object interaction, it does not include low-level manipulation tasks or capabllit-orriented evaluation. Moreover, the benchmark is limited in scale, containing only 328 test instances.
B. Future Research Directions
While EMBODIEDBENCH represents a significant step forward in evaluating MLLM-based embodied agents, several challenges remain, offering rich opportunities for future research. Below, we outline potential research directions:
Expanding Task Diversity. Current benchmarks for MLLM-basee embodied agents are still limited in task diversity. Future research could explore more realistic and complex environments with different action levels, such as autonomous driving (Gulino et al., 2024; Ma et al., 2024a; Gao et al., 2024a), multi-agent collaboration (Liu et al., 2024d), and human-agent interaction (Chang et al., 2024). These scenarios would better assess the agents' adaptability and generalization capabilities in real-world settings.
17

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Low-Level Tasks and Spatial Reasoning. Our findings show that current MLLM-based agents struggle with spatial reasoning and low-level control. Future research could improve these capabilities by better integrating spatial reasoning with low-level action planning, including 3D visual grounding (Chen et al., 2024a; Cheng et al., 2024) and alignment (Ahn et al.. 2022; Yang et al., 2024d).
Long-Horizon Planning. Long-horizon planning is still challenging for embodied agents. Future research can study techniques like hierarchical planning (Song et al., 2023; Ajay al., 2023), memory-augmentee methods (Sarch et al., 2024a), and world models (Mazzaglia et al., 2024) enhance their ability to plan and execute complex, multi-step tasks more effectively.
Multi-step/MMutti-vieew Image Understanding. Our experiments show that current MLLMs struggle with multi-step and multi-view image inputs. Future research could improve multi-frame and multi-view comprehension, temporal reasoning, and spatial awareness enhance MLLM agents' visual perception and reasoning. One promising direction is leveraging video pretraining (Madan et al., 2024; Wang et al., 2024) better equip embodied agents for these challenges.
Visual In-context Learning (ICL). Our experiments confirm the effectiveness of visual ICL (Zhou et al., 2024b; Sarch et al., 2024b) in embodied decision-making. This approach promising because it enables adaptability and versatility without fine-tume fine-tuning, allowing better use of off-the-shelf MLLMs. However, designing more effective visual ICL methods for embodied tasks remains an open problem for future research.
Training Multimodal Embodied Agents. While our work focuses on evaluation, fine-tuning MLLMs for embodied tasks could significantly enhance their performance (Mu et al., 2024; Szot et al., 2024; Zawalski et al., 2024; Zhai et al., 2025). Future research can explore embodied pretraining, imitation learning, and both offline and online reinforcement learning (Sun, 2023) to better optimize MILMs for embodied learning approaches that seamlessly integrate perception, reasoning, and action could reduce the need for designing complex agent frameworks, leading to more adaptive and generalizable agents.
Additionally, developing end-to tond
Robustness and Generalization of MILMM Agents. Ensuring real-world applicability requires a thorough study of MLLM agents' robustness and generalization capabilities. While related studies are emerging in other domains (Zou et al., 2024; Xu et al., 2024; Yang et al., 2023b; 2024b; Zhang et al., 2024b), research on MLLM agents remains limited. Potential methods involve incorporating adversarial settings (Liu et al., 2024b: Wu et al., 2024a), dynamically generated environments (Wang et al., 2023c), or domain shifts (Chattopadhyay et al., 2021) to assess and enhance the ability of embodied agents to perform reliably in varying conditions.
By exploring these directions, the field can move closer to realizing the full potential of MLLM- embodied agents in real-world applications.
C. Details about EMBODIEDBENCE Environments and Datasets
Below, we provide detailed descriptions of four environments and their corresponding datasets. Please note that the maximum number of environment steps varies by task: 30 steps for high-level tasks (EB-ALFRED and EB-Navigation), 20 steps for EBa-avigation, and 15 steps for ETB-Manipulation. In addition to task completion and exceeding the maximum step limit, we introduce two additional stopping conditions: (1) Invalid Action Limit: If the model generates more than 10 invalid actions in single trajectory, indicating a lack of understanding and difficulty in producing valid actions. (2) Empty Plan Generation: If the model generates an empty plan because it incorrectly assumes the task is complete. This issue mainly occurs in high-level tasks, and once it happens, the model tends to keep generating empty plans without making progress. These additional stopping conditions help reduce unnecessary computational costs and improve evaluation efficiency.
C.1. EB-ALFRED
Task Description. We develop the EB-ALFRED tasks based on the ALFRED dataset and the A12-THOR simulator, which are well-regarded within the embodied AI community for their diverse houschold tasks and scenes. These tasks aim to evaluate an agent's ability to organize and execute sequences of high-level actions in household scenarios, such as "Put washed lettuce in the refrigerator." " Each task in ALFRED can be described using the Planning Domain Definition Language (PDDL., which helps assess the agent's s success in completing the task or subgoals. The ALFRED dataset includes task
18

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
types, Pick Place, Stack & Place, Pick Two & Place, Clean & Place, Heat Place, Cool& Place, and Examine in Light. Our simulator is based on Lota-Bench implementation for 8 high-level action types: "pick up", "open", "close", "turn on", "turn off", "slice", "put down", and "find". Each action can be parameterized with specific object to form an action, "g.."fin an apple" or "pick up an apple". The simulation offers an egocentric view and text feedback on the validity of action execution and potential reasons for any invalid actions. For example, it may indicate "failure to pick up an object because another object is already being held."
Despite its strengths, task type due to the inability to handle multiple instances of one object type. (2) Some actions lead to incorrect task execution, such as the "put down" action erroneously placing object on top of the sink instead of inside it, causing correct action but unsuccessful outcome. (3) Additionally, some instructions in the original ALFRED dataset suffer from low quality, We observe the erroneous use of "potato" in task related to "tomato", which prevents agents from successfully completing the tasks due to these incorrect instructions.
's simulator has three notable limitations: (1) does not support the Pick Two & Place
To enhance the simulation, we implemented several improvements. Firstly, we introduced support for multi-instance settings in ALFRED by appending index suffixes to objects, such as "find a cabinet 2," to accommodate multiple instances of the same object type. Therefore, we can support all 7 task types in ALFRED. Given the dynamic number of objects in the ALFRED dataset. made the action space of EB-AEEEDD dynamic, ranging from 171 to 298 actions. To minimize redundancy in the action space, we merge all "put down" actions into single action, since only one object can be held at a time. Additionally, we manually corrected bugs in the original simulation and improved the quality of language instructions to ensure tasks are solvable and actions can be executed more accurately. These enhancements make EB-ALFRED a high-quality benchmark for evaluating embodied agents.
Dataset Collection. Following Lota-Bench (Choi et al., 2024), we use the valid seen set from the ALFRED dataset. We first partition the dataset based on the number of steps in the oracle policy. Specifically, we select 50 samples from the subset with fewer than 15 steps, carefully refining their instructions to minimize ambiguity and improve task solvability. The commonsense and complex instruction subsets are primarily derived from this base subset, with GPT-4o augmentation tailored to specific capabilities. Additionally, we select 50 tasks with more than 15 steps to form the long-horizon subset. The visual appearance and spatial awareness subsets are chosen directly from the original dataset based on language descriptions of color/shape, or relative positions. In total, EB-ALFRED comprises 300 testing instances, evenly distributed across six subsets (50 instances each).
C.2. EB-Habitat
Task Description. EB-Habitat is developed based on the Language Rearrangement benchmark (Szot et al., 2023), featuring 282 diverse language instruction templates designed for robotic rearrangement tasks. It leverages the Habitat 2.0 simulator (Szot et al., 2021) and includes object data from the YCB dataset (Calli et al., 2015) and ReplicaCAD (Szot et al., 2021). The benchmark focuses on planning and executing 70 high-level skills achieve user-defined goals, such as "Find a toy airplane and move it to the right counter." These skills are categorized into five action types: "navigation", "pick", "place", "open", and "close", each parameterized by specific objects.
Unlike ALFRED, which permits navigation to any object, EB-Habitat constrains navigation to receptacle-type objects, requiring robots to visit multiple locations to locate target items. Task and subgoal completion are evaluated using PDDL, with agents receiving visual input and textual feedback similar to ALFRED. Given its broad range of language instructions and distinct navigation constraints, EB Habitat serves as a complementary counterpart to EB-ALFRED, expanding the scope of our high-level embodied tasks.
Dataset Collection. Habitat already provides fine-grained evaluation datasets with multiple subsets. We reorganize the subsets to formulate our dataset. Specifically, merge "new scenes". "novel objects", and "instruction rephrasing" to form our base subset; we use the "context" set as our commonsense subset; we merge the "conditional instructions" and "irrelevant instruction text" as our complex instruction subset; we use the "referring expressions" as our visual appearance subset; we use the "spatial relationship" as our spatial awareness subset; we merge the "multiple rearrangements" and "multiple objects" as our Long-horizon subset. Then, we sample 50 instances from each subset to form our EB- B--Habitat dataset, resulting in a total of 300 testing instances.
19

---

EMBODIEDBENCH, Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
C.3. E.B-Navigation
Task Description. EB-Navigation is an evaluation suite built on A12-THOR, designed to assess the navigation capabilities of embodied agents. In each task, the agent is placed at starting position and must use visual observations and behavior feedback to execute low-level actions. The goal is to locate a target object and navigate to its vicinity. The agent's action space consists of seven actions that are executable by physical robots: (1) Move forward/backward by Ar. (2) Move rightward/lefwward by Ay. (3) Rotate to the right/left by AO degrees. (4) Tilt the camera upward/downward by App degrees. At the start of each task, the agent is provided with a textual description of the action space, where each action is mapped to a unique index. Then, the agent selects an action by outputting the corresponding index, which the environment then executes.
At the beginning of each step, the environment provides the agent with first-person visual observation. Using this visual input, the agent performs planning and decision-making to choose its next action. After executing action, the environment evaluates its validity. For example, it checks for collisions or obstacles that might cause the action to fail. The environment then provides this valid or invalid signal as feedback to the agent. This signal is the only feedback the agent receives, as it is feasible to obtain in real-world scenarios. Together with the visual observations, this feedback equips the agent with sufficient information to perform navigation tasks effectively.
Dataset Collection. We constructed the dataset based on the original dataset provided by A12-THOR. In A12-THOR (Kolve et al., 2017), there are diverse scenes including environments such as kitchens, living rooms, and bedrooms, we designed a total of 90 navigation tasks, one for each scene. Each task dataset includes the following information: (I) Initial Robot Pose: Including its (z. y, 2) coordinates and initial orientation. (2) Target object information: Specifying the object type, ID and the 3D coordinates of the object's center. (4) Language navigation instruction: A human-readable instruction specifying the target object the agent needs to navigate to. We ensure the validness of the task dataset through the implementation of the following characteristics: (1) Initial distance: The agent's s starting position is carefully constrained to be at least a certain adjustable distance (denoted as a) from the target object. This adjustable allows users to customize the number of navigation steps required for each task. (2) Target object accessibility: All target objects are exposed in the environment, reachable without requiring the agent to leave the ground. (3) Task completion conditions: A task is considered complete if the agent reaches a position within a specified distance threshold from the target object or if the maximum number of steps is exceeded. Additionally, the dataset includes an automated allows users to create custom task datasets by specifying parameters such the target object type, initial distance threshold, and random seed for each scene. This flexibility ensures the dataset can be adapted to various research needs and scenarios.
script. This script
For the capability-oorinteed subsets, we begin by sampling 60 instances from the original 90 tasks to form the base subset. We then use GPT- to perform instruction augmentation, generating more complex instructions and incorporating common sense knowledge to create the complex instruction and common sense subsets. The visual appearance subset is manually curated to include detailed descriptions of the target object's color and shape. Finally, the long horizon subset is constructed by ensuring the target object is not visible in the agent's 's initial view, requiring extended navigation to locate it. In total, we collect 300 testing instances across these 5 subsets (excluding the spatial awareness subset).
C.4. EB-Manipulation
Task description. EBB-Manipulation is an extension of VLMBench (Zheng al., 2022) using the CoppeliaSim simulator (Rohmer et al., 2013) to control 7-DoF Franka Emika Panda robotic arm. EB-Manipulation includes four task categories: (1) Pick & Place Objects, (2) Stack Objects, (3) Shape Sorter Placement, and (4) Table Wiping, each with randomly varied instances in color, position, shape, and orientation for diverse evaluation. The action space is a 7-dimensional vector. The simulator processes these actions and performs automatic motion planning to achieve the desired position. To facilitate motion planning, the environment operates in ABS EE_ JOSN.PLAA. WORLD.FRAME mode, ensuring automatic trajectory execution from the current pose to the target pose. This simplifies the agent's 's role in predicting keypoints necessary for task completion.
Direct low-level manipulation is challenging for MLLMs due to insufficient domain-speciffe training. To overcome this, we implemented enhancements. (1) Action space discretizationYYin et al., 2024), which divides the position component into 100 bins and the orientation component into 120 bins, enabling valid actions to take forms like [x. y, 2, roll, pitch, you, gripper] = [57, 61, 10, 60, 25, 1]. Here, the first three dimensions (X, Y, Z) range from to 100, while the next three (Pitch, Yaw, Gripper) range from 0 to 120. The gripper state remains binary (0.0 or 1.0). By discretizing the originally continuous action space, the model can predict actions using integer values, reducing complexity
20

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
for MLLMs. (2) Additional information like YOLO (Redmon, 2016) detection boxes with index markers and 3D object pose estimation for indexed objects, reducing the need for precise 3D location. Instead, the agent can focus perceiving and reasoning about each object's relationship to the indexed objects. With these improvements and additional in-context examples, our MLLM agent effectively tackles complex low-level manipulation tasks.
At each step, the environment provides a front-view visual observation capturing a wooden table, a robotic arm positioned at the center corner, and multiple objects placed on the table. Each object is enclosed within a detection box labeled with a numerical index, sorted descending order based on its Y-coordinate. A 3D XYZ coordinate system is displayed the robot's frame origin for spatial reference. The field of view (FOV) and image resolution are configurable, offering flexibility in visual input settings. Additionally, all visible objects in the scene are provided with discrete 3D coordinates, sorted in descending order based on their Y-coordinate, and labeled with object index (e.g., "object 1*). This setup requires the agent to understand the correlation between objects mentioned in the instruction and their corresponding object indices. Using this position information, the agent can plan and execute action sequences to achieve the manipulation goal. To ensure validity, the environment evaluates each action, preventing constraint violations such as invalid trajectories or out-of-range movements. The validity signal serves as the sole feedback mechanism.
Dataset Collection. For the base and spatial subsets, we select and curate samples from the VLMBench dataset. To generate instructions for each subset, we provide GPT-4o with 10 in-context examples. The common sense, visual appearance, and complex instruction subsets are derived from the base subset, with modifications designed assess specific capabilities. The visual appearance subset consists of 36 tasks, as the table wiping task is excluded due to the inability to distinguish objects based on appearance. Each of the remaining 4 subsets comprises 48 tasks evenly distributed across four categories, with 12 tasks per category. In total, MBB-Manipllation consists of 228 testing instances.
D. Model Versions
Table 4 lists the versions full names of the models used in our experiments. We accessed proprietary models through API calls and open-source models via local deployment using Imdeploy (Contributors, 2023) and vilm (Kwon et al., 2023).
E. Definitions and Examples of Capability-oriented Subsets
As listed in Table 5, we provide definitions and examples of six capability-oriented subsets in EMBODIEDBENCH.
21

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Model Name
Creator
Full Name
GPT-4o
OpenAI
gpt-40-2024-08-06
GPT-4o-mini
OpenAl
gpD-4o-mini-2222-07-11
Claude-3.7-Sonne
Anthropic
claude-3-7ssomeee22255011
Haude-3.5-Somet
Anthropic
claud-335--omnee222241222
Gemini-1.5 1.o-oo
Google
gemini-1.5-pro
Gemini-2.0-flash
Google
gemini-2.0-ftash-exp
Gemini-1.5-fae
Google
gemini-l.5-flash
Qwen-VL-Max
Qwen
qwen-vllmaax-2225--1-5
Llama-3.22 3.2-90B- -isi--ins
Meta
meta- metaa llmma//Laam333329999--- -er------iiioo-nttrreee
Llama-3.2-111 1B-1 Ins
Meta
InterVL22. 5-78B
OpenGVLab
OpenGVVLablmneem1122.55555 5-7B
Inter VL2 5-38B
OpenGVLab
OpenG\VLaa//meeem\Y2225555 5-38B
InternVL2_5-88
OpenGVLab
QpennGVVLab//neee/22222-888
InternVL3--8B
OpenGVLab
OjennGVLabbInneem\LL3333888
InternVL3-338
OpenGVLab
OpenG Vaabllnneem\L3333388
Intern VL3-8B
OpenGVLab
OpenGGVLabb/nneem2L33888
Qwen2-V1--72B-In
Qwen
Qwan//Qwen22VV--22---nnntc
Owen2-VL-7B-Inn
Qwen
QwennQQwen2-VL-77B-nnsttu
Owen2.5-VVL-72B-Ins
Qwen
Qweni(Qwent255VV--22B--nnmm
Qwen2.5-VLL-7B-Ins
Qwen
Qwenn/Qwean2555-2-77-1nnnnnn
Ovis2-34B
AIDC-AI
AIDC-A1/Ovis2-34B
Ovis2-16B
AIDC-AI
AIDC-A1/Ovis2-16B
gemma-3-27b-it
Google
google/gemma-332271
gemma-3-12b-it
Google
googlelgemma-3-12b-it
Table 4. Full names of MLLMs used experiments.
22

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Table 5. Definitions and examples of six capability oriented subsets in EMBODIEDBENCH. Four environments AB-FRDD EB BB-Habitat, EBBManniplattion, and EBBNavigation are abbreviated as ALF, Hab, Man, and Nav, respectively.
Subset Name
Instruction Example
Description
ALF; Put washed lettuce in the refrigerator.
Hab: Move one of the pear items to the indicated sofa.
Base
Instructions used describe basic tasks.
Man: Pick up the star and place into the silver con- tainer.
Nav: Novigate the pillow in the room and be as close as possible to it.
ALF; Place washed teafy green vegetable in recepta- ele that can keep it fresh for several days.
Common Sense
Refer to objects indirectly using common sense knowledge.
Hab: Prepare for a game by delivering something to play with to the TV stand.
Man: Pick up the bright object that usually appears in night sky alongside the moon and place it into the silver box used for storing things.
Nav: I'm feeling thirsty and need small container to hold water or coffee. Please navigute that object and stay near it.
ALF: For freshness, place the washed lettuce in the re- frigerator. This way, it ready for any delightful recipe ideas you have.
Add longer relevant or irrelevant context to obscure the instruction. This is used to evaluate the ability of understanding complex instructions.
Complex Instruction
Hab: When you find the fridge door open, go ahead and move an bowl the sofa; otherwise, transport an hammer to the sofa.
Man: The objects on the desk seem perfect for children to play with. Con you now pick up the star and place it into the silver container? We'r 'r riyiing up.
Nav: The rhythoic ticking of the kitchen clock blends with the occastonal drip from the faucet. There's 's small pile of onions on the table, freshly chopped. Please move towards the stove burner for me. The kitchen has comforting hum
ALF: Put two spray bottles in the cabinet under the sink against the wall.
Refer objects by their location relative to other recep- tacles or objects.
Spatial Awareness
Hab: Move a spatula from the right counter to the right
receptacle of the left counter.
Man: Pick up the left object and place it into the front container.
ALF: Put a knife a blue container onto the black
Visual Appearance
table in the corner.
Refer objects indirectly by their visual appeurance.
Hab: Deliver small red object with green top to intended a large gruy piece of furniture with backrest by physically moving it there.
Man: Put the green object with five evenly spaced points into the sorting container.
Nav: Find the rectangular yellowish object with a soft and smooth surface.
ALF: Pick up knife, slice apple, put knife in bowl, heat slice of apple in microwave, put apple slice on table. Hab: Move the rubriks cube to the left counter, the wrench to the left counter, and the bowl to the brown table.
Describe lask that requires long sequence of actions to complete. For EB-naiigation, the instruction sim- ilar the base subset but the location of object not visible initialization.
Long Horizon
Nav: Navigate to the Toaster the room and as
close as possible to it.
23

---

EMBODIEDBENCH. Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Table Subgoal success rates subsets of EB-ALFRED and EB-Habital with the best proprietary model in bold and open-source model underlines per column.
Model
EB-ALFRED
EB-Habitat
Avg
Base
Common Complex
Visual
Spatiol
Long
Base
Common Complex
Visual
Spatial
Long
Proprictary MIZ.ss
GFT-4a
65.1
74.0
60.3
74.0
58.3
61.3
62.5
70.7
90.7
56.0
68.0
75.2
62.1
72.2
GPP-4o-mini
34.3
47.8
35.3
43.5
33.3
29.0
17.0
44.0
77.5
32.5
42.0
33.1
57.8
21.3
Claude 3.5 Sonnet
65.3
72.0
66.0
76.7
63.0
59.7
54.5
70.8
97.5
68.5
79.5
72.0
43.8
63.3
Gemini- Pro
67.4
74.3
66.7
76.5
62.8
59.0
65.0
61.0
92.5
53.5
49.5
59.4
50.0
61.2
Semiini-..0-fiaad
56.3
65.7
51.3
58.3
50.7
50.0
62.0
48.2
82.0
39.5
43.0
39.0
49.6
36.2
Gemini- 1.5-Baah Blah
46.1
49.5
45.2
60.2
48.3
32.2
41.5
46.8
79.0
33.0
50.0
41.2
55.5
22.0
GPT-4o (Lang)
65.6
67.7
70.3
77.0
59.7
54.0
65.0
66.7
85.2
58.5
67.5
79.2
62.1
47.7
GPT-4o-mini (Lang)
40.1
44.8
41.2
54.2
36,0
24.7
39.5
48.1
85.8
39.0
43.5
39.0
56.8
24.5
Open-Soure Source MLLMs
Llama: 3.2.a-3.2-00 3.2-90B Vision las
37.6
43.7
37.3
49.2
35.3
36.0
24.0
50.6
94.5
32.5
53.0
39.7
59.6
24.3
Llama-3.--1IB. VisiVisiio-al las
19.7
29.7
13.0
25.7
28.7
9.3
12.0
33.2
72.0
23.8
36.5
16.2
39.7
11.2
IniemVL2.5-70
41.0
42.3
35.3
43.3
35.7
40.3
49.0
55.2
82.0
43.0
39.0
63.9
45.1
38,2
IniemVL2 L12-38L 5-38B
31.3
37.3
33.0
38.3
25.3
17.3
36.5
44.0
61.5
32.5
49.0
39.5
46,3
35.0
IntemVL2 SB
2.0
4.0
6.0
2.0
0.0
0.0
0.0
19.4
40.2
11.5
11.0
16.0
30.7
7.3
Owen2 VL-7VB--LBBB Las
38.7
45.3
33.3
44.7
35.7
33.0
40.0
42.2
72.0
33.0
39.0
37.0
52.0
20,3
Qwen2-VLL 11-78tm
5.2
8.3
5,3
7.0
1.7
3.3
5.5
26.1
53.3
9.0
24.0
25.8
41.2
3.2
F. Additional Experiment Results
To thoroughly evaluate the performance of MILMS as agents within EMBODIEDBENCH. we present additional metric results, including subgoal success (Appendix F.1) and average step counts (Appendix F.2), and conduct series of ablation studies. These ablation studies, spanning from Appendix F.3 to Appendix F.7, focus five critical factors: (1) varying camera resolutions, (2) the use of detection boxes, (3) multi-step images, (4) multi-view images, and (5) visual in-context learning. In the subsequent sections, we systematically analyze each of these factors, offering insights into their effectiveness and potential limitations.
F.1 Subgoal Success Rate
In addition to the task success rates presented in Table 2, we further analyze the subgoal success rates for ligh-level tasks (EB-ALFRED and EB-Habitat), as detailed in Table 6. Given the use of symbolic expressions (Planning Domain Definition Language, PDDL) in high-level tasks, calculating subgoal success rates is straightforward. For instance, a task success condition can be expressed as "condition A and condition B." Completing condition alone results 50% subgoal success rate, even though the final task success rate remains 0%.
The results in Table 6 generally align with those in Table 2. For most models, the subgoal success rates are higher than their final task success rates, which is expected. Notably, Gemini- 1.5-Pro achieves higher subgoal success rates than Claude- 3.5-Sonnet on the EB-ALFRED benchmark, despite Gemini-1.5-2
having a lower final task success rate. Additionally, GPT-4o demonstrates subgoal performance comparable to Claude-3.5, with a gap of less than 0.2 both environments, despite substantial gap in their final task success rates. These findings suggest that while models demonstrate better ability achieve subgoals, completing the final task remains significant challenge. Additionally, the capability to achieve subgoals may slightly differ from the ability accomplish the entire task. Since our primary objective is to achieve the full task, future research should focus on developing strategies improve the final task success rate.
F.2. Average Planner and Environment Steps
This section presents the results of average planner steps and environment steps, which quantify the number of model inferences and interactions with the environment, respectively. Since employ multi-step planning strategy, the number of environment steps exceeds that of planner steps. However, is important to note that neither planner steps nor environment steps serve as precise metrics for evaluating agent performance, unlike the success rate. This is because the agent may generate empty plans or produce more than 10 invalid actions, potentially triggering early termination. Consequently. fewer steps do not always indicate superior planning performance. Nevertheless, meaningful insights can still be derived from Table 7 and Table 8:
24

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
The multi-step planning strategy demonstrates significant efficiency in most cases, reducing average planner steps by around 50% 80% compared to average environment steps. This is particularly evident in the EB-Manipulation task, where the average planner step for GPT-4o is 2.6, and the average environment step is 12.9, resulting in nearly 80% fewer model inferences. This highlights the model's ability to generate long action sequences and we can effectively leverage this capability to minimize costs. Such efficiency is especially advantageous when utilizing expensive large proprictary MLLMs.
Despite the inherent inaccuracies in average step counts, it is still possible to observe that more capable models tend to achieve smaller average planner and environment steps. For instance, Claude-3.5-Sonme achieves the lowest planner and environment steps in both EB-ALFFED and FB-Habitat tasks, while GPT-4o records the lowest average planner and environment steps in EBB-Maniullttion Additionally, larger models generally require fewer steps than their smaller counterparts, as evidenced by the comparison between GPT-4o and GPT-4o-mini, as well as Gemini- Aemini--15-PP4 1.5-Pro and Gemini-2.0-Flass
Model
EB-ALFRED
EB-Habitat
Avg Planner Step
Avg Env Steps
Avg Planner Step
Avg Eny Steps
GPT-4o
4.4
16.3
5.5
13.1
GPT-4o-mini
7.7
20.6
7.4
18.8
Claude-3.5-Sonee
4.0
12.1
4.2
10.9
Gemini-1.5
3,9
15.7
5.4
12.6
Gemini-2.0-flash
4.4
16.3
6.8
14.8
Llama-3.2-90B-
1..2-90B--iisio-Ins Ins
7.3
16.7
7.3
16.2
Intern 5-78B
5.5
13.9
6.3
14.1
6.2
16.8
6.6
14.3
Qwen2-VL-72B-In
6.1
13.7
6.8
14.2
Table 7. Average planner steps and environment steps in EB-ALFRED and EB- 3B-Habitat for different models.
Model
EB-Navigation
EB-Manipulation
Avg Planner Step
Avg Env Steps
Avg Planner Step
Avg Eny Steps
GPT-4o
6.2
15.5
2.6
12.9
GPT-4o-mini
7.6
17.5
3.4
14.7
Claude-3.5-Sonne
6.2
15.6
2.7
13.3
Semini--.5-PP.
8.8
16.5
2.7
13.4
Gemini-2.0-flash
9.2
16.0
2.8
14.0
Llama-3.2-90B- 1.....2-99B-iiiio--i
7.5
17.4
3.0
13.9
Intern V1.2 5-78B
13.2
17.3
2.9
13.5
InterVLL2 5-78B-MPO
6.2
16.5
2.9
13.5
Qwen2-VL--72B-Ins
11.1
17.8
2.9
13.9
Table 8. Average planner steps and environment steps in EB -B--avigation and EB EB-Manipulation for different models.
F.3. Camera Resolution
As shown in Figure 7, we tested three camera resolutions 300xx300, 500x500, and 700x700 on EB-ALFRED, EB- Manipulation, and EB-Navigat tasks. Our results reveal a desendepnndnnt pattern: for EB-ALFRED, where vision plays a secondary role, increasing the resolution slightly improves performance for both GPT-4o and Claude-3.5-Sone S-Snnet,, with
25

---

EMBODIEDBENCH Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
accuracy gains of 2% 4%. In contrast, for EB-Manipulttion and EB-Navigation, resolution performance achieved 500x500. This suggests that while low-resolution images may lack the fine details needed for task execution, overly high resolutions introduce unnecessary complexity, making it harder for MLLMs to focus relevant information. These findings underscore the importance of choosing the right resolution when deploying MLLM-bar based embodied agents.
critical, with the best
camera resolutions
camara resolutions Manipulation
Different camera resolutions Navigation
74.3
Success Rate (%6
66.0
66.0
Success Rate (16)
60.0
Success Rate
27.1
29.2
51.7
48.3
40.3
10
22.9
20
GPT-40
Claude-3.5 Claude-3.--Somme 5-Sonnet
CPT40
Claude3.5-Soneet
Claude-3.5-Sone
Figure Impact of different camera resolutions on EMBODIEDBENCH.
F.4. Detection Boxes
Figure illustrates the impact of using detection (bounding) boxes. The results show that detection boxes are beneficial for both AB-ALFRED and EB-Manipulation, enhancing object recognition and interaction for models such as GPT-4o and Claude-3.5-Somme In particular, Manipuaiullttion experiences a significant performance boost of nearly 10%.
In contrast, detection boxes tend hinder performance in EBB-Navigation. This is likely because they obscure visual cues that are critical for effective path planning, leading to lower success rates. To investigate further, we conducted an additional experiment on EB-Navigation using only single bounding box around the target object. As shown in Figure 9, this approach avoids the visual clutter caused by multiple boxes. The results, presented in Table 9, demonstrate that using single detection box consistently improves accuracy. To better reflect real-world conditions, Eav-aiigation omits detection boxes by default, requiring the MLLM agent to detect and recognize objects autonomously.
These findings highlight the importance tailoring visual augmentation strategies the specific requirements of each task. Consequently, enable detection boxes by default only for EIB-Manipulation, while disabling them for all other task groups.
Impact Detection Baxes for ALFRED
Imoact of Detection Boxes for Ee-Manipulation
Impact of Detection Boxes for EB- R--Navigation
100
wyo
detection boses
w/ detection bose:
wt detection boxes
wio detection boves
Success Rate (%)
76.0
72.0 0
40
30.6
37.5
64.0
27.1
29.2
Successo Mater
46.7
SuCCORs
43-
20
30
30
10:
GPT-4o
Coude-3.5-Sernee
GPT-40
Cloude 3.5 5-Sonnet
GFT-43
Claudo-3.5 3.5-Sonnet
Figure 8. Impact of detection boxes on EMBODIEDBENCH.
Table 9. Comparing Different Detection Box Strategics in EB IB-Navigation.
Model
No Box
One Box
Multi Box
GPT-4o
61.7
68.3
53.3
Claude-3.5-Sonnet
46.7
58.3
48.3
26

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Figure 9. Illustration of Multiple (left) or Single (right) Delection Boxes in EB-Navigation
F.5. Multi-step Images
Using sequences of images a common approach to address partial observation. As shown in Figure and 11, observation images from the previous two environment steps are also included in addition to the planner's original visual input. We explore the effectiveness of multi-step images- sequential frames shown in Figure 12-where the latest three images included as input. Surprisingly. adding temporal context does not improve decision-making instead, it leads to decline in performance, particularly for EB-Manipun CBB-Manipulation. This may be due to the models' struggle to interpret the relationship between multiple sequential images and their current state. These results emphasize the challenges of effectively utilizing temporal continuity in vision-language tasks.
Step t-2
Step t-1
Step
Figure 10. Multi-step observation example in EB-aiiggtionn
Step t-2
Step t-1
Step
Figure II. Multi-step observation example in MB-Manipulation
27

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driver Embodied Agents
Impact of Multi-step Images for EB-Manipulation
Impact of Multi-step Images for EB-Navigation
wy mulli-step
w/n muiti-step
30
wi milli-eep
wic mulli-step
Success Rate (%)
Surress Rote (56)
60
61.7
30
29.4
40
20
20
GPT-40
Clude-3355 3.5-onnttonnttt
GFT-0o
Clauce-3 3.5-oonnet
Figure 12. Impact of multi-step images EMBODIEDBENCIL
F6. Multi-view Images
In addition to multi-step images, another approach is to incorporate multi-view images from different cameras at the same time step. As shown in Figure 13, the planner receives images from two different viewpoints as input. For EBB-avigation, the input consists a front-view image and a top-down view image. For M--napppllaiinn the planner receives a front-view image and a wrist-view image. evaluate whether multi-view images enhance performance EBB-Navgation, we present the results in Figure 14. Surprisingly, using multi-view data also results in performance decline, particularly for GPT-4o. While multiple viewpoints theoretically offer richer spatial context, GPT-4o and Claude-3.5-Sone seem to struggle with effectively integrating and leveraging these additional perspectives. This limitation may arise from challenges in multi-view feature fusion or the increased complexity of the input.
EB-Manipulation and
EB- -aviiiggtti
EB- EB-Manipulation
Front view
Top-down view
Front view
Wrist view
Figure 13. Multi-view observation example in EB- EB-Navigation (left) and EB EB--Manpuulation (right).
Impact of Multi-view Images for EB- Manipulation
Impact of Multi-view Images for EB- Navigation
50
wi multi-iiew
win multi-v ew
80
My mu tiviee
web multi-view
Success Rate (96)
39.5
Success Rate (%6)
60
61.7
31.3
50.0
48.7
20
GPT-40
Cluce-3.5-Somee
GPT-40
Claude-3 5-Sonnet
Figure 14. Impact of multi-view images on EMBODIEDBENCH.
28

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Visual in-context learning example for EB-Navigation
Planning Step
("reesoning": can see garbage can upper left corner of the inage, next to the kitchen sink. To move there, we can go forward-left, but since there's a kitchen counter directly ahead, we should go left first. Following the strategy, can go by first moving leftward.", "output": "[[3, "Move leftward 0.25'l, [3, 'Move leftward by 0.25']]", senv_feebbacke "Last action is executed successfully.",
Planning Step 2
("reasoning"; "From the secene, I see that by noving leftward, we are getting closer to the garbage can, Now, the garbage can is in front of me, slightly to the left. And there's Following the strategy, can go by first noving forward then moving leftward.", "out "I[0, "Move forward by 0.25'I, [0, 'Move forward by 0.25 [0, "Move forward by 0.25'I, (0, 'Move leftward by 0.25 0,25"]]",
large area
"env cnn__fedbback. "Last action is executed success fuccessfully.""
Planning Step 3
see the garbage can is very close to us, still to our front-left. Moving teftward that there still space in
("reasoning"; "From the inage
night be blocked but front of me to get closer to the garbage can, Following the strategy, we can take about two steps forward then one step left to reach the garbage can. "output": "[[3, 'Move forward by 0.25*), [3, 'Move forward by 0.25'], 'Move leftward by 0.25
:eedbackb:;" "Success")
Visual in-context learning example for EB-Manipulation
("visual state,, destteddecciitiion "From left to right, can see purple cylinder at [45, 13, 18], blue noon at [72, 20, 18],
green star at [50, 42, 171, purple cube at [36, 42, 18),
yellow container at [70, 43, 17] and purple container [50, 77, 17).,,,reasonnii pick the star located [50,
and renldeeellciion "The task is to
17] and place the yellow container located at 170, 43, 17]. The plan involves moving the gripper to the star's position, closing the gripper to pick it moving to the yellow container's position, and opening the gripper to release the cube into the container. " "executable p "[[50, 42, 17, 6, 1), [50, 42, 17, 6, 61, 36, 0), [47, 41, 26, 6, 61, 36, 8], [69, 39, 24, 88, 32, 77, 01, [69, 39, 24, 87, 32, 77, 111"}
36,
Figure 15. Visual in-context learning examples for EB-Navigttion & EBB-Manipulation
F.7. Visual In-context Learning (ICL)
Previous research has mainly focused on text-based in-context learning (ICL) demonstrations, this study, we explore the impact of visual ICL for embodied agents by including image observations part of the in-context examples for TBB-Manipulation. This approach helps the model better grasp the connection between successful low-level actions and the positions of objects in the image. We provide two visual ICL examples in Figure 15, where the planner receives images corresponding to the textual in-context examples. To avoid overwhelming the model with excessive visual input, we limit the number of examples to two, which might slightly reduce performance compared to the main results without visual ICL. As illustrated in Figure 16, the results show that visual ICI. significantly outperforms language-only ICL, with particularly
29

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Impact of Visual ICL for EB-Manipulation
Impact of Visual ICL for EB-Navigation
Visual
Visual
Visual ICL
Veo Visual ICL
66.7
Surcees (%6)
Stccess Ratr (%6
25.4
30
27.1
0
GP
Claide-3
GPT-46
Claude-3 Sonnet
Figure 16. Impact of visual in-context learning EMBODIEDBENCH.
impressive gains in manipulation tasks. For instance, Saaude-3.5-Some achieves 16.7% improvement in performance. These findings highlight the potential of visual ICL promising direction for future research in vision-driven embodied agents.
E88 Additional Ablation Study Conclusion
Overall, ablation studies reveal that while certain visual enhanceemens--such moderate resolution increases, bounding- box detection, and visual in-context learning- be beneficial, others-like extreme high-resolution inputs, multi- step/multi-vies images, or detection boxes for navigation may have limited or even negative effects. These findings highlight that the effectiveness of visual strategies heavily depends on the specific task and how additional visual information integrated. Future research should focus on developing more advanced fusion techniques for embodied agents to better optimize the use of diverse visual inputs from multiple images.
G. Further Discussion on Chat History as Input for EB-Navigation
Among the four environments in EMBODIEDBENCH EBBNavigation is the most sensitive to historical input. Specifically, we observe that incorporating chat history-i.e. prior conversations paired with images, rather than the multi-step image setting studied in Appendix F.5, affects different models in varying ways.
Table 10 summarizes the results. We draw two key observations: (1) proprietary MI.LMs tend to benefit from the inclusion of chat history, particularly long-horizon tasks. (2) open-source models exhibit mixed preferences. For example, Llama-3.2 and the Intern VL series perform better with chat history, whereas Qwen2-VL. VL models perform best with single-step input, using only the summarized "interaction history" in the other environments.
While we do not have definitive explanation for this discrepancy, we hypothesize that some MILMs may have been trained on large amount of multi-turn multimodal dialogue data, influencing their capability to this input format. Due to the lack of a clear conclusion, we adopt chat history only for -Bviagigt and do not use as a global design choice.
H. Error Definitions and Additional Analysis
H.1. Error Type Definition
In this section. define the types of errors and sub-errors encountered. We categorize errors into three main types: perception errors, reasoning errors, and planning errors. Each error type corresponds to specific stage in our agent pipeline. For example, perception errors occur during the visual state description stage, reasoning errors arise in the reflection and reasoning stages, and planning errors occur during the language plan and executable plan generation stages. A detailed breakdown of sub-errors for each error type provided in Table 1I.
H.2. Error Analysis for EB-Navigation
In evaluating the performance of MILMs on navigation tasks, we identified three main types of errors: perception errors, reasoning errors, and planning errors. These errors significantly hinder the model's ability to successfully navigate to the target object.
30

---

EMBODIEDBENCH Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Table 10. Comparison of performance in EB-Navigation with and without chat history. "wo CH" indicates that the model does not chat history as imput.
EB-Navigation
Model
Avg
Base
Common Complex
Visual
Long
Proprietary MLLMs
GPT-4o
57.7
55.0
60.0
58.3
60.0
55.0
GPT-4o w/o CH
45.3
63.3
58.3
51.7
38.3
15.0
Claude-3.-SSomet
45.0
50.0
61.7
50.0
36.7
26.7
Claudc-3.--Somnct CH
39.3
53.3
48.3
36.7
41.7
16.7
Claude-3.5-Sonee
44.7
66.7
51.7
41.7
36.7
26.7
Claude-3.5-Somee
36.3
45.0
40.0
38.3
46.7
11.7
Gemini-I.5-Prt
24.3
23.3
25.0
25.0
28.3
20.0
Gemini- 1.5-Pro w/o CH
21.3
25.0
25.0
23.3
28.3
Gemini-2.0-faan
48.7
63.3
65.0
50.0
51.7
13.3
Gemini-2.0-flush w/o CH
36.0
48.3
35.0
43.3
43.3
10.0
Open-Source Source MLLMS
Llama-3.2-9000 -os--n--issss
30.0
48.3
23.3
38.3
33.3
6.7
3.2-0-3.--90B-- Vision-Ins w/o CH
22.3
35.0
31.7
18.3
18.3
8.3
1lama-3.2-118 2--1BB--880 Vision-Ins
21.4
23.3
21.7
26.7
18.3
17.0
1amma-3.211B--isionn w/o CH
17.3
28.3
21.7
15.0
18.3
3.3
InterVLL 5-78B
30.7
36.7
38.3
33.3
21.7
23.3
Intem w/o CH
29.3
40.0
28.3
35.0
26.7
16.7
21.3
35.0
23.3
21.7
26.7
0.0
InternVL2 5-8B
5.7
8.3
6.7
6.7
Dwen2-VL-72B--B
21.2
26.7
30.0
28.3
16.0
Qwen2-V1 Vi.n2--V.-72B-Imss w/o CH
36.3
36.7
40.0
50.0
36.7
18.3
Qwen2-VL--B-inn
14.0
26.7
10.0
15.0
15.0
3,3
Qwen2-VL-7B-Ins w/o CH
16.3
20.0
20.0
30.0
8.3
3.3
hallucination 4%
inaccurate action 23%
wrong recog. 22%
planning error 23%
perception 31%
spatial understanding 4%
reasoning 46%
reflection error 23%
spatial reasoning 23%
insufficient exploration %
Figure 17. Error Analysis on Naviiiatttinn.
Perception Errors. The first category involves the model's s ability to interpret visual observations and recognize the
spatial position of the target object. We observed two common failure patterns: (1) Wrong Recognition: In some cases, the
31

---

EMBODIEDBENCH, Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Error Type
Definition
Perception Errors
Hallucination
Perceiving objects or attributes that are not present in the visual input
Wrong Recognition
Incorrectly identifying object types or attributes
Spatial Understanding
Misjudging object positions / depths 3D space
Reasoning Errors
Spatial Reasoning
Failure to understand reason about spatial relationships
Insufficient Exploration
Only giving a suboptimal exploration strategy through reasoning
Wrong Termination Decision Reflection Error
Ending task execution before completing the goal
Failing to realize previous errors or adapt plans using environmental feedback
Planning Errors
Inaccurate Action
Executing actions with incorrect parameters poses Omitting necessary actions in sequential plans Attempting physically impossible interactions Misaligning action names with wrong action IDs
Missing Steps
Invalid Action Action ID Mismatch
Table 11, Error Taxonomy with Definitions
model failed to identify the target object even when was present in the visual input. This suggests limitations in object recognition, possibly due to inadequate feature extraction or attention mechanisms. (2)aalluciattion of the Target Object: In other instances, the model incorrectly claimed to have detected the target object when it was not actually present. This issue is particularly problematic, leads to premature conclusions and incorrect decisions instead of further exploration. Ideally, the model should acknowledge its inability to locate the target and continue navigating appropriately.
Reasoning Errors. The second category arises from the model's limitations in reasoning. This problem appears in two main ways: flawed logical reasoning about possible paths, even when visual observations are accurate; and weak reflection on spatial structure after failed attempts or feedback from previous steps. These issues indicate a lack of strong 3D spatial reasoning. The model often struggles to build a coherent 3D representation from sequential 2D observations, resulting in poor movement decisions. Furthermore, the model not good at refining its actions based on ongoing feedback.
Planning Errors. Even when the model correctly identified the general direction, it often struggled with movement precision ("inaccurate actions").. For instance, it might overshoot the target by taking three steps instead of two. This highlights problems with spatial quantification, the model's distance estimation and movement execution frequently did not align with real-world needs.
These main categories of errors reveal significant limitations in the navigation capabilities of currett MLLM- -bssed agents. To address these challenges, improvements are needed in several areas: strengthening object recognition, reducing hallucinations, enhancing 3D spatial reasoning, and aligning the model output with the action space to generate accurate plans. These advancements would enable more reliable and efficient autonomous navigation.
H.3. Format Errors
In addition to the aforementioned error types, we also observe output format errors when generating JSON. Specifically, smaller-scale models can fail to produce valid JSON files. Table 12 shows the number format errors across all six subsets, revealing that even proprictary models are not immune these errors. clear trend emerges: larger models tend to have fewer format errors, while smaller models are more prone to such issues. This highlights the need for further alignment of small models to improve the accuracy of structured outputs. Additionally, EB-ALFRED exhibits a higher number of errors, largely due to its greater complexity and the increased number of action steps in a trajectory
32

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Model Name
EB-ALFRED
EB-Habitat
GPT--4o
0.0067
0.0067
GPT-4o-mini
0.0867
0.0200
Claude-3.5-Sonnet
1.5400
0.0000
Gemini-I..-Pro
0.0000
0.0000
Gemini-2.0-flass
0.0000
0,0000
Llama-3.2-90B- --soo---isson--nns
1.8033
0.0233
Llama-3.2-15 ---
1.6767
1.3233
InternVVL2 5-78B
1.1467
0.0000
Intern Vit 5-38B
2.4600
1.7433
Intern V1.2 5-8B
8.0400
4.3967
Qwen2-VL-72B-Inns Qwen2-VL-7B-Ing
1.6100
0.0767
1.6933
0.7067
Table 12. Format error number per trajectory in EB-ALFRED EB-Habitat across subsets.
I. Input of the Vision-driven Embodied Agent
I.1. Prompts
We provide the agent input prompts used as textual input to MILMSs for all four environments.
33

---

EMBODIEDBENCH:
Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Prompt EB-ALFRED
## You are robot operating home. Given a task, you must accomplish the task using a defined of actions achieve the desired outcome.
## Action Descriptions and Validity Rules
Find: Parameterized by the name of the receptacle to navigate to. So long as the object is present in the scene, this
skill is always valid.
Pick up: Parameterized by the name of the object to pick. Only valid the robot close to the object, not holding
another object, and the object is not inside a closed receptacle.
Put down: Parameterized by the name of the object put down nearby receptacle. Only valid if the robot holding object.
Drop: Parameterized by the name of the object to put down. It is different from the Put down action, as this does not guarantee the held object will be put into specified receptacle.
Open: Parameterized by the name of the receptacle to open. Only valid if the receptacle is closed and the robot is
close to the receptacle.
Close: Parameterized by the name of the receptacle to close. Only valid if the receptacle open and the robot
close to the receptacle.
Turn on: Parameterized by the of the object on. Only valid if the object is turned off and the robot close to the object.
close to the object.
Turn off: Parameterized by the name of the object to turn off. Only valid if the object is turned on and the robot is
Slice: Parameterized by the name of the object to slice. Only valid if the object is sliceable and the robot is close to
the object.
## The available action id (0 (len(SKILL SET) 1}) and action names are: {SKILL SET).
[ICL EXAMPLES}
## Guidelines
1. **Output Plan**: Avoid generating empty plan. Each plan should include no more than 20 actions.
** Visibilityy** Always locate a visible object by the 'find' action before interacting with it.
** *ctiion Guidelines**; Make sure to match the action name and its corresponding action id in the output.
Avoid performing actions that do not meet the defined validity criteria. For instance, you want to put an object in receptacle, use put down' rather than 'drop' actions.
4. ** Prevent Repeating Action Sequences* Do not repeatedly execute the same action or sequence of actions. Try to modify the action sequence because previous actions do not lead to success.
5. **ultiple Instances There may be multiple instances of the same object, distinguished by an index following their names, e.g., Cabinet.2, Cabinet.3. You can explore these instances if you do not find the desired object the current receptacle.
** **fflection on History and #eedback**; Use interaction history and feedback from the environment to refine and improve your current plan.
If the last action is invalid, reflect the reason, such not adhering action rules missing preliminary actions, and adjust your plan accordingly.
(ACTION HISTORY ENVIRONMENT FEEDBACK (if available))
## Now the human instruction is: (TASK INSTRUCTION] You are supposed output in json. You need describe the current visual state from the image, output your reasoning steps, and plan. At the end, output the action id (0- (lenLSKILL SET) 1}) from the available actions to execute.
34

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Prompt for EB- 2B-Habitat
## You are a robot operating in home. Given a task, you must accomplish the task using a defined set of actions to achieve the desired outcome.
## Action Descriptions and Validity Rules
Navigation: Parameterized by the name of the receptacle to navigate to. So long the receptacle is present in the scene, this skill is always valid.
Pick: Parameterized by the name of the object to pick. Only valid if the robot is close to the object, not holding
another object, and the object is not inside closed receptacle.
Place: Parameterized by the name of the receptacle to place the object on. Only valid the robot close to
receptacle and is holding an object.
Open: Parameterized by the name of the receptacle to open. Only valid if the receptacle is closed and the robot is
close to the receptacle.
Close: Parameterized by the name of the receptacle to close. Only valid if the receptacle is open and the robot is close to the receptacle.
## The available action (0 66) and action names are: {SKILL SET}.
{ICL EXAMPLES]
## Guidelines
1. *Output Plan**: Avoid generating empty plan. Each plan should include more than 20 actions.
2. VVsibility**: If an object is not currently visible, use the "Navigation" action to locate it or its receptacle before attempting other operations.
** **Acioon V Make sure to match the action name and its corresponding action id in the output.
Avoid performing actions that do not meet the defined validity criteria.
4. ** Prevennt Repeating Action Sequences** Do repeatedly execute the same action sequence of actions.
Try to modify the action sequence because previous actions do not lead to success.
5. **Mltiple Instances There may be multiple instances of the same object, distinguished by an index following their names, c.g., cabinet 2, cabinet 3. You can explore these instances if you do not find the desired object in the current receptacle.
** **feeection on History and Feedback**; Use interaction history and feedback from the environment to refine and enhance your current strategies and actions. the last action is invalid, reflect on the reason, such as not adhering to action rules missing preliminary actions, and adjust your plan accordingly.
(ACTION HISTORY & ENVIRONMENT FEEDBACK (if available))
## Now the human instruction is: (TASK INSTRUCTION; You are supposed to output in json. You need to describe the current visual state from the image, output your reasoning steps, and plan. At the end, output the action id 69) from the available actions to execute.
35

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Prompt for -B-Vaiigttion at step
##f You are a robot operating in a home. You can do various tasks and output a sequence of actions to accomplish a given task with images of your status.
## The available action id (0 7) and action names are: action id O: Move forward by 0.25,
action id 1: Move backward by 0.25,
action id 2: Move rightward by 0.25,
action 3: Move leftward by 0.25, action Rotate
right by 90 degrees,
action 5: Rotate the left by 90 degrees,
action id 6: Tilt the camera upward by 30 degrees,
action id 7: Tilt the camera downward by 30 degrees
Strategy
Locate the Target Object Type: Clearly describe the spatial location of the target object from the ob- servation image (i.e. on the front left side, a few steps from the current standing point).
2. Navigate by *** Using Move forward and Move right/left the main strategy *** since any point can be reached through combination of those. When planning for movement, reason based target object's location and obstacles around you.
3. Focus on the primary goal: Only address invalid action when it blocks you from moving closer in the direction to target object. In other words, do not overly focus on correcting invalid actions when direct movement toward the target object can still bring you closer.
4. Use Rotation Sparingly *** only when you lose track of the target object and it's not in your view. If so, plan nothing but ONE ROTATION at step until that object appears in your view. After the target object appears, start navigation and avoid using rotation until you lose sight of the target again.
*** Do not complete the task too early until you can not move any closer to object, i.e. try to be as close as possible.
{ICL EXAMPLES}
## Now the human instruction is: (TASK INSTRUCTION]. To achieve the task, 1. Reason about the current visual state and your final goal, and 2. Reflect on the effect of previous actions. 3. Summarize how you learned from the Strategy and Examples provided.
Aim for about 2 actions in this step. !!!Notice: You cannot assess the situation until the whole plan in this planning step finished and executed, so plan accordingly.
At last, output the action id(s) (0 7) from the available actions to execute.
The input given to you is first-person view observation. Plan accordingly based on the visual observa- tion.
36

---

EMBODIEDBENCH: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Prompt for -B-vaiggttion at remaining steps
##f You are a robot operating in a home. You can do various tasks and output a sequence of actions to accomplish a given task with images of your status.
## The available action id (0 7) and action names are: action id O: Move forward by 0.25,
action id 1: Move backward by 0.25,
action id 2: Move rightward by 0.25,
action 3: Move leftward by 0.25,
action 4: Rotate
right by 90 degrees,
action 5: Rotate the left by 90 degrees,
action id 6: Tilt the camera upward by 30 degrees,
action id 7: Tilt the camera downward by 30 degrees
Strategy
Locate the Target Object Type: Clearly describe the spatial location of the target object from the ob- servation image (i.e. on the front left side, a few steps from the current standing point).
2. Navigate by *** Using Move forward and Move right/left the main strategy *** since any point can be reached through combination of those. When planning for movement, reason based target object's location and obstacles around you.
3. Focus on the primary goal: Only address invalid action when it blocks you from moving closer in the direction to target object. In other words, do not overly focus on correcting invalid actions when direct movement toward the target object can still bring you closer.
4. Use Rotation Sparingly *** only when you lose track of the target object and it's not in your view. If so, plan nothing but ONE ROTATION at step until that object appears in your view. After the target object appears, start navigation and avoid using rotation until you lose sight of the target again.
*** Do not complete task too early until you can not move any closer to the object, i.e. try to be as close as possible.
{ICL EXAMPLES]
## Now the human instruction is: (TASK INSTRUCTION]:
(ACTION HISTORY ENVIRONMENT FEEDBACK (if available))
To achieve the task, 1. Reason about the current visual state and your final goal, and 2. Reflect on the effect of previous actions. 3. Summarize how you learned from the Strategy and Examples provided.
Aim for about 5-6 actions in this step to be closer to the target object. !!!Notice: You cannot assess the situation until the whole plan in this planning step is finished and executed, so plan accordingly. At last, output the action id(s) (0 7) from the available actions to execute.
The input given to you is first-person view observation. Plan accordingly based on the visual observa- tion.
37

---

EMBODIEDBENCH, Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Prompt for 1a-Manppulation
## You are a Franka Panda robot with a parallel gripper. You can perform various tasks and output sequence of gripper actions to accomplish a given task with images of your status. The input space, output action space, and color space are defined as follows:
**
Input Space **
Each input object is represented as a 3D discrete position in the following format: [X, Y, Z].
There is red XYZ coordinate frame located in the top-left corner of the table. The X-Y plane is the table surface. The allowed range of X, Y, is [O, 100].
Objects are ordered by Y in ascending order.
**
Output Action Space **
Each output action is represented 7D discrete gripper action in the following format: [X, Y, Z, Roll, Pitch, Yaw, Gripper].
X. Y, Z are the 3D discrete positions of the gripper in the environment. It follows the same coordinate system as the input object coordinates.
The allowed range of X, Y, Z is [0, 100].
Roll, Pitch, and Yaw are the 3D discrete orientations of the gripper in the environment, represented as discrete
Euler Angles.
The allowed range of Roll, Pitch, and Yaw is [0, 120] and each unit represents 3 degrees.
Gripper state is 0 for close and I for open.
Color space **
Each object can be described using one of the colors below:
["red", "maroon", "lime", "green", "blue", "navy", "yellow", "cyan", "magenta", "silver", "gray", "olive", "purple", "teal", "azure", "violet", "rose", "black", "white"],
Below are some examples to guide you in completing the task.
[ICL EXAMPLES]
## Now you are supposed follow the above examples to generate sequence of discrete gripper ac- tions that completes the below human instruction.
Human Instruction: (TASK INSTRUCTION)
Input: (TASK INPUT)
Output gripper actions: {ACTION HISTORY & ENVIRONMENT FEEDBACK (if available)
38

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
1.2. Skill Sets
Below are the skill sets for EB-ALFRED and EB-Habitat. Note that the objects for EB-ALFRED vary depending on the scene, and the example provided here is illustrative. In contrast, the skill set for OB-Habitat remains static.
The skill sets (or action spaces) for EB-Navigation and EB-Manipulation are already included in the planner input prompt. For detailed prompts, please refer to Appendix I.I.
Action Type
Target Object
Find
AlarmClock, Apple, Apple_2, Apple_3, ArmChair, BasketBall, Bathtub, Bed, Book,
Box, Bread, Bread 2, ButterKnife, ButterKnife 2, Cabinet, Cabinet 10, Cabinet 2, Cabinet_3, Cabinet_4, Cabinet_5, Cabinet.6, 6, Cabinet_7, Cabinet_ 8, Cabinet Can- dle, Cart, CellPhone, CD, Chair, Cloth, CoffeeMaachnne, Table CounterTop, CounterTop 2, Counter'Top 3, CreditCard, Cup, Cup 2, Cup 3, Desk, DeskI ammp Dish- Sponge, DishSponge 2, DiningTable, Dresser, Drawer, Drawer_2, Drawer_3, Drawer-4, Drawer 5, Drawer 6, Egg, Faucet, FloorLamp, Fork, Fork 2, Fork 3, Fridge, Garbage- Can, Glassbottle, HandTTowel, Kettle, Kettle_2, _3, KeyChain, Knife, Knife_2, Ladle, Laptop, Lettuce, Lettuce 2, Microwave, Mug, Mug 2, Mug 3, Newspaper, Ot- toman, Pan, Pan.2, Pan_3, PepperShaker, PepperShaker.2 22 PepperShaker 33, Pencil, Pen, Pillow, Plate, Plunger, Potato, Potato RemoteControl. Safe, SaltShaker, Salt- Shaker_2, Shelf, SideTTable, Sink, SoapBar, SoapBottle, SoapBottle_2, 2, Sofa, Spatula, Spatula SprayBottle, Statue, StoveBurner StoveBurner StoveBurner Stove- Burner 4, TennisRacket, TissueBox, Tomato, Toilet, ToiletPaper, ToilerPaperHaanger, Vase, Watch, WateringCan, WineBottle
Pick up
AlarmClock, Apple, BaseballBat, BasketBall, Book, Bowl, Box, Bread, ButterKnife, Candle, CD, CellPhone, Cloth, CreditCard, Cup, DishSponge, Egg, Fork, Glassbottle, HandTowwl, Kettle, KeyChain, Knife, Ladle, Laptop. Lettuce, Mug. Newspaper, Pan, Pen, Pencil, PepperShaker, Plate. Plunger, Potato, RemoteControl, SaltShaker, SoapBar, SoapBottle, Spatula, SprayBottle Spoon, Statue, TennisRacket, TissueBox, Tomato, Vase, Watch, WateringCan, WineBottle
Put down
Object in hand
Drop
Object hand
Open
Box, Cabinet, Cabinet_10, Cabinet_2, Cabinet_3, Cabinet_4, Cabinet_5, Cabinet_6,
Cabinet 7, Cabinet 8, Cabinet 9, Drawer, Drawer 2, Drawer 3, Drawer 4, Drawer 5,
Drawer 6, Fridge, Laptop, Microwave, Safe
Close
Box, Cabinet, Cabinet. 10, Cabinet Cabinet.3, Cabinet 4, Cabinet.5, Cabinet.6 6, Cabinet 7, Cabinet 8, Cabinet 9, Drawer, Drawer Drawer Drawer 4, Drawer 5, Drawer_6, Fridge, Laptop, Microwave, Safe
Turn on
DeskLamp, Faucet, FloorLamp, Microwave
Turn off
Las Faucet, FloorLamp, Microwave
Slice
Apple, Bread, Lettuce, Potato, Tomato
Table 13. The skill set for EB-ALFRED
39

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Action Type
Target Object
Navigate to
Cabinet 4, Cabinet 5, Cabinet 6, Cabinet Chair 1, Left counter in the kitchen, Left drawer of the kitchen counter, Refrigerator, Refrigerator push point, Right counter in the kitchen, Right drawer of the kitchen counter, Sink in the kitchen, Sofa, Table 1, Table 2, TV stand
Pick up
Apple, Ball, Banana, Block, Book, Bowl, Box, Can, Clamp, Cleanser, Cup, Drill, Hammer, Knife, Lego, Lemon, Lid, Mug, Orange, Padlock, Peach, Pear, Plate, Plum, Rubik's s cube, Scissors, Screwdriver, Spatula, Spoon, Sponge, Strawberry, Toy airplane, Wrench
Place
Chair 1, Left counter in the kitchen, Left drawer of the kitchen counter, Refrigerator, Right counter in the kitchen, Right drawer of the kitchen counter, Sink in the kitchen, Sofa, Table 1, Table 2, TV stand
Open
Cabinet 4, Cabinet 5, Cabinet 6, Cabinet 7, Refrigerator
Close
Cabinet 4, Cabinet 5, Cabinet 6, Cabinet 7, Refrigerator
Table 14. The skill set for EB Habitat
40

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
1.3. In-context examples
The in-context examples provided the agent are detailed below. Each environment includes one to ten representative examples, with the complete set in our source code.
Specifically, EB-ALFRED utilizes 10 examples from the training set of the original ALFRED dataset for all subsets except the long-horizon subsets. We make sure the 10 examples diversely cover all task types of ALFRED, For the long-horizon subsets, we include different demonstrations closely related to the task types "pick heat then place in.iiinrrreee recep" and woock..col.thenppacc.. recep.. It's important to note that the in-context examples do not include scenarios with multiple instances of the same object. Additionally, we ensure the examples do not involve variable action indices (i.e., for actions with IDs below 162, object names remain static without suffixes, while actions with IDs 162 higher involve changing objects).
Similarly. EB-Habitat features 10 demonstrations from the 'train' set of the original Language Rearrangement dataset (Szot et al., 2023) across all subsets. For EB-Navigation, we provide 3 demonstrations for all subsets, ensuring they are distinct from the testing instances. In EB-Manipulation, the number of examples varies depending on task complexity: 2 examples for Pick & Place, examples for Stacking and Shape Sorting, and examples for Table Wiping, All demonstrations are carefully selected to differ from the testing instances.

---

EMBODIEDBENCH, Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
In-context examples for EB-ALFRED tasks excluding long horizon tasks
## Example 1:
Human instruction: Pick up the alarm clock and turn on the lamp. Output:
"Language plan": : "First, locate the alarm clock in the recn. Once found, carefully pick up the alarm clock. Next, identify the location of the desk lamp. After locating the desk lamp, proceed to turn on.", "executable_plan""
("action_id", 57, "action_nanc name": "find a AlarmClock" h ("action_id": 98, "action_name": "pick up the AlarmClock" b ["action_id" 63, "action periion_name:: "find Desklamp i"action_ia"; 151, nact;oo_namee; "turn on the DeskLamp"
## Example 2: Human instruction: Examine credit card by the light of a tall lamp. Output:
ponguage_plan"" "Start by locating the credit card in the room. Once you have found the credit card, proceed to pick it up. Then, search for the floor lmmp and find its location. After finding the floor lamp, turn it to illuminate the area so you can examine the credit card eftectively.", "executable_plan""
("action_id": 56, "action_name"" "find a CreditCard" h ("action_id"; 96, "action_name" "pick up the CraditCard" b, ("action_id"; 27, "action_name" "find a Flooriamp" }, ("action_ia" 153, "action name": "turn on the FloorLamp"
## Example 3: Human instruction: Use lamp light to look at the remote. Output:
"1 aanuuge_plan"" "First, locate the remote control in the room. Once the remote control is found, pick it up carefully. Next, search for the floor lamp and determine its location. After you have located the floor lamp, turn it provide sufficient light for examining the renote control.", "executable_plan""
("action_id"; 68, "action_name"" "find a RemoteContzol" },
["action_ia"; 116, "action name": "pick up the RemoteCentrol" ) ["action_ia"; 27, paction_names "gind a Flooriamp" ,, ("action_id"; 153, "action_name"" "turn on the Floorlamm" }
42

---

EMBODIEDBENCH, Comprchensive Benchmarking Multi-modal Large Language Models for Yision-Driven Embodied Agents
In-context example for EB-ALFRED long horizon tasks
## Example 1:
Human instruction: Place the microwaved slice apple on to the white table. Output:
"Language :lan:: "Locate an apple and pick it up, find dining table and put the apple down, look for knife and pick it up, find the apple and slice it. After slicing, need find place like the dining table put the knife down After that, find an apple slice and pick it up, search for a microwave and open it, put the apple in the microwave and close it, turn on the microwave to heat the apple slice, turn off the microwave and open it, pick up the microwaved apple slice, close the microwave, find the white table, and place the microwaved apple slice on white table.", "executable_plan" [
I"action_ia": 22, "action_name" "find a Apple" b ("action_ic": 107, "astion_name": "pick up the Apple" h ("action_ic": 18, "action_name": "find a Dininglable" h ("action_id": 133, "actrio_namm": "put down the object in hand" ) "'attio__ia: 34, "action__aaee "find a Knife" ,
["action_ia". 127, "action_ namen__amme:._. "pick the Knite" b
|"action_ia" 22, "action Bariiion_mmm___. "find Apple"
("action_ic": 160, "action_name": "slice the Apple" b,
("action_id"; 18, "action_name"; "find a DiningTable" h
("action_ic": 133, "action_namm" "put down the object in hand" )
("action_ic": 22, "action_name" "find a Apple" ), ("action_ic": 107, "action_name": "pick up the Apple" } ("action_id": 38, "action_name": "find a Microwave" }
("action_id": 143,
"actiio_name": "cpen the Microwave" h
["action_ia": 133,
"action_name": "put down the object in hand" )
''action_ia": 144,
"action_name": "close the Microwave"
"action_name": "turn on the Microwave" 1 ("action_id": 150,
|'action_ia": 149,
"action_name"" "turn off the Microwave" }, ("action_id": 143,
"action_name": "cpen the Microwave" h
("action_ic": 22,
"action_name": "find Apple" ) "action :ariion__amm": "pick up the Apple" b ("action_iat' ia": 144, "action :ame:::___mm:::___ "close the Microwave" b
"aation_ia", 107,
"aattio_id"" 18, "action :eriion_name:: "find a DiningTable" }
i"action_id"; 133, "action_name": "put down the object in hand"
43

---

EMBODIEDBENCH, Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driver Embodied Agents
In-context examples for EB-Habitat
## Example 1:
Human instruction: Put both a toy airplane and bowl onto the black table. Output:
"Language plan": : "To achieve the goal, the robot must locate the toy airplane and the bowl in the room and then move each item to the black table. There are two robot identifies table as the black table.
tables in the room, Therefore, the objective is to place both objects on table 1. The plan is as follows: first, navigate to the sofa, pick up the airplane, move to table 1, and place the airplane there. Then, proceed to table 2, where the bowl might be, pick up the bowl, return table 1, and "exeoutable_plan":
the bowl there.",
("action_ia"; 12, "action_name": "navigate to the sofa" b ["action_ia": 47, "action_name": "pick up the toy airplane" }
("action_id": 6, "action_name": "navigate to table 1" h ("action_ic"; 50, "action_name". "place
b (Maction_id": 7, "actrin_name": "navigate to table 2" b
table
["action 42, "action name": "pick up the bowl" b
("action ia": 6, "action name": "
"navigate to table 1" h
['action_ia" ia"; 50, "action name": "place table 1"
## Example 2:
Human instruction: I made mistake and left the fridge open. Can you assist by closing it?
Output:
"language plan": : "The objective for the rebot to close the refrigerator. To do so, the rebot first navigates to the refrigerator and then closes it.", "executable_plan" I ("action_iá"; 13, "amtion_name"; "navigate to the refrigerator" ) ("action_id"; 61, "ame;" "clese refrigerater" }
## Example 3: Human instruction: Displace the sponge from the left counter. Output:
"Ianguage_plan"" "To accomplish this task, the robot should navigate to the left counter to pick up the sponge. It should then move to another location, such table and place the sponge there.", "exeoutable_plan":
("action_id": 11, "action_name": "navigate to the left counter in the kitchen"),
("action_id" 28, "action_name"" "pick up the sponge" h
("action_id"; 7, "action_name": "navigate to table 2" b
ia"; 51, "action name": "place table 2
44

---

EMBODIEDBENCH. Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
In-context examples for Navigattion
## Example 1:
Human Instruction: Navigate to the floorlamp and be as close as possible.
Reasoning and reflection: I can see the floorlmap is left side in front of me, next the TVStand. Around me is Cabinet my left side. Following the guideline, since the floorlamp is left side, can approach it by series of moving forward and left rightward. Since currently there is an obstacle at my left, will first move forward twice.
Executable plans: [[O, Move forward by 0.25]. [O, Move forward by 0.25]]
Environment Feedback: Step executed successfully; Step 2 executed successfully;
Reasoning and reflection: can now get closer to the floorlamp. It is still located in front of the left. I will try move forward and move leftward in series.
Executable plans: [[O. Move forward by 0.25], [3, Move leftward by 0.25], [0, Move forward by 0.25), [3, Move leftward by 0.25], [0, Move forward by 0.25]]
Environment Feedback: Step is executed successfully; Step 4 is executed successfully; Step 5 is exe- cuted successfully; Step 6 executed successfully; Step is Invalid, as a Standardlskandeeight is blocking you from moving forward.
Reasoning and reflection: am lot closer to the floorlamp by a series forward and leftward movement. It is still in front of me. However, the last action is invalid, showing that am blocked from moving forward. That means there is an obstacle in front of me and can see is probably a small table. Following the strategy, can get away by first moving to the left and move forward.
Executable plans: [[3, Move leftward by 0.25], [3, Move leftward by 0.25], [0, Move forward by 0.25], [0, Move forward by 0.25]]
Environment Feedback: Step 8 is executed successfully; Step 9 is executed successfully; Step 10 is exe- cuted successfully; Step 11 is executed successfully;
Reasoning and reflection: can see that the floorlamp is at the right side of me, slight the front. I think it in effect consecutively moving leftward. Also, have got away with the obstable. Now need to move right and will probably be close the floorlamp.
Executable plans: [[2, Move rightward by 0.25), [2, Move rightward by 0.25]]]
Environment Feedback: Step 12 is executed successfully; Step 13 is executed successfully.
Reasoning and reflection: Now can see I am almost standing right next to the floorlamp. can complete the task.
Executable plans: [[8, Complete the current task]]
Task Success
45

---

EMBODIEDBENCH, Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
In-context examples for MM-nipullttion
## Example 1:
Human instruction: Pick up the star and place it into the yellow container.
Input: (' ('object 1': [45, 13, 18], 'object 2 I72, 20, 18], 'object 3 [50, 42, 171, 'ojject 4 (36, 42, 18), 'object 5 [70, 43, 17), 'object 6'
[50, 77, 171}
Output:
"visual state description"; "From left to right, I can see purple cylinder [45, 13, 181, blue moon at [72, 20, 18), a green star at [50, 42, 17), a purple cube at [36, 42, 18], yellow container at [70, 43, 17] and purple container argasoning_an__rfleeeiiii
[50, 77, 17].",
"The task is to pick up the star located [50, 42, 17] and place it into the yellow container located at [70, 43, 17). The plan involves moving the gripper to the star's position, closing the gripper to pick it up, reving to the yellow container's position, and opening the gripper to release the cube into the container.", "executable :
150, 42, 17, 6, 61, 36, 11,
150, 42, 17, 6, 61, 36, 01,
[47, 41, 26, 6, 61, 36, 0],
169, 39, 24, 88, 32, 77, 01,
[69, 39, 24, 87, 32, 77, 1]
## Example 2:
Human instruction: Stack the red star and the olive star sequence.
Input: {' o'oject 1': [42, 24, 19), 'object 68, 19), 'object 4': [39, 73, 18]]
[54, 53, 19), 'object 3 [65,
Output:
crisual_statc_deee detate_descripptonn: "From left to right, I can see a purple star at [42, 24, 19], an alive star at [54, 53, 19], a grey star at [65, 68, 19] and red star at [39, 73, 18].",
" aeeasonin__nndrreelleet..
need understand the instruction first. the objects in sequence, the zed star at 139, y3, 181 should be at the botton and the olive star at 154, 53, 19] should be placed on top. The plan involves moving the gripper to the olive star's position, closing the gripper to pick it up, meving to the top of the red star, and opening the gripper to release the olive star onto the red star.",
stack
"executable_plan""
[54, 51, 28, 3, 66, 37, 1],
[54, 54, 20, 3, 66, 37, 0], [54, 51, 30, 66, 37, 0], [39, 74, 29, 3, 66, 7, 01,
[39, 74, 26, 3, 66, 7, 1]
46

---

EMBODIEDBENCH:
Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
In-context examples for M--aippullaiinn
## Example 3:
Human instruction: Put the red star into the shape sorter.
Input: (' ('object 1': [65, 11, 17], 'object 2 [50, 38, 17], 'object 3 [40, 64, 371, 'object 4': [45, 83, 171}
Output:
"From left to right, I can see purple star at [65, 11, 171, a red star at [50, 38, 17], shape sorter at [40, 64, 37] and green star at [45, 83, 171.". Baaiinngnnnn___ee wawbsoninn_addreeleeettte the shape sorter first. I can see that the star hole is at the center of the shape sorter, which is at 140, 64, 371. Then, the task is to pick up the red star at 150, 38, 17] and put it into the star hole on the shape sorter. The plan involves moving the gripper to pick it up, moving upwards, moving to the top of the star hole, releasing the star into the hole.",
"
"I need identify the location of the
hole
the star's position, closing the gripper
"executable_plan"
[
[51, 38, 25, 119, 59, 26, 11,
[51, 28, 17, 119, 59, 26, 0], [51, 38, 27, 119, 59, 26, 0), [40, 65, 49, 0, 60, 45, 0], [40, 65, 41, 0, 60, 45, 1]
## Example 4:
Human instruction: Wipe the horizontal area.
Input: ('object 1': [33, 36, 19), 'object 2 [52, 38, 15], 'object 3': [59, 68, 15]}
Output:
"visual state caaatt__escription. "From left to right, I can see yellow sponge at 133, 36, 191, green rectangle at 159, 68, 15).", areasoning_anddreeeeetiiii
152, 38, 15] and orange rectangle
"I need to identify which area is horizontal. first. I can see that the orange rectangle area at [59, 68, 15] is horizontal since it is parallel to the green arrow representing the y-axis of the coordinate system annotated in the image. Then, the task is to pick up the sponge (33, 36, 19] and use to wipe the horizontal area at [53, 68, 15]. The plan involves moving the gripper to the sponge's position, closing the gripper to pick it up, moving to the side of the horizontal area, and moving the sponge along the main direction of the horizontal area to wipe it.", "executable planuttableppaa":: plan":
[32, 34, 25, 0, 60, 34, 1]
[32, 34, 17, 0, 60, 34, 0],
[32, 34, 27, 0, 60, 34, 0],
160, 80, 18, 61, 31, 01,
[61, 54, 17, 0, 61, 31, 01
47

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
1.4. Output JSON Schema Below the JSON schemas that guide the output structure of MLLMs.
Output JSON Schema for EB-ALFRED, EB-Habitat, and EB-Navigation
"type": "object",
"properties": secill_sttte__eeeseee
{
"type": "string", "description": "Description of current state from the visual image"
"type": "string",
"descripticn": "Summarize the histery of interactions and any available envi feedback. Additionally, provide reasoning as to why the last action or plan failed and did not finish the task."
Language plan":
"type": "string",
"description": "The list of acticns to achieve the user instruction. Each actien is started by the step number and the action name."
"executable_plan"
"type": "array",
"description": "A list of actions needed to achieve the user instruction, with each action having an action ID and name. Do not output an empty list."
"items":
"type": "object", "properties"; { "action_id"
"type": "integer",
"cescription": "The action ID to select from the availsble actions given by the prompt"
action_name"
"type": "string",
"écscription"; "The name of the action"
"required": ["action_id", "acticn_name")
irequrred"
grssual_stat__deeeee "reasoniingaad__eefles "language_plaa plan", "executable_plan"
48

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Yision-Driven Embodied Agents
Output JSON Schema for MB-Manppulation
"type": "object", "properties":
{
"type": "string",
"description"; "Describe the coler and shape of each object in the detection box in nurerical order in the image. 'hen provide the 30 coordinates of the objects chosen from input."
b,
"reasoning and reflection":
"type": "string",
"descripticn": "Reason about the everall plan that needs to be taken the target objects, and reflect on the previous actions taken if available." "
"lanquage_plan" {
"type": "string", "descripticn": list natural language actions achieve the instruction. Each language action is started by the step number and the language action name."
"executable_plan""
"type": "array",
"description"; "A list of discrete actions needed to achieve the user instruction, with each discrete action being 7-dimensional discrete actien.",
"items":
"type"; "object", "propertics";
Maction": {
"type": "string",
"description": "The 7-dimensional discrete action in the format list given by the prompt."
"required"; ["action"]
},
"required";
@visual_staae stateettttee...eeee staae_description.
"reasoning_and htt.riig_annreeeleetiiooo.. reflection",
"language_plan", "exccutable_plas"

---

EMBODIEDBENCH Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
J. Supplementary Case Studies of Successful Planning
In this section, we present successful planning examples for (laude-3.5-Sonnee VL2.512L...578B. GPT-4o, and Gemini- 1.5-pro across EB-ALFRED, EB- Habitat, EB-Navigation, and 1BB-Maniuulation. Refer Figures 18, 19, 20, and 21 for detailed reasoning and planning.
Human Instruction: slice a tomato, heat it up the microwave, and place it in the sink.
Invalid action
2. find a Knife
3. pick up the Knife
4, find a Tomato
5. slice the Tomato
6. find Sink
7. put down the object hand
8, find a Tomato
9. pick up the Tomato
10. find a Microwave
11. open Microwave
12, put down the object in hand
turn Microwave
15. open Microwave Microwave
18. pick up Tomato
19. close the Microwave
20. find Sink
21. put down the hand
Planning Step 1 (env step 1):
Visual state description: The image shows a kitchen environment with white cabinets stove. There are two copper-colored pots visible on the stovetop. The mounted above. floor
countertop appears
white, and there's beige brown color.
appears
Reasoning Reflection: Based the visible kitchen environment, we need locate necessary tools ingredients first. We'll then the microwave that's visible in the scene, and finally locate the sink for the final placement. The plan needs follow the sequence of first preparing the tomato, then heating it, and finally transferring
knife slice tomato,
its destination.
Planning Step 2 (env step env step 21):
image shows kitchen environment with white cabinets Visual state description: copper-colored pots visible the stovetop. The and stove. There
countertop appears
white, and there's microwave mounted above. The floor appears to be a beige brown color.
Reasoning Reflection: Based the visible kitchen environment, need execute sequence of actions to slice, heat, and move tomato. We'll need to first locate necessary tools like knife, then find tomato to slice it. After slicing, we'l 11 need to heat it in the visible microwave, and finally transfer it to a sink. From previous examples, I understand we need to put down tools like knives before picking up food items for safety.
Figure 18, Planning example of Claude-3 3,50 5-Sonnett in EB-AFRED.
50

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Human Instruction: Move the spoon the brown table, the sponge the brown table, and the cleanser the black table.
navigate left counter in the kitchen
pick spoon
navigate table
4. place
navigate stand
6, pick the sponge
7. navigate to the counter kitchen
8. pick up the sponge
9. navigate to the
10. place at the table
11. navigate to the TV stand
pick up
navigate pick left counter the kitchen
navigate table 1
16. place
Planning Step (env step env step 6)
Visual state description: The image shows a kitchen with wooden flooring, refrigerator, a sink, and several cabinets. There are no visible objects such as spoon, sponge, or cleanser in the immediate view.
The task requires moving spoon, sponge, and cleanser to specific tables. Since these objects
Reasoning Reflection:
visible the current view, the robot needs to navigate to locations where these objects might found. The robot will start by navigating to the left counter
kitchen to search for spoon.
Planning Step 2 (env step env step 12)
Visual state description: The image shows a room with blue carpet, can, and small object on the floor. There is cabinet and table visible in the scene.
Reasoning Reflection:
The robot has successfully moved the spoon the table (table stand was unsuccessful because
2). However, the attempt pick up the sponge from the sponge was not near the robot. robot needs to locate the sponge and the cleanser to complete the task.
Planning Step 3 (env step 13 env step 16):
Visual state description: The image shows a room with a dark blue floor and a black wall. There is
the floor and small object near it. The robot arm is visible in the upper right corner of the image.
Reasoning Reflection: The robot has successfully moved the spoon and sponge brown table (table 2). However, the last attempt pick up the cleanser from was unsuccessful because the cleanser was not near the robot. The robot needs to locate the cleanser before attempting to pick it up.
stand
Figure 19, Planning example in EB-Habitat for Inter VL2.5-78B.

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Human Instruction: navigate
Pot room as close possible it.
Move forward by 0.25
2. Move forward by 0.25
Move forward by 0.25
Move leftward by 0.25
Move forward 0.25
Move leftward by 0.25
7. Move forward by 0.25
& Move leftward by 0.25
9. Move forward by 0.25
10. Move forward by 0.25
11. Move leftward by 0.25
12. Rotate to the right degrees
forward by forward by leftward forward by
Planning Step (env step step 2):
The pot is located
countertop the front left of my position. approach it by moving forward and leftward. will start with moving forward towards the pot clear."
since the path ahead
Planning Step (env step env step 5):
moved forward successfully, which brought me closer to the pot. The pot is now more directly in front and slightly to the left. The approach has been effective so far, following the planned movements, without any obstacles blocking my path.
Planning Step (env step 6):
successfully moved closer
directly front the left countertop reached closest proximity possible. Continuing adjust my position
pot.
slightly leftward and forward will get close possible.
Planming Step (env step
step 8):
I've moved forward and left get closer pot, my last left edge or obstacle. I can still move forward to get
blocked
by the baseboard. This suggests I'm closer to the pot since it's in front.
Planning Step (env step env step 11):
managed to move forward but my sideward movements were blocked by the baseboard, indicating limitation this spot. Further approaches will more successful by moving forward and then attempting lateral moves align with the pot.
Planning Step (env step
step 16):
forward several times towards where the pot located. Despite attempts left get closer, these actions repeatedly blocked by obstacles. This indicates misalignment with the countertop edge preventing approach. Since I've lost sight of the pot, rotation is required.
Planning Step 7 (step 17):
After rotating, the pot appeared on the counter. I've moved forward and then left to approach it. I can now see it clearly in my current position, slightly ahead.
Figure 20. Planning example of GPT-4o in EB-Navigation.
52

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Human Instruction: Stack the front star on top of the right cylinder.
(47, 66, 60,
2. [47, 69, 18, D, 60, 0, 0]
3. [47, 69, 28, 0, 60, 0, 0]
4. [50, 89, 28, 0, 60, 0)
5. [50,, 89, 25, 0, 60, 0,
6. (33,65, 27, 60,
7. [33, 65, 19,0, 60, 0]
8. [33, 65, 29, O, 60, 0, (51, 90, 29, 0, 0, 10. [51, 90, 21, 0, 0.
Planning Step (env step 1 step 5): visual description: gray gear [33, 64, 19], yellow star [47, 69, 18], and navy cylinder at [5e, 89, 20].
left right,
a magenta cylinder [47, 31, 20],
stack the front star on top the right cylinder. The yellow star at [47, 69, 18] should be picked up and placed on the navy cylinder at [50, 89, 28]. The plan involves moving the gripper to the yellow star's position, closing the gripper to pick it up, moving to the top of the navy cylinder, and opening the gripper to release the yellow star onto the navy cylinder.
Reasoning Reflection: The instruction
Planning Step (env step env step 10):
Visual state description: From left to right, I can see a magenta cylinder at [47, 31, 20], a gray star at [33, 65, 19], yellow star [se, 88, 25], and blue cylinder at [51, 90, 28].
Reasoning & Reflection: The task is to stack the front star on top of the right cylinder. The front star is the gray star at [33, 65, 19], and the right cylinder is the blue cylinder at [51, 90, 20]. The plan involves moving the gripper to the gray star's position, closing the gripper to pick it up, moving to the top of the blue cylinder, and opening the gripper to release the gray star onto the blue cylinder.
Figure 21. Planning example Gemini-1.55 1.5-pr- Ma-Maanpullaiin.
53

---

EMBODIEDBENCH, Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
K. Supplementary Case Studies of Unsuccessful Planning
also present failure cases across different tasks, highlighting key challenges in perception, reasoning, and planning, In EB-ALFRED, we showcase planning error where the model fails to generate an effective executable plan accomplish the task. In EB-Manipulation. execution. In EB- -B-aaiigttion, we illustrate reasoning error where the model struggles to interpret spatial relationships. These examples provide insights into the limitations of current models and highlight areas for improvement in object recognition, planning, and spatial reasoning. Refer to Figures 22, 23, and 24 for details.
analyze perception error where the agent misidentifies key objects, leading to incorrect
Human Instruction: move two books From the bed to the desk.
find Bed
2. find Book
3. pick the Book
find Desk
S. put down the object
find Bed
7.d Book
8. pick up the Book
9.find Dask
10. put down the
11. empty plan
Planning Step (env step
step 10):
Visual state description: The image shows room corner with red trash bin, light switch on the wall, and a closed black door. The floor is carpeted, and there is a small part of a table visible on the left.
Reasoning Reflection: The task is to move two books from the bed the desk. However, Therefore, first step is will involve finding the bed, picking up the books,
the current inage does
bed, books, or
locate these objects, The finding the desk, and placing the books it.
Language Plan: First, locate the bed in the roos. Once the bed is found, identify and pick up the first book. Then, locate the desk and place the book on it. Repeat the process for the second book by returning the bed, picking up the second book, and placing the desk.
Executable
("action id":32,
"action "action_name". name": "find Bed"),
("action id":54,
"action samoon_name=: "Find Book"),
("action id"::4, pection_name" "find Desk"),
"pick up the Book"),
("action ""action_id":49, 1d":49,
("action_id" id":133,
"action_nanee "put down the object in hand"),
("action socttion_d":32,,
Daccion_name" "find a Bed"),
("action "aatioo_idd:54 i5"
"action_name "find Book"),
("action 19":94,
"action pection_name": "pick up Book"),
("action id":49,
"action name": "find Desk"),
("action_id" 11":
"action_nane" : "put down the object in hand")
Planning Step 2 (env step 11):
Visual description: The image shows black desk with 1amp, mug, pen, keychain, book titled 'Clean Code', and three tissue boxes. There is chair next to the desk.
Reasoning & Reflection: The task was to move two books from the bed to the desk. The previous actions successfully moved two books to the desk. The task is complete as per the instruction, and no further actions are needed.
Language Plan: successfully. further actions required.
of noving books from the bed the desk has been completed
Figure 22. Planning Error Example in EB-ALFRED: The agent was supposed locate "Book.2" by the 7th action but instead continued interacting with the first book.
54

---

EMBODIEDBENCH: Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Human Instruction: Pick up the star and place it into the blue container.
55, O, 90,
2. [34, 55, 60, 90,
55, 27, 0, 90.0
4. [68, 27, 60, 90,
[68, 17, 27,0, 60, 90,
(26, 16, o, 60, 90,1]
7. [26, 16, 17, 60, 90, 0)
[26, 16, 27, O, 90, 0]
9. (67, 27, 60, 90, 0]
10. (67, 18, 27, 0, 60, 90,
18, 17,0, [66,
[66, 60, 0) 14. 126, 14, 27, 0, 60, 90, 15. (26, 27, 0, 60, 90,
Planning Step (env step env step 5)
Visual state description: From left right, can see magenta cylinder at [26, 16, 18], blue container [68, 17, 17], green star at [34, 55, 17], an olive container [57, 63, 17).
Reasoning Reflection:
task is to pick up the star located at [34, 55, 17] and place into the blue container located at [68, 17, 17]. The plan involves moving the gripper to the star's position,, closing the gripper pick it up, moving to the blue container's position, and opening the gripper to release the star into the container.
Planning
(env step
step 10)
Visual state description: From left right, 17], blue container at [67, 17], and green container at [57, 63, 17].
pink star [26, 16,
task is pick up the star located at [26, 16, 17] and place into the blue container located [67, 18, 17]. The plan involves moving the gripper moving the blue container's position, and opening gripper release the star into the container.
Reasoning Reflection:
star's position, closing the gripper pick up,
Planning Step 3 (env step 11 env step 15):
Visual state description: From left to right, I can see blue container at [26, 14, 17], purple star at [66, 18, 17], and green container at [57, 63, 17].
The task is to pick up the star located at [66, 18, 17] and place it into the blue container located at [26, 14, 17]. The plan involves moving the gripper to the star's position, closing the gripper to pick it
Reasoning & Reflection:
moving to the blue container's position, and opening the gripper to release the star into the container.
Figure 23. Perception Error Example in FB-Manipulation: the agent erroneously observed color of ohject.
55

---

EMBODIEDBENCH. Comprchensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Human Instruction: navigate to Pot room close possible to
Move
Move for 0.25
leftward
for 0.25
for 0.25
for 0.25
for 0.25 Invalid
for 0.25 Invalid
forward
leftward 0.25
Move leftward
Move 0.25
12. Tilt the camera downward dogrees
for 0.25 Invalid
0.25
upward by 30 degrees
Invalid
Invalid
Invalidd
13. Move backward
Move rightward for 0.25
17.
for 0.25
right by 90 degrees
0.25
left by 9D degrees
left by 90 degrees
19. Move forward 0.25
20. Move forward for 0.25
Planning Step (env step 1 env step 2):
My goal is to locate the pot this kitchen. Given the current view, there's no pot visible directly. I should move forward to explore more of my surroundings, as moving forward will help me view more of countertop and other areas not fully visible right now.
Planning Step (env step 6):
The previous action sequence moving forward eventually led invalid due to being blocked by the countertop. This indicates proximity the target. The is front bit the left, requiring leftward and forward movements to closer.
Planning Step (env step
step 20):
The pot visible now, slightly blocked by the counter. In order to closer, should navigate around obstacles and
right. Previous attempts resulted being
path aligns effectively towards target, adjusting the direction needed avoid obstructions.
Figure 24. Reasoning Error Example in EB- -B-vavigttion: the agent recognized it blocked by the countertop but failed attempt navigating around it.
56


Reviews:
[{"review_id": "I45ynAnpY1", "reviewer": "Reviewer_yZa7", "summary": "The authors present EmbodiedBench - a set of diverse tasks and environment to evaluate MLLMs for embodied agents. They present high-level benchmark environments - EB-Habitat, EB-ALFRED and  low-level benchmark environments - EB-Navigation and EB-Manipulation. The tasks are also divided into basic task solving, common sense, complex instructions, spatial awareness, visual perception and long-term planning.\n\nThey evaluate 13 different MLLMs on this benchmark, showing that high-level tasks are easier than lower-level tasks, and that visual cues are more important for low-level tasks than higher-level tasks. They perform ablations showing image resolution should be reasonable, multi-step images harm the MLLMs, and visual in-context learning helps. They also perform error analysis showing the reasons for failures in specific aspects of robotics (planning, perception, reasoning).", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.\n\n**Q1: No real-world evaluations. It is hard to say how aligned with the real-world performance**\n\n**A1:** We agree with the reviewer regarding the gap between simulated and real-world tasks. While real-world evaluation is crucial for assessing model performance in practical scenarios, there is a trade-off between reproducibility, cost, safety, and real-world applicability. Simulated benchmarks offer a standardized, easily reproducible environment, reducing the time, financial cost, and safety risks for researchers to replicate results [1,2]. **In our revision, we will include a discussion about this limitation in the main paper.**\n    \n[1] Evaluating Real-World Robot Manipulation Policies in Simulation. CoRL 2024.\n\n[2] Visualagentbench: Towards large multimodal models as visual foundation agents. ICLR 2025.\n    \n**Q2: How to prevent failure of plans? Any qualitative examples?** \n\n**A2:** In EmbodiedBench, we allow MLLM agents to generate multiple actions at once. If a failure occurs (e.g., invalid actions or unmet goals), they **replan** using the latest image and interaction history. **In Appendix F, we provided four qualitative examples of our MLLM agents (Figures 13–16)**, demonstrating their ability to replan effectively.\n    \n**Q3: Some results are repeated and can be (optionally) removed from appendix**\n\n**A3:** In Appendix D, we opt to include both the results from the main paper and additional results to provide a clearer trend across different tasks. \n\n**Q4: Using panoramic images for multi-view**\n\n**A4:** We included a panoramic image in our multi-view ablation for EB-Navigation. It provides a top-down perspective, capturing the entire scene. However, we found that it can mislead the agent, negatively impacting overall performance. Examples of the multi-view setup are shown in Figure 1 of https://anonymous.4open.science/r/rebuttal-3568/rebuttal_file.pdf.\n\n**Q5: Train policies on this benchmark and show performance ... this is not a necessity since the purpose of the benchmark is to evaluate MLLMs.**\n\n**A5:** We agree that further fine-tuning is a promising direction, but a key challenge is the lack of training data. Currently, the only available embodied planning dataset in our setting is ALFRED, which lacks the structured perception and reasoning. To address this, we are collecting trajectories using our agent framework on the base subset while reserving other subsets for evaluation. This will benefit future fine-tuning MLLM research, and we aim to release the dataset soon.\n    \n**Q6: Difference between the complex instruction understanding and long-horizon tasks. Any intersection?**\n\n**A6:** The two subsets target different challenges:  \n- \"Complex Instruction\" **adds longer, relevant/irrelevant context**, making user instructions harder to interpret while keeping task complexity similar to the base subset.  \n- \"Long Horizon\" **increases task difficulty** by requiring more steps to complete.  \n\nIn EB-ALFRED, GPT-4o takes an average of 13.4 steps (base), 14.2 steps (complex instruction), and 23.9 steps (long-horizon), showing their differences. **We ensure no subset overlap in our benchmark design**.\n\n**Q7: How is the data generated for EB-Navigation? Which Python program was used? I did not find this in appendix.** \n \n**A7:** We describe the data generation process in **Appendix B.3** but will clarify it further in our revision. The EB-Navigation dataset consists of: 1. scene and object information, 2. initial robot position and pose, 3. target object information, 4. language instruction. We create the dataset using a Python program that ensures the validity of the (1,2,3,4) combinations. The process follows these steps:\n- Step 1: Scene and Object Initialization  \nWe use 90 scenes from AI2-THOR-supported scenes. Each scene initializes a set of objects.\n- Step 2: Target Object Selection  \nIn each scene, we iterate over potential target objects, excluding those inside receptacles or with multiple instances to reduce ambiguity.  \n- Step 3: Agent Position and Pose Determination  \nFor each target object, we use AI2-THOR’s `GetInteractablePoses` to randomly sample a valid agent position, ensuring a distance at least 2.5 meters from the target. The agent's pose is set to either include the target object in view (e.g., base subset) or keep it out of view (long horizon).  \n- Step 4: Instruction Generation and Augmentation  \nBased on predefined templates (e.g., \"Move towards the {target object} and stay near it\"), we apply GPT-4o to augment linguistic diversity and preserve subset requirements.  \n\nAfter executing the above data generation, we select 60 tasks for each subset to form the EB-Navigation dataset.\n\n**Q8: Typos**\n\n**A8:** Thank you for pointing out these typos. We have carefully corrected them in our revised manuscript.", "claims_and_evidence": "1. **EmbodiedBench is a useful tool for evaluating MLLMs in embodied settings**: The results are very useful and they help use understand the performance of difference LLMs in different settings and types of tasks. The insights are interesting and would help further research on this domain. One specific issue that I find with this approach is that all data is in simulation, and there are no real-world episodes, or evaluations. It is hard to say how aligned with the real-world performance are the results.\n2. **MLLMs excel at high-level, struggle at low-level tasks**: This claim is shown by the performance on EB-ALFRED and EB-Habitat vs EB-Navigation and EB-Manipulation. I believe this is an interesting insight.\n3. **Visual cues are necessary for low-level tasks:** This makes sense, since precision is key in such tasks. It is interesting that the authors empirically find this for MLLMs using their benchmark.", "methods_and_evaluation_criteria": "1. They create a benchmark over different environments and tasks to evaluate MLLMs for embodied AI. I think this is a sound approach and casts a wide-net for MLLM evaluation.\n2. They describe the data creation in detail, and it is well-grounded in prior works. The task subsets are well thought.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Agent Design: The agent design uses single-step images for efficiency and provide valid skill sets for each task to the MLLM. For manipulation, they provide markers and boxes. I think this is a fair design, considering MLLMs struggle with multi-step input.\n- Task Planning: This design is particularly interesting to me. Instead of doing per-step planning, they plan multiple steps in a single go and let the MLLM decide the number of steps. How do the authors prevent failure of plans? Are there any qualitative examples of this? It would be nice to discuss this if possible.\n- Ablations: I think the ablations are also careful and evaluate several aspects of the approach. The results show importance of feedback, in-context learning, camera resolution, detection boxes (for manipulation), multi-step input, visual in-context learning.\n- Error analysis: They show interesting error analysis and show subtypes of error, along with discussion on why different LLMs might fail on different subtypes.", "relation_to_broader_scientific_literature": "The paper presents an interesting benchmark to evaluate MLLMs for Embodied AI agents and is one of the first ones to do so. The insights are pretty interesting and the evaluation/analysis is comprehensive. I think this will help understand MLLMs better for robotics, especially since it is a up-and-coming field.", "essential_references_not_discussed": "Can not think of anything that might be missing. They consider a good amount of related benchmarks and show a comparison table.", "other_strengths_and_weaknesses": "The paper is pretty interesting overall, has nice insights and analysis. It adds value to the community, and might be helpful in evaluating future MLLMs. It would be nice to have some real-world evaluations, or something that shows that the benchmark performance translates.", "other_comments_or_suggestions": "- Line 215: Incorrect quotes around \"navigate\"\n- Line 257, Col 2: \"constained\"-> contained or constrained?\n- Line 306, Col 2: Incorrect quotes around \"Lang\"\n- Line 386: Mentions \"multi-view\", which is in ablation. Have the authors tried using panoramic images from multi-view?\n- Is it possible to train some e2e or modular policies on this benchmark and show how they perform on this benchmark? Might be interesting to compare the best MLLM with such policies. I agree that this is not a necessity since the purpose of the benchmark is to evaluate MLLMs.\n- Section 6 title should be: \"Conclusion\"", "questions_for_authors": "1. How is the complex instruction understanding task different from long-horizon? Is there an intersection?\n2. How is the data generated for EB-Navigation? Which Python program what used? I did not find this in appendix.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HjtducCcNp", "reviewer": "Reviewer_k7jC", "summary": "This paper proposes embodiedbench, a benchmark for evaluating MLLMs' capability in a diverse set of embodied tasks. Specifically, the tasks range from high-level semantic tasks to low-level tasks with atomic actions. Furthermore, the tasks under different simulators are classified into different subsets, to evaluate agents' capability in common sense reasoning, complex instruction following, spatial awareness, visual perception and long-term planning. This paper benchmarks the performance of multiple open-source MLLMs and close-source MLLMs for embodied tasks, providing interesting insights in how MLLMs perform at high-level tasks and low-level manipulation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.\n\n\n**Q1: MLLMs performance might relate to the prompt being used, and one single prompt is used for all the models. Exploring how MLLMs work with a different set of prompts might bring more insights and robust conclusion about how well different MLLM work for embodied tasks. Some ablations for the language prompt provided in Sec.5.3 can partially mitigate this issue.**\n\n\n**A1:** We agree that prompt design plays an important role in MLLM agent performance. In Section 5.3, we analyzed the effects of textual environmental feedback and the number of in-context examples. To further investigate prompt robustness, we conducted additional studies on the EB-ALFRED \"Base\" subset, evaluating the following variations:\n\n1. **\"No Guideline\"** – Removing instructional guidelines intended to assist model generation (see Page 19).\n2. **\"Prompt v2\"** – Rewriting the original prompts using GPT-4o to rephrase and restructure while preserving similar information.\n\nThe results show that removing guidelines has no effect on model performance, while \"Prompt v2\" causes a slight 4% performance drop. **This suggests that our MLLM agent is relatively robust to prompt modifications** compared to its sensitivity to environmental feedback (around 10% drop) and in-context examples (more than 20% drop).\n\n\n|  | GPT-4o  |GPT-4o (no guideline)   |  GPT-4o (prompt v2)  | Claude-3.5-Sonnet  | Claude-3.5-Sonnet (no guideline)  |  Claude-3.5-Sonnet (prompt v2) |\n| -------- | -------- | -------- |-------- | -------- |-------- | -------- |\n|  EB-ALFRED (Base)   |   64  |  64  | 60 | 72 |   72  |68 |\n\n\nFurthermore, we also examined the impact of reasoning in in-context examples (i.e., ReAct prompting [1]). Removing this reasoning step leads to an 8% performance drop for GPT-4o and a 4% drop for Claude-3.5-Sonnet, indicating that reasoning within in-context examples is more impactful than prompt rephrasing or guidelines, though still secondary to environmental feedback and in-context examples.\n\n|  | GPT-4o  |GPT-4o (w/o ReAct)   |   Claude-3.5-Sonnet  | Claude-3.5-Sonnet (w/o ReAct)   |\n| -------- | -------- | -------- |-------- | -------- |\n|  EB-ALFRED (Base)   |   64  |  56  | 72 |  68  |\n\nIn summary, our MLLM agent framework is robust to prompt rephrasing and guideline removal. However, environmental feedback and in-context examples, as discussed in Section 5.3, have a much greater impact on performance.\n\n[1] React: Synergizing reasoning and acting in language models. ICLR, 2023.", "claims_and_evidence": "Strengths:\n1. This paper proposes a benchmark for evaluating MLLM's capability in diverse set of embodied tasks, covering a wider range of tasks compared with previous research. \n2. The design of including both high-level semantics and low-level tasks are essential, which is also supported with the large performance difference observed for MLLMs. \n3. Further classifying the agents' capability into different categories can bring more insights while doing evaluation.", "methods_and_evaluation_criteria": "Strengths:\n1. Benchmarking both open-source MLLMs and close-source MLLMs on the proposed embodiedbench, which serves as a good start point for futher work. \n\nWeakness:\n1. MLLMs performance might relate to the prompt being used, and one single prompt is used for all the models. Exploring how MLLMs work with a different set of prompts might bring more insights and robust conclusion about how well different MLLM work for embodied tasks. Some ablations for the language prompt provided in Sec.5.3 can partially mitigate this issue.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Strengths:\n1. Detailed ablations for both language-centric analysis and vision-centric analysis.\n2. Interesting findings indicating that vision is crucial for embodied tasks with low-level actions.", "relation_to_broader_scientific_literature": "The proposed benchmark can be very crucial for future development of MLLMs for embodied AI tasks, especially in low-level manipulation category.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8UfdIowf6a", "reviewer": "Reviewer_R7Zw", "summary": "This paper introduces EmbodiedBench, a comprehensive benchmark for evaluating vision-driven embodied agents based on multi-modal large language models (MLLMs). The benchmark features 1,128 testing instances across four environments, covering both high-level semantic tasks and low-level atomic actions, with six meticulously curated subsets evaluating essential agent capabilities such as common sense reasoning, complex instruction following, spatial awareness, visual perception, and long-term planning. Through extensive experiments, the authors evaluate 13 leading proprietary and open-source MLLMs within EmbodiedBench, revealing that MLLMs excel at high-level tasks but struggle with low-level manipulation, with the best model, GPT-4o, scoring only 28.9% on average.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.\n\n**Q1: The absence of evaluation in real-world physical environments ... Embodied intelligence is largely meant to operate in the real world, and I would suggest that the authors add some real-world experiments, or some discussion.**\n\n**A1:** We agree with the reviewer on the importance of real-world evaluation. However, there is an inherent trade-off between reproducibility, cost, safety, and real-world applicability. While real-world testing is crucial for practical deployment, simulated benchmarks provide a standardized and easily reproducible environment, reducing the time, financial burden, and safety risks associated with real-world evaluation [1,2]. EmbodiedBench is a step forward in enabling evaluating MLLM agents in diverse simulated embodied tasks. Future research could benefit from more realistic and complex embodied simulations [3] or standardized and cost-effective real-world test suites [4,5]. **In our revision, we will add a discussion on this limitation at the end of the main paper.** \n    \n[1] Evaluating Real-World Robot Manipulation Policies in Simulation. CoRL 2024.\n\n[2] Visualagentbench: Towards large multimodal models as visual foundation agents. ICLR 2025.\n\n[3] Behavior-1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. ArXiv, 2024\n\n[4] Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. RSS 2023.\n\n[5] Mobile aloha: Learning bimanual mobile manipulation using low-cost whole-body teleoperation. CORL 2024. \n\n**Q2: The review includes a very large number of MLLMs, but the VLA model is missing. I suggest adding some experiments.**\n\n**A2:** We appreciate the reviewer’s suggestion. **We reviewed VLAs in paragraph 3, Appendix A**. While VLA models such as Octo, OpenVLA, and RDT-1B have shown strong performance in manipulation tasks, there is currently no open-source VLA model that can handle both high-level household tasks and low-level navigation & manipulation simultaneously. Consequently, there is no directly comparable VLA model for our evaluation.\n\nHowever, to explore their capabilities, we conducted experiments on three pretrained VLA models (Octo, OpenVLA, and RDT-1B) using the EB-Manipulation (Base) subset. All models achieved a **0% success rate**. This outcome can be attributed to the distribution shift: existing robotic foundation models [6][7] are primarily trained on real-world datasets such as Open X-Embodiment. The large domain gap between these datasets and our simulator environment prevents these models from generalizing effectively without fine-tuning. This reinforces the need for a more general framework, as proposed in our paper, that enables adaptation to diverse tasks without fine-tuning.  \n\nWhile a few VLA models [8][9] have been developed for the same simulator we use, they are not publicly available. Furthermore, other available models [10][11] have input formats that differ significantly from our environment (e.g., proprioception mismatch), making direct evaluation infeasible. The only available VLA model trained in the same simulator as ours is 6D-CLIPort [12], which processes multi-view observations and language instructions to generate 6-DoF actions. Below are the evaluation results on EB-Manipulation:\n\n| EB-Manipulation | Base  | Common |   Complex  | Spatial   | Visual | Avg |\n| - | - | - |- | - | - |- |\n|  6D-CLIPort   |   8.3  |  6.3  | 2.1 |  16.7  | 16.7 | 10.0 |\n\nThe model performs well on tasks that rely on visual understanding and spatial reasoning. However, its success rate drops significantly on tasks requiring common sense reasoning or understanding complex instructions, suggesting that action-grounded VLA models may struggle in these areas. In contrast, our MLLM-based agent framework achieves higher performance, with GPT-4o reaching an average score of 28.9. This result highlights the effectiveness of our MLLM agent framework and the potential of general MLLMs as embodied agents.\n    \n[6] OpenVLA: An Open-Source Vision-Language-Action Model. CORL 2024.\n\n[7] RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation, ICLR 2025.\n\n[8] MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation. Arxiv 2025.\n\n[9] HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation, ICLR 2025.\n\n[10] RVT-2: Learning Precise Manipulation from Few Examples. RSS 2024.\n\n[11] SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation. Arxiv 2025.\n\n[12] VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation, NeurIPS 2022.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper significantly advances the evaluation of multimodal macromodels", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n1. EmbodiedBench is the first benchmark to comprehensively evaluate MLLM-based embodied agents across multiple environments and task levels, providing a standardized platform for comparison.\n\n2. The benchmark includes six capability-oriented subsets that allow for detailed analysis of different agent capabilities, offering valuable insights into model limitations.\n\n3. The authors develop a unified agent framework that effectively integrates egocentric visual perception, few-shot in-context examples, interaction history, and environment feedback for decision-making.\n\n4.  The paper presents thorough experiments with 13 state-of-the-art MLLMs, providing valuable insights into their performance on various embodied tasks.\n\nWeaknesses:\n\n1.  The absence of evaluation in real-world physical environments. The entire benchmark is implemented in a virtual environment, which raises the question of whether the results of the virtual benchmark measurements reflect the capabilities of the model in the real world. Embodied intelligence is largely meant to operate in the real world, and I would suggest that the authors add some real-world experiments, or some discussion.\n\n2. The review includes a very large number of MLLMs, but the VLA (Vision-Language-Action) model is missing. I suggest adding some experiments.", "other_comments_or_suggestions": "See Weaknesses", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "10EGbaUqkW", "reviewer": "Reviewer_MUiH", "summary": "The paper proposes a powerful and comprehensive benchmark, EmbodiedBench, for both high-level and low-level actions in embodied intelligence. It consists of four distinct subdatasets, ranging from high-level semantic tasks to low-level metric tasks, with each subdataset having its own focus. To build the entire benchmark, the authors first collect results from other datasets and utilize a simulator for data generation. They then correct and refine the limitations of the original datasets. Moreover, the tasks are categorized into six types, covering common and challenging embodied intelligence tasks. Additionally, evaluations conducted on both proprietary and open-source models demonstrate that this dataset presents significant challenges.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions. \n\nThe anounymous link for figures is https://anonymous.4open.science/r/rebuttal-3568/rebuttal_file.pdf. We use \"4o\" to refer to GPT-4o and \"Claude\" to refer to Claude-3.5-Sonnet.\n\n**Q1: Detection box in EB-Manipulation. (1) How are they provided to MLLMs? (2) Do color or thickness impact performance? (3) Why does performance drop in EB-Navigation?**\n\n**A1:** (1) Detection boxes are directly drawn on images to guide the model’s focus on relevant regions. An example is on the right side of **Figure 1 in the anonymous link**.\n\n(2) Our additional ablation studies on EB-Manipulation base subset show:\n- All detection boxes improve performance over no box.\n- Box color has minimal impact; even the color similar to desk (yellow) only causes a slight drop (4.2%).\n- Increasing thickness (1px->2 or 3px) slightly reduces performance (<6.2%).\n\n|  | Default (red, 1px) | 2px  | 3px  |  black  | yellow  | no Box |\n| - | - | - | - |- | - | - |\n|  Claude | 37.5 | 31.3 | 33.3 | 37.5 | 33.3 | 29.2  |\n\n(3) **Figure 2 in the anonymous link** shows that in EB-Navigation, multiple detection boxes can obscure distant objects, reducing visibility. In contrast, EB-Manipulation keeps objects at a fixed distance, ensuring clear visibility. As a result, detection boxes affect the two environments differently.\n\nIn EB-Navigation, we tested using **a single box only on the target object** that can reduce obstruction, showing consistently improved accuracy. To better reflect real-world scenarios, EB-Navigation omits detection boxes by default, requiring the MLLM agent to detect and recognize objects. \n\n| Model | No Box | One Box | Multi Box |\n|-|-|-|  -|\n| 4o | 61.7 | 68.3 | 53.3  |\n| Claude | 46.7 | 58.3 | 48.3  |\n \n**Q2: Differences between high-level and low-level trajectories. Do they vary in difficulty or instructions?**\n\n**A2:** \"High-level\" and \"low-level\" refer to different action representations based on their executability in robotic systems (see Section 3, Paragraph 1). **The trajectory structure and instructions are consistent across all high-level and low-level tasks; the key difference lies in how actions are represented—either as high-level abstractions or low-level primitives.**  \n\nIn terms of difficulty, our results show that low-level tasks are more challenging for MLLM agents. This is because they require stronger perception and spatial awareness, which remain limitations for current MLLMs.\n\n**Q3: How effective is ChatGPT in data generation? How is accuracy verified? Is there a human review process? How do we ensure the accuracy of the generated data?**\n\n**A3**: **Our dataset preparation combines GPT-4o and human annotation to ensure high quality**. GPT-4o is used not for full data generation but to enhance linguage instruction diversity.\n\nFor example, in EB-ALFRED, task descriptions (PDDL) and instructions for the base subset are sampled from ALFRED. For other subsets (e.g., \"Common Sense\"), we craft 10 examples to guide GPT-4o in generating augmented instructions. To ensure accuracy, we manually review all instructions for correctness and coherence with PDDL descriptions, revising or discarding invalid data. This human-in-the-loop approach ensures dataset reliability.\n\n**Q4: How accurate is the multi-step planner? An ablation study comparing it with single-step execution.**\n\n**A4:** The multi-step planner is crucial for improving performance while reducing API/inference costs. To assess its impact, we compared multi-step and single-step execution. Results show significant performance drops with single-step execution on EB-ALFRED base subset. These results confirm the importance of multi-step planning.\n    \n|  | Default | Single Step |  \n|-|-|-|  \n| 4o   | 64 | 36 |  \n| Claude    | 72 | 62 |  \n\n**Q5: Evaluation of converting multi-step images into text as in-context information.**\n\n**A5:** We tested incorporating multi-step observation descriptions into the context. While this method did not improve GPT-4o’s performance, it led to a 4% gain for Claude-3.5 on EB-ALFRED (Base). We plan to offer this as an optional feature in our code release.\n    \n|  | Default | w/ Image Descriptions |  \n|-|-|-|  \n| 4o | 64 | 64 |  \n| Claude | 72 | 76 |  \n\n**Q6: Limitations of simulation in real-world scenarios.**\n\n**A6:** We acknowledge the limitations of simulations in capturing real-world challenges. Please refer to **Q1 of Reviewer yZa7** for a detailed discussion. We will add a discussion on the limitations in the revision.\n\n**Q7: Evaluation of qwen2.5-VL models.**\n\n**A7:** We evaluated Qwen2.5-VL models on EmbodiedBench and observed notable improvements over Qwen2-VL. Qwen2.5-VL-72B achieves an overall score of 34.7, surpassing previous open-source SOTA, InternVL2.5-78B (33.9). We will include evaluations of more recently released MLLMs in our updated manuscript.", "claims_and_evidence": "None.", "methods_and_evaluation_criteria": "1. In EB-Manipulation, I am a bit confused about the necessity of providing the detection box to the MLLM. (1) In what manner is the box provided to the MLLM? Is it drawn directly on the image, and have other forms been considered? (2) Does the color of the box or the thickness of the box's outline impact the enhancement effect? (3) Why does adding the box lead to a decrease in performance for the navigation subtask? I think it would be helpful to analyze this further in relation to the task's inputs and outputs. I saw some discussion about the detection box in the supplementary materials, but it did not fully resolve my confusion.\n2. Please explain the differences between high-level and low-level trajectories. Are there differences in difficulty or the way instructions are expressed?\n3. What is the effectiveness of using ChatGPT directly to generate data? How can accuracy be verified? Is there a human review process involved? If I believe that ChatGPT may have difficulty understanding spatial concepts (as referenced in the final evaluation table), how can we ensure the accuracy of the data it generates?", "theoretical_claims": "None.", "experimental_designs_or_analyses": "1. In Task Planner, how accurate is it for the agent to execute multiple steps at once? Is there an ablation study comparing this with single-step execution?\n2. For multi-image inputs, I understand that due to the limitations of large multimodal models, directly adding multiple images might lead to suboptimal results. Is there a comparative experiment that converts the information from multiple images into text and provides it to the model in an in-context manner?", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Weakness1: The entire benchmark is centered around scenarios in a simulator, but does not address potential limitations that may be encountered in real-world scenarios.", "other_comments_or_suggestions": "I suggest including the evaluation of the qwen2.5-VL 7B and 72B models in the final version of the paper.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces EmbodiedBench, a benchmark designed to evaluate the capabilities of multi-modal large language models (MLLMs) in the context of embodied AI tasks. Different MLLMs are evaluated on this benchmark, providing valuable insights into how these models perform across various tasks. 

Strengths:
The paper introduces a novel and valuable benchmark for evaluating MLLMs across a broad set of tasks, from high-level reasoning to low-level manipulation. Thorough evaluations across various environments are conducted, providing insights into the strengths and weaknesses of different MLLMs. It presents new findings, such as the importance of visual cues in low-level tasks and the challenges MLLMs face in long-term planning and complex instruction understanding.

Weaknesses:
The paper uses a single prompt across all models, which might influence model performance. 

All reviewers are positive towards this paper, clearly an acceptance.
Decision: Accept (oral)