HomieBot: an Adaptive System for Embodied Mobile Manipulation in Open Environments

Authors: Dongping Li, Tielong Cai, Tianci Tang, Wenhao Chai, Katherine Rose Driggs-Campbell, Hongwei Wang, Gaoang Wang
Primary area: datasets and benchmarks
Keywords: Embodied Agent, Embodied AI Benchmark, Mobile Manipulation

Abstract:
Embodied Mobile Manipulation in Open Environments (EMMOE) is the challenge that agents understanding user instructions and executing long-horizon everyday tasks in home environments. This challenge encompasses task planning, decision-making, navigation and manipulation, and is crucial to develop a powerful home assistant capable of autonomously completing daily tasks. However, the absence of a holistic benchmark, data incompatibility between large language models (LLMs) and mobile manipulation tasks, the lack of a comprehensive framework, and insufficient dynamic adaptation mechanisms all continue to hinder its development. To address these issues, we propose EMMOE, the first unified benchmark that simultaneously evaluates high-level planners and low-level policies, and new metrics for more diverse evaluation. Additionally, we manually collect EMMOE-100, the first everyday task dataset featuring detailed decision-making processes, Chain-of-Thought (CoT) outputs, feedback from low-level execution and a trainable data format for Large Multimodal Models (LMMs). Furthermore, we design HomieBot, a sophisticated agent system which integrates LMM with Direct Preference Optimization (DPO) as the high-level planner, small navigation and manipulation models as the low-level executor. Finally, we demonstrate HomieBot's performance and methods for evaluating different models and policies.

Decision: reject
Reviewer scores: 5, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730353275119

thread_type:
official_review

<thread_object>
creator: reviewer_tyvz

<content>
summary:
The paper introduces a unified benchmark named “EMMOE” designed to simultaneously evaluate both high-level planning and low-level control abilities of embodied agents. This benchmark comprises 100 complex everyday tasks based on scenarios from the Replica dataset, including long-horizon tasks, open-ended tasks with multiple possible outcomes, logical tasks, etc. To better assess agent performance, the authors propose three new metrics: Task Progress, Success End Rate, and Success Re-plan Rate. Additionally, the paper provides training data using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), featuring Chain-of-Thought (CoT) outputs and replan processes. A baseline agent, HomieBot, is introduced, featuring a Large Multimodal Model (LMM) for high-level planning and several task-specific models for low-level control. HomieBot achieves a 31.8% success rate on training tasks and 20% on test tasks.

strengths:
1. The objectives of the study are clearly stated.
2. The related work cited in this article is extensive.
3. The provision of the datasets (including CoT, replanning, etc.) is beneficial for future embodied AI research.

weaknesses:
1. The article lacks clear organization.
     - In Section 3, it is inappropriate to describe the dataset for model training (Section 3.2, Data Augmentation for SFT and DPO) here. This dataset is intended for training the embodied agents, not for constructing the benchmark tasks.
     - The work aims to evaluate high-level planning and low-level execution abilities. However, there is no individual assessment of these abilities (e.g., the success rate of high-level planning, low-level execution, etc.). The relationship between the proposed evaluation metrics (TP, SER, SRR) and these abilities is unclear.
    - The problem is not well defined. It would be beneficial to provide a problem formulation regarding high-level planning and low-level execution in the paper.
   - Some content in the article is unclear. For instance, on page 4, line 207, the statement “if subtask i fails but subtask i + 1 succeeds” lacks clarity regarding the relationship between subtask i and subtask i+1. From my understanding, subtask i and i+1 are the same task but with different model responses (actions). This needs to be clarified.
2. There are a lot of mistakes in the paper,
   - In Section 3.3, Equation 1 appears to be incorrect. For different keypaths $k_i$, the nodes to be checked are also different. Therefore, the equation should be $TP=max_{{k_i}\in K_T} (\frac{len(k_i^{check})}{len(k_i)})$.
   - On page 6, line 278, M is defined as the set of all tasks, and $t \in M$, so $t$ should be a single task in the task space. However, the author also states “t is a task trajectory,” which conflates the concepts of task and trajectory.
   - The criterion “Success End Rate” is not reasonable. For example, if an agent never outputs “End” but can complete some tasks, the equation will be infinite. The correct equation should be $\frac{\text{number of successful tasks whose last output is End}}{\text{number of successful tasks}}$'.
   - On page 4, line 318, $i$ is not defined.
   - Spelling errors: On page 4, “controll” should be “control”.
3. The experimental section needs improvement. There is even no evaluation of the model before fine-tuning.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730437122614

thread_type:
official_review

<thread_object>
creator: reviewer_ammf

<content>
summary:
This paper presents a benchmark of 100 tasks in home robotics, collected within the Habitat simulator using human demonstrations annotated with detailed reasoning explanations. The benchmark evaluates models using success rate, success end rate, and success replan rate. The authors train Video-LLaVA on this dataset with either SFT (supervised fine-tuning) or DPO (Direct Preference Optimization) and find that SFT+DPO performs best on the train set across all metrics, while SFT-only achieves the highest scores on the test set.

strengths:
- Introduction of a new benchmark for home robotics with a diverse set of 100 tasks.
- Collection of detailed annotations to capture the reasoning process and a diverse set of tasks in home environments.
- Strong metrics to evaluate embodied execution, focusing on nuanced aspects of embodied execution.
- Propose a dataset for SFT and DPO fine-tuning from egocentric trajectories.

weaknesses:
- Limited baseline comparisons, which reduces the clarity of contributions relative to existing methods.
- Lack of diverse model comparisons, including text-only and zero-shot multi-image baselines (e.g., Qwen2VL, GPT-4o), and no modular vs. end-to-end performance analysis.
- Insufficient analysis of task-specific challenges and bottlenecks within the benchmark; error analysis and reasoning failure modes are explored to some degree, but not in detail.
- No discussion of potential real-to-sim discrepancies for the manipulation models, which could impact model generalization.

questions:
- Could the authors provide stronger baseline comparisons, especially with text-only and zero-shot multi-image models?
- How do modular models compare to end-to-end approaches on this benchmark?
- Can the authors analyze the primary failures observed by the model, and give qualitative analysis of what are the major challenges in the benchmark for existing models?
- Is there any observed real-to-sim gap affecting the manipulation models, and if so, how does it impact performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668286405

thread_type:
official_review

<thread_object>
creator: reviewer_vzga

<content>
summary:
This work focuses on studying embodied AI instruction following mobile manipulation in simulated environments. To this end, the paper proposes the EMMOE-100 dataset and benchmark and HOMIEBOT as a system integration solution to it. Some unique contributions involve that it is a dataset of human-controlled embodied agent trajectories:
1. Decomposition into subtasks
1. Annotated reasoning process with every execution
1. Replanning for some failed subtasks
The paper also introduces metrics to measure Task Progress, Success end rate, and success replan rate and presents results for SFT and SFT+DPO on the Video-LlaVA model.

strengths:
1. The paper introduces the human controlled embodied agent trajectories, unlike the use of PDDL in ALFRED and other RL benchmarks.
1. The focus seems to be on open-ended long horizon questions, multiple ways of solving it, annotating sub-goals and deliberately collecting replanning trajectories.

weaknesses:
1. Overclaim: it does not seem like the first to unify the high-level and low-level. This has been discussed in Robotics Task and Motion planning (TAMP) literature and has been introduced as baselines in previous simulator benchmarks. Connection with previous work and clarity on the unique contribution of this work is needed.
1. Lack of failure analysis: The paper does not discuss the limitations and failures modes of the combined high-level and low-level modules. 
1. Limited discussion: It is unclear why the success rate for HOMIEBOT is so low in training and test tasks. 
1. Clarity: Paper is not well formatted and has lots of typos (line 180 control, 185-186 between)
1. Limited novelty: with discrete action space, the work looks very similar to ALFRED (Shridhar et al.) and FiLM (Min et al.) with a few enhancements.

questions:
1. Why not use continuous action space? Habitat Replica allows for continuous action space. HM3D 
2. How is the Success end rate, and success re-plan rate measured? How is the count for end and replan computed for each trajectory?
3. Why is “trainable data format” listed as an important contribution compared to the existing datasets? 
4. How does the high-level planner know when and where low-level execution fails?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707739457

thread_type:
official_review

<thread_object>
creator: reviewer_pilp

<content>
summary:
The paper introduces three main contributions:

1. EMMOE: A unified benchmark for evaluating both high-level planners and low-level policies in embodied mobile manipulation tasks.

2. EMMOE-100: A dataset with 100 complex everyday tasks featuring task-planning processes and COT outputs

3. HOMIEBOT: An intergrated agent system combining high-level planner and low-level executors.

strengths:
1. The new evaluation metrics are novel, potentially providing a valuable resource for the research community and practical applications.

weaknesses:
1. **Insufficient Methodology Details**
   - The data processing and augmentation methods are not adequately explained:
     * The "fixed-format conversation data" conversion process in SFT Augmentation is not described
     * No clear explanation of how the uniform script processes the EMMOE-100 data
     * The DPO Augmentation section lacks clear algorithmic description or flowcharts

2. **Limited Experimental Evaluation**
   - Inadequate baseline comparisons and empirical analysis:
     * No evaluation of existing RL algorithms or LM-based agents on the EMMOE benchmark
     * Only one system (HOMIEBOT) is evaluated, lacking comparative analysis
     * The relatively low success rates (31.8% training, 20% testing) are not thoroughly analyzed
     * Missing ablation studies or detailed error analysis to understand performance bottlenecks

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732357515393

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for your interest in and suggestions for our work. We are also sorry for the shortcomings in our manuscript due to time constraints and lack of experience, which may have caused inconvenience to you, we are grateful for your tolerance and guidance, which have helped us complete this manuscript. In the updated version, we primarily make the following changes based on the reviewers' feedback and mark them in orange:

* **Paper Writing and Organization:** We move the Data Augmentation part to Section 4.1, add Section 3.1 for more precise system definitions, rewrite Section 4.4 to emphasize the relationship between new metrics and results, add Section 4.5 for detailed performance analysis. We also correct writing errors, remove redundant descriptions, and move related work to Appendix A due to the page limitation.

* **Methodology Details:** We refine the method descriptions and correct mistakes mentioned by reviewers for clarity and accuracy. Additionally, we add more detailed task demonstrations, metric calculations, code implementations for data augmentation, data samples, visualization processes for system and experimental details in Appendix B~H.

* **Comparison Experiments:** We add evaluations of GPT-4o[1], Gemini-1.5-pro[2], Qwen2-VL-7B[3], MiniCPM-V 2.6[4] on the EMMOE dataset, evaluation settings and model deployment can be found in Section 4.3 and Appendix H.1. The analysis of the results can be found in Section 4.4.

* **Performance Analysis:** We refine the error definitions in Section 4.5 and conduct comprehensive error statistics to further explore bottlenecks in the current challenge, we also show how to use HomieBot to evaluate both high-level planners and low-level policies, the results are in Figure 4, Table 4, and Appendix H.2. We also conduct several case studies in Appendix I.

* **Supplimentary Materials:** We add more anonymized codes, scripts and data demonstrations in our supplimentary materials. After further organization, all codes and data will be open-sourced to make contributions to the community and ensure the reproducibility.

At the same time, we would also like to clarify that:

**Our contributions remain unchanged**, which can be listed:

* EMMOE, the first unified benchmark for both high-level and low-level embodied tasks with three novel metrics for more advanced evaluation.
* EMMOE-100, the first everyday task dataset featuring CoT outputs, diverse task design, re-plan processes, SFT and DPO dataset built on it.
* HOMIEBOT, a sophisticated agent system which integrates models at different levels, multiple error detection and adaptation mechanisms to complete EMMOE tasks.

Our **core contributions are the new benchmark, metrics and evaluation methods**, followed by the design of HomieBot. Our primary requirement for HomieBot is to robustly perform EMMOE tasks. In revisions, **all methods and data used are the same as before**(including EMMOE-100, SFT and DPO augmentation, HLP and LLE frame), everything from the previous version is included in the new version. **Our revisions focus mainly on experimental comparisons and further analysis of the results, addressing the shortcomings mentioned by the reviewers and highlighting our features**. 
The additional materials include plenty of original code or data samples. I believe these updates are comprehensive, and we hope to engage in further discussion with you. We welcome any criticism and feedback!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733079136086

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all the reviewers once again for your valuable suggestions and the time and effort dedicated to our work. As the discussion phase comes to a close, we hope to receive further feedback on the article's content and engage in in-depth discussions with you. We are committed to continuously polishing our work and answering all questions you may have, and we will respond as quickly as possible after receiving feedback!
</content>
</thread_object>

</thread>


Metareview: