Title: OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
Venue: NeurIPS 2025
Accepted: yes
Keywords: Embodied AI

Mobile Manipulation

Agentic Data Synthesis

Vision-Language Model

LLM Agent
Abstract: The rapid progress of navigation, manipulation, and vision models has made mobile manipulators capable in many specialized tasks. 
However, the open-world mobile manipulation (OWMM) task remains a challenge due to the need for generalization to open-ended instructions and environments, as well as the systematic complexity to integrate high-level decision making with low-level robot control based on both global scene understanding and current agent state. To address this complexity, we propose a novel multi-modal agent architecture that maintains multi-view scene frames and agent states for decision-making and controls the robot by function calling.
A second challenge is the hallucination from domain shift. To enhance the agent performance, we further introduce an agentic data synthesis pipeline for the OWMM task to adapt the VLM model to our task domain with instruction fine-tuning. We highlight our fine-tuned OWMM-VLM as the first dedicated foundation model for mobile manipulators with global scene understanding, robot state tracking, and multi-modal action generation in a unified model. Through experiments, we demonstrate that our model achieves SOTA performance compared to other foundation models including GPT-4o and strong zero-shot generalization in real world.
The project page is at https://hhyhrhy.github.io/owmm-agent-project.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes OWMM-Agent, a vision-language agent for open-world mobile manipulation. The model, OWMM-VLM, is fine-tuned on a large synthetic dataset and performs high-level action planning (e.g., pick, place, navigate) based on natural language, multi-view images, and robot state history. It interfaces with classical planners for execution. Experimental results show that the proposed method significantly outperforms proposed baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Propose a unified vision-language agent to support multimodal training\n2. Introduces a scalable synthetic data generation pipeline covering four grounding tasks (pick, place, navigation, and scene search).\n3. Both simulation-based evaluations and real-world zero-shot tests\n\nWeaknesses:\n\n1. The number of real-world test cases is very limited, which undermines the statistical significance of the reported results.\n2. The model design may introduce significant computational bottlenecks, raising concerns for real-world deployment efficiency.\n3. Some claims lack sufficient clarification (see questions)\n\nQUESTIONS:\n1. Could you elaborate on how the scene is discretized and how the pose graph is constructed during the pre-mapping phase? Specifically, what strategy do you use to select camera poses and ensure sufficient scene coverage?\n\n2. How does your setting differ from goal-oriented vision-language navigation tasks such as REVERIE [1] and SOON [2]? Since your method has access to pre-captured global scene information (i.e., posed frames), would it be fair to consider your setup a simplified variant of those tasks?\n\n3. If I understand correctly, the pre-captured scene-level posed frames are included as part of the model's input. Could you clarify how the number of these frames affects GPU memory consumption and inference time? In the case of large or complex scenes, would this design lead to significant computational overhead?\n\n4. You mention the method can handle dynamic environments. Could you provide any design details or experimental results to support this claim?\n\n5. What types of visual hallucination occur in this navigation task when using foundation models, and how does your method address them?\n\n6. Writting issue: \n    - Line 217, Table 5 => Table 1\n\n[1] Qi, Y., Wu, Q., Anderson, P., Wang, X., Wang, W. Y., Shen, C., & Hengel, A. V. D. (2020). Reverie: Remote embodied visual referring expression in real indoor environments. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 9982-9991).\n\n[2] Zhu, F., Liang, X., Zhu, Y., Yu, Q., Chang, X., & Liang, X. (2021). Soon: Scenario oriented object navigation with graph-based exploration. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 12689-12699).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed OWMM-Agent, which performs open-world mobile manipulation tasks. The core of the system is a VLM that is fine-tuned using a large-scale synthetic dataset generated in simulation. This model is capable of handling multiple sub-tasks by reasoning over multimodal inputs, including long-term scene memory and short-term robot state history. The finetuning enables state-tracking of the task, resulting higher task success rate. The proposed model shows strong performance in decision making and grounding, for both simulation and real-world environment.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper is well-written and presented the problem and methodology cleary.\n- The fine-tuned OWMM-VLM effectively integrates vision and language for embodied reasoning.\n- Extensive experiments are conducted, including comparisons with strong baselines, ablation studies, and analysis of training data effects.\n\n\n\n## Weakness\n\n- Failure analysis needed: While the proposed method shows promising results, the full task success rate is notably low, even for the best-performing model with huge number of parameters. I understand this is a difficult task, and since there are multiple subtasks so low success rate is kind of natural considering accumulation of errors. However, a detailed failure analysis would help the reader better understand the primary bottlenecks and guide future improvements.\n- It would be beneficial to explicitly discuss why the proposed model avoids dead loops. I think the fine-tuned VLM model and the robot history helps avoid repetition, so it can naturally avoid dead loop while other can't. (Please correct me if I'm wrong). It would be nice to clarify about this with some examples.\n- The overall system appears to operate in open-loop control. While the VLM can understand the current state and make the decision based on it, but it is unclear how the system handles failures.  What happens if the picking object fails? Is the fine-tuned VLM model able to recover itself?  A discussion or experiment showing whether the VLM can detect and recover from failure would be valuable.\n\nQUESTIONS:\n- The gap between object image retrieval (88.56%) and goal image retrieval (30.39%) is quite large. Could the authors elaborate on potential reasons for this discrepancy? For example, are goal receptacles more visually ambiguous, less salient, or more linguistically diverse?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a VLM-based agent for open-vocabulary mobile manipulation. The authors formulate the instruction-following problem as letting the VLM reason over history observations, scene graphs (pose graphs), and egocentric observations. To induce reasoning capabilities, the authors selected several proxy tasks for training the VLM-agent and generated the datasets correspondingly. The results show improvement on both real-world and synthetic environments.\n\nSTRENGTHS AND WEAKNESSES:\n[+] The overall goal of this paper is well-motivated and is indeed an important problem to be studied.\n\n[+] The proposed dataset and the agent model are beneficial for OWMM-related research.\n\n[+] The authors tested not only in synthetic environments but also in real-world scenarios, indicating solid system design for real-world robots.\n\n[-] One major concern about this paper is the potentially missing details about the whole pipeline. As the idea of using graphs and VLMs for reasoning in open-vocabulary mobile manipulation tasks has already been discussed before, the distinction between this paper lies in the details of the modules that improve the model performance (e.g. CoT design, pose graph construction for sim and real environments, etc.). However, both the main paper and the appendix lack sufficient discussion on these details, which hinders further understanding of the current good performance of the model. This also makes the current dataset construction and experimental analyses a bit plain without much insights in addition to performance improvements.\n\n[-] I do feel the current version of paper writing could benefit from further revision and refinement.\n\nQUESTIONS:\nSee the strength & weakness section. Currently it's missing some details and therefore I'm leaning towards a borderline reject decision (with the hope that the revision/rebuttal could significantly improve clarity on model design and dataset construction).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes OWMM-Agent, a novel vision-language model (VLM)-driven agent for open-world mobile manipulation (OWMM) tasks. The core idea is to reformulate OWMM as a multi-turn, multi-image, and multi-modal reasoning problem. The proposed architecture comprises:
1. A unified VLM (OWMM-VLM) that performs global scene understanding, agent state tracking, and high-level action generation from multi-modal inputs;
2. An agentic data synthesis pipeline that generates scalable, instruction-conditioned training episodes in simulation, significantly reducing human annotation needs.
Experimental results show state-of-the-art performance across both simulation and real-world settings, demonstrating strong zero-shot generalization and outperforming GPT-4o-based baselines.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The idea of unifying scene understanding, state tracking, and affordance prediction into a VLM agent is novel and well-motivated.
2. Strong generalization to real-world environments, using only simulation data, demonstrates practical significance.
3. Experimental results are extensive: they include both single-step evaluations (with clear affordance grounding metrics) and episodic evaluation, including real-world robot experiments.
Weaknesses:
1. The assumption of a pre-mapping phase (pose graph and RGB map) weakens the claim of “open world.”

QUESTIONS:
1. Pre-Mapping Assumption: The system assumes a pre-built pose graph and scene images. Could the model be extended to perform online mapping and actuation in tandem?
2. Action Format: The model uses bounding boxes for affordance grounding. Are there failure cases where bounding boxes are insufficient ?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-05