OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
Authors: Junting Chen
Haotian Liang
Lingxiao Du
Weiyun Wang
Mengkang Hu
Yao Mu
Wenhai Wang
Jifeng Dai
Ping Luo
Wenqi Shao
Lin Shao
Venue: NeurIPS 2025 poster ()
Keywords: Embodied AI
Mobile Manipulation
Agentic Data Synthesis
Vision-Language Model
LLM Agent
Primary area: applications
TL;DR: We introduce a novel embodied VLM agent with a VLM fine-tuned by agentic data synthesis for open-world mobile manipulation, unifying scene understanding, state tracking, and action generation for state-of-the-art results.

Abstract: The rapid progress of navigation, manipulation, and vision models has made mobile manipulators capable in many specialized tasks. 
However, the open-world mobile manipulation (OWMM) task remains a challenge due to the need for generalization to open-ended instructions and environments, as well as the systematic complexity to integrate high-level decision making with low-level robot control based on both global scene understanding and current agent state. To address this complexity, we propose a novel multi-modal agent architecture that maintains multi-view scene frames and agent states for decision-making and controls the robot by function calling.
A second challenge is the hallucination from domain shift. To enhance the agent performance, we further introduce an agentic data synthesis pipeline for the OWMM task to adapt the VLM model to our task domain with instruction fine-tuning. We highlight our fine-tuned OWMM-VLM as the first dedicated foundation model for mobile manipulators with global scene understanding, robot state tracking, and multi-modal action generation in a unified model. Through experiments, we demonstrate that our model achieves SOTA performance compared to other foundation models including GPT-4o and strong zero-shot generalization in real world.
The project page is at https://hhyhrhy.github.io/owmm-agent-project.

OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
Junting Chen 12+
Haotian Liangl3. 13*
Lingxiao Du Weiyun Wang' Mengkang Hu
Yao 2u
Wenhai Wang*
Jifeng Dai
Ping Luo Wenqi Shao Lin Shao
Shanghai Al Laboratory
2School of Computing, National University of Singapore *USTC
"The Univeristy of Hongkong
$Shanghai Jiaotong University 6 Tsinghua University
Abstract
The rapid progress of navigation, manipulation, and vision models has made mobile manipulators capable in many specialized tasks. However, the open-world mobile manipulation (OWMM) task remains a challenge due to the need for generalization to open-ended instructions and environments, as well as the systematic complexity to integrate high-level decision making with low-level robot control based on both global scene understanding and current agent state. To address this complexity, we propose a novel molti-modal agent architecture that maintains multi-view scene frames and agent states for decision fecision-making and controls the robot by function calling. second challenge is the hallucination from domain shift. To enhance the agent performance, OWMM task to adapt the VLM model to our task domain with instruction fine- tuning. We highlight our fine-tuned OWMM-VLM as the first dedicated foundation model for mobile manipulators with global scene understanding. robot state track- ing, and multi-modal action generation unified model. Through experiments, we demonstrate that our model achieves SOTA performance compared to other foun- dation models including GPT-4o and strong zero-shot generalization real world. The project page is at https: httts:://h/pp:: hthps://bhyhry... github. 1o/ owmm agent project
further introduce agentic data synthesis pipeline for the
Embodied VLM Agent
1. Leng term Enviroment Memory (Tmages)
Retrieved Images with associated
Multimodal Memory
2. Transient Robot State Memory (Tests)
map poses relevannt tasks
So
S
3
VLM
Timi
0p
Ap
frame
to
positiea
02
A
Nuv
0 A3
Multimodal Tools
Please clean
rible
place back fruits
bock bowlss
t=2
t=3
Figure 1: OWMM-Agent Operates Fetch Robot for Tidying Task. OWMM-Agent receives natural language instructions and leverages both long-term environment memory (scene images) and transient robot state memory (textual summary) to generate sequential multi-modal actions to finish the task. By multi-turn, multi-image, and multi-modal VLM reasoning, the agent conducts global scene aware reasoning, updates robot state memory, and actuates itself to desired coordinates without any other learning- baanid models in unstructured environments.
Equaa Contributions Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1 Introduction
The vision of generalist home assistant robots has brought open-world mobile manipulation (OWMM) to the forefront of embodied AI research 37 40 43 28]. OWMM tasks require mobile manipu- lators to interpret open-ended natural language instructions and operate in unstructured, previously unseen environments. Although advancements in navigation, manipulation, and vision models have effectively enabled mobile manipulators to perform many specialized tasks under constraints, achieving robust autonomy in these settings remains challenging.
A central difficulty in OWMM is the need for comprehensive global scene understanding and reasoning conditioned on natural language instructions and agent state. On one hand, prior approaches often construct 2D semantic maps [30] or 3D semantic fields with CLIP-based features [19 28).
retrieving targets by computing embedding distances between the semantic map and language instructions. While these methods have enabled progress, they are limited by the capacity of embedding models, which can struggle with complex, compositional instructions, compared to foundational generative models like large language models (LLM) or laionnlanguage models (VLM). Additionally, they often require time-consuming dense 3D reconstruction, making them less suitable for complex, open-ended and dynamic environments. On the other hand, the recent advances L.L.Ms and VLMs, with strong generalization capability, versatility, and reasoning capability, offer promising opportunities and potentially a fundamental pathway to solve all sorts of scene understanding, task planning, and robot control issues in open-world intelligent robot systems 20 1SI
Based on the aforermentioeed observations, we propose novel VLM agent framework, OWMM-Agent to address these challenges and leverage the power of VLMs for OWMM task. More specifically, we formulate the high-level OWMM task for the internal VLM model as a multi-turn, muulti-imaee. and multi-modal reasoning problem. The VLM model generates end-to-end chain-of-thouglle (CoT) thinking process, tracked agent states, and multi-modal actions with coordinates based on all raw multi-modal input. Then the agent calls the coordinate-based planners to actuate the robot. Our approach built on two insights: 1) We do not need the detailed geometric representation of the environment for instruction conttciiioononiiiineee global scene understanding, and we could easily access precise and even dynamic geometric information when the robot moves to the task-relevant local region. 2) By leveraging the strong vision-language grounding capabilities, we can effectively bridge the high-level reasoning process in language and low-level robot control targets in coordinates, with the help of 2D-to-3D reverse projection.
However, directly applying pre-trained VLMs to our embodied agent presents challenges of domain shift: 1) Rare grounding tasks: Robotic planners and controllers require multi-modal inputs, including both tools and coordinates in the visual space for robot control. The base models could be powerful for object-centric grounding such as detecting novel objects, but they suffer in other grounding tasks including detecting non-blocked navigable areas in the ego-centric image. 2) State tracking: The agent must infer and track its own state from observations and history records to make contextually appropriate decisions. 3) Embodiment priors: Effective becision-making in egocentric settings demands strong embodiment-dependde constraints, such as maximum reach for picking actions.
priors, such as knowledge of the robot's kinematic
To address the problem of domain adaptation, we further introduce an agentic data synthesis pipeline tailored for OWMM, to generate large-scale and instruction-driven episodes that teach the VLM agent to track its state, reason over multi-view observations, and generate multi-modal action affordances grounded in both the global scene and the agent's ss embodiment. This pipeline minimizes human annotation effort by utilizing predefined task sequence templates and ground-truth symbolic world representations from simulation. With extensive experiments in simulation, we demonstrate that OWMM-VLL consistently outperforms baseline models. In the real-world experiment, we find that our model has strong zero-shot generalization to real-world observations, with 27/30- = 90% action generation success rate on our fetch robot in the lab environment, even being fine-tuned on the simulated data. We also provide ablation studies on models and dataset analysis to provide insights into the model design and training data construction. In summary, our contributions are as follows:
We propose OWMM-Agent unified VI.M- agent architecture for open-world mobile manip- ulation, capable of global scene understanding, state tracking, and end-to-end action generation.
2

---

We introduce a Aumulation-based agentic data synthesis pipeline that enables scalable data collection for instruction fine-tuning for domain adaptation with minimized human effort, with detailed analysis on the quality of the generated dataset.
We introduce a foundation model for OWMM, capable of multi-image reasoning and executable multi-modal action generation, with extensive experiments analyzing the model's performance.
Related Works
Open World Mobile Manipulation
opea-vocab Mobile Manipulation (OVMM) focuses on navigating and manipulating novel objects in unseen environments with language instructions. Referred to as Open Vocabulary Mobile Manipulation (OVMM) by 40 43 19 or Open World Mobile Manipulation (OWMM) by [29 31 we use the term OWMM for this paper. In our formulation, "open-world" refers semantic diversity- the ability to generalize to unseen scenes, novel object categories, and diverse instances- rether than unconstrained physical exploration without any prior environmental information. This terminology aligns with the HOMERobot challenge [40]
The original OWMM baseline and Melnik et al. 2 assume the agent starts without scene observation and must explore to build a representation for decision-making. Recent works [99
28 43 suggest
two-stage approach: first using SLAM 77 to create 3D semantic maps, then performing OVMM models like GPT-4V and GPT-4o [[]].
using open-vocabulee
Zhi et al. 43 introduces COME-robot, a closed-loop OVMM framework using GPT-4V for reasoning and replanning. producing code for preset funetions and object captions as in Code-as- Poolccy [17]. Unlike relying pre-trained skill models requiring inputs like skill names and object captions, our model directly produces target positions for position-baat based motion planners and controllers.
Vision-Language Navigation and Interactive Embodied Tasks
Our work is related to the broader vision-language navigation (VLN) literature, which has explored goal-oriented navigation and manipulation in indoor environments. REVERIE [27] introduced remote embodied visual referring expressions. requiring agents to navigate and ground target objects based on natural language descriptions. SOON
44 proposed scenario-oriented object navigation with graph-based exploration strategies. These works primarily focus navigation with discrete action presented a benchmark for interpreting grounded proposed modular methods for following instructions in language. Recent work has also explored topological planning with transformers for VLN 4. Our
spaces and viewpoint selection. ALFRED instructions for everyday tasks, while FILM [4
31
OWMM setting differs from these VLN works key aspects: (1) Continuous action space: We continuous positional control requiring precise affordance grounding rather than discrete viewpoint selection. (2) Direct object interaction: Our agent physically manipulates objects with low-level joint control, whereas VLN tasks use simplified interactions or only predict bounding boxes. (3) Sim- to-real transfer: Our framework is validated on real robotic hardware. (4) Unified reasoning: We formulate the task as multi-modal, multi-turn reasoning within single VLM, integrating perception, planning, and action generation.
Large Foundational Models for Robotics
Recent advances in large fundamental models show significant potential in robotic control and generalization. One major research focus is to adapt pre-trained Visual Language Model (VLM) to robot scenario. RoboPoint [42] introduces synthetic data pipeline for instruction-tunin
VI.Ms in robotics, supporting accurate spatial affordance prediction in object manipulation and navigation. MOKA [18] uses novel VLM approach in robotic manipulation with point-based affordance and motion representation, using visual prompts to turn key points and waypoint predictions into visual question-answering tasks for VLMs. Our proposed model OWMM-VLM also falls into this category.
The other popular research topic is Assion-Language Action (VLA) models, focusing on using relatively smaller transformer backbones to directly generate robotic actions with high frequency. VLe [1] is 7B-parameter open-source model trained on 970,000 real-world demonstrations using Ilama 2 K4 architecture, excelling in general manipulation tasks. Octo 33 advances generalist robot policies, handling language commands and goal images while adapting quickly to new inputs and actions with standard GPUs. To || presents flow fotthing architecture based
3

---

pre-trained VLM, excelling in dexterous tasks. These models mark significant progress in making robotic systems more versatile, scalable, and adaptable to different trajectories.
Methodology
In this section, introduce the definition of OWMM in section 3.1 After that, claborate the agent framework in section[.22 Finally, the training method of OWMM-VLM presented in section 3.3
The overview of our method is shown in Figure 2
World
Graph of Posed Frames
Real-time Ego-centric Trame
Instruction
Nooot Ofoecte losu ke bieed Bonatakke ause siasma flacer
OWAD
Histora
AIFE
Chain-of Thought Reasoning with Multimodal Action
E+1
Ege -mnri
Reanmbie
Norioval and eljest Neod Rad
1+2
Zibr odvour
ego-centric franso
Lafe pill
Hare tnirgoole
Wntinggioo nege Ca placeolpet
Learning-free positional-hhved robot planners
Figure 2: The Overview OWMM Agent Framework. The left panel represents the world space, including graph of posed frames generated during the pre-mapping phase and real-time egocentric frame captured by the robot. The right panel showcases the Agent Space, where OWMM- MMM processes task instructions, robot history, and visual inputs perform thain-of-thougeee reasoning and generate high-level actions with region coordinates, which are then sent to robot planners for navigation and manipulation.
3.1 OWMM Task Definition
Following the common OVMM/OWMM problem setting 40 19 37], the robot needs to follow the instruction in the pattern of Move (A) (in (B)) and place it on/in (C>", where (A) (B) (C) are novel objects/initial receptacles/gont receptacles in the unseen environment from the training data.
Following the problem setting in 19 28 we assume pre-mapping phase separating active explo- ration and the SLAM module from the OWMM task focus. This is practical, as most robotic vacuums automate room mapping before cleaning. The pre-mapping implementation details are provided in Appendix C.T.
Thus, we introduce a pose graph G and associated RGB images I as the output of the pre-mapping stage the basis of [40), and define our OWMM problem as follows: In an OWMM task episode of max timestep T, at each timestep 0 < t T. an agent takes inputs composed of 1) natural language instruction L: 2) a pre-mapping camera pose graph G (V, E} of poses, where
V lu...., n, edges are not used;
3) and associated RGB images I (Jo,..., In), each image 7 33xwx taken head camera view pose 1; in G;
4) the agent's 's current head camera RGB image If and depth image DC.

---

With these imputs, an agent needs to generate a low-level continuous action that directly actuates the robot kinematically, including joint velocities the robot arm and the base velocity of the robot. Let's F Pagent note the logical function of the agent policy model, and we have
a = F Fagent (C. G,I,I, Di,x)) x
where X stands for the robot state time t.
3.2 OWMM Agent
Running large VLM models at 25Hz and gathering sufficient data for training a generalist VLA model from open-set language and visual observations remain challenging. To address this latency issue in the OWMM agent, we have the large VLM to produce high-level actions. The agent employs unified VLM model Felm to convert visual and lingual inputs into action types and positional commands, using a classical planner for navigation and a motion planner for manipulation, similar to (ekep[10]. The model's output represents high-level action At spanning several simulation steps, while planners resolve trajectories and low-level actions a, for each step.
Ac: He Fulm (C, G,I, 1i Hz-1),
a, A, (x,, DÇ),
where He, H are the high-level robot history, updated by the VLM model itself. a, A, Di) indicates that the high-level action itself can be converted to executable code with the action handle linked different planners and positional targets. In this regard, part of the high-level action At can be seen as a special type of language model program, as proposed in [17]. Then the linked planner
takes the state of the robot x,, and point clouds converted from depth map Dł as an additional input to calculate the low-level action d:-
To translate high-level action A, into low-level action at, the agent has path planner 8 for navigation and a motion planner [32] for arm manipulation. These planners generate waypoints that satisfy mechanical constraints for base chassis and arm joints through sampling-basee methods. There is also a gripper controller to grasp/ungrasp the object. The high-level actions that aim to actuate the robot will be associated with planners and controllers through predefined functions.
3.3 OWMM-VLM model
IntemL Qwen2.5-32B
( JSON
Based
Foed
The history indictto that am clour outter viow, cun see Therdiee, Liced pick the Nordle Ware Original Bandt Pun.
Projection MLP
Langunge Tokenizer
Iniern VIT-300M:
Proepp:
Artion Information: [162, 55%, ass 737]
zan raange single ribe
Task Instruction Mova Nortie Ware Origital listt ectele ouper aboudd is pou
Semmartratien: The rad has dartal and have rasigatal to Lateal Storage Calrnet, Brow Spenar and picced up the Nundic Warn Urigaed tlenet thas.
Roterr Histers The
and
cko Luteral Suraas Calrned.... cirrage emags 3 <iruge To <irage Robor's corver viee is <irage S6.
Model Output
Frozen
Finetuned
Ordered Images
Instruction Input
comprising ViT, 2-layer projection MLP, and a LLM. During training, VIT parameters are frozen while the projection MIP and the L.M parameters are trainable. The model is required generate multi-modal actions in JSON format conditioned on scene images, task instructions, and robot history.
Figure 3: Overview of OWMM-VLM. Our model is fine-tuned InternVL-2.5(53)
Intuitively, a VLM model requires three multi-modal capabilities accomplish the OWMM task:
(1) Image Retrieval. Given the graph of posed frames and an egocentric frame, the VLM model needs to retrieve posed frame that contains the relative objects or receptacles that the robot needs to navigate to.
5

---

(2) Ego-centric Decision-making. Given multiple posed frames and an egocentric frame, the VLM model needs to decide which action conduct based on the task context, robot history, and current egocentric observation. This capability is closely associated with the idea of spatial intelligence [9) that VLM models should understand the spatial relationship between themselves and the scene objects in order to make decisions on actions.
(3) A Aorraance Grounding. If the agent decides to interact with the near surroundings perceived in the egocentric frame, it should also generate the target positions that correspond to the intention of the task.
Following this insight, we train a versatile VLM model that takes the task instruction L. multimodal observations I, I;, and history H4-1, and generates all high-level actions. We design four types of high-level actions: 1) Posed image retrieval, 2) Navigate to point, 3) Pick, and 4) Place, which are associated with planners and the grip controller. However, due to the extended time horizon of the OWMM task, simply generating the executable action is insufficient. We instruct the VLM model to monitor the state through robot history and to infer the subsequent action by considering both the history and the present observations. Figure 3 demonstrates our model architecture as well as its input and output.
3.3.1 Chain-of-Thought Reasoning Design
To address the visual hallucination challenges faced by pre-trained VLMs OWMM tasks, we design structured Chain-of- Thought (CoT) reasoning framework. During our baseline evaluation, we observed three key hallucination-related failure modes: (i) Error location outputs: Base models achieve very low affordance success rates (0.05-0.18) due to incorrect object detection or mismatched bounding-bol box outputs; (ii) Multi-image hallucination: Models frequently hallucinate on images where target objects are absent, achieving only 1.27% accuracy on 8-image retrieval; (iii) Insufficient long-hop hong-horizon reasoning: Models fall into dead loops due to limited ability to track historical context and robot state.
Our CoT reasoning approach generates structured reasoning chains that include: 1) Task instruction reasoning and summarization for decision-makinn 2) Perception and grounding of current egocentric- view and scene images, integrating visual information to support task decisions (e.g.. determining whether the agent is close enough to object for interaction); 3) Task decision oulpul with execution targets/coordinates as bounding boxes; 4) Summarization of decisions and actions for next-step input.
This design allows OWMM-VLM to acquire patterns for task comprehension, scene perception, and eecision-making from structured training data. Crucially, the model summarizes historical context after each decision, enabling each subsequent step to jointly reason prior history and current observations. This structured reasoning, learned through supervised fine-tuning on CoT-annotated data, enables the model to track task progress and avoid repetitive actions dead loops. As shown in Appendix G removing reasoning and summarization capabilities leads to substantial performance degradation across all metrics.
For more details on model implementation, see Appendix
4
Dataset
In this section, we elaborate on our data construction pipeline and quality verification method in section 4.1 A detailed analysis of the data is provided in section 4.2
4.1
Agentic Data Synthesis Pipeline
Effective OWMM-VLM model training requires comprehensive ground-truth annotations cover- ing navigation, object grasping, and manipulation affordances with full contextual understanding. Previous research [42 21 often generates question-answer pairs from images or videos, but lacks comprehensive action sequence representations and necessary affordance information for multi-step reasoning.
To address this challenge, we developed an automated agentic data synthesis pipeline that generates both action sequences and structured chain-of-thouge annotations. Our pipeline consists of four key stages:

---

Table 1: Dataset Overview for Instruction Fine-tuning. Our dataset consists of four subsets, each correspond- ing to one of the four primary task actions: Pick, Place, Navigate Point, and Search Scene Frame. The dataset is designed to encompass diverse scenarios and objects, ensuring comprehensive coverage of open world mobile manipulation tasks.
Task Action
Pick
Place
Nav point
Search scene frame
Data Size
64.7K
68.9K
59.6K
378.8K
Task Description
Move Arm Hammer Di- aper Pail Refills 12 Pack from Brunel-style stool to the white 2-scater sofa.
Move Shark from the Conlay kitchen to the comfortable sofa.
Move wood block from the 7-picce dining set with grey chairs to Low kitchen element, Natural element.
Move screwdriver from the Modem Indus- Dresser, Natural Ma- terial to the Stacked shelf system.
Context Description
I have embarked on my steadily advancing toward the Brunel-style bar stool, where the Arm Hammer Diaper Pail Refills 12 Pack MFWkmoweejt situated.
have embarked on my successfully nav- igated to the Conlay kitchen, retrieving the Shark with case. Now. am inching clnser to the cozy baven the fostable sofa, where I will soon place Shark.
The task has started and navigated piece dining set with grey chairs and picked the wood block am getring closer to Low kitchen element. Natural element where should place wood black.
The jourscy has com- menced, and have suc- cessfully navigated to Modern Industrial Dresser, Natural Material. where have now picked up the screwdriver.
Action Information
[168. 239, 967]]
11447, 539, 999, 9991]
1|246, 666, 285, 705]1
Task Planning with PDDL Using Habitat simulation [26.. we first construct symbolic task plans to complete OWMM tasks based Planning Domain Definition Language (PDDL. [22 PDDL provides a structured representation of preconditions, actions, and effects, enabling systematic generation of valid task sequences. This ensures that all generated episodes logically consistent and executable.
Stage 1:
Stage 2: Trajectory Execution and Data Collection We direct the robot execute task sequences within the simulator, recording key information at each step: robot coordinates, current action, object and receptacle positions, and camera extrinsic parameters. To enhance data collection efficiency, we do not initially collect RGB images at every timestep, but rather mark keyframe candidates during execution.
Stage 3: Keyframe Selection and Filtering We apply strategic keyframe selection to ensure data quality. For navigation, we select steps where the target receptacle is visible as start points and robot stopping points endpoints, sampling waypoints at intervals. For pick and place actions, select frames where the target object/receptacle is both visible and reachable by the robotic arm.
Stage 4: Chain-of-TT -hiought-TThught Annotation Generation We construct structured of-in-of-thought annotations based on predefined templates that incorporate: (i) lask instruction reasoning for decision context; (ii) perception reasoning about current observations; (iii) action decisions with affordance grounding (bounding boxes for navigation/pick/place image IDs for scene retrieval); (iv) summariza- tion of the current step for historical context. The summarization from each step is systematically incorporated into the "Robot's History" for the next step's 's question, enabling temporal reasoning.
To enhance linguistic diversity, we use GPT-4o mini to paraphrase the reasoning and summarization components while preserving the structured format and action annotations. This increases robustness to natural language variation without compromising annotation precision.
Finally, we collect scene graph frames for each episode by sampling robot head-view images at task- relevant locations (initial and goal receptacles) and additional random positions for scene coverage, as detailed in Appendix C.T Additional implementation details of the data synthesis pipeline are
provided in Appendix D
7

---

4.2 Dataset Analysis
We used 143 scenes from The Habitat Synthetic Scenes Dataset (HSSD) 13 and combined objects from YCB Objects [3] and Google Scanned Objects 66 create dataset with 157 unique manipula- tion objects and 1,471 1 receptacles from selected scenes. In each scene, objects were randomly placed for the robot pick and relocate another receptacle, with 400 episodes sampled per scene.
Following the data synthesis pipeline described in section 4.1 we collected and filtered episodes from each scene, ultimately gathering 21,046 valid episodes with approximately 572K total annotations. As shown in Table the final dataset comprises: i) pick action dataset of 64.7K image-texI pairs, ii) place action dataset of 68.9K image-text pairs, iii) navigation dataset of 59.6K image-text pairs, and iv) search scene frame dataset with 378.8K multi-image-tex pairs.
In our datasets, also apply re-labeling process for objects and receptacles, unlike HomeRobot's fixed criteria(40), We kept the original object labels and used GPT-4o to rewrite receptacle labels. These labels were diverse and descriptive, suited for open- open-world woold scenarios.
5 Experiments
In this section, we present the evaluation results in both simulation and real-world data. We present the experimental results of single-step evaluation for WWMM-VLM in our simulated benchmark in section 5.1 and episodic evaluation for the OWMM-Agent in our simulated benchmark in section 5.2 We then present the real-world evaluation in section 5.3 Due to the page limit, we discuss the data scaling law and how data diversity impacts the model performance in Appendix D.2] For the ablation study model design, such the choice of generating bounding boxes rather than points, please see Appendix G We further provide the qualitative comparisons of different models in Appendix [] Additional analysis including failure mode categorization (Appendix H and computational efficiency with varying frame counts (Appendix]) [ are also available the appendix.
Table 2: Single-step evaluation of VLM models on OWMM core multi-modal capabilities. The DWMM-VLMM338 model achieves the best performance across all metrics, demonstrating its superior ability to integrate scene understanding, decision-making and action generation. Since PIVOT and RoboPoint are designed for single image, we also report the single image grounding results for fairness.
Model/ Task Score
Ego-centric Decision- making?
Image Retrieval*
Affordance Grounding (objectit
Affordance Grounding (reeeptaclo)
Affordance
Time
Grounding
Consumptionis)
(navigation)
OWMM-VI.M-BR MAW--VIIM-38///oo (ours)
97.85%
87.54%
(-9.
0.94(=0.19)
0.88(1:.11)
36.58
WWMM-VLMMBB(ouny)
96.72%
79.04%
0.93(=0.11)
0.91(2::20)
0.83(#0 21)
16.58
GPT-4e
48.53%
46.46%
0.56 10.38)
0.35(100.0)
0.07
21)
160.74
Intern12.55888
17.52%
1.27%
0.09(10.19)
0.18(10.31)
0.14(
16.06
GPT 4o+PIVOT
25
52.72%
55.38%
0.67 (L0.38)
0.45(10.44)
0.05( ..05(1..18))
22.91
GPT HPT--4o+Robopiing
+
49.56%
49.72%
0.64(20.41)
0.38(=0042)
0.06(20.20)
14.19
Test of Single Image Grounding(")
Robopoint +
0.91 33)
0.83(#0.11)
0.72(40.11
PIVOT(GPT-fo) 35
0.86(20.13
0.84(#0.12)
0.74(#20113)
Table 3: Agent success rate in OWMM Task. OWMM-VILM-38B model consistently outperforms
others across all metries. Method
Task
Image Re- Objjec)
Robot close to Object
Object Picked
Image
Robot close to Goal
Dead Loop
trae))l
OWMM-VT VL.M-.18((MMM
21.90%
88.56%
84.64%
38.56%
30.39% %
23.53
0/308
OWMM. VI. (ours)
9.45%
81.43%
74.59%
17.92%
15.96%
10.42%
0/3088
GPT-4o+PIVOT
0.33%
55.15%
10.13%
0.65%
0.33%
0.00%
195/308
GPT-4o-Robopoint
56.86%
1.31%
0.00%
Experiment
lenient distance tolerance
OWMM: VLM VI.M-38B(urr)) 38B(ours)
51.52%
89.23
88.22%
62.96%
51.52%
44.78%
0/308
OWMM- VLMM-B (ours)
38.59%
83.22%
$2.35%
39.92%
0/308
GPT-1o+PIVOT
1.68%
60.27%
12.12%
5,39%
1.68%
1.35%
204/308
GPT Ro-Robppain
3.03%
52.86%.
10.10%
4,04%
2.69%
1.35%
209/308

---

5.1 Single-step Evaluation
In the single-step evaluation, we assess three core VLM capabilities for the OWMM task: 1) Egocentric Decision-makinn making: We evaluate the success rate of choosing correct action categories. 2) Image Retrieval: We measure the image retrieval success rate. 3) Affordance Grounding: Instead OWMM-VI MLM outputs a bounding box, from which we compute the center as the target point. With the target point, we compute the score for affordance grounding by s Σ, 1 vatia x (1 norm pwr__dist)) where 1 vatia(() is the indicator function of whether the model generates: an action matched with ground truth and a valid bounding box or point on the i th test case. 1 if both conditions are satisfied simultaneously, and 0 otherwise. norm_dise E [o, 1] is the distance between the predicted target point and the ground truth point, normalized by the diagonal of the image. In short, E [0, 1] measures VLM's s ability to generate accurate grounding with the correct format. Higher scores indicate better performance.
of predicting points directly like in [2 25
Regarding the baseline methods, we have evaluated both 1) multitasking foundation VLM models, including GPT-4o [1] and Intern 21.-2.5-8B that share the same unified input and output configuration as ours and 2) modularized agent with multiple models, including GPT-4o+PIVOT235 and GPT- 4o+Robopoint For Robopoint and PIVOT, which specialize in grounding. GPT-4o serves as the higher-level module for decision-man decision-making making and image retrieval. If GPT-4o's s actions need grounding, its outputs are combined with task details as input to Robopoint and PIVOT for grounding.
The results are reported in Table Our model excels in decision-making, achieving state-of-the-an results in image retrieval and affordance grounding. GPT-4o and Intern VL2.5, as generalist models, perform poorly at affordance grounding. In contrast, RoboPoint and Pivot that concentrated on affordance grounding, exhibit capabilities on par with our model in this task, indicating that existing specialized approaches already provide good effect on robot's 's action affordance.
Moreover, our model demonstrates marked improvement over GPT-4o in decision-maker tasks. This advantage directly translates into higher overall accuracy compared to methods that employ GPT-4o as the agent. In other words, using the data from our data synthesis pipeline conduct supervised fine-tuning yields a significant enhancement in robotic decision-making performance.
5.2 Episodic Evaluation
In episodic evaluation, we assess how well cach model completes OWMM task episode in the simulator. Task success is measured by placing objects in goal receptacles using distance thresholds of 0.85m or 1.7m. The 0.85m threshold relates to half the average diagonal length of goal receptacles' 3D bounding boxes in our test set.
Additionally, we introduce three metrics to assess subgoals: 1) Image retrieval: Success rate in locating object and goal receptacles from multiple posed images. 2) Object Picked: The success rate of the robot grasping item when its end effector is either within 0.15m or 0.8m of the target, with the latter matching standard HomeRobot setups [40]. 3) Robot close to: The success rate of robot staying within 1.5m or 2.0m of the object or goal receptacle before picking or placing. Additionally, we propose the "dead loop" metric to quantify the number of cyclic stagnations occurring during test episodes. As mentioned in 5.1 GPT-4o may erroneously output image retrieval decisions when the expected action is navigation, thereby inducing cyclic stagnation. Detailed experimental results presented in Table 3 See Appendix (FFendxxxFFffooro extra details about evaluation settings. We provide comprehensive failure mode analysis categorizing 100 failed episodes in Appendix H
5.3
Real world Evaluation
In our real-robot experiments, adopted the mobile manipulation system described in Robi Butler[35] within real-world home environment. For safety reasons, we cannot allow the agent to fully operate the fetch robot in the real world. When OWMM-VLM generates multi-modal action to execute, the agent prompts the visualization of the action and waits for human confirmation, and the fetch robot only executes the action with human consent.
We first had the robot navigate through the scene with human control to perform SLAM process. We then select 10 test samples from the sequence of the robot's 's head view during its run. We used human operators to judge the model's 's output according to several criteria: whether the chosen action was correct, whether the predicted affordance was accurate, and whether the target was reachable, among

---

other factors. The results of these experiments are presented in Table The results show that the model trained on synthetically generated data in the simulator also demonstrates strong zero-shot generalization capability in real-world scenarios. Table presents the agent action prediction result on real-world data.
Table 4: Real world single evaluation. OWMM-VLM-388 model achieved the best performance, and DWMM- M-MM-L------ VLM-8B model also outperformed the baseline. While the baseline model demonstrated relatively strong affordance grounding capabilities for objects, its poor performance in action decision- making led to incorrect navigation.
Method
Image
Affordance
Affordance Grounding(auvvigaioon)
Total Acc
Retrieval
Groundingobjet&keeeptadd
OWMM-VLM-38EB VLM333B(UrS))))
7/10
10/10
10/10
90.00 %
OWMM-VIAM VL-MBB (ours)
5/10
10/10
9/10
80 0.%00%
PIVOT GPT Robbpppiin
8/10
6/10
0/10
46.67%
8/10
6/10
0/10
46.67%
Table 5: Demonstration single step evaluation in real world. These demos showease OWMM-VLM VL-M38B- outputs, illustrating that even though its training data are drawn entirely from our data-synthesis approach in the simulator, the model delivers outstanding decision-makinn and affordance-groumee performance in real-world settings.
Model's Outpur Action
Pick
Place
Nav to point
Task Description
Move the NutriSoy Bean Milk Box from the Minimalist Black Workstation Desk the White Rectangular Office Mocting Table.
Move the banana from the black desk the White Rechangular Office Meeting Table.
Move the chip box from Genuine Leather Sofa to the white table.
Context Description
The task has started I am getting closer to the Minimalist Black Workstation Desk where the NutriSoy Bean Milk located.
The task started and have navigated the black desk and picked up the banona. I am getting closer the White Reciangular Office Meeting Table where should place the banana.
The task has started and I am getting closer Genuine Leather Sofa where the chip box
Action Information
[576, 263, 769, 548]]
[[0, 445, 1000, 999]]
[[539, 978, 578, 999]]
Conclusion
In this paper, we introduced OWMM-Agent, a novel agent architecture featuring the DWMM-VLM. a vision- languala model fine-tuned via a simulation-based agentic data synthesis pipeline for Open- World Mobile Manipulation (OWMM) tasks. This approach enables the VLM to learn state tracking, multi-view reasoning, and multi-modal action generation grounded in global scene understanding and agent embodiment. Extensive experiments demonstrated that our DWMM-VLMM particularly the 38B variant, achieves state-of-the-ard performance in single-step multi-modal capabilities like egocentric socision-making and affordance grounding, outperforming generalist VLMs and specialized robotics models. Episodic evaluations in simulated environments further confirmed the OWMM-Ageni s superior success rates and robustness against common failure modes like dead loops, while real-world tests on a Fetch robot indicated strong zero-shot generalization. Ablation studies underscored the importance of our design choices, such as bounding box prediction and integrated reasoning, and revealed that while data scaling is crucial, egocentric spatial intelligence can be learned effectively even with limited object and scene diversity data volume is sufficient. Future work will focus on addressing limitations like pre-mapping reliance and enhancing cross-embodiment adaptability for more complex manipulation tasks. Please also refer to the appendix for discussions about the potential impact of this research in Appendix A and extended discussions on limitations in Appendix B
10

---

Acknowledgements
We sincerely thank Anxing Xiao and David Hsu from the National University of Singapore for their crucial support and guidance in the successful real-world deployment of this project, as well as their generosity in providing the Fetch robot.
For the funding support, this paper is partially supported by the National Key R&D Program of China No.20222D0161000.
References
[1] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. piO: A issionlanguaage-ctiinn flow model for general robot control. arXiv preprint 241:::2:10.244444
2024.
[2] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297 2024.
[3] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[4] Kevin Chen, Junshen K. Chen, Jo Chuang, Marynel Vazquez, and Silvio Savarese. Topological planning with transformers for vision-and -ada--uud- navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11766- 11266, 2021,
[5] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 242.02241..05271, 2024.
[6] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Rey- mann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Roboties and Automation (ICRA), pages
IEEE, 2022.
[7] Hugh Durrant-Whii Whyte and Tim Bailey, Simultancous localization and mapping: part i. IEEE robotics & automation magazine, 13(2):99-110, 2006.
[8] Enric Galceran and Marc Carreras. A survey on coverage path planning for robotics. Robotics and Autonomous systems, 61 227-1276, 2013.
[9] Giorgio Grisetti, Cyrill Stachniss, and Wolfram Burgard. Improved techniques for grid mapping with pao-blackwellizer particle filters. In IEEE Transactions on Robotics, volume 23, pages 34-46, 2007.
[10) Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.016622 2024.
[11] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
[12] Charles C Kemp, Aaron Edsinger, Henry M Clever, and Blaine Matulevich. The design of stretch: A compact, lightweight mobile manipulator for indoor human environments. In 2022 International Conference on Robotics and Automation (ICRA), pages 3150-3157. IEEE, 2022.
[13] Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X Chang, and Manolis Savva. Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16384-16393, 2024.

---

[14] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, al. Openvla: An open-source vision-language-actiin model, arXiv preprint 2XXi::2406009246 2024.
[15] Yeseung Kim, Dohyun Kim, Jieun Choi, Jisang Park, Nayoung Oh, and Dachyung Park. A survey on integration of large language models with intelligent robots. Intelligent Service Robotics, 17(5):0011-110 2024.
[16] Junyi Li, Weijian Wang, Haotian Zhang, Haotian Liu, Deva Ramanan, and Yong Jae Lee. Inference-time image token compression for vision-language models. arXiv preprint, 2024.
[17] Jacky Liang, Wenlong Huang. Fei Xia, Peng Xu, Karol Hausman, Brian lchter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500, IEEE, 2023.
[18] Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulle robotic manipulation through baske visual prompting. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024.
[19] Peiqi Liu, Yaswanth Orr,, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Ok-robot: What really matters in integrating open- ppen-knowledge models for robotics. arXiv preprint arXiv:2401. 12202, 2024.
[20] Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey vision- language-action models for embodied ai. arXiv preprint 240.:22000.1099.. 2024.
[21] Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 16488-16498, 2024.
[22] Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. Pddl-the planning domain definition language. 1998. https//////
[23] Andrew Melnik, Michael Bütteer, Leon Harz, Lyon Brown, Gora Chand Nandi, Arjun PS, Gaurav Kumar Yadav, Rahul Kala, and Robert Haschke. Uniteam: Open vocabulary mobile manipulation challenge. arXiv preprint arXiv:2312.086711 2023.
[24] So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhut- dinov. Film: Following instructions in language with modular methods. In International Conference on Learning Representations (ICLR), 2022.
[25] Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872, 2024.
[26] Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung- Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13721
2023.
[27] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang. William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9982 9991, 2020.
(28) Dicong Qiu, Wenzong Ma, Zhenfu Pan, Hui Xiong, and Junwei Liang. Open-vocabulary mobile manipulation in unseen dynamic environments with 3d semantic maps. arXiv preprint arXiv:2406. /8/15, 2024.
[29] Ri-Zhao Qiu, Yafei Hu, Ge Yang, Yuchen Song, Yang Fu, Jianglong Ye, Jiteng Mu, Ruihan Yang, Nikolay Atanasov, Sebastian Scherer, et al. Learning generalizable feature fields for mobile manipulation. arXiv preprint arXiv:2403.07563 2024.
12

---

[30] Fabian Schmalstieg, Daniel Honerkamp, Tim Welschehold, and Abhinav Valada. Learning hierarchical interactive multi-object search for mobile manipulation. IEEE Robotics and Automation Letters, 2023.
[31] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10740-10749, 2020.
[32] Ioan Sucan, Mark Moll, and Lydia E Kavraki, The open motion planning library. IEEE Robotics & Automation Magazine, 19(4):72-88) 0222-22,22 2012.
[33] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. 12213, 2024.
[34] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 09288, 2023.
[35] Anxing Xiao, Nuwan Janaka, Tianrun Hu, Anshul Gupta, Kaixin Li, Cunjun Yu, and David Hsu. Robi butler: Multimodal remote interaction with household robot assistant, 2025. URL
https://arxiv.orr org. abs 20548
[36] Chenlin Xie, Zhenhua Dong, Prateek Sharma, and Tianyi Chen. Advancing Ilm inference efficiency through quantization-aware training. arXiv preprint, 2024.
[37] Haoyu Xiong, Russell Mendonca, Kenneth Shaw, and Deepak Pathak. Adaptive mobile manipulation for articulated objects in the open world. arXiv preprint urXiv:2401.14403, 2024.
[38] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
[39] Jihan Yang, Shusheng Yang. Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412. 14171, 2024.
[40] Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yen Yang. Vidhi Jain, Alex William Clegg, John Turner, Zsolt Kira, Manolis Savva, Angel Chang, Devendra Singh Chaplot. Dhruv Batra, Roozbeh Mottaghi, Yonatan Bisk, and Chris Paxton. Homerobot: Open vocab mobile manipulation. In Conference on Robot Learning, 2023. URL http::tttt//aaeeeee www.aiiww.aaaaaaaaa..o.aaoo httggabb/23006. 2t0.//23........... 11565
[41] Sriram Yenamandra, Arun Ramachandran, Mukul Khanna, Karmesh Yadav, Jay Vakil, An- drew Melnik, Michael Büttner, Leon Harz, Lyon Brown, Gora Chand Nandi, et al. Towards open-world mobile manipulation in homes: Lessons from the neurips 2023 homerobot open vocabulary mobile manipulation challenge. arXiv preprint 2eXiv:2407.06933 2024.
[42] Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406. 10721, 2024.
[43] Peiyuan Zhi, Zhiyuan Zhang, Muzhi Han, Zeyu Zhang, Zhitian Li, Ziyuan Jiao, Baoxiong Jia, and Siyuan Huang. Closed-loop open-vocabulary mobile manipulation with gpt-4v. arXiv preprint 2xXv:2404.100200 2024.
[44] Fengda Zhu, Xiwen Liang, Yi Zhu, Qizhi Yu, Xiaojun Chang, and Xiaodan Liang. Soon: Scenario oriented object navigation with graph-based exploration. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12689- 12699, 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main claims, including the proposal of a novel multi-modal agent architecture for open-world mobile manipulation (OWMM), the introduction of an agentic data synthesis pipeline for instruction fine-tuning a vision-language model (VLM), and achieving state-of titte-of-hheann the-art art performance validated by extensive experiments. These claims align well with the detailed methodology and experimental results presented in the paper (Sections 3...
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper has a whole section Bito to discuss the limitations of our pipeline of OWMM-Agent as well as the fine-tuned OWMM- VLM model.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
14

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper does not present theoretical results requiring formal assumptions or proofs. focuses on trying to a solve a significant real wollw problem by system design, data synthesis, model fine-tuning, and analysis on evaluation.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides detailed descriptions of the dataset construction 4.1 model architecture 3.3 tinettunig procedure C experimental setups [3.1] both in simulation and real-world environments. Besides, the project repository in the simulation is open-sourced and anonymous in the given link. These information is enough reproduce the results. However, we also admit that for the real-world experiment, depending on the hardware setup, the embodiments, the real environment, the reproduction is challenging.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
15

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The data and code open-sourced and anonymous in the given link in abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/gust cerec/gguiis//eeeee Stte///ouiiniiioooooooo.oooooeae
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies the experimental settings/details in section 5 with addi-
tional settings in Appendices and F and additional ablation experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
16

---

Answer: [Yes]
Justification: For continuous-valiie metrics, we report the standard deviation as errоr bars in Table [ Table O and Table For binary success metrics, error bars are suitable and are therefore not reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard crror of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides the compute resources used for training, the GPUs and training hours in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://///eaaaessccc/pllll//eeaa
Answer: [Yes]
Justification: The paper follows standard research ethics in robotics and machine learning. Specifically, this research does not involve human interaction nor collect human data.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
17

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: This paper presents the broader impact of this project in Appendix A with ethical concerns, societal impact discussed in detail.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not release models data with known high misuse risks and does not discuss safeguards. The incremental data used in instruction fine-tuning are synthesized from open-sourced simulator. The used base model Intern VL-2.5 (5) is also open-sourced.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
18

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites all datasets and models used, such as Habitat simulation, YCB and Google Scanned Objects datasets, InternVL-22. model, and others, with appropriate references. Licenses are not explicitly stated but standard academic citations are provided.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces large-scale synthesized dataset for instruction fine- tuning OWMM- MLM, with detailed descriptions of dataset construction, size, content, and annotation process in section 4.1 and D Besides, the dataset is also released in the link provided abstract.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or human subjects research.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
19

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve human subjects research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
The paper explicitly describes the use large vision- language models (VLMs) as core components of the OWMM agent architecture and fine-tuning process. We also fol- low the neurips L.M policy to describe the usage of VL.Ms both in methodology section 3.3 and in experimental implementations section 5.1
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy httt:p/:/mnarrrpssecc//oofrreeee for what should or should not be described.
20

---

Appendix
A
Impact Statement
This work contributes to the long-term vision of creating generalist household robots capable of assisting with daily activities in homes and other human-centric spaces. Ethically, deploying such systems raises considerations regarding safety, privacy, and workforce displacement. Ensuring safe interactions with humans and securing data used for training are critical priorities. In addition, while automation may replace certain household jobs, it also creates opportunities for new roles in robot design, deployment, and maintenance. Future societal implications include increased accessibility to robotic assistance for individuals with disabilities limitations through continued research into adaptability and real-world robustness, OWMM- MLM can pave the way toward more inclusive and effective robotic solutions for societal benefit.
aging populations. By addressing current
B Limitations
this work, we have proposed novel embodied agent architecture with foundational VLM model to address the open-world mobile manipulation problem. However, we also identify some limitations of our approach.
Pre-mapping:
Although our method does not require 3D reconstruction of the environment, we still assume pre-mapping phase with camera pose graph and 2D occupancy map for path planning in navigation.
Complex manipulation: Following the grasping setup in 40 agent and model can be directly applied robot with suction as end effector. However, our model fells short in the circumstances when the robot needs to control complex end effectors like dexhands.
Cross-embodiment: As demonstrated spatial understanding and reasoning. However, when deploying the model onto other robots with different mechanical compositions such as maximum arm stretch distance, our model could fail, i.c. the cross-cmbodde emboodmebbd issue.
experiments, our model learns the object-scale prior for
C Implementation Details
Regarding the model's architecture, we have trained two variants consisting of billion and 38 billion parameters, based the pre-trained model from IL-em_V--2.555.. The 8B model is composed of ViT-30M and IntemLLM-2.5-7BB
and the 38B model is composed of Intern ViT-6B and OWMM dataset. The OWMM-VLNM
Qwen2.5 38] We directly finetune the base model model is trained to autoregressively generate the response tokens consisting of the output action and its corresponding task context in JSON format. Specifically, only adjust the parameters in MLP and LLM. As for the training time, OWMM-VLM-8D is trained on 8X NVIDIA A 100 GPUs for about 7 hours, and OWMM-VLM-3880
freeze the parameters in VIT and
trained on 24X NVIDIA A100 GPUs for about 18 hours. Both our models were trained for epoch. For the testing, we deploy OWMM-VZA MLM and RoboPoint 42 locally and use the openAI API access GPT-4o and 25V
C.1 Pre-mapping
This section provides comprehensive implementation details for the pre-mapping stage and camera pose selection, which essential for reproducing approach.
Simulation Environment In simulated experiments, we utilize the navigation mesh provided by Habitat environment, where navigable locations are represented mesh triangles. This pre-computed
21

---

navmesh serves map for localization and navigation, similar to game engines. The navmesh provides collision-free free navigation space and enables efficient path planning for the mobile base.
Real-World Deployment In the lab environment, run RobiButler [35 over Fetch Lidar input for the pre-mapping stage and localize the robot geometrically. More specifically, RobiButler uses Gmapping [9] algorithm to compute the 2D occupancy map from lidar data. The map is further used to localize the robot and camera poses during task execution. This approach is practical and widely adopted in commercial robotic systems such robotic vacuum cleaners.
C.2 Camera Pose Selection
Considering the context length of current VLMs, we select 8 views from the pre-mapping stage plus 1 current egocentric view our experiments, as demonstrated Figure This design balances scene coverage with computational efficiency.
Simulation As detailed in Section 4.1, we employ a strategic sampling approach:
First, we position the robot the location of the receptacle where objects were initially located and at the goal receptacle, sampling the robot's 's head-view images. This ensures coverage of task-critical locations.
Subsequently, we randomly position the robot and capture its head-view images to ensure sufficient scene coverage while maintaining computational efficiency.
3. This sampling strategy guarantees that both the initial and goal regions are represented in the pose graph while providing diverse viewpoints of the environment.
Real Robot Deployment In real robot experiments, we extract keyframes from the ROS bag RGB data and manually select the frames related to the task to guarantee enough scene coverage. This manual selection serves as a proxy for a ground-truth frame selection mechanism.
We acknowledge that autonomous and online frame selection from videos are critical for full autonomy but is beyond the scope of the current paper. This could represent promising direction for future work to enable fully autonomous mobile manipulation in dynamic environments.
D Details of Datasets
sectionDetails of Datasets
D.1 Extra Dataset Construction Details
Our evaluation pipeline is constructed using the HomeRobot [40] framework, which serves software structure designed to enable comprehensive benchmarking in both simulated and real- wolldworld settings. Specifically, we use the simulation part of HomeRobot project, built on Habitat platform 226
with 200 scenes, 150 categories, and 7,892 object instances.
The original episodic data in HomeRobot are generated with Stretch Robot [12), which has special telescopic arm instead of a normal articulated arm with rotary joints. This adds additional difficulty in base control as it requires the mobile chassis to rotate accurately align the with the target object for manipulation. However, the baseline VLMS and methods we are going to compare with are designed for robots with conventional articulated arms 42 25 providing a broad range of chassis poses that allow for successful arm manipulation.
Therefore, recreate the OWMM episodic training and testing datasets in the simulation using the Fetch Robot, which is a mobile robot equipped with a standard articulated arm and has also been integrated into the Habitat platform.
We partitioned the scenes into training and testing sets using a ratio of 113:30. Besides, we allocated 157 objects between the training and validation sets with a ratio of 137:20, ensuring that the testing set contained entirely unseen objects. This division resulted and 4k testing data entries, establishing robust dataset for training and testing in our OWMM task.
total of 152k training data entries
22

---

Keyframe Sampling Strategy In the dataset construction pipeline, we first sample key information each step. This information included the robot's coordinates, current action, the positions objects and receptacles, and the extrinsic parameters of the robot's 's head-view camera. In particular, at this stage, we did not collect the robot's head-view images to enhance the data collection efficiency. We recollected the robot's head-view images of these steps within the simulator after applying the selection strategy.
For navigation actions, among all steps that the robot is moving, we select the step that the receptacle visible from the robot's 's head-view image as the start point of the navigation action. The point at which the robot stops moving is considered the end point of the navigation action. Within these steps, we sample the waypoint step data at specified intervals.
For grasp and manipulation actions, we select the first three frames during which the robot executes the action the pre-defined action data.
Data Filtering Pipeline The data filtering pipeline ensures the following criteria:
For navigation actions, both the receptacle and the next waypoint are within the robot's head-view image.
For grasp actions, the object to be grasped is reachable by the robotic arm, and the object is within the robot's head-view image.
For manipulation actions, the receptacle intended for object placement is reachable by the robotic arm, and the receptacle is within the robot's head-view image.
After filtering, approximately 10% of initially collected frames are rejected, ensuring high annotation quality.
Linguistic Diversity Enhancement To enhance the diversity of the dataset, we paraphrased reasoning and summarization parts of the answers using GPT-4o mini, while keeping the action annotations and affordance bounding boxes unchanged. This process increases linguistic diversity by generating 3-5 paraphrases per template, resulting in more robust language understanding without additional manual annotation effort.
sppon Crayons
cone MD
COLE
table
rable
Uigo StaraO e desk
HIGEE Lack
Gamen
eego
Color
Bowl
wou lock
Lightts tand
ball
Bal Ballet Flats
Snake
de
dressen
pepo
Ink
Clamp
Coffee
ha
Canon
Tie
ountt
tr
Souter
Tdgepack
large Friends
Thbillas
Leather S to
Whit L
a
Tagebedd
&
chair
cabinet
bookCase
box
Leathe
toy
d
Lane
OZ
eolered SCr screwdriver
Magento mi Pirennn
b
Wooden
pon Sale
(a) Object Categories
(b) Receptacle Categories
Figure 4: Word Cloud Distribution of Objects and Receptaeles in our dataset. Object categories show diverse household items (tools, food containers, toys), while receptacles include varied furniture types (tables, shelves, cabinets) with rich descriptive labels.
D.2 Analysis on the training data
This analysis tries to answer two questions: How does the diversity of objects and environments affect the model's performance on unseen objects and environments in the test set? We examine dataset diversity using three 45k-sample sets: 100% scenes and objects, 100% scenes with 30% objects, and 30% scenes with 100% objects. We control the total number of training samples while changing the number of object instances or scenes appearing in the training data. 2) How does the model's performance change as the training data scales up? For data scaling, we use five data sizes: Ok (no fine-tuning), 15k (10%), 45k (30%). 76k (50%), and 152k (100%). Ok, we give the Internyl-2.5 -BB model limited input-output pairs, allowing it to generate structured outputs via
23

---

1.0
Task Score
0.6
0.4
Image Retrieval
Ego-centric Decision-making
Affordance Grounding (Average)
0.0
0%
10%
30%
50%
100%
Training Data Size (%)
Figure 5: OVMM-VLM-8B Sub-task Performance with the Increase of Training Data Size. The task scores consistently improve as the training data size increases.
in-context learning. We evaluate the performance in image retrieval, egocentric decision-making. and three affordance grounding subtasks. Results are shown in Table 6nd Figure 5
The results for the first question show that object and scene diversity have negligible effects on metric fluctuations remain within 5% range. For the second quesion, data scaling is crucial for enhancing OWMM-VLM's performance. As seen Figure increasing the dataset from Ok to 152k samples shows logarithmic improvement, especially at lower sizes (Ok to 15k, and 15k to 45k). However, benefits diminish near 152k. While larger datasets aid generalization, marginal gains decrease beyond threshold. As performance gains plateau, egocentric decision making approaches a success rate of 1.0, whereas image retrieval lingers at approximately 0.8. This difference is likely due to the model's limited capacity with 8 billion parameters. We also draw two extra observations from the experiment:
multi-modal capabilities,
1) The embodiment prior for deciding the current action based on the ego-centric RGB image, especially how close the robot should to interact with the target objects, can be learned data-driven approach.
2) The ability comprehend multiple images the multimodal context length may present one of the bottlenecks for VLM models to function as the core cognitive model for intelligent robots, particularly when scene-level understanding is essential.
Table 6: Results with different data diversity data scales. The best performance across training sets with different scales is indicated with bold font. Besides, underline highlights the best performance across three 4am-le training sets with different diversity.
Data Composition/ Task Score
Ego-centric Decision- makingt
Image Retrieval!
Affordance Grounding (object)
Affordance Grounding (receptacle)
Affordance
Grounding
(navigation)]
0k(0%)
17.52%
1.27%
0.05
0.18(=0.31)
0.14
15k(10%)
73.27%
41.36%
0.69(10.43)
0..2(=0.20)
0.45(11.41)
45k(30%))
91.01%
70.68%
0.88(110.2)
0.84( 3.333)
0.74(10.31)
45k/100% sccne 30% object)
91.56%
71.95%
0.87 26)
0.89(=0.22 23)
0.72(+0.33)
45k(30% scene 100% object)
88.96%
69.12%
0.87(110.2)
0.84(20.31)
0.69(10.36)
76k(5056) 152k(
95.79%
76.20%
0.91(1.0.1))
0.88(11..2
0.84(+0.20)
79.04%
0.93(10.14)
(29002))20)
0.63(10.21)
E Details of Baseline Setting
As Robopoint and PIVOT are designed for single-image QA task, we adjusted some settings to enable them fully utilizing their capabilities under the OWMM task.

---

E.1 Single Image Grounding
For the single-step evaluation, we first extracted robot's task instruction from the original prompt of the current step. Based on the ground truth action of the current step and whether the robot picks up an object, we designed new task instructions, as shown in Table 7 For Robopoint, we appended the
Table Redefined Task Instructions. [object item), (target rec) and (goal rec) are from robot's task instruction.
Ground Truth Action Pick
New Task Instruction
The robot needs pick {object item) on (target rec)
Place
The robot needs to place (object item) on (goal rec)
Nav to point(object picked)
The robot needs to navigate closer to the [goal rec) for placing (object item)
Nav to picked)
point(object not
The robot needs to navigate closer to the (target rec) for picking (object item)
following context: "Find few spots for robot execute the action. Your answer should be formatted as a list of tuples, i.e. [(x1, y1), (x2, y2), ...]. where each tuple contains the x and y coordinates of point satisfying the conditions above. The coordinates should be between 0 and indicating the
normalized pixel locations of the points in the image." This configuration aligns with Robopoint's 's original settings.
For PIVOT, we configured the following parameters: n sampless sotsss_iiitio,.t n samples_ _otllle__p__,______ n_ i_iters=2.. In evaluation settings, the imput consists of single RGB image and task instructions, randomly sample initial points in the image from a 2D Gaussian distribution. The distribution is parameterized with mean of (256, 256) and standard deviation of (100, 100).
E.2 Agent Setting
We employed GPT-4o for agent Ceastruction.PP--S first receives our instruction inputs and returns JSON-formatted responses. When gpt's output action is "search scene frame". we directly adopt GPT-4o's 's response as the agent's current-step output. For actions "nav to point", "pick", or "place", the system sends both the action name and robot's current-view RGB image (single frame) to Robopoin/PIVOOT for action affordance. The reformulated task instruction sent to Robopoint/PIVOT follows this template:
"The robot needs to taask_intrruction Now the robot needs to ppp__oouut__ctionn crobot_history )"
where (task instruction) is the original task instruction, (gptt [appnnnpptt_ttppppppeee_e output_action) is gpt's output ac- is the summarization of previous step. In single-step evaluation, Robopoint and PIVOT process these new task instructions using the same methodology described in Appendix [E. episodic evaluation, we transmit depth information to PIVOT while maintaining consistency with its original configuration.
F
Extra Details of Episodic Evaluation
As mentioned in section 5.2 we designed the following metrics for episodic evaluation. More detailed specifications of these metrics are outlined below:
Object to Goal Distance: We used the object to goal distance as the metric to determine whether objects are successfully placed in goal receptacles. To establish appropriate thresholds, we first calculated the 3D bounding box diagonal distances of all goal receptacles in the test set, filtering out those with distances less than 0.75m or greater than 3m. 3m. am.Table show some examples of goal receptacles in our test sut.eubsey, we computed the average diagonal distance (1.7m) from remaining valid receptacles. Based on this value, we selected half of the average (0.85m) as the strict threshold criterion and the full average (1.7m) threshold approach ensures successful placement recognition when robots position objects near goal anceptacles,and reasonable constraint boundaries to prevent excessive leniency evaluation.
the relaxed threshold criterion. This
25

---

Table 8: Example Goal Receptacles in our Test Set. The numbers in the figure represent the diagonal distances of the 3D bounding boxes of the receptacles. This indicates that there significant variation in the sizes of the goal receptacles in the test set.
Circular table, Small oak 0.789m
Hisa Wooden Console 1.655m
Silver Picardy Bod 2.504m
tall sideboard 2.931m
F.1 Simulation
For simulation step, the robot state delta is calculated by forward kinematics, as implemented by the Habitat 3.0 environment 26). The robot state and observations updates can be expressed mathematically as:
X = fx(x,, a,, At)
1+1 Di+1 fots
where stands for robot current state (e.g., joint angles, positions), a, stands for velocities, and fx represents the kinematic model function that computes the next robot state within the discretized time step of duration At. fots is the observation model function, decided by the sensor link forward kinematics function and camera model.
G Ablation Study on OWMM-VLM
The ablation study evaluates the contributions of the components of the OWMM- model. We focus on grounding output formats, comparing the bounding box and point coordinate, and we assess examine the beam
the inclusion of reasoning and summarization in the outputs. Furthermore, search option provided by the base model Internvl-2.5-8B(J] The results are in Table
From the table, we have these observations and indications:
Beam Search. Beam search decoding algorithm widely used in language generation, main- taining a beam number of top candidate sequences at each step. Beam search enhances Ego-centric ppcision-making and Affordance Grounding tasks, with minimal impact on Image Retrieval, but increases temporal and spatial overhead in inference, especially on the 38B variant. Hence, its effect briefly shown only the ablation study.
2) Grounding Format. Replacing bounding box predictions with direct output coordinates reduces performance in Affordance Grounding, especially for objects (0.9251 0.6542) and receptacles (0.9060 - 0.6479). It is postulated that the large-scale visual grounding data in the pre-trained model allow model to utilize this prior knowledge. The consistency in output format between the base model and the instruction fine-tuning dataset aids the training process.
3) Reasoning and Summarization. Removing reasoning and summarization capabilities leads to the worst performance across most metrics, with decrease in Image Retrieval (0.7904 - 0.6586) and Ego-centric Decision-mal makinng (0.9672 - 0.9049). This highlights the critical role of reasoning and summarization maintaining contextual coherence and task understanding.
Table 9: Ablation Study on OWMM-VLM. The best performance is indicated with bold font.
Model Ablation
Ego-centric Decision- making?
Image Retrieval!
Affordance Grounding (object)?
Affordance Grounding (recoptacle)?
Affordance
Grounding (navigation)
OWMM-VLM-BR
96.72%
79.04%
0.93 3033++0000 14)
0.91(20..20
0.83( +0.21
beam search
97.30 %
78.47%
0.97 140.14)
0.85( +0.18)
culpul
10.17)
reasoning and summarization
90.49%
65.86%
0.88 24)
0.831 10.33)
0.82[ 20.24)
26

---

H
Failure Mode Analysis
To better understand the limitations and bottlenecks of our system, conducted comprchensive failure analysis on 100 randomly selected failed episodes from the evaluation logs. Since the current evaluation pipeline does not support automatic failure analysis, manually reviewed the action sequences and categorized failures into four distinct types:
H.1 Failure Categories
Ego-centric Decision Making Error (36 %) These failures occur when the model makes incorrect pick/place navigation action decisions. Typical scenarios include:
The robot close enough to the target but outputs a navigation action, leading it move away and lose the target from view.
The robot is not close enough to interact but outputs pick place action, resulting in failed execution.
The model selects an inappropriate action type given the current spatial relationship between the robot and target objects.
Image Retrieval Error (28%) The retrieved image ID is incorrect, meaning the model selects posed frame that does not contain the target object or receptacle. This can result from:
Confusion between visually similar objects or receptacles.
Incorrect interpretation of the language instruction.
Hallucination when reasoning over multiple candidate images.
Affordance Grounding (Object/Receptacle) Error (15%) The robot is positioned close enough the target, but the center point of the output bounding box does intersect with the target object/receptacle, causing a failed pick or place action. Alternatively, the grounded object/recepacle incorrect despite correct spatial positioning.
Affordance Grounding (Navigation) Error (21%) The output bounding box for navigation corresponds to unreachable region (e.g., blocked by obstacles or outside the navigable space), or causes the target object/receptacee to disappear from the robot's egocentric view after navigation completes.
From this analysis, we draw two key conclusions:
Bottleneck in object picking/placing actions: While our model achieves strong performance in early stages (88.56% on object image retrieval, 84.64% on robot navigation to object), performance degrades significantly in later stages (38.56% object picking. 30.39% goal image retrieval). This suggests that manipulation affordance grounding remains critical challenge.
Error accumulation across subtasks: Failed early actions compound, making later subtasks increasingly difficult. This cascading effect accounts for the performance gap between single-step and episodic evaluations, as errors in object retrieval or navigation prevent the robot from ever reaching the manipulation stage.
I Computational Efficiency Analysis
To address concerns about real-world deployment efficiency and scalability to large scenes, conducted experiments evaluating GPU memory consumption and inference time with varying numbers of input frames.
We evaluated two model configurations:
OWMM- -LM-M- Single-step inference on a single A100-40G GPU with 2+1, 4+1, 8+1, and 16+1 frames (posed frames egocentric frame). The result is shown in Table I0
OWMM- VLM-38B: Single-step inference 4x 4xAA1000 A100-40G GPUs with 8+1, 16+1, 32+1, and 64+1 frames using parallel inference. The result is shown in Table 11
27

---

Table 10: OWMM-VLM-8B inference performance with varying frame counts on single A100-40G
GPU.
Input Frames
Prompt Tokens
Time (s)
Memory (GB)
2+1
3409.19
3.66
17.44
4+1
6057.19
3.67
17.77
8+1 (default)
11353.19
4.84
18.43
16+1
21945.19
7.09
19.75
Table 11: OWMM-VLM-388 inference performance with varying frame counts on 4x AX0000-00G GPUs with parallel inference.
Input Frames
Prompt Tokens
Time (8)
Memory (GB)
8+1 (default)
2810.37
4.39
98.22
16+1
4922.37
5.04
99.13
32+1
9146.37
7.34
100.65
64+1
17594.37
15.33
104.30
Scalability Both models demonstrate reasonable scalability: inference time grows sub-linearly with the number of frames due to efficient multi-image processing in modern VI.Ms. For the 8B model, doubling frames from 8+1 to 16+1 increases inference time by only 46% (4.84s 7.09s).
Memory Efficiency Memory consumption grows modestly with additional frames, the visual tokens are processed through the same encoder. The 8B model remains under 20GB even with 16+1 frames, while the 38B model stays around 100GB across all tested configurations.
Potential Optimization for Real-time Deployment While these timings reflect comprehensive processing (multi-image processing, chain-of-thought reasoning, and action generation), real-time deployment could benefit from optimization techniques:
Image token compression [6] Methods that select text-relevant image patch tokens signifi-
cantly reduce input token counts.
Model quantization [36]: Quantization-awarc approaches can accelerate inference while main-
taining performance.
Dynamic frame selection: As discussed in Appendix C.I adaptive frame selection can maintain
scene coverage while controlling computational costs.
J Qualitative Evaluation
We provide the qualitative evaluation of our OWMM-VLM model compared to other baseline models.
28

---

point
Table 12: Single step Qualitative Evaluation. The table demonstrates the single step qualitative evaluation
results: represent the ground truth; represent GPT-4o; represents RoboPoint; represents PIVOT; represents Intern VL base model; represent ours
29

---

Task Description
Context Description
Response
Move Schleich Allosaurus from the Uphol- stered Sofa Brown and Gold Accent Cabinet.
Task just started.
Heavy Long pads the Dark Wooden Tall Open Bathroom Cabinet to the moliiunctional games ta- ble.
navigated to Dark Wooden Tall Open Bathroom Cab- inct and picked up the Tena Pads Heavy Long pads.
Move hammer from the Multiple Drawer Short Boy to the Dark Wooden Tall Open Bathrom
The task has started and have navigated to Multiple Drawer Short Boy and picked up the
Move 065-b cups from the Unch metal and stool Magnolia Home Foundry Console Table.
The task has started and have navigated Unch bur picked up the 065-b cups.
Table 13: Single step Qualitative Evaluation Search Scene Frame. The table demonstrates the single step qualitative evaluation search frame results: represent the ground truth: represent GPT-4o; represents Inter base model; represent ours
30


Reviews:
[{"review_id": "j66YwgjfW7", "reviewer": "Reviewer_8pbX", "summary": "This paper proposes OWMM-Agent, a novel vision-language model (VLM)-driven agent for open-world mobile manipulation (OWMM) tasks. The core idea is to reformulate OWMM as a multi-turn, multi-image, and multi-modal reasoning problem. The proposed architecture comprises:\n1. A unified VLM (OWMM-VLM) that performs global scene understanding, agent state tracking, and high-level action generation from multi-modal inputs;\n2. An agentic data synthesis pipeline that generates scalable, instruction-conditioned training episodes in simulation, significantly reducing human annotation needs.\nExperimental results show state-of-the-art performance across both simulation and real-world settings, demonstrating strong zero-shot generalization and outperforming GPT-4o-based baselines.", "questions": "1. Pre-Mapping Assumption: The system assumes a pre-built pose graph and scene images. Could the model be extended to perform online mapping and actuation in tandem?\n2. Action Format: The model uses bounding boxes for affordance grounding. Are there failure cases where bounding boxes are insufficient ?", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of unifying scene understanding, state tracking, and affordance prediction into a VLM agent is novel and well-motivated.\n2. Strong generalization to real-world environments, using only simulation data, demonstrates practical significance.\n3. Experimental results are extensive: they include both single-step evaluations (with clear affordance grounding metrics) and episodic evaluation, including real-world robot experiments.\nWeaknesses:\n1. The assumption of a pre-mapping phase (pose graph and RGB map) weakens the claim of “open world.”", "was_revised": true, "final_justification": "Pre-mapping Assumption: The authors provided a thoughtful and technically grounded response to the concern regarding the pre-mapping requirement. While the current system assumes a pre-mapped environment, the authors clarified that the system can be adapted to an online mapping setup with minimal changes. I find this clarification satisfactory and believe it resolves the primary concern.\n\n\"Open-World\" Claim: The authors clarified that their use of the term “open world” refers to semantic diversity (e.g., unseen scenes, object categories, and instances), rather than unconstrained physical exploration. This aligns with common usage in prior work (e.g., HOMERobot challenge) and was transparently acknowledged in the paper’s limitations. I accept this framing, though a more prominent clarification in the main paper (rather than appendix) would have been helpful.\n\nAction Format: The authors conducted an ablation study comparing bounding box vs. point-based action outputs, showing that box-based actions consistently perform better. Their explanation—pretraining alignment and robustness to noise—is reasonable. While rare failures exist, these are due to recognition issues rather than the representation itself. I consider this issue resolved.\n\nThe authors were responsive and precise in their rebuttal, providing additional implementation details and future plans. The raised concerns were addressed satisfactorily, and the paper’s core contributions remain novel. I maintain a positive score.", "rebuttal": "We sincerely thank the reviewer for the time and effort devoted to evaluating our paper and for recognizing the strengths of our work, in particular the unified VLM agent, simulated agentic data‑synthesis, and extensive experiments. Apart from positive feedback, the reviewer also raised some concerns about the pre‑mapping assumption and action format. We would like to address these concerns below:\n***\n## Response to Weaknesses\n\n### Pre‑mapping and the “open‑world” claim\n\nWe agree that assuming a pre‑mapping phase may appear to narrow the scope of “open world”, as we have also discussed in **Appendix B Limitations (line 455)**.  However,  in our formulation *open world* primarily refers to the semantic breadth of environments instances, with 200 scenes, 150 categories, and 7892 object instances, especially when the agent is tested under novel environments and objects, as detailed in **AppendixD.1 (line 477)**. We inherit this setting from previous works such as[1] from Meta, as discussed in **Section 2 Related Works (line 82)**.\n\nBesides, we think the current system can be adapted to online system without pre-mapping. For example, running on an online SLAM system, the posed‑image graph we maintain can also be updated online while the agent acts: because scene images are sampled from the egocentric stream, the agent can resample from the most recent frames during execution and replace images tied to already‑executed actions. To increase the diversity and utility of the maintained image set, we can re‑score candidate frames and refresh highly redundant views with more informative ones. *We will add this discussion—and an ablation comparing **static** vs. **online‑refreshed** image graphs—to the appendix of the final version.*\n\n[1] Yenamandra, Sriram, et al. \"Towards open-world mobile manipulation in homes: Lessons from the neurips 2023 homerobot open vocabulary mobile manipulation challenge.\" Arxiv 2024.\n***\n## Answer to Questions\n\n### Pre‑Mapping Assumption — Can the model be extended to online mapping?\n\nYes. Our agent can be extended to **jointly update the posed‑image graph during execution** with minimal changes:\n\n- **Online refresh mechanism.** At each decision step, we resample from the newest egocentric frames captured during action execution and **swap out** frames corresponding to completed waypoints.\n- **Lightweight redundancy control.** We compute similarity scores between candidate frames and the current graph using efficient visual encoders (e.g., EfficientNet[2] or CLIP[3]) and **replace near‑duplicates** to preserve coverage and context without imposing heavy SLAM overhead.\n- **Practicality.** This keeps the memory/latency budget stable because we cap the graph size and trade redundant views for fresher, task‑relevant views.\n\nWe will include implementation details (resampling policy, replacement thresholding) and a small‑scale study in the appendix, comparing online refresh vs. static pre‑mapping in terms of success rate and per‑step latency.\n***\n### Action Format — Are there failure cases where bounding boxes are insufficient?\n\nIn **Appendix G**, we conduct an ablation on the action‑output representation (bounding box vs. point coordinates). Across OWMM tasks, **bounding boxes consistently yield higher success rates** than point‑coordinate outputs. We hypothesize two reasons: (i) the InternVL‑family pre‑training includes box‑grounded supervision, which better aligns with box outputs; and (ii) boxes carry local spatial context (extent/scale) that is more robust to calibration noise and detection jitter than a single point. **The detailed experiment results are in Table9 in Appendix G (line 604)**.\n\n**Failure cases.** We do observe *rare* failures with box outputs, typically when the target is an **uncommon or long‑tail object** (e.g., *Schleich Allosaurus*), where the error stems from **recognition/grounding** rather than the box representation itself.\n***\nWe appreciate the reviewer’s constructive feedback. We will incorporate the above clarifications in the final version. We also consider adapting the system to an online mapping setting could be meaningful future work. We are also looking forward to more discussions in the next phase.\n\n[2] Tan, M. & Le, Q. *EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.* ICML, 2019.\n\n[3] Radford, A. et al. *Learning Transferable Visual Models From Natural Language Supervision.* ICML, 2021."}, {"review_id": "wokKZiXvOP", "reviewer": "Reviewer_QZc6", "summary": "This paper proposes a VLM-based agent for open-vocabulary mobile manipulation. The authors formulate the instruction-following problem as letting the VLM reason over history observations, scene graphs (pose graphs), and egocentric observations. To induce reasoning capabilities, the authors selected several proxy tasks for training the VLM-agent and generated the datasets correspondingly. The results show improvement on both real-world and synthetic environments.", "questions": "See the strength & weakness section. Currently it's missing some details and therefore I'm leaning towards a borderline reject decision (with the hope that the revision/rebuttal could significantly improve clarity on model design and dataset construction).", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[+] The overall goal of this paper is well-motivated and is indeed an important problem to be studied.\n\n[+] The proposed dataset and the agent model are beneficial for OWMM-related research.\n\n[+] The authors tested not only in synthetic environments but also in real-world scenarios, indicating solid system design for real-world robots.\n\n[-] One major concern about this paper is the potentially missing details about the whole pipeline. As the idea of using graphs and VLMs for reasoning in open-vocabulary mobile manipulation tasks has already been discussed before, the distinction between this paper lies in the details of the modules that improve the model performance (e.g. CoT design, pose graph construction for sim and real environments, etc.). However, both the main paper and the appendix lack sufficient discussion on these details, which hinders further understanding of the current good performance of the model. This also makes the current dataset construction and experimental analyses a bit plain without much insights in addition to performance improvements.\n\n[-] I do feel the current version of paper writing could benefit from further revision and refinement.", "was_revised": true, "final_justification": "The rebuttal helps address major concerns on technical details, therefore I'm increasing my score to borderline accept.", "rebuttal": "We really thank the reviewer for the thoughtful review of our paper. We appreciate your acknowledgment of the well-motivated research goal, the benefits of this paper for related research community, and the value of real-world experiments. Apart from the positive comments, the reviewer also raises some concerns about technical details about the module design. Although we would like to provide a revision for the questions mentioned to provide more details, this option is not allowed in the review program of this year. Instead, in the sections below, we will try our best to address these concerns.\n***\n## Chain-of-Thought (CoT) Design\n\nThe CoT design is briefly demonstrated in Table 1 (line 183) of our paper, and we have uploaded examples in the supplementary materials. Besides, we have also provided an ablation experiment about the CoT process in **Appendix G (line 601)**, indicating that  performing reasoning during model generation and maintaining a summarization of historical information are very important for the success rate of the OWMM task; after removing them, the success rate drops significantly.But we also acknowledge the writing ambuity here and we plan to add a section to **Section 3.3** with the following details.\n\nOur CoT reasoning approach builds upon recent advances in embodied reasoning for robotics. Our OWMM-VLM generates structured reasoning chains that include: 1) Reasoning of the task instruction and summarization, and making decisions accordingly; 2) Performing perception, grounding, and task‑specific interpretation for the current egocentric-view image and the scene images. The model integrates visual information to support task decisions, for example, it determines whether the agent is close enough to an object in the current view, and then decides whether to perform pick/place or continue navigation; 3)Outputting the task decision and the execution targets/coordinates in the form of bounding boxes; 4)Summarizing the decision made at the current step and the actions executed; this summary is fed into the next step as input.\n\nThe key insight of this design is that augmenting outputs with Chain‑of‑Thought (CoT), OWMM-VLM acquires patterns for task comprehension, scene perception, and decision‑making from structured training data. OWMM-VLM models also summarize historical context after each decision, allowing every subsequent step to jointly reason over prior history and current observations.\n***\n## Pose Graph Construction for Sim and Real Environments\n\nWe really thank the reviewer for pointing out the missing details of pose graph contruction, as we also think thiese details are important for reproduction and real-world deployment. We aso We acknowledge the pose graph of images as multi-modal memory as one important agent design for global scene understanding, while meeting the VLM input data requirement as common VLM base models only accept texts and images as input. However, we do not claim pose graph construction method as one of our technical contributions, as in the Section 1 Introduction (line 68). As explained in **Section 3.1 line 136**, \"... we assume a pre-mapping phase separating active exploration and the SLAM module from the OWMM task focus. This is practical, as most robotic vacuums automate room mapping before cleaning.\"\n\nWe now provide more details about the implementation of pose graph construction. Basically, it involves pre-mapping stage and camera pose selection to cover the task-relavant images.\n\n**Pre-mapping Stage**: In simulated experiments, we get pre-mapping using the navigation mesh[1] provided by Habitat environment, where navigable locations are represented as mesh triangles. This pre-computed navmesh serves as a \"map\" for localization and navigation, similar to game engines. In the lab environment, we run RobiButler[2] over Fetch Lidar input for the pre-mapping stage and localize the robot geometrically. More specifically, RobiButler uses Gmapping[3] algorithm to compute the 2D occupancy map from lidar. The map is further used to localize the robot and camera poses.\n\n**Camera Pose Selection**: Considering the context length of current VLMs, we select 8 views from pre-mapping stage and 1 current ego-centric view in our experiments, as demonstrated in Figure 2. In simulation, as detailed in **Section 4.1 line 205**, we \"first set the robot at the location of the receptacle where objects were initially located and the goal receptacle, sampling the robot's head-view images. Subsequently, we randomly positioned the robot and captured its head-view images.\" This ensures sufficient scene coverage while maintaining computational efficiency. In the real robot, we extract the keyframes from the ROS bag RGB data and manually select the frames related to the task to guarantee enough scene coverage.\n\nBesides, although actively selecting related frames from videos is beyond the contributions of this paper, there are some related works trying to address this issue. VideoAgent[4] and the follow-up works proposed to perform video understanding by retrieving task-relavant frames and then conducting a normal VLM reasoning process. Here we assume we have a ground truth VideoAgent and focus on downstream tasks.\n\nAs in **Appendix B Limitations (line 455),** We acknowledge this is one of the limitations of our method and we plan to add more discussions to the final version of our paper to provide more insight. Addressing this limitation could be future work.\n\n[1] Arkin, Ronald C. \"Path planning for a vision-based autonomous robot.\" *Mobile robots I*. Vol. 727. SPIE, 1987.\n\n[2] Xiao, Anxing, et al. \"Robi Butler: Multimodal Remote Interaction with a Household Robot Assistant.\" ICRA 2025.\n\n[3] G. Grisetti, et al., “Improved techniques for grid mapping with rao-blackwellized particle filters,” IEEE transactions on Robotics, 2007\n\n[4] Wang, Xiaohan, et al. \"Videoagent: Long-form video understanding with large language model as agent.\" ECCV 2024.\n***\n## Experimental Analysis and Dataset Construction\n\nFor this section, we kindly disagree that \"dataset construction and experimental analyses a bit plain without much insights in addition to performance improvements\". In fact, we have provided rich details about dataset construction pipeline and the underlying insight. Besides, we have also conducted extensive ablation studies on both training data compositions and model designs.\n\n### Dataset Construction Insights\n\nOur agentic data synthesis pipeline offers several key insights for the robotics community:\n\nAutomated Template-Based Reasoning Framework: We demonstrate that high-quality robotic training data can be generated with minimal human annotation through the combination of following components: 1) Systematic PDDL-based task sequences and ground-truth symbolic world representations. This addresses a critical scalability challenge in robotics data collection, as detailed in **Section 4.1 (line 197)**. 2) Domain-Specific Re-labeling Strategy: The PDDL-based system can only generate symbolic state descriptions with poor diversity in predicates and label names based on the model dataset. We introduced a GPT-4o-based re-labeling process that generates diverse descriptions for object labels and states, suited for open language instruction. This insight shows how foundation models can be leveraged to enhance dataset diversity without manual effort, as described in **Section 4.2 (line 220)**.\n\n### Experimental Analysis Insights\n\nBesides, our experimental analyses provide several non-trivial insights for 1) what is a good agentic fine-tuning dataset on our domain-specific task and 2) what are the effective components of our VLM model.\n\nIn **Appendix** **D.2 Analysis on the training data (line 514)**, we studied scaling law in fine-tuning and dataset diversity. We systematically analyzed how object and scene diversity affect model performance, discovering that diversity has negligible effects on multi-modal capabilities (fluctuations within 5% range), while data volume shows logarithmic improvement patterns. This finding challenges conventional wisdom about dataset diversity requirements in robotics, and could be contributed to that most grounding capabilties derive from the base model and its large pre-training data. Besides, we also present two speculations from the observation in **Appendix D.2 (line 533)**: 1) Distance-based decision making prior can be learned in a data-driven way and 2) multi-image retrieval presents a bottleneck in our task. These insights could have broader implications for the field.\n\nIn **Appendix G Ablation Study on OWMM-VLM (line 585),** we conducted the ablation study and found that 1) Grounding format matters. Our ablation study shows that bounding box prediction significantly outperforms direct coordinate prediction (0.93 vs 0.65 for object grounding), revealing that consistency with pre-trained VLM representations is crucial for effective fine-tuning. 2) CoT matters. We demonstrate that removing reasoning and summarization capabilities leads to substantial performance degradation across all metrics, providing concrete evidence for the importance of structured reasoning in embodied AI systems.\n***\n## Revision Plan\n\nWe acknowledge that the current version could benefit from revision and will address this through:\n1. Enhanced Technical Detail: We will expand Section 3 with more comprehensive implementation details for CoT design, pose graph construction and place them in the appendinx. \n2. Improved Experimental Analysis: We will expand the analysis of our experiemnts to provide deeper insights into our design choices and their impact on performance. \n***\nWe are really grateful for the valuable suggestions from the reviewer and we hope our response addresses your concerns. We are also looking forward to your responses and more discussions in the next phase."}, {"review_id": "0qM5dp7gqk", "reviewer": "Reviewer_hsT1", "summary": "This paper proposed OWMM-Agent, which performs open-world mobile manipulation tasks. The core of the system is a VLM that is fine-tuned using a large-scale synthetic dataset generated in simulation. This model is capable of handling multiple sub-tasks by reasoning over multimodal inputs, including long-term scene memory and short-term robot state history. The finetuning enables state-tracking of the task, resulting higher task success rate. The proposed model shows strong performance in decision making and grounding, for both simulation and real-world environment.", "questions": "- The gap between object image retrieval (88.56%) and goal image retrieval (30.39%) is quite large. Could the authors elaborate on potential reasons for this discrepancy? For example, are goal receptacles more visually ambiguous, less salient, or more linguistically diverse?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper is well-written and presented the problem and methodology cleary.\n- The fine-tuned OWMM-VLM effectively integrates vision and language for embodied reasoning.\n- Extensive experiments are conducted, including comparisons with strong baselines, ablation studies, and analysis of training data effects.\n\n\n\n## Weakness\n\n- Failure analysis needed: While the proposed method shows promising results, the full task success rate is notably low, even for the best-performing model with huge number of parameters. I understand this is a difficult task, and since there are multiple subtasks so low success rate is kind of natural considering accumulation of errors. However, a detailed failure analysis would help the reader better understand the primary bottlenecks and guide future improvements.\n- It would be beneficial to explicitly discuss why the proposed model avoids dead loops. I think the fine-tuned VLM model and the robot history helps avoid repetition, so it can naturally avoid dead loop while other can't. (Please correct me if I'm wrong). It would be nice to clarify about this with some examples.\n- The overall system appears to operate in open-loop control. While the VLM can understand the current state and make the decision based on it, but it is unclear how the system handles failures.  What happens if the picking object fails? Is the fine-tuned VLM model able to recover itself?  A discussion or experiment showing whether the VLM can detect and recover from failure would be valuable.", "was_revised": true, "final_justification": "I encourage the authors to revise the paper in line with the rebuttal and the reviewers’ comments.\nThis work presents a valuable contribution, but its presentation and framing could be improved.\nI also share reviewer 8pbX’s concern regarding the “open-world” claim.\n\nI believe these issues can be addressed through revisions for the final version. \nAlso, the rebuttal and the authors’ responses have resolved my concerns and questions.\nTherefore, I maintain my original score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful evaluation and for recognizing the clarity of the writing, the embodied reasoning enabled by **OWMM‑VLM**, and the breadth of experiments. Below we address the requests on failure analysis, loop avoidance, and failure recovery, and then answer the question on the object–goal retrieval gap:  \n***\n### Failure analysis in episodic evaluation\n\nWe acknowledge that detailed failure analysis is crucial for understanding system limitations, and we plan to add more analysis discussions in the final version.  In fact, in **Table 3 in Section 5 (line 230)**, we have provided the cascaded success rate on the episodes, meaning until which step in the episode the agent is still successful, as detailed in **Section 5.2 (line 259)**.  And we have two key observations:\n\n1) **Bottleneck exists on object picking/placing action.** While our model achieves strong performance in the early stages (88.56% on object image retrieval, 84.64% on robot navigation to object), performance degrades significantly in later stages (38.56% object picking, 30.39% goal image retrieval)\n\n2) **Error accumulation matters.** Failed early actions compound, making later subtasks increasingly difficult.\n\nWe will include this analysis in the revision and we plan to further dive into the first observation to check whether the object interaction action fails because of decision making (choosing wrong action) or grouding. We will keep you informed in the following discussion phase.\n\nWe acknowledge that detailed failure analysis is crucial for understanding system limitations, and we plan to add more analysis discussions in the final version.  In fact, in **Table 3 in Section 5 (line 230)**, we have provided the cascaded success rate on the episodes, meaning until which step in the episode the agent is still successful, as detailed in **Section 5.2 (line 259)**.  And we have two key observations:\n\n1) **Bottleneck exists on object picking/placing action.** While our model achieves strong performance in the early stages (88.56% on object image retrieval, 84.64% on robot navigation to object), performance degrades significantly in later stages (38.56% object picking, 30.39% goal image retrieval)\n\n2) **Error accumulation matters.** Failed early actions compound, making later subtasks increasingly difficult.\n\nBesides, we further dive into episodic action sequences to analyze their failure reasons. Since the current evaluation pipeline does not support automatically failure case analysis, we randomly selected 100 failed samples from evaluation logs, and categorized their failures into four types:\n\n- **Ego-centric Decision Making Error**: Errors in pick/place or navigation actions, typically caused by the robot either being close enough but outputting a navigation action, leading it to move away and lose the target, or not being close enough but outputting a pick or place action.\n- **Image Retrieval Error**: The retrieved image ID is incorrect.\n- **Affordance Grounding (object/receptacle) Error**: The robot is close enough, but the center point of the output bounding box does not intersect with the target object/receptacle, causing a failed pick or place action, or the grounded object/receptacle is incorrect.\n- **Affordance Grounding (navigation) Error**: The output bounding box corresponds to an unreachable region or causes the target object/receptacle to disappear from the robot’s egocentric view.\n\nThe evaluation results are as follows:\n\n| Model/Error ratio | Ego-centric Decision making Error（pick/place or navigation） | Image<br>Retrieval Error | Affordacne grounding(object/ receptacle) Error | Affordacne grounding(navigation) Error |\n| --- | --- | --- | --- | --- |\n| OWMM-VLM-38B | 36/100 | 28/100 | 15/100 | 21/100 |\n\n\nWe will include these analyses and more visualizations in the revision appendix to provide more insight about the experimental results. We sincerely thank the reviewer for this constructive suggestion.\n***\n### Why our model avoids dead loops\n\nWe really appreciate this insightful question and we mostly agree with the reviewer's interpretation of this phenomenon. We performed supervised fine-tuning (SFT) on a dataset that includes chain-of-thought (CoT) annotations. The CoT component enables the model to learn to understand tasks and historical information, make decisions by incorporating situational context, and summarize past information. With this capability improved, the model is better at solving sequential tasks: it can keep track of which stage it is currently at in completing the sequence, thereby avoiding repeatedly outputting the exact same action.\n***\n### Failure Recovery\n\nThis is a good point that touches one of the **limitations of our current approach**. In fact, our system currently operates in open-loop mode where the VLM generates high-level actions based on observations but doesn't explicitly verify action success or implement recovery strategies. As discussed in the last point of deal loops, the current data synthesis pipeline focuses more on sequential reasoning and multi-modal action output, rather than failure recovery.\n\nHowever, we think to further strengthen failure recovery behavior and reduce error accumulation could be important **future work**. Recent progress on LLM Reinformance Learning [1] and observed \"Aha moment\"[2] reveal that this failure recovery behavior can be injected by an RL finetuning process: (i) explicitly synthesizing error‑restart training episodes in simulation; and (ii) exploring RL with language feedback for recovery policies.\n\nWe would like to add this discussion to the Appendix B to discuss more about its limitations and possible future work.\n***\n### Gap between object image retrieval and goal image retrieval\n\nThe figures the reviewer cites are drawn from different evaluation regimes. In our single‑step evaluation, the standalone ability of **object retrieval** and **goal (receptacle) retrieval** is comparable, the experiment results is presented at Table2.\n\nBy contrast, the numbers in the episodic evaluation presented at Table3 compute success as “successful episodes / total episodes.” Under this protocol, **goal retrieval** success is conditioned on all preceding sub‑tasks, including a successful pick. For **OWMM‑VLM‑38B**, the episode‑level success for **goal image retrieval** is **30.39%**, and the preceding **object picked** success is **38.56%**. This confirms our point: the apparent gap at the episode level largely reflects upstream execution ceilings and error compounding, not an intrinsic disparity in standalone retrieval capability.\n***\nWe appreciate the reviewer’s constructive suggestions. We will incorporate the expanded failure analysis, the loop‑avoidance clarification with examples, the failure‑recovery discussion and ablation, and the clarified, conditioned retrieval metrics in the appendices and corresponding sections of the final version. We are also looking forward to your response or follow-up discussion in the next stage.\n\n[1] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" Arxiv 2025.\n\n[2] Zhou, Hengguang, et al. \"R1-Zero's\" Aha Moment\" in Visual Reasoning on a 2B Non-SFT Model.\" Arxiv 2025."}, {"review_id": "Qa2y46trrS", "reviewer": "Reviewer_v5R9", "summary": "This paper proposes OWMM-Agent, a vision-language agent for open-world mobile manipulation. The model, OWMM-VLM, is fine-tuned on a large synthetic dataset and performs high-level action planning (e.g., pick, place, navigate) based on natural language, multi-view images, and robot state history. It interfaces with classical planners for execution. Experimental results show that the proposed method significantly outperforms proposed baselines.", "questions": "1. Could you elaborate on how the scene is discretized and how the pose graph is constructed during the pre-mapping phase? Specifically, what strategy do you use to select camera poses and ensure sufficient scene coverage?\n\n2. How does your setting differ from goal-oriented vision-language navigation tasks such as REVERIE [1] and SOON [2]? Since your method has access to pre-captured global scene information (i.e., posed frames), would it be fair to consider your setup a simplified variant of those tasks?\n\n3. If I understand correctly, the pre-captured scene-level posed frames are included as part of the model's input. Could you clarify how the number of these frames affects GPU memory consumption and inference time? In the case of large or complex scenes, would this design lead to significant computational overhead?\n\n4. You mention the method can handle dynamic environments. Could you provide any design details or experimental results to support this claim?\n\n5. What types of visual hallucination occur in this navigation task when using foundation models, and how does your method address them?\n\n6. Writting issue: \n    - Line 217, Table 5 => Table 1\n\n[1] Qi, Y., Wu, Q., Anderson, P., Wang, X., Wang, W. Y., Shen, C., & Hengel, A. V. D. (2020). Reverie: Remote embodied visual referring expression in real indoor environments. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 9982-9991).\n\n[2] Zhu, F., Liang, X., Zhu, Y., Yu, Q., Chang, X., & Liang, X. (2021). Soon: Scenario oriented object navigation with graph-based exploration. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 12689-12699).", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Propose a unified vision-language agent to support multimodal training\n2. Introduces a scalable synthetic data generation pipeline covering four grounding tasks (pick, place, navigation, and scene search).\n3. Both simulation-based evaluations and real-world zero-shot tests\n\nWeaknesses:\n\n1. The number of real-world test cases is very limited, which undermines the statistical significance of the reported results.\n2. The model design may introduce significant computational bottlenecks, raising concerns for real-world deployment efficiency.\n3. Some claims lack sufficient clarification (see questions)", "was_revised": true, "final_justification": "Several concerns were addressed, but key issues remain. \n\n1. The paper’s distinction from existing VLN work is unclear. \n2. The claimed link between visual hallucination and synthetic data design is weak. The method design is overly intuitive and trivial.\n\nI will maintain my score and defer these concerns to the AC for thorough consideration.", "rebuttal": "We really thank the reviewer for the time and efforts invested in this thoughtful evaluation of our paper. We would also appreciate the recognition of our work's strengths, particularly the acknowledgment of the unified agent framework and data synthesis pipeline. Apart from the positive comments, we would like to address the concerns and questions raised by the reviewer:\n***\n## Response to Weaknesses\n\n### Limited Real-World Test Cases\n\nOur real-world experiments, conducted in from a single lab environment, were designed as a proof-of-concept to demonstrate two ideas: 1) the proposed embodied **agent architecture** can be deployed onto the real mobile manipulation robot in a reasonable way; 2) the fine-tuned **VLM models** from synthesized dataset has zero-shot generalization capability  from simulation to reality. \n***\n### Computational bottlenecks and real-world deployment efficiency\n\nWhile these timings reflect comprehensive processing including multi-image processing, chain-of-thought reasoning and action generation, we acknowledge that real-time deployment requires optimization. Although the optimization of VLM inference is beyond the scope, we provide some analysis of this bottleneck and possible solutions. One clear bottleneck we observe is that whole-scene understanding involves the tokenization of multiple images in the scene, which composes most of the input tokens to VLM model. The image token compression approaches could alleviate this issue. For example, [1] proposed a method to compress image tokens by selecting the text-relevant image patch tokens. Besides, applying model quantization to VLM models could also accelerate inference efficiency, such as [2].\n\n[1] Li, Kevin Y., et al. \"Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters.\" arXiv  (2024).\n\n[2] Xie, Jingjing, et al. \"Advancing multimodal large language models with quantization-aware scale learning for efficient adaptation.\" ACM MM. 2024.\n***\n## Answer to Questions\n\n### Scene Discretization and Pose Graph Construction\n\nWe really thank the reviewer for pointing out this question. We also think this is a key concern in real-world system design. We have also acknowledged the pre-mapping stage as one of our main limitations in **Appendix B Limitations** (line 455), specifically, \"... we **assume a pre-mapping phase** with a camera pose graph and 2D occupancy map for path planning in navigation ...\". \n\n**Pose Graph Construction in Pre-mapping Stage**: In simulated experiments, we get the pre-mappming using the navigation mesh[3] provided by Habitat environment, where navigable locations are represented as mesh triangles. In the lab environment, we run RobiButler[4] over Fetch Lidar input for the pre-mapping stage and localize the robot geometrically.\n\n**Camera Pose Selection**: Considering the context length of current VLMs, we select 8 views from pre-mapping stage and 1 current ego-centric view in our experiments, as demonstrated in Figure 2. In simulation, as detailed in **Section 4.1 line 205**, we \"first set the robot at the location of the receptacle where objects were initially located and the goal receptacle, sampling the robot's head-view images. Subsequently, we randomly positioned the robot and captured its head-view images.\" This ensures sufficient scene coverage while maintaining computational efficiency. In the real robot, we extract the keyframes from the ROS bag RGB data and manually select the frames related to the task to guarantee the scene coverage.\n\nBesides, although actively selecting related frames from videos is beyond the contributions of this paper, there are some related works trying to address this issue. VideoAgent[5] and the follow-up works proposed to perform video understanding by retrieving task-relavant frames and then conducting a normal VLM reasoning process. Here we assume we have a ground truth VideoAgent and focus on downstream tasks.\n\nWe acknowledge this is one of the limitations of our method and we plan to add more discussions to the final version of our paper to provide more insight. Addressing this limitation could be future work.\n\n[3] Arkin, Ronald C. \"Path planning for a vision-based autonomous robot.\" *Mobile robots I*. Vol. 727. SPIE, 1987.\n\n[4] Xiao, Anxing, et al. \"Robi Butler: Multimodal Remote Interaction with a Household Robot Assistant.\" ICRA 2025.\n\n[5] Wang, Xiaohan, et al. \"Videoagent: Long-form video understanding with large language model as agent.\" ECCV 2024.\n***\n### Comparison with REVERIE and SOON\n\nWe thank the reviewer for the question about the positioning of our paper. Our setting differs fundamentally from REVERIE and SOON in several key aspects:\n\nOverall, these two papers can be categorized to Visual Language Navigation tasks. In comparison, this paper focuses on Open World/ Vocabulary Mobile Manipulation as introduced in **Section 2 Related Works (line 77)**. Specifically, our approach differs in three major ways : 1) **Exploration vs. VLM Reasoning on Pre-mapping**: Unlike REVERIE/SOON which focuses on more efficient online exploration, our agent focuses on task-relevant frame retrieval and action decision making based on pre-mapping information; 2) **Manipulation Integration**: Unlike REVERIE/SOON, We address both navigation and manipulation in a unified framework 3) **Low-level Action Predicion**: REVERIE/SOON uses pre-defined view points as navigation action space. This action is not flexible under ego-centric agent, especially for mobile manipulation, in which task the robot needs to carefully choose a navigation point not too distant or too close so that the object falls into the arm workspace. In comparison, our agent can choose a viewpoint to navigate to and can also **generate finer action including moving to any point or grasping any region in ego-centric space**. You can refer to **Table 1 (line 183)** for more details.\n\n***\n### How does number of frames affects GPU consumption and inference time?\n\nWe thank the reviewer for this question that is important for real-world deployment. We managed to add the new experiment to test the GPU consumption and inference time on two settings:\n\n1) Evaluation of single-step inference of OWMM-VLM-8B on single A100 GPU with [2+1; 4+1; 8+1; 16+1] frames as image input.\n2) Evaluation of single-step inference of OWMM-VLM-32B on 4 X A100 GPU with [8+1; 16+1; 32+1; 64+1] frames as image input. Since the larger VLM model cannot be deployed on a single A100-40G GPU, we use parallel inference on 4 GPUs, which is common in VLM  deployment.\n\nThe result is below:\n\n| Input Frames | 2+1 |  |  | 4+1 |  |  | 8+1 (default) |  |  | 16+1 |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Model/ Setting | Prompt token | Time avg(s) | Memory | Prompt token | Time avg(s) | Memory | Prompt token | Time avg(s) | Memory | Prompt token | Time avg(s) | Memory |\n| OWMMVLM-8B | 3409.19 | 3.6573 | 17+0.44=17.44G | 6057.19 | 3.6723 | 17+0.77=17.77G | 11353.19 | 4.8412 | 17+1.43=18.43G | 21945.19 | 7.0883 | 17+2.75=19.75G |\n\n| Input Frames | 8+1 (default) |  |  | 16+1 |  |  | 32+1 |  |  | 64+1 |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Model/ Setting | Prompt token | Time avg(s) | Memory | Prompt token | Time avg(s) | Memory | Prompt token | Time avg(s) | Memory | Prompt token | Time avg(s) | Memory |\n| OWMMVLM-38B | 2810.37 | 4.394 | 97+1.22=98.22G | 4922.37 | 5.0403 | 97+2.13=99.13G | 9146.37 | 7.3397 | 97+3.65=100.65G | 17594.37 | 15.3297 | 97+7.30=104.30G |\n\n\nI would like to put more analysis here, but we are **reaching character limitations** with the rest of the questions. Besides, in the case of large or complex scenes, we would propose to use the dynamic frame selection scheme as we discussed in answer to question 1. We think this question is meaningful and we would like to add these new experiments and analysis to the appendix in the final version of this paper.\n***\n### Dynamic Environment Handling\n\nWe thank the reviewer for this insightful question, though we do not claim \"the method can handle the dynamic environments\". Instead, we guess that the closest statement in the paper could be \"Additionally, they often require time-consuming dense 3D reconstruction, **making them less suitable for complex, open-ended and dynamic environments**\" in line 32.\n\nCompared to the 3D reconstruction-based methods, we believe our proposed scene representation of a graph of posed images better suits dynamic scenarios. For example, in a human-robot cohabiting environment, where the scene is changed by the human subject, the robot agent can detect the changes by its observation and replace the outdated frame with the new observation. In comparison, the 3D reconstruction-based methods need to use more complex algorithms to update the 3D representation incrementally, which would be computationally costly.\n\nWe also acknowledge that how this architecture should be applied to a dynamic environment is an open and challenging question.\n***\n### Visual Hallucination in Navigation Tasks\n\nWe would really like to share more of our insights on this question, but unfortunately, we have reached the limitation of response. In short, we have met with all sorts of visual hallucinations on image retrieval, grounding, and misalignment between images and texts. This is one of our key motivations to design the data synthesis pipeline, as in the abstract. This could be inferred from **Table 2 (line 230)**, where we compare the base model (InternVL2.5-8B) with the model after instruction fine-tuning (OWMM-VLM-8B). Besides, we will add more discussions and analysis in the appendix in the final version.\n***\nWe really thank the reviewer's dedication and profession in evaluation and we hope our response addresses your concerns. We are also looking forward to your responses and more discussions in the next phase."}]

Author rebuttal:

Meta-review: ## Summary

This paper introduces OWMM-Agent, a novel framework for open-world mobile manipulation. The core contributions are a unified vision-language model (OWMM-VLM) that integrates scene understanding, state tracking, and action generation, and a scalable, simulation-based agentic data synthesis pipeline to fine-tune the model. This approach aims to address the challenges of generalization and system complexity in mobile manipulation by reformulating the task as a multi-modal, multi-turn reasoning problem. The authors demonstrate state-of-the-art performance in both simulation and real-world zero-shot scenarios.

The paper received a mixed set of reviews, with three reviewers ultimately leaning towards acceptance and one recommending rejection.

## Strengths identified by the reviewers
- The unified framework that combines perception, reasoning, and action generation into a single VLM is novel and well-motivated.

- The agentic data synthesis pipeline is a significant contribution, enabling scalable data generation and demonstrating strong sim-to-real transfer.

- The experimental evaluation is extensive, covering both simulation and real-world tests with strong baseline comparisons.

## Weaknesses
- Reviewer 8pbX (Accept) and Reviewer hsT1 (Accept) raised concerns about the pre-mapping assumption, the "open-world" claim, and the need for more detailed failure analysis. The authors provided thorough rebuttals, including additional ablation studies and clarifications on their methodology and claims. Both reviewers were fully satisfied with the responses, confirmed their concerns were resolved, and maintained their positive scores.

- Reviewer QZc6 (Borderline Accept) initially found the paper lacking in technical details regarding the implementation pipeline (e.g., CoT design, pose graph construction). The authors' rebuttal supplied these missing details, which addressed the reviewer's concerns and led them to raise their score from borderline reject to borderline accept.

- Reviewer v5R9 (Borderline Reject) raised more fundamental concerns about the paper's novelty and positioning within the existing Vision-Language Navigation (VLN) literature. Despite a detailed, multi-part rebuttal and follow-up discussion from the authors, the reviewer remained unconvinced. They argued that the task setting was a simplified variant of existing VLN tasks and that the connection between the stated problem of "visual hallucination" and the proposed data synthesis solution was not sufficiently justified. The reviewer maintained their negative score, deferring the final decision to the Area Chair.

## Recommendation Justification
This paper tackles the challenging and important problem of open-world mobile manipulation. The consensus among the majority of reviewers is that the proposed unified agent architecture and the agentic data synthesis pipeline are novel and significant contributions. The authors' ability to achieve strong zero-shot performance on a real robot using only synthetic data is a notable result with high practical impact for the robotics and embodied AI communities.

The primary dissenting opinion from Reviewer v5R9 centers on the paper's positioning relative to prior VLN work. While this is a valid point and the paper would benefit from a more detailed discussion of this context, I believe the work's focus on integrating both navigation and manipulation with continuous control and its strong sim-to-real results distinguish it sufficiently. The contributions in creating a capable, open-source foundational model and dataset for mobile manipulation are valuable for the field.

The paper is technically solid, the experiments are thorough, and the contributions have the potential for high impact. While the concerns of Reviewer v5R9 are noted, the overwhelming positive feedback and the successful resolution of most issues lead me to believe the paper is a clear accept.
Decision: Accept (poster)