HomeRobot: Open-Vocabulary Mobile Manipulation
Authors: Sriram Yenamandra
Arun Ramachandran
Karmesh Yadav
Austin S Wang
Mukul Khanna
Theophile Gervet
Tsung-Yen Yang
Vidhi Jain
Alexander Clegg
John M Turner
Zsolt Kira
Manolis Savva
Angel X Chang
Devendra Singh Chaplot
Dhruv Batra
Roozbeh Mottaghi
Yonatan Bisk
Chris Paxton
Venue: CoRL 2023 Poster (2023)
Keywords: benchmark
mobile manipulation
sim2real

Abstract: HomeRobot (noun): An affordable compliant robot that navigates homes and manipulates a wide range of objects in order to complete everyday tasks.

Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object in any unseen environment, and placing it in a commanded location. This is a foundational challenge for robots to be useful assistants in human environments, because it involves tackling sub-problems from across robotics: perception, language understanding, navigation, and manipulation are all essential to OVMM. In addition, integration of the solutions to these sub-problems poses its own substantial challenges. To drive research in this area, we introduce the HomeRobot OVMM benchmark, where an agent navigates household environments to grasp novel objects and place them on target receptacles. HomeRobot has two components: a simulation component, which uses a large and diverse curated object set in new, high-quality multi-room home environments; and a real-world component, providing a software stack for the low-cost Hello Robot Stretch to encourage replication of real-world experiments across labs. We implement both reinforcement learning and heuristic (model-based) baselines and show evidence of sim-to-real transfer. Our baselines achieve a 20% success rate in the real world; our experiments identify ways future research work improve performance. See videos on our website: https://home-robot-ovmm.github.io/.

L HomeRRbot:
Mobile Manipulation
Sriram Yenamandra" 1
Arun Ramachandran" 1 Theophile G
Karmesh Yadav*1,2
Austin Wang'
Mukul Khanna'
Tsung-Yer Yen Yang² Zsolt Kira
Vidhi Jain
Alexander William Clegg
John Turner
Manolis Savva?
Angel Chang"
Devendra Singh Chaplot?
Dhruv Batra 1.2 Chris Paxton
Roozbeh Mottaghi
Yonatan Bisk
Georgia Tech
FFIRR, Meta AI homerobot-infoo Honerobotinn@@ggglgggoo... @goglegroupss.com
Carnegie Mellon
SSmonn Fraser
Abstract: HomeRobot E (noun): An affordable compliant robot that navigates homes and manipulates wide range of objects in order complete everyday tasks.
Open- Veen-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object in any unseen environment, and placing it in commanded location. This is a foundational challenge for robots to be useful assistants in human environments, because it involves tackling sub-problems from across robotics: perception, language understanding, navigation, and manipulation are all essential to OVMM. addition, integration of the solutions to these sub-problems poses its own substantial challenges. To drive research in this area, we introduce the HomeRobot OVMM benchmark, where an agent navigates household environments grasp novel objects and place them target receptacles. HomeRobot has two components: simulation component, which uses large and diverse curated object set in new, high-quality multi-room home environments; and real-world component, providing a software stack for the low-cost Hello Robot Stretch to encourage replication of real-world experiments across labs. We implement both reinforcement learning and heuristic smodel-based. baselines and show evidence of sim-to-real transfer of the nav and place skills. Our baselines achieve 20% success rate in the real world; our experiments identify ways future work improve performance. See videos on our website: https://vmmm.iituub..
Keywords: Sim-to-real, benchmarking robot learning, mobile manipulation
Introduction
The aspiration to develop household robotic assistants has served as north star for roboticists since the beginning of the field. The pursuit of this vision has spawned multiple areas of research within robotics from vision to manipulation, and has led to increasingly complex tasks and benchmarks. useful household assistant requires creating capable mobile manipulator that understands a wide variety of objects, how interact with the environment, and how to intelligently explore a world with limited sensing. This has separately motivated research in diverse areas like navigation [1, 2], service robotics [3-5], language understanding [6, 7] and task and motion planning [8]. We refer to this guiding problem as Open- Vocabulary Mobile Manipulation (OVMM): useful robot will be able to find and move arbitrary objects from place to place in an arbitrary home.
Prior work does not tackle mobile manipulation large, continuous, real-world environments. Instead, generally simplifies the setting significantly, e.g. by using discrete action spaces, limited object sets, small, single-room environments that are easily explored. However, recent developments tying language and vision have enabled robots to generalize beyond specific categories [9-13], often through multi-modal models such as CLIP [14]. Further, comparison across methods has remained difficult and reproduction of results across labs impossible, since many aspects of the
7th Conference Robot Learning 2023), Atlanta, USA.

---

Find Object on Start Receptacle
Pick Object from Start Receptacle
Find Goal Receplacle
Place Object on Goal Receptacle
REAL
Tay
Move toy animal from chair table
SMM
Savig Gat
Move pitcher from drawer to serving cart
Open- Vocabulary Mobile Manipulation requires agents to search for previously unseen object at a particular location, and move it to the correct receptacle.
Figure 1:
settings (environments, and robots) have not been standardized. This is especially important now, as wave of research projects have begun show promising results in complex, open-vocabulary navigation [9, 15, 11, 12, 16] and manipulation [17, 10, 18] again wide range of robots and settings, and still limited to single-room environments. Clearly, now is the time when we need common platform and benchmarks to drive the field forward.
In this work, we define Npen-Vocabulary Mobile Manipulation as key task for in-home robotics and provide benchmarks and infrastructure, both in simulation and the real world, build and evaluate full-stack integrated mobile manipulation systems, in wide variety of human-centric environments, with open object sets. Our benchmark will further reproducible research in this setting, and the fact that we support arbitrary objects will enable the results to be deployed in variety of real-world environments.
OVMM: We propose the first reproducible mobile-manipulkae benchmark for the real world, with associated simulation component. In simulation, interactive 3D scenes [19] instantiated in the AI Habitat simulator [20, 21] create large number of challenging, multi-room OVMM problems with a wide variety of objects curated from variety of sources. Some of these objects' categories have been seen during training; others have not. In the real world, we create an equivalent benchmark, also with mix of seen and unseen object categories, in a controlled apartment environment. We use the Hello Robot Stretch [22]: an affordable and compliant platform for household and social robotics that is already in use over universities and industry research labs. Fig. shows instantiations of our OVMM task in both the real-world benchmark and in simulation. We have controlled real-world test environment, and plan to the real-world benchmark yearly to assess progress on this challenging problem. Real-world benchmarking will be run as part of the NeurlPS 2023 HomeRobot OVMM competition [23].
dataset of 200 human-authored
HomeRoboo: We also propose HomeRobot,' 1 software framework to facilitate extensive bench- marking both simulated and physical environments. It comprises identical APIs that are imple- mented across both settings, enabling researchers conduct experiments that can be replicated in both simulated and real-world environments. Table I compares HomeRobot OVMM to the literature. Notably, HomeRobot provides robotics stack for the Hello Robot Stretch which supports range of capabilities in both simulation and the real world, and is not restricted to just the OVMM task. Our library also supports a number of sub-tasks, including manipulation learning [24], continuous learning [25], navigation [26], and object-goal navigation [2].

---

Object
Continunus Actions
Rohoties Stack
Open Licensing
Scenes
Cats
Inst.
Sim2Real
Manipulation
Room Rearrangement
[28]
120
118
118
Habital ObjectNav Challenge
216
6
7,599
TDW-Transport
[30]
50
112
X
36
VirtualHome
[31]
308
1,066
X
X6
ALFRED
[6]
120
84
84
X
X
Habitat 2.0 HAB
[21]
105
20
20
X
ProTTHOR
[32]
10,000
108
1,633
X
RoboTHOR
[33]
75
43
731
X
*
Behavior- IK
[34]
50
1,265
5,215
ManiSkill-2
[35] 2,000
2,000
E OVMM + HomeRobot
200 150
7,892
Table 1:
Comparisons of our proposed benchmark with prior work. We provide a large number of environments and unique objects, focusing on manipulable objects, with a continuous action space. Uniquely, also provide multi-p real-world robotics stack, with demonstrated sim-to-real capabilities, allowing others to reproduce and deploy their own solutions. Additional nuances in footnote Partial availability Noot available CCapability available
In this paper, we HomeRobot to compare two families of approaches: a heuristic solution, using a motion planner shown to work for real-world object search [2], and reinforcement learning (RL) solution, which learns how to navigate to objects given depth and predicted object segmentation. We use the open-vocabulary object detector DETIC [27] to provide object segmentation for both the heuristic and RL policies. We observe that while the RL methods moved the object more efficiently if an object was visible, the heuristic planner was better at long-horizon exploration. We also substantial drop in performance when switching from ground-truth segmentation to DETIC segmentation. This highlights the importance of the HomeRobot OVMM challenge, only through viewing the problem holistically integrating perception, planning, and action can we build general-purpose home assistants.
To summarize, in this paper, we define Open--ocateeeeeee Vocabulary Mobile Manipulation as new, crucial simulation environment, with multiple,
task for the roboties community in Sec. 3. We provide multi-room interactive environments and wide range of objects. We implement a robotics library called HomeRobot which provides baseline policies implementing this in both the simulation and the real world. We describe real-worke benchmark in controlled environment, and show how current baselines perform in simulation and in the real world under different conditions. We plan to initially run this world benchmark as Neurips 2023 competition [23].
2 Related Work
We discuss work related to challenges and reproducibility of robotics research in more detail, but continue the discussion of datasets and simulators in Appendix A.
Challenges. There have been several challenges aiming benchmark robotic systems different tasks. These challenges provided great testbed for ranking different systems. However, in most of the challenges (e.g., [36-39, 3]), the participants create their own robotic platform making fair comparison of the algorithms difficult. There are also challenges where the organizers provide the robotic platform to the participants (e.g., [40]). However, changing the task during the periodic evaluations made difficult to track progress over time. Our aim is to have a real world benchmark using a standard hardware that sustainable at least for few years.
Reproducibility of robotics research. Standardized robotics benchmarks have been pursued for long time, often by open-soureing robot designs or introducing low-cost robots [41-49]. However, the environments in which these robots are used vary dramatically, leading to evaluation of components (e.g., object navigation, SLAM) in isolation, instead of as components of a larger system that
'ALFRED uses object masks for interaction. ObjectNav uses scans, not full object meshes. ProcThor scenes procedurally generated, this has the benefit that the potential number of environments unbounded.

---

SIM
REAL
Figure 2: A low-cost home robot performing tasks in both a simulated and a real-world environment. We provide both (1) challenging simulated tasks, wherein mobile manipulator robot must find and grasp multiple seen and unseen objects, and (2) a corresponding real-world robotics stack to allow others reproduce this research and evaluation to produce useful home robot assistants.
may not benefit from those changes. The HomeRobot stack enables end-to-end benchmarking of individual components by providing a full robotics stack, with multiple implementations of different sub-modules. The simplicity helps move beyond standardized sets of objects (e.g., (50-52]) to common set of robots, objects, and environments. Ours is the only benchmark to provide a broadly capable robotics stack for implementing and sharing robotics code; this is similar to projects like PyRobot [53], which doesn't 't also provide strong simulation benchmark.
Real World Benchmarks. RoboTHOR [33] provides common set of scenes and objects for benchmarking navigation. RB2 [54] ranks different manipulation algorithms in a local setting. TOTO [55] takes a step further by providing a training dataset and running the experiments for the users. However, training and testing happen in the same environments and are limited to tabletop manipulation. Finally, the NIST Task Board [56] is successful challenge for fine-grained manipulation skills [57], also limited to tabletop context. Kadian et al. [58] propose the Habitat- PyRobot bridge (HaPy) to allow real-world testing the locobot robot; their framework is limited to navigation, and doesn't provide generally-usefut robotics stack with visualizations, debugging, motion planners, tooling, etc.
3 Open-Vocabulat Mobile Manipulation
Formally, our task is set up as instructions of the form: "Move (object) from the istart_receptacll to the (goal_reccol rgoal__eceptaclee"" The object is a small and manipulable household object (e.g., cup, stuffed toy. or box). By contrast, start_recee seart_receptacle and goal_receptacle are large pieces of furniture, which have surfaces upon which objects can be placed. The robot is placed in unknown single-floor home environment such as apartment- and must, given the language names of starrt_rceettale,, object, and geal_receptacle, pick up an object that is known to be on start_rrarcooo start_recoppaale and move it to any valid poal_receptacll start_recese start_receptaclee is always available, to help agents know where to look for the object.
The agent is successful the specified object indeed moved from a start_ _eeartteceeetcleeeee on which it began the episode, to any valid seal_recepttclee We give partial credit for each step the robot accomplishes: finding the start_ree ceaat_eeceptccee with the object, picking up the object, finding the goal_ receptacle, and placing the object on the goal_ rec_rrecetacle. There can be multiple valid objects that satisfy each query.
Crucially, we need and develop both (1) simulation version of the Bgen-Vocabulary Mobile Manipulation problem, for reproducibility, training, and fast iteration, and (2) real-robot stack with corresponding real-world benchmark. We compare the two in Fig. 2. Our simulated environments allow for varied, long-horizon task experimentation; our real-world HomeRobot stack allows for

---

experimenting with real data, and we design set of real-world tests to evaluate the performance of our learned and heuristic baselines.
The Robot. We use the Hello Robot Stretch [22] with Dex the mobile manipulation platform. because it (1) relatively affordable S25, 000 USD, (2) offers 6 DoF manipulation, and (3) is human safe and human-sizen making it safe to test in labs [24, 11] and homes [2], and can reach most places a human would expect a robot to go. For a breakdown of hardware choices, see Sec. H.1.
Objects. These are split into seen vs. wnseen categories and instances. In particular, at test time we look at unseen instances of seen or unseen categories; i.e. no seen manipulable object from training appears during evaluation. Agents must pick and place any requested object.
Receptacles. We include common household receptacles (e.g. tables, chairs, sofas) in our dataset; unlike with manipulable objects, all possible receptacle categories are seen during training.
Scenes. We have both a simulated scene dataset and a fixed set of real-world scenes with specific furniture arrangements and objects. In both simulated and real scenes, we use a mixture of objects from previously-seen categories, and objects from wnseen categories as the goal object for Open- Vecn-Vocayllary Mobile Manipulation task. We hold out validation and test scenes, which do not appear in the training data; while some receptacles may re-appear, they will be at previously unseen locations, and target object instances will be unseen.
Scoring. We compute success for each stage: finding object on start_receptaree picking up object, finding goal_receptacee and placing object on the goal. Overall success is true if all four stages agents receive point for each successive stage accomplished, normalized by the number of stages. More details in Appendix C.
successfully
accomplished. We compute partial success as tie-breakes in which
3.1 Simulation Dataset
The Habitat Synthetic Scenes Dataset (HSSD) [19] consists of 200+ human-authored 3D home scenes containing over 18k 3D models of real-world objects. Like most real houses, these scenes are cluttered with furniture and other objects placed into realistic architectural layouts, making navigation and manipulation similarly difficult the real world. We used subset of HSSD [19] consisting of 60 scenes for which additional metadata and simulation structures were authored to support rearrangement 4. For our experiments, these are divided into train, validation, and test splits of 38, 12, and 10 scenes each, following the splits in the original HSSD paper [19].
Objects and Receptacles. We aggregate objects from A12- Thor [59], Amazon Amazon-Berkeley Objects [60], Google Scanned OB- jects [61] and the HSSD [19] dataset to create large and diverse dataset of real-world robot problems. In total, we annotated 2,535 ob- jects from 129 total categories. We identified different categories of receptacles which appear in the HSSD dataset [19].
Figure 3: HSSD scenes.
We construct our final set of furniture receptacle objects by first automatically labeling stable areas on top of receptacles, then manually refining and processing these in order to remove invalid or in- accessible receptacles. addition, collision proxy meshes were automatically generated and in many cases manually corrected to support physically ac- curate procedural placement of object arrangements.
SC, SI
SC, UI
UC, UI
Total
Cats
85
64
44
129
Insts
1,363
748
424
2,535
Table 2: # of objects in the sim for each split of (S)een and (U)nseen (I)nstance and (C)ategory.
All 200+ scenes with rearrangement support will be released soon.
5

---

Desktop
Ossevvclera (Goak cua)
Prodited Semantle
All model code
(perception policies)
High-lavel planning code
Heuristic policies
Visualizatione (e.g. RVIZ)
Mobile Manipulator
Router
Runs SLAM for robot pose Open-source low- level control stack
Internet connection
Wired to desktop
Wireless
HoboeRobot is a simple, casy-to-set-up library that works in multiple environments and requires only relatively affordable hardware. Computationally intensive operations performed on a desktop PC with GPU, and a dedicated consumer-grade router provides a network interface to a robot running low-level control and SLAM.
Figure 4:
Episode Generation. We generate episodes consisting of varying object arrangements and particular values for object, start_ seaart_rrceptacee, and goal_receptacle. which allow agent to successfully move about and interact with the world. In the case of Open- vece-Voocbbllayy Mobile Manipulation, this task is particularly challenging because we have to place objects in locations that are navigable, meaning that the robot can get to them, reachable, meaning its arm can make it to these locations, and from which we can navigate to a navigable, reachable goal receptacle. For full episode generation details see App. D.2.
Training and Validation Split. Training episodes consist of objects from the large pool of seen instances of seen categories (SC,SI). In contrast, we use unseen instances of seen object categories (SC, UZ) and unseen instances of unseen categories (UC, UZ) for validation and test episodes. Two- thirds of the categories were randomly designated as seen, and two-thirds of the objects in the seen category were randomly marked as seen instances. Splits are in Table and the distribution objects across categories is in App. Fig. 6.
3.2 Real World Benchmark
Real-world experiments are performed in a controlled 3-room apartment environment, with a sofa, kitchen table, counter with bar, and TV stand, among other features. We documented the positioning of various objects and the robot start position, in order ensure reproducibility across trials. Images of various layouts of the test apartment are included in Fig. and task execution is shown in Fig. 16.
During real-world testing, we selected object instances that did not appear in simulation training, split between classes that did and did not appear. We used eight different categories: five seen (Cup, Bowl, Stuffed Toy. Medicine Bottle, and Toy Animal), and three unseen (Rubik's 's cube, Toy Drill, and Lemon). We performed 20 experiments for each of our two different baselines and with seven different receptacle classes: Cabinet, Chair, Couch, Counter, Sink, Stool, Table.
The HomeRobbot Library
To facilitate research on these challenging problems, we open-source the HomeRobot library, which implements navigation and manipulation capabilities supporting Hello Robot's 's Stretch [22]. In our setup, it is assumed that users have access to a mobile manipulator and an NVIDIA GPU-powered workstation. The mobile manipulator runs the low-level controller and the localization module, while the desktop runs the high-level perception and planning stack(Fig. 4). The robot and desktop are connected using off-the-shelf router". The key features of stack include:
Ouu experiments used NetGear Nighthawk router.

---

Transferability; Unified state and action spaces between simulation & real-world settings for each task, providing an easy way to control a robot with either high-level action spaces (e.g., pre-made grasping policies) or low-level continuous joint control.
Modularity: Perception and action components to support high-level states (e.g. semantic maps, segmented point clouds) and high-level actions (e.g. go to goal position, pick up target object). Baseline Agents: Policies that use these capabilities to provide basic functionality for OVMM.
4.1 Baseline Agent Implementation
Crucially, we provide baselines and tools that enable researchers to effectively explore the Open- Vocabulary Mobile Manipulation task. We include two types of baselines in HomeRobot: a heuristic baseline, using motion planning [2] and simple rules for manipulation; and reinforcement learning baseline. In addition, we have implemented example projects from several recently released papers, testing different capabilities such as object-goal navigation [1, 2]. skill learning [24], continual learning [25], and image instance navigation [26]. We implement a high-level policy called OVMMAgent which calls a sequence of skills one after the other. These skills are:
find goal_ receptacc... Gaze: Move close enough to an object to grasp it, and orient head to get good view of the object. The goal of the gaze action is to improve the success rate of grasping.
FindOhj/FindRe:: Locate an object on a ceaat_receptaclee
Pick: Pick up the object. We provide high-level action for this, since we do not simulate the gripper interaction in Habitat. However, our library is compatible with a range of learned grasping skills and supports learning policies for grasping.
Place: Move to location in the environment and place the object on top of the goal_
Specifically, OVMMAgent is Maatem that calls FindObj. Gaze, Pick. FindRec, and Place in that order, where Pick is grasping policy provided by the robot library in the real world. The other skills are created using the approaches given below:
Heuristic. We implement a version using only off-the-shelf learned models and heuristics, noting that previous work in mobile manipulation has used these models great effect (e.g. [62]). Here, DETIC [63] provides masks for open- open-vocabulary set of objects as appropriate for each skill. The start_receptaare example of the heuristic navigation and place policy being executed in the real world (App. E).
object.goal laagggoall_eecpee receptacle for each episode is given. Fig. 16 shows an
RL. We train the four skills our modified version Habitat [21] as policies which predict actions given depth, ground truth semantic segmentation and priopreceptive sensors (i.e. joints, gripper state), using DDPPO [64]. While RGB is available in our simulation, baseline policies do not directly utilize it; instead, they rely predicted segmentation from Detic [27] test time.
5 Results
We first evaluate the two baselines in our simulated benchmark, followed by evaluation in real- world, held-out test apartment. These results highlight the significance of OVMM as a challenging new benchmark, encompassing numerous essential challenges that arise when deploying robots in real-world environments.
We break down the results by sub-task in addition reporting the overall performance in Tables and 4. The columns FindObj. Pick and FindRee refer to the first 3 phases of the task mentioned in the scoring section (Sec. 3), and succeeding in the final Place phase leads to successful episode.
Simulation. We evaluate the baselines on held-out scenes, with objects from unseen instances of seen classes, and unseen instances of unseen classes, as described in See. 3. We show results with two different perception systems: Ground Truth segmentation, where we use the segmentation input directly from the simulator, and DETIC segmentation [27], where the RGB images from the simulator are passed through DETIC, an open-vocabulary object detector.
7

---

Simulation Results
Skill
Partial Success Rates
Overall Success Rale
Partial Success Metric
Perception
Navigation
Gaze
Place
FindObj
Pick
FindRec
Ground Truth
Heuristic
None Heuristic
54.1
48.5
31.5
5.1
34.8
Heuristic
RL
56.5
51.5
42.3
13.2
40.9
None
Heuristic
65.4
43.7
42.8
RI.
RI
RI.
66.6
61.1
50.9
14.8
48.3
DETIC (27)
Heuristic
None
Heuristic
28.7
15.2
5.3
0.4
12.4
Heuristic
RL
29.4
13.2
5.8
12.2
None
Heuristic
21.9
6.0
21.7
10.2
6.2
9.6
Table 3: Partial and overall success rate (SR) (in %) for different combinations of skills and perception systems. The partial SR for each skill is dependent the previous skill's SR. 'he partial SR for the place skill is the same as the overall SR. The partial success metric is calculated by averaging the partial SRs. One of the main causes of failures for our baseline systems was perception; ground-truid perception is notably better. Both RL and heuristic skills struggled to navigate tightly constrained multi-room environments and successfully place objects.
We report results on HomeRobot OVMMin Table 3. RL policies outperformed heuristic methods for both navigation and placement tasks. However, all policies declined in performance when using DETIC instead of ground truth segmentation. Heuristic policies exhibited less degradation in performance compared to RL policies: when using DETIC, the heuristic FindObj policy even outperforms RL. We attribute this to the heuristic policy's ability to incorporate noisy predictions by constructing 2D semantic map, which helps handle small objects that are prone to misclassification. Furthermore, using the learned gaze policy led improved pick performance, except when used in combination with the Heuristic with DETIC perception. Example simulation trajectories can be found in Appendix Figure 18, and comparisons of seen versus unseen categories in Appendix G.2.
Real World. Finally, we conducted a series of experiments in a real-world held-out apartment setting. performed a total 20 episodes, utilizing a combination seen and unseen object classes as our target objects. The results of these experiments are presented in Table RL performed slightly better than the Heuristic baseline, successfully completing 1 extra episode and achieving success rate of 20%. This difference primarily stemmed from the pick and place sub-tasks. In the pick task, the RI Gaze skill plays crucial role in achieving better alignment between the agent and the target object, which leads to more successful grasping, Similarly, the RL place skill demonstrated more precision, ensuring that the object stayed closer to the surface of the receptacle.
Both simulation and real-world results show the baselines are promising, but insufficient, for Mobile Manipulation. DE- TIC [27] caused many failures due to misclas- sification, both simulation and the real world. Further, RL navigation was on par or better than heuristic policies in both sim and real. Although our RL place policy performed better in sim than heuristic place, it needs further improvement in the real world. Gaining the advantages of web-
Real World
FindOhj Pick FindRec Overall Success
Heuristic Only Only
70
35
30
15
70
45
30
20
Table 4: Success Rate (in %) for heuristic and RL. baselines in the real world OVMM task. In both cases, the grasping action is executed as described in Sec. 4; but initial conditions of the robot such its position relative the object or to other obstacles may cause various failures.
scale pretrained lanion-l models like DETIC, but tuned to our agents may be crucial for improving performance.
Conclusions and Future Work
We proposed a combined simulation and real-world benchmark to enable progress on the important problem of cecnn-ooabullayy Mobile Manipulation. We ran extensive experiments showing promising simulation and real-world results from two baselines: heuristic baseline based on a state-of-the-an motion planner [2] and a reinforcement learning baseline trained with DDPPO [64], In the future, we hope to improve the complexity of the problem space, adding more complex natural language and multi-step commands, and provide end-to-end basclines instead of modular policies.

---

7 Acknowledgements
We would like to thank Andrew Szot for help with Habitat policy training, Santhosh Kumar Ra- makrishnan with help on Stretch Object navigation in simulation and on the real robot, and Eric Undersander for help with improving Habitat rendering. Priyam Parashar, Xiaohan Zhang, and Jay Vakil helped with testing on Stretch and real-world scene setup.
We would also like to thank the whole Hello Robot team, but especially Binit Shah and Blaine Mat- ulevich for their help with the robots, and Aaron Edsinger and Charlie Kemp for helpful discussions.
The Georgia Tech effort was supported in part by ONR YIP and ARO PECASE. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies endorsements, either expressed implied, of the U.S. Government, any sponsor.
References
[1] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and E. Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv, 2020.
[2] T. Gervet, S. Chintala, D. Batra, J. Malik, and D. S. Chaplot. Navigating to objects in the real world. arXiv, 2022.
[3] T. Wisspeintner, T. Van Der Zant, L. Iocchi, and S. Schiffer. Robocup@ home: Scientific competition and benchmarking for domestic service robots. Interaction Studies, 2009.
[4] J. Bohren, R. B. Rusu, E. G. Jones, E. Marder-Eppstein, C. Pantofaru, M. Wise, L. Mösenlech- ner, W. Meeussen, and S. Holzer. Towards autonomous robotic butlers: Lessons learned with the pr2. In ICRA, 2011.
[5] W. Burgard, A. B. Cremers, D. Fox, D. Hähnel, G. Lakemeyer, D. Schulz, W. Steiner, and S. Thrun. Experiences with an interactive museum tour-guide robot. Artificial intelligence, 1999.
[6] M. Shridhar, Thomason, D. Gordon, Y. Bisk, Han, R. Mottaghi, L. Zettlemoyer, and Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, 2020.
[7] X. Puig, K. Ra, M. Boben, J. Li, T. Wang. S. Fidler, and A. Torralba. Virtualhome: Simulating household activities via programs. In CVPR, 2018.
[8] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez Integrated task and motion planning. Anmual Review of Control, Robotics, and Autonomous Systems, 4:265 293,, 2021.
[9] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler. Open-vocabulayy queryable scene representations for real world planning. arXiv, 2022.
[10] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q Vuong, P. Wohlhart, B. Zitkovich, F. Xia, C. Finn, et al. Open-world object manipulation using pre-trained vision-language models. arXiv, 2023.
[1I] N. M. M. Shafiullah, C. Paxton, L. Pinto, Chintala, and A. Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. arXiv, 2022.
[12] B. Bolte, A. Wang. J. Yang, M. Mukadam, M. Kalakrishnan, and C. Paxton. Usa-net: Unified semantic and affordance representations for robot memory. arXiv, 2023.

---

[13] A. Curtis,
X. Fang, L. P. Kaelbling, T. Lozano- Pérez, and C. R. Garrett. Long-horizon manipulation of unknown objects via task and motion planning with estimated affordances. In ICRA, 2022.
[14] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In ICML, 2021.
[15] K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, S. Li, G. Iyer, S. Saryazdi, N. Keetha, A. Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv, 2023.
[16] J. Krantz, E. Wijmans, A. Majundar, D. Batra, and S. Lee. Beyond the nav-graph: Vision and language navigation in continuous environments. In European Conference on Computer Vision (ECCV), 2020.
[17] W. Liu, Hermans, S. Chermova, and C. Paxton, Structdiffusion: Object-centric diffusion for semantic rearrangement of novel objects. arXiv, 2022.
[18] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv, 2023.
[19] M. Khanna*, Y. Mao*, H. Jiang, S. Haresh, B. Shacklett, D. Batra, A. Clegg, E. Undersander,
A. X. Chang, and M. Savva. Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of
3D Scene Scale and Realism Tradcoffs for ObjectGoal Navigation. arXiv preprint, 2023.
[20] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun,
J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied AI Research. ICCV, 2019.
[21] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. In NeurIPS, 2021.
[22] C. C. Kemp, A. Edsinger, H. M. Clever, and B. Matulevich. The design of stretch: A compact, lightweight mobile manipulator for indoor human environments. In ICRA, 2022.
[23] S. Yenamandra, A. Ramachandran, M. Khanna, K. Yadav, D. S. Chaplot, G. Chhablani, A. Clegg, T. Gervet, V. Jain, R. Partsey, R. Ramrakhya, A. Szot, T.-Y. Yang, A. Edsinger, C. Kemp. B. Shah, Z. Kira, D. Batra, R. Mottaghi, Y. Bisk, and C. Paxton. Homerobot open vocab mobile manipulation challenge 2023. aetps://aihabital ceeeit.or//chhhe http//hhhleeng/2233... homerobot_ovmm/ 2023.
[24] P. Parashar, J. Vakil, S. Powers, and C. Paxton. Spatial-language language attention policies for efficient robot learning. arXiv, 2023.
[25] S. Powers, A. Gupta, and C. Paxton. Evaluating continual learning on home robot, 2023.
[26] J. Krantz, T. Gervet, K. Yadav, A. Wang, C. Paxton, R. Mottaghi, D. Batra, J. Malik, S. Lee, and D. Chaplot. Navigating to objects specified by images. arXiv, 2023.
[27] X. Zhou, R. Girdhar, Joulin, P. Krähenbühl, and I. Misra. Detecting twenty-thousand classes using image-level supervision. In ECCV, 2022.
[28] I. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi. Visual room rearrangement. In CVPR,
2021.
[29] K. Yadav, Krantz, R. Ramrakhya, S. K. Ramakrishnan, Yang, A. Wang, J. Turner, A. Gokaslan, V.P. Berges, R. Mootaghi, O. Maksymets, A. X. Chang, M. Savva, Clegg, D. Chaplot, and D. Batra. Habitat challenge 2023. sctps://aihabitat org/ challenge/ 2023/, 2023.
10

---

[30] C. Gan, J. Schwartz. S. Alter, D. Mrowca, M. Schrimpf. J. Traer, J. De Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, et al. Threedworld: A platform for interactive multi-modal physical simulation. NeurlPS Datasets and Benchmarks Track, 2021.
[31] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba. Virtualhome: Simulating household activities via programs. In CVPR, 2018.
[32] M. Deitke, E. VanderBilt, A. Herrasti, L. Weihs, J. Salvador, K. Ehsani, W. Han, E. Kolve, A. Farhadi, A. Kembhavi, and R. Mottaghi. Procthor: Large-scale embodied ai using procedural generation. In NeurlPS, 2022.
[33] M. Deitke, W. Han, A. Herrasti, A. Kembhavi, E. Kolve, R. Mottaghi, J. Salvador, D. Schwenk, E. VanderBilt, M. Wallingford, L.. Weihs, M. Yatskar, and A. Farhadi. RoboTHOR: An Open Simulation-to-Real Embodied AI Platform. In CVPR, 2020.
[34] C. Li, R. Zhang. J. Wong, C. Gokmen, S. Srivastava, Martin-Martin, C. Wang, Levine, M. Lingelbach, J. Sun, et al. Behavior-Ik: Ik: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In CoRL, 2023.
[35] T. Mu, Z. Ling, F. Xiang, D. C. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. In NeurIPS Datasets and Benchmarks Track, 2021.
[36] E. Krotkov, D. Hackett, L. Jackel, M. Perschbacher, J. Pippine, J. Strauss, G. Pratt, and Orlowski. The darpa roboties challenge finals: Results and perspectives. The DARPA Robotics Challenge Finals: Humanoid Robots To The Rescue, 2018.
[37] G. Seetharaman, A. Lakhotia, and E. P. Blasch. Unmanned vehicles come of age: The darpa grand challenge. Computer, 2006.
[38] M. Buehler, K. lagnemma, and S. Singh. The DARPA urban challenge: autonomous vehicles in city traffic. Springer Berlin, Heidelberg, 2009.
[39] N. Correll, K. E. Bekris, D. Berenson, O. Brock, A. Causo. K. Hauser, K. Okada, A. Rodriguez, J. M. Romano, and P. R. Wurman. Analysis and observations from the first amazon picking challenge. IEEE Transactions on Automation Science and Engineering, 2016.
[40] L. D. Jackel, R. Krotkov, M. Perschbacher, J. Pippine, and C. Sullivan. The darpa lagr program: Goals, challenges, methodology, and phase i results. Journal of Field Robotics, 2006.
[41] M. Müller and V. Koltun. Openbot: Turning smartphones into robots. In ICRA, 2021.
[42] N. Kau, A. Schultz, N. Ferrante, and P. Slade. Stanford doggo: An open-source, quasi-direct- drive quadruped. In ICRA, 2019.
[43] F. Grimminger, A. Meduri, M. Khadiv, J. Viereck, M. Wüthrich, M. Naveau, V. Berenz, S. Heim, F. Widmaier, J. Fiene, A. Badri--Srööitz, and L. Righetti. An open torque-controlled modular robot architecture for legged locomotion research. IEEE Robotics and Automation Letters, 2019.
[44] B. Yang, J. Zhang, V. H. Pong, S. Levine, and D. Jayaraman. Replab: A reproducible low-cost arm benchmark platform for robotic learning. arXiv, 2019.
[45] D. V. Gealy, S. McKinley, B. Yi, P. Wu, P. R. Downey, G. Balke, A. Zhao, M. Guo, R. Thomas- son, A. Sinclair, P. Cuellar, Z. McCarthy, and P. Abbeel. Quasi-direct drive for low-cost compliant robotic manipulation. In ICRA, 2019.
[46] M. Wüthrich, F. Widmaier, Grimminger, Joshi, V. Agrawal, B. Hammoud, M. Khadiv, M. Bogdanovic, V. Berenz, J. Viereck, M. Naveau, L. Righetti, B. Schölkopf, and S. Bauer. Trifinger: An open-source robot for learning dexterity. In CoRL, 2020.

---

[47] M. Ahn, H. Zhu, K. Hartikainen, H. Ponte, A. Gupta, S. Levine, and V. Kumar. ROBEL: RObotics BEnchmarks for Learning with low-cost robots. In CoRL,, 2019.
[48] A. Murali, T. Chen, K. V. Alwala, D. Gandhi, L. Pinto, S. Gupta, and A. K. Gupta. Pyrobot: An open-source robotics framework for research and benchmarking. arXiv, 2019.
[49] L. Paull, J. Tani, H. Ahn, J. Alonso-Mora. L. Carlone, M. Cáp, Y. F. Chen, C. Choi, J. Dusek, Y. Fang, D. Hochener, S. Liu, M. M. Novitzky, I. F. Okuyama, J. Pazis, G. Rosman, V. Varric- chio, H.-C. Wang, D. S. Yershov, H. Zhao, M. R. Benjamin, C. Carr, M. '. Zuber, S. Karaman, E. Frazzoli, D. D. Vecchio, D. Rus, J. P. How, J. J. Leonard, and A. Censi. Duckietown: An open, inexpensive and flexible platform for autonomy education and research. In ICRA, 2017.
[50] Calli, A. Singh, A. Walsman, Srinivasa, P. Abbeel, and M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In ICRA, 2015.
[51] D. Morrison, P. Corke, and J. Leitner. Egad! evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation. IEEE Robotics and Automation Letters, 2020.
[52] B. Yang, P. E. Lancaster, S. S. Srinivasa, and J. R. Smith. Benchmarking robot manipulation with the rubik'ss 's cube. IEEE Robotics and Automation Letters, 2020.
[53] Murali, Chen, V. Alwala, D. Gandhi, Pinto, Gupta, and A. Gupta. Pyrobot: An open-source robotics framework for research and benchmarking. arXiv, 2019.
[54] S. Dasari, J. Wang. J. Hong, S. Bahl, Y. Lin, A. S. Wang, A. Thankaraj. K. S. Chahal, B. Çalli, S. Gupta, D. Held, L. Pinto, D. Pathak, V. Kumar, and A. Gupta. Rb2: Robotic manipulation benchmarking with twist. arXiv, 2022.
[55] G. Zhou, V. Dean, M. K. Srirama, A. Rajeswaran, J. Pari, K. B. Hatch, A. Jain, T. Yu, P. Abbeel, L. Pinto, Finn, and A. Gupta. Train offline, test online: real robot learning benchmark. arXiv, 2022.
[56] K. Kimble, K. Van Wyk, J. Falco, E. Messina, Y. Sun, M. Shibata, W. Uemura, and Y. Yokoko- hji. Benchmarking protocols for evaluating small parts robotic assembly systems. IEEE Robotics and Automation Letters, 2020.
[57] W. Lian, T. Kelch, D. Holz, A. Norton, and Schaal. Benchmarking off-the-shelf solutions to robotic assembly tasks. IROS, 2021.
[58] A. Kadian, Truong, A. Gokaslan, A. Clegg, E. Wijmans. S. Lee, M. Savva, S. Chernova, and D. Batra. Are making real progress in simulated environments? measuring the sim2real gap in embodied visual navigation. arXiv, 2019.
[59] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. AI2-THOR, an interactive 3d environment for visual AL. arXiv, 2017.
[60] J. Collins, S. Goel, A. Luthra, I. Xu, K. Deng, X. Zhang, T. F. Y. Vicente, H. Arora, T. Diderik- M. Guillaumin, et al. Abo: Dataset and benchmarks for real-world 3d object understanding. In CVPR, 2022.
[61] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T. B. McHugh, and V. Vanhoucke. Google scanned objects: A high-quality dataset 3d scanned houschold items. In ICRA, 2022.
[62] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng Song, Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv, 2023.
12

---

[63] T. Zhou, R. Tucker, J. Flynn. G. Fyffe, and N. Snavely. Stereo magnification: Learning view synthesis using multiplane images. SIGGRAPH, 2018.
[64] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. In ICLR, 2019.
[65] J. Deng, W. Dong, R. Socher, L.-. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[66] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
[67] Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. Piqa: Reasoning about physical common- sense in natural language. In AAAI, 2020.
[68] M. Sap, Rashkin, D. Chen, R. LeBras, and Y. Choi. SocialIQA: Commonsense reasoning about social interactions. EMNLP, 2019.
[69] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[70] S. Keisuke, I.. B. Ronan, B. Chandra, and C. Yejin, Winogrande: An adversarial winograd schema challenge at scale. AAAI, 2019.
[71] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[72] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
[73] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In ICRA, 2016.
[74] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. IJRR, 2018.
[75] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng. Do as i can and not as say: Grounding language in robotic affordances. CoRL, 2022.
[76] Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, Tung. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In CoRL, 2018.
[77] P. Sharma, L. Mohan, L. Pinto, and A. K. Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In CoRI., 2018.
[78] J. Mahler, Liang, Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metries. arXiv, 2017.
[79] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn. Robonet: Large-scale multi-robot learning, arXiv, 2019.
13

---

[80] A. Gupta, A. Murali, D. P. Gandhi, and L. Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. In NeurIPS, 2018.
[81] P. Anderson, Q. Wu, D. Teney, Bruce, M. Johnson, N. Sünderhauf, I. D. Reid, S. Gould, and van den Hengel. vision-and-languus navigation: Interpreting -araallg navigation instructions in real environments. CVPR, 2017.
[82] H. Team. Habitat CVPR challenge, 2019. URL httpps//aahaabiaat.oooo/bbb 2019/.
[83] F. Xia, W. B. Shen, C. Li, P. Kasimbeg, M. Tchapmi, A. Toshev, R. Martín-Martín, and S. Savarese. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments. IEEE Robotics and Automation Letters, 2020.
[84] Chen, U. Jain, C. Schissler, S. V. Gari, 7.. Al-Halah, V. K. Ithapu, P. Robinson, and Grauman. Soundspaces: Audio-visual navigation 3d environments. In ECCV, 2020.
[85] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and
R. Mottaghi. Manipulathor: A framework for visual object manipulation. In CVPR, 2021.
[86] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRI, 2019.
[87] S. James, Z. Ma, D. R. Arrojo, and A. Davison. Rlbench: The robot learning benchmark learning environment. IEEE Robotics and Automation Letters, 2020.
[88] A. Ku, Anderson, R. Patel, E. Ie, and J. Baldridge. Room-across-room: Multilingual vision-andd-lnguuage navigation with dense spatiotemporal grounding. In EMNLP, 2020.
[89] A. Padmakumar, J. Thomason, A. Shrivastava, P. Lange, A. Narayan-Chen, S. Gella, R. Pi- ramithu, G. Tur, and D. Hakkani-Tur. TEACh: Task-driven embodied agents that chat. In AAAI, 2022.
[90] X. Gao, Q. Gao, R. Gong, K. Lin, G. Thattai, and S. Sukhatme. Dialfred: Dialogue- agents for embodied instruction following. IEEE Robotics and Automation Letters, 2022.
[91] Szot, K. Yadav, A. Clegg, V.-P. Berges, A. Gokaslan, Chang, M. Savva, Z. Kira, and D. Batra. Habitat rearrangement challenge. https://aihabican org/ htggcchhllenge22222.. rearrange, 2022.
[92] C. M. Kim, M. Danielczuk, Huang, and K. Goldberg. Simulation of parallel-jaw grasping using incremental potential contact models. ICRA, 2022.
[93] D. Hall, B. Talbot, and N. Sünderhauf. The robotic vision challenges. https:// 2022.
nikosuenderhauf http://..o//rooottoo//saoooaaaaa
[94] A. Kurenkov, R. Martin-Martín, J. Ichnowski. K. Goldberg, and S. Savarese. Semantic and geometric modeling with neural message passing in 3d scene graphs for hierarchical mechanical search. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11227-1122 IEEE, 2021.
[95] K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, Reid, and N. Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. arXiv preprint arXiv:2307.00 30..0.63...,.. 2023.
[96] J. Chen, G. Li. S. Kumar, B. Ghanem, and F, Yu. How not train your dragon: Training-free embodied object goal navigation with semantic frontiers. arXiv preprint 265.923305.16925, 2023.
14

---

[97] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet Efficient 6-dof grasp generation in cluttered scenes. In ICRA, 2021.
[98] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox. 6-dof grasping for target-driven object manipulation in clutter. In ICRA, 2020.
[99] H.-S. Fang, Wang. M. Gou, and C. Lu. Craspnet-lbiilion A large-scale benchmark for general object grasping. In CVPR, 2020.
[100] C. R. Garrett, C. Paxton, T. Lozano-Pérez, L. P. Kaelbling, and D. Fox. Online replanning in belief space for partially observable task and motion problems. ICRA, 2020.
[101] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In ICCV, 2019.
[102] Paxton, C. Xie, Hermans, and D. Fox. Predicting stable configurations for semantic placement of novel objects. In CoRL, 2022.
[103] S. Kohlbrecher, J. Meyer, O. von Stryk, and U. Klingauf. A flexible and scalable slam system with full motion estimation. In Proc. IEEE International Symposium on Safery, Security and Rescue Robotics (SSRR). IEEE, November 2011.
[104] J. J. Kuffner and S. M. Valale. Rrt-connect: An efficient approach to single-query path planning. ICRA, 2000.
[105] D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In NeurlPS, 2020.
[106] Yamauchi. A mootteer-based approach for autonomous exploration. In IEEE International Symposium on Computational Intelligence in Robotics and Automation, 1997.
[107] J. A. Sethian. Fast marching methods. SIAM review, 1999.
[108] D. S. Chaplot, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning to explore using active neural mapping. ICLR, 2020.
[109] M. Caron, Touvron, I. Misra, Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. CVPR, 2021.
[110] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y Lo, et al. Segment anything. arXiv preprint arXiv:2304.02644, 2023.
[111] K. M. Jatavallabhula, S. Saryazdi, G. Iyer, and L. Paull. gradslam: Automagically differentiable slam. arXiv preprint 1910. 10672, 2019.
[112] Q. Gu, A. Kuwajerwala, S. Morin, M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. arXiv preprint arXiv:2309.16650 2023.
[113] M. Quigley, K. Conley, Gerkey, J. Faust, Foote, J. Leibs, R. Wheeler, A. Ng. et al. Ros: an open-source robot operating system. In ICRA Workshop on Open Source Software, 2009.
15

---

Appendix
Table of Contents
A Extended Related Work
16
B Limitations
17
C Metrics
18
C.I Simulation Success Metries
18
C.2 Real World Success Metrics
18
D Simulation Details
19
D.1 Object Categories Appearing in the Scene Dataset
19
D.2
Episode Generation Details
21
D.3 Diversity in Receptacle Instances
22
D.4 Scene Clutter Complexity
22
D.5 Improved scene visuals
23
D.6
Action Space Implementation
24
E
HomeRobot Implementation Details
24
E.1 Pose Information
25
E.2
Low-Level Control for Navigation
25
E.3
Heuristic Grasping Policy
25
E,4 Heuristic Placement Policy
26
E.5 Navigation Planning
27
E.6
Navigation Limitations
28
F
Reinforcement Learning Baseline
28
F11
Action Space
29
F2
Observation Space
29
F3
Training Setup
29
F.4
ConceptFusion
31
G
Additional Analysis
31
G.1
Number of steps taken in each stage by different baselines
32
G.2
Performance on Seen vs. Unscen Object Categorics
33
H Hardware Setup
34
H.1
Hardware Choice
34
H.2
Robot Sctup
35
H.3 Visualizing The Robot
37
H.4 Using The Stretch: Navigation vs. Position Mode
37
A Extended Related Work
difficult to justice to the rich embodied AI, natural language, computer vision, machine learning, and robotics communities that have addressed aspects of the work presented here. The following extends some of the discussion from the main manuscript about important advances that the community has made.
16

---

Benchmarks have helped the community focus their efforts and fairly compare system performance. For example, the YCB objects [50] allowed for direct comparison of results across manipulators and models. While benchmarks and leaderboards are comparatively rare in robotics [49, 56, 33, 54, 63, 3. 3,3,, they have been hugely influential machine learning (e.g. ImageNet [65], GLUE [66], and various language benchmarks [67-70]. COCO [71], and SQuAD [72]). In robotics, competitions such RoboCup @ Home [3], the Amazon Picking Challenge [39], and the NIST task board [56] prevalent and influential as alternative, but generally systems aren't reproducible across teams.
Datasets, In addition to the environments referenced in Table 1, offline datasets including robot interactions with scenes have been used widely to train models. These datasets are typically obtained using robots alone (e.g.. [73, 74]). by teleoperation (e.g.. [75, 76)) or buman-robot demonstration (e.g., [77]). Previous work such [78] aim to collect large-scale datasets while works such as [79] consider scaling across multiple embodiments. [80] take step further by collecting robot data in unstructured environments. Unlike these works, we do not limit our users specific dataset. Instead, we provide a simulator with various scenes that can generate large-scale consistent data for training. Also, note that we test the models in unseen environments, while most the mentioned works use the same environment for training and testing.
Simulation benchmarks. The embodied AI community has provided various benchmarks in simulation platforms for tasks such as navigation [1, 81-84], object manipulation [85, 35, 86, 87], instruction following [6, 88-90], room rearrangement [28, 91], grasping [92] and SLAM [93]. While these benchmarks ensure reproducibility and fair comparison of different methods, there is always a gap between simulation and reality since it is infeasible to model all details of the real world in simulation. Our benchmark, in contrast, enables fair comparison of different methods and reproducibility of the results in the real world. Additionally, previous benchmarks often operate in a simplified discrete action space [20, 6). even forcing that structure on the real world [2].
Robotics benchmarking. Robotics benchmarks must contend with the diversity of hardware, morphology, and resources across labs. One solution is simulation [87, 59, 35, 20, 21, 83, 86, 6], which can provide reproducible and fair evaluations. However, the sim-to-real gap means simulation results may not be indicative of progress in the real world [2]. Another proposed solution robotic competitions such as RoboCup @ Home [3], the Amazon Picking Challenge [39], and the NIST task board [56]. However, participants typically use their own hardware, making it difficult conduct fair comparisons of the different underlying methods, and means results are not transferable different labs or settings. This is also a large barrier to entry to these competitions.
Exploration of unseen environments. Various papers have looked at object search in different home environments. Gervet al. [2] use a mixture of heuristic/model based planning and reinforcement learning to achieve strong results in a variety of real-world environments; importantly their planning- based methods perform competititvely to the best learned methods, and much better than end-to-end reinforcement learning real tasks. Also promising is graph-based exploration. Kurenkov et al. [94] propose hierarchical mechanical search, based on 3d scene graph representation, for exploring environments; although unlike in our Open- -ocaablayyyl Mobile Manipulation task, they assume such a scene graph exists. Similarly, SayPlan [95] performs search over a compex scene graph using a large language model; however, this approach also does not look into iteratively constructing this scene graph on new scenes. While there is active work in iteratively exploring and building scene graphs and other hierarchical representations (e.g. (96)), there not yet strongly established methods in this space.
B Limitations
Due to simulation limitations, we don't physically simulate grasping in the current version of the benchmark, which is why we provide separate policy for this in the real world. Grasping is well-studied problem [97-99], but simulations that train useful real-world grasp systems require special consideration. We also currently consider full natural language queries out-of-scope. Finally,
17

---

we did not evaluate many motion planners (see Sec. E.2), performed lesk-and-motinn planning with replanning, as would be ideal for a long horizon task [100].
C
Metrics
We informally defined our scoring metries in Sec. 3. Here, we provide formal definitions of partial success metrics.
C.1
Simulation Success Metrics
Success in simulation is defined per stage as:
FindObj: Successful if the agent reaches within 0.1m of a viewpoint of the target object on start_receptacle, and at least 0.1% of the pixels in its camera frame belong object instance.
Pick: Successful if FindObj succeeded, the agent enables the gripper at an instant where an object instance is visible and its end-effector reaches within 0.8m of target object. We magically snap the object to the agent's gripper in simulation.
FendRec: Successful if Pick succeeded, and the agent reaches within Om of a viewpoint of haall_recettccl. and at least 0.1 of the pixels its camera frame belong to the object containing a valid receptacle.
Successful if FindRec succeeded, the agent releases the object and subsequently the object stays in contact with the threshold of 5e-3 m/s and 5e-2 rad/s respectively for 50 contiguous steps. Further, the agent should not collide with the scene while attempting to place the object.
Place:
with linear and angular velocities below
An episode is considered to have succeeded it succeeds in all 4 stages within 1250 steps.
Better pick success condition. We plan to use more realistic grasping condition in simulation. We try replacing the magic snap simulation with stricter condition that requires the agent to move its arm near the object without colliding with the scene or other objects. Additionally, we tested baseline (Figure 5) that performs top-down grasps resembling resorting to side-ways ways grasps when the object is farther. While this bascline succeeds in reaching the object starting from object viewpoint 79% of the time, it does so without colliding only 47% of the time.
real-world grasping policy, and
C.2 Real World Success Metrics
Success in real world is defined per stage as:
FindObj: Successful if the agent reaches within 1m of the target object on start_receptacle and the object is visible in the RGB image from the camera.
Pick: Successful if FindObj succeeded and the agent successfully picks up the object from the start_receptacle.
FindRec: Successful if Pick succeeded, and the agent reaches within Im of a receptacc,,,,,,, and the goal_receptaci_ is visible the RGB image from the camera.
Place: Successful if FindRec succeeded and the agent places object goal_
and the object settles down on the goal_ seal_receptacll stably.
Given that the scene use in the real world is much smaller than the apartments the simulation, allow the agent to act in the environment for 300 timesteps. The episode is considered to have succeeded if it succeeds in all 4 stages.
18

---

Top-down grasp success
Sideways grasp success
Grasp failures
Figure 5: A few success and failure cases for simple grasping policy under the new grasp condition that requires the agent's arm to reach near the object without colliding. The agent resorts to sideways grasps when the object can't 't be reached via top- down grasp that bends the gripper. Most grasping failures are because of the collisions with the scene.
D Simulation Details
D.1 Object Categories Appearing the Scene Dataset
action_ action_figure, android_figure apple, backpack, baseballbat, baaket, basketball, bath_towe towel, aattery_chargee board_game, book, bottle, boul, box, bread, bundt_pan, outter_dish, c-clamp, cake_pan, can, can_opener, candle, candle_holder, candy_bar, canister, carrying_case caaserole, cellphone, clock, cloth, credit_card, cup, cushion, dish, doll, dumbbell, egg, electric.kettle, electronic_cable file_sorter, folder, fork, gaming_console, glass, hammer, hand_towel, handbag, hard_drive, hat, helmet, jar, jug, kettle, keychain, knife, ladle, lamp, laptop, laptop_cover, laptop_stand, lettuce, lunch_box, milk_frother_cam monitor_stant mouse_pad, multiport_hub neuspaper, pan, pen, pencil_case, phone_stand, picture_frame, pitcher, plant_container, plant_saucer, plate, plunger, pot, potato, ramekin, remote, salt_and Betttaandpeppe__shaeee scissors, screadriver, shoe, soap, soap_dish, soap_ dispenser, spatula, apectacles, spicemill, sponge, spoon, spray_bottle, squeezer, statue, deuffed_toy, sushi_mat, tape, teapot, tennis_racquet, tissue_box toiletry, tomato, toy_airplane, toy_animal, toy_bee, toy_cactus, toy_ www.construction.cs joy_fire_trued foodfood, truift toy_lamp, toy toy-pineapple, toy_rattle, toy tray, toy_refrigerator, toy_sink, toy_sofa, toy_swing, toy_table, toy_vehicle, wine_bot bottle
vase, titeo_game_cartridge. watch, watering_can,
In Fig. 7 we show some of the examples of selection of these categories from the training and validation/test splits.
19

---

cretion
Sear Catugarias
Single seen instance categories; watch. loy swing. toy pincapple. loy food, loy fire truck. toy cactus, sushi mat, stuffed toy, spoon, tacles, spatula, seap dish. driver. scisons, ramekinn pitcher. mouse pad, monitor stand, milk frother cup. lunch
Pistrr frame
Interoos
10
Meffeed ingeenner
botte
pem
tiatoo
cardo
can ppiteral
hub
Retion figurss
W (and towel
ponbaine
laptop cover, lamp, ladle, keychain. hat. handbaag. hammer, fork. folder. sorier, carrying case. candy har, candle holder, cake pan, c- clamp, butter dish, bundt bath lowel, baskeiball
10
pia Kes unliphooe
Sprcemil
tsaiii recrual Kofti
elosblo Knttlo
tipe
Laging stang
vase
Cotegorios
10
olate
post
leftuce
toilstyy
plant satiear
sail and papper shaker
Single instance categories: wa- lering can, video game cartridge. utensil holder cup, toy table, toy sofa. toy sink, toy refrigerator, toy lamp, tuy iruits, tay construction hee, tissue box, squeezer, soap. helmet, electronic cable. dull, disht cun opener, batlery charger, backpack
clstth
pencil caste
hard divve
newspaper
antraid figure
10
contnter
plunger
remado
banket
wine bottle
toy aipians
phano stand
can
toy mitle
uponge
Jar
Figure 6: Number of objects across different splits, for both seen categories and unseen categories. We divide objects between categories which appear in training data seen categories and those that do not unseen categories. The goal of Open- Open-Vocabulans Vocabulary Mobile Manipulation is to be able to find and manipulate objects specified by language.
cushion
cup
pan
vase
plate
plant saucer
Example objects in our object dataset across 6 categories. The cushion, cup. and pan categories are in the train split, and the vase, plate, and plant saucer are the validation and sets.
Figure 7:
20

---

D.2 Episode Generation Details
Figure 8: Visualization of the navigable geometry (top row) and top-down views of example scenes from the Habitat Synthetic Scenes Dataset (HSSD) [19]. We use the computed navigable area to efficiently generate large number of episodes for the Open- Vocabulary Mobile Manipulation task. Object placement positions are sampled to be near navigable areas of the map, atop one a large variety of different receptacles, such that the robot reach them.
When generating episodes, we find the largest indoor navigable area in each scene, and then filter the list of all receptacles from this scene that are too small for object placement. Fig. 8 shows the navigable islands in several of our scenes (top row), and corresponding top-down views of each scene in the bottom row. We then sample objects according to the current split (train, validation, or test). We run physics to ensure that objects are placed in stable locations. Then we select objects randomly from the appropriate set, as determined by the current split.
Figure 9: First-person view from different precomputed viewpoints in our episode dataset. These viewpoints are used as goals for training navigation skills, and are used in the initialization of the placement and gaze/grasping skills as well. The purple mesh indicates receptacle surface. Finally, generate set of candidate viewpoints, shown in Fig, 9, which represent navigable locations to which the robot can move for each receptacle. These are used for training specific skills, such as navigation to receptacles. Each viewpoint corresponds to particular start_receptar receptacle or roal__receptcce, and represents nearby location where the robot can see the receptacle and within 1.5 meters. Fig. 10 gives examples of where these viewpoints are created.
Navmesh: We precompute navigable scene geometry done in [20] for faster collision checks of the agent with the scene. The "mesh" comprising this navigable geometry is referred to navmesh.
21

---

Figure 10: Viewpoints created for an object during episode generation. The gray area the navigable region of the scene. The big red dot and the black box are the object's s center and bounding box respectively. The surrounding dots are viewpoint candidates: red dots were rejected because they weren't t navigable, and blue dots were rejected because they were too far from the object. The green dots are final set viewpoints.
Figure 11: The variation in instances belonging the "table" category in our dataset.
Number of objects: This is dynamically set per scene to 1.5-2x the total available receptacle area in m For example, if the total receptacle surface area for seene is 10m?, then 15-20 objects will be placed. The exact number of objects will be randomly selected per episode to be in this range.
The full set of included receptacles in simulation is: bathtub, bed, bench, cabinet, chair, chest_or of of_drfwers_ couch, counter, alling_cabinet hamper, , serving cart, shelves, shoe_rack, sink, stand, stool, table, toilet, trunk, wardrobe, & washer_dryer
D.3 Diversity in Receptacle Instances
The instances within each receptacle category exhibit substantial variability. Figure 11 shows a few different receptacles from our dataset belonging to the "table" category.
D.4 Scene Clutter Complexity
Our procedural placement of target and distractor objects creates diverse and interesting scenarios that require reasoning over which direction to approach receptacles, stable placement clutter, open vocabulary object detection under occlusion which makes the task quite challenging. Figure 12 shows few examples of clutter surrounding target objects in scenes.
22

---

Statue on chair
Spicemill on table
Lunchbox on cabinet
Figure 12: A few examples of clutter surrounding the target object in our simulation settings.
Figure 13:
Here we present the improvements in scene visuals with Horizon-based Ambient Occlusion (HBAO) and expanded Physics-based Rendering (PBR) material support added to the Habitat renderer. The top row shows images from the default renderer whereas the bottom row shows the improved renderings.
D.5 Improved scene visuals
We rewrote and expanded the existing Physically- -aasee Based Rendering shader (PBR) and added Horizon- based Ambient Occlusion (HBAO) to the Habitat renderer, which led to notable improvements in viewing quality which were necessary for using the HSSD [19] dataset.
Rewrote PBR and Image Based Lighting (IBL) base calculations.
Added multi-layer material support covering
XER_materrials_clearroo
KHR_materials_epp sppcuuaar,, [HR_materials_ion and wwwaaatrrialssaiisttttoeoe for both direct and indirect (IBL. lighting.
Added tangent frame synthesis precomputed tangents are not provided.
Added HDR Environment map support for IBL.
present comparisons between default Habitat visuals and improved renderings in Figure 13.
We also benchmark the ObjectNav training speeds of a DDPPO-basse RL agent with and without the improved rendering and present the results in 14, We see that the improvement in scene lighting and
23

---

perf/pp
perf/ (ps, window tag perl/ppa_ window
1i perl/pp
340
m
385
Without HBAO and PBR With HBAO and PBR
330
320
315
370
310
310
Figure 14: Minor drop in FPS with improved scene rendering: Here, we benchmark the training speeds (through FPS numbers) of two ObjectNav training runs with and without the HBAO and PBR-based improved scene visuals. We observe that the improved rendering leads to a very small drop in FPS from around 340 to 330 (3 % drop).
rendering comes at the cost only a 3% dip in training FPS (decreasing from around 340 to around 330).
D.6 Action Space Implementation
We look at different choices of action space for navigation agents, either making discrete continuous predictions about where to move next. Our expectation from prior work might be that the discrete action space would be notably easier for agents to work with.
Discrete, Previous benchmarks often operate in a fully discrete action space [20, 6]. even in the real world [2]. We implement translation of steps 0. 0.25m forward.
discrete actions, with fixed in-place rotation left and right, and
Continuous. Our continuous action space is implemented as teleporting agent, where the robot needs to move around by predicting a local waypoint. Our robot's low-level controllers are expected to be able to get the robot to this location, in lieu of simulating full physics for the agent.
In simulation, this is implemented if the robot will go into collision with any objects if moved towards the new location, and move it to the closest valid location instead.
check against the navmesh
the navmesh determine
E HomeRobot Implementation Details
Here, describe more specifies for how implemented the heuristic policies provided as baseline to accelerate home assistant robot research.
Although there exists a considerable body of prior research looking at learning specific grasping [101, 98, 99, 97] or placement [102, 17] skills, we found that was easiest to implement heuristic policies with low CPU/GPU requirements and high interpretability. Other recent works have similarly used heuristic grasping and placement policies to great affect (e.g. TidyBot [62]).
There are three different repositories within the open-source HomeRobot library:
home_robote Shared components such as Environment interfaces, controllers, detection and segmentation modules.
Rome_robot__stn Simulation stack with Environments based on Habitat.
Come_robot_her Hardware stack with server processes that runs the robot, client API that runs the GPU workstation, and Environments built using the client API.

---

Most policies are implemented in the core home_robor library. Within HomeRobot, we also divide functionality between Agents and Environments, similar to how many reinforcement learning benchmarks are set up [20].
Agents contain all of the necessary code execute policies. We implement agents which mixture of heuristic policies and policies learned on our scene dataset via reinforcement learning.
Environments provide common logie; they provide Observations to the Agent, and function which allows them to apply their action to the (real or simulated) environment.
E.1
Pose Information
We get the global robot pose from Hector SLAM [103] on the Hello Robot Stretch [22], which is used when creating 2d semantic maps for model-based navigation policies.
E.2 Low-Level Control for Navigation
The Hello Stretch software provides native interface for controlling the linear and angular velocities of the differential
robot base. While we do expose an interface for users to control these velocities directly, it is desireable to have desired short-term goals as more intuitive action space for policies, and to make them update-able at any instant to allow for replanning.
Thus, we implemented a velocity controller that produces continuous velocity commands that moves the robot to an input goal pose. The controller operates in a heuristic manner: by rotating the robot so that it faces the goal position at all times while moving towards the goal position, and then rotating to reach the goal orientation once goal position is reached. The velocities to induce these motions are inferred with trapezoidal velocity profile and some conditional checks to prevent from overshooting the goal.
Limitations The Fast Marching Method- motion planning from prior work [2] that we describe in Sec. E.2. It assumes the agent is a cylinder, and therefore is much more limited in where can navigate than, e.g., sampling based motion planner like RRT-connect [104] which can take orientation into account. In addition, our semantic mapping requires list of classes for use with DETIC [27]; instead, it would be good to use fully ppen-vocabulary scene representation like CLIP-Fields [11], ConceptFusion [15], USA-Net [12], which would also improve motion planning significantly.
E.3 Heuristic Grasping Policy
Grasping tests in various lab environments. To provide strong baseline, we tuned the grasp policy to be highly reliable given the Stretch's viewpoint, on variety of objects. Numerous powerful grasp generation models have been proposed in the literature, such GraspNet- IBillion [99], 6-DOF GraspNet [98], and Contact-GraspNer [97]. However, for transparency, repro-
Figure 15:
25

---

An example of the robot navigating to goal_receptacle (sofa) and using the heuristic place policy to put down the object (stuffed animal). Heuristic policies provide an interpretable and easily extended baseline.
Figure 16:
ducibility, and ease of installation, we implement a simple, heuristic grasping policy, which assumes a parallel gripper performing top-down grasps. Heuristic grasp policies appear throughout robotics research (e.g. in TidyBot [62]). In our case, the heuristic policy voxelizes the point cloud, and chooses areas at the top of the object where points exist, surrounded by free space, in order grasp. Fig. 15 shows the simple grasp policy in action and additional details are presented in Sec. E.3. This policy works well on a wide variety of objects, and we saw comparable performance to the state-of-the-art open-source grasping models we tested [97, 99].
The intuition to identify areas where the gripper fingers can descend unobstructed into two sides of a physical part of the object, which do through a simple voxelization scheme. We take the top 10% of points in object, voxelize fixed resolution of 0.5cm, and choose grasps with free voxels (where fingers can go) on either side of occupied voxels. In practice, this achieved high success rates on a variety of real objects.
The procedure is as follows:
Given a target object point cloud, convert the point cloud into voxels size 0.5 cm.
2. Select top 10% occupied voxels with the highest Z coordinates.
3. Project the selected voxels into 2-D grid.
4. Consider grasps centered around each occupied voxel, and identify three regions: two where the gripper fingers will be and one representing the space between the fingers.
5. Score each grasp based on 1) how occupied the region between the fingers is, and 2) how empty the two surrounding regions are.
6. Perform smoothing on the grasp scores to reject outliers (done by multiplying scores with adjacent scores).
7. Output grasps with final scores above some threshold.
We compared this policy to other methods like ContactGraspnet [97], 6-DoF Graspnet [98, 101], and Graspnet 1-Billion [99]. We saw more intermittent failures due to sensor noise using these pretrained methods, even after adapting the grasp offsets to fit to the Hello Robot Stretch's gripper geometry. In the end, we leave training better grasp policies to future work.
E.4 Heuristic Placement Policy
As with grasping. a number of works on stable placement of objects have been proposed the litera- ture [102, 17]. To provide a reasonable baseline, we implement heuristic placement strategy that assumes that the end-receptacle is at least barely visible when it takes over; projects the segmentation mask onto the point cloud and chooses voxel on the top of the object. Fig. 16 shows an example of the place policy being executed in the real world.
26

---

Specifically, our heuristic policy is implemented as such:
1. Detect the end-receptacle in egocentric RGB observations (using DETIC [27]), project predicted image segment to a 3D point cloud using depth, and transform point cloud to robot base coordinates using camera height and tilt.
2. Estimate placement point: Randomly sample 50 points on the point cloud and choose one that is at the center the biggest (point cloud) slab for placing objects. This is done by scoring each point based on the number of surrounding points in the X/Y plane (Z. is up) within 3 height threshold.
3. Rotate robot for to be facing the placement point, then move robot forward it is more than 38.5 em away (length of retracted arm + approximate length of the Stretch gripper).
4. Re-estimate placement point from this new robot position.
5. Accordingly, set arm's extension and lift values to have the gripper be a few cm above placement position. Then, release the object to land on the receptacle.
E,5
Navigation Planning
Our heuristic baseline extends prior work [2], which was shown to work in wide range of human environments. We tune it for navigating close to other objects and extended it to work in our continuous action space challenging navigation aspects not present in the original paper. The baseline has three components:
Semantic Mapping Module. The semantic map stores relevant objects, explored regions, and obstacles. To construct the map, we predict semantic categories and segmentation masks of objects from first-person observations. We Detie [27] for object detection and instance segmentation and backproject first-person semantic segmentation into point cloud using the perceived depth, bin it into a 3D semantic voxel map, and finally sum over the height to compute a 2D semantic map.
We keep track of objects detected, obstacles, and explored areas in an explicit metric map of the environment from [105]. Concretely, it a binary
M M matrix where M x M is the map size and K is the number of map channels. Each cell of this spatial map corresponds to 25 cm² (5 x 5 cm) in the physical world. Map channels K C' + 4 where C' is the number of semantic object categories, and the remaining 4 channels represent the obstacles, the explored area, and the agent's current and past locations. An entry in the map is one if the cell contains an object of particular semantic category, an obstacle, or is explored, and zero otherwise. The map is initialized with all zeros at the beginning of an episode and the agent starts at the center of the map facing east.
We explore the environment with heuristic frontier- based exploration policy [106]. This heuristic selects as the goal the point closest to the robot in geodesic distance within the boundary between the explored and unexplored region of the map.
Frontier Exploration Policy.
Navigation Planner. Given a long-term goal output by the frontier exploration policy, we use the Fast Marching Method [107] as in [105] to plan a path and the first low-level action along this path deterministically. Although the semantic exploration policy acts at coarse time scale, the planner acts at a fine time scale: every step update the map and replan the path to the long-term goal. The robot attempts to plan to goals if they have been seen; if it cannot get within a certain distance of the goal objects, then it will instead plan to point on the frontier.
Navigating to objects on start_reeeeeee receptaccee..eeee Since small objects (e.g. action action_figure, apple) can be hard to locate from a distance, we leverage the typically larger start_ cecert_reeeetclleee goals for finding objects. We make the following changes the original planning policy [108]:
1. If object and start_ start_receptaall co-oceur in at least one cell of the semantic map, plan to reach the object
2. If the object is not found but start start_receptacle appears in the semantic map after exclud- ing the regions within Im of the agent's 's past locations, plan reach the start_receptaee
27

---

Find Toy Animal
Pick Up Toy Animal
Dining Table
Place On Dining Table
Find Stuffed Animal
Pick Up Stuffed Animal
Find Sofa
Sofa
Figure 17: Real-world examples (also see Fig 2). Our system able to find held-out objects in an unseen environment and navigate to receptacles in order to place them, all with no information about the world at all, other than the relevant classes. However, we see this performance is highly dependent on perception performance for now; many real-world examples also fail due to near-mis cear-miss collisions.
3. Otherwise, plan to reach the closest frontier
In step 2, we exclude the regions that the agent has been close to, to prevent it from re- re-visiting already visited instances of start_ startt_recettacl..
E.6 Navigation Limitations
We implemented a navigation system that was previously used in extensive real-world experiments [2]. but needed to tune it extensively for it to get close enough to objects to grasp and manipulate them. The original version by Gervet et al. [2] was focused on finding very large objects from limited set of only six classes. Ours supports many more, but as result, tuning it to both be able to grasp objects and avoid collisions in all cases is difficult.
This is partly because the planner is a discrete planner based on the Fast Marching Method [107], which cannot take orientation into account and relies on a 5cm discretization of the world. ampling- based motion planners like RRT-CConect [104], or like that used in the Task and Motion Planning literature [100, 8], may offer better solutions. Alternately, we could explore optimization- based planners specifically designed for open- open- vocabulary navigation planning, as has recently been pro- posed [12].
Our navigation policy relies on accurate pose information from Hector SLAM [103], and unfortunately does not handle dynamic obstacles. It also models the robot's location as a cylinder; the Stretch's center rotation slightly offset from the center of this cylinder, which is not currently accounted for. Again, sampling-han planners might be better here.
F Reinforcement Learning Baseline
We train four different RL policies: FindObject, FindReceptacle, GazeAtObject, and PlaceObject.
28

---

F1
Action Space
F.1.1
Navigation Skills
FindObject and FindReceptacle are, collectively, navigation skills. For these two skills, we use the discrete action space, as mentioned in Sec. D.6. In our experiments, we found the discrete action space was better exploration and easier to train.
F11.2 Manipulation Skills
For manipulation skills, using continuous action space to give the skills fine grained control. In the real world, low-level controllers have limits the distance the robot can move in any particular step. Thus, in simulation, we limit our base action space by only allowing forward motions between 10-25 cm, or turning by 5-30 degrees a single step. The head tilt, pan and gripper's ' yaw, roll and pitch can be changed by at most 0.02-0.1 radians a single step. The arm's 's extension and lift can be changed by at most 2- 2-10cm single step. We learn by teleporting the base and arm to the target locations.
F.2 Observation Space
Policies have access to depth from the robot head camera, and semantic segmentation, well as the robot's pose relative to the starting pose (from SLAM in the real world), camera pose, and the robot's joint states, including the gripper. RGB image is available to the agent but not used during training.
F.3
Training Setup
All skills are trained using a slack reward of -0.005 per step, incentivizing completion of task using minimum number of steps. For faster training, learn our policies using images with reduced resolution of 160x122 120 (compared to Stretch's 's original resolution of 640x480).
F331 Navigation Skills
We train FindObject and FindReceptacle policies for the agent to reach a candidate object or a candidate target receptacle respectively. The training procedure is the same for both skills. We pass in the CLIP [14] embedding corresponding with the goal object, as well segmentation masks corresponding with the detected target objects. The agent is spawned arbitrarily, but at least meters from the target, and must move until within 0.1 meters of goal "viewpoint," where the object is visible.
Input observations: Robot head camera depth, ground-truth semantic segmentation for all receptacle categories (receptacle segmentation), robot's pose relative to the starting pose, joint sensor giving states of camera and arm joints. We implement object-level dropout for the semantic segmentation mask, where each object has probability of 0.5 of being left out of the mask. In addition, the input observation space includes the following:
Goal specification: For FindObject, we pass in the CLIP embedding of the target object and the start receptacle category. For FindReceptacle, we pass in the goal receptacle category.
During training, the simulator provides ground truth goal object segmentation; on the real robot, these are predicted by DETIC [27]. For FindObject, we pass in two channels: one showing all instances of candidate objects, one showing all instances of candidate start receptacles. For FindReceptacle, we pass a single channel showing all instances of candidate goal receptacles. We implement similar object-level dropout procedure here we did for the receptacle segmentation.
Goal segmentation images:
The agent is spawned at least 3m away from candidate object receptacle. It starts in "navigation mode," " with the robot's 's head facing forward.
Initial state:
29

---

Actions:
The policy predicts translation and rotation waypoints, as well as a discrete stop action.
Success condition: The agent should call the discrete stop action when it reaches within 0.5m of a goal view point. The agent should be facing the target: the angle between agent's heading direction and the ray from robot center of the closest candidate object should be more than 15 degrees.
Assume at time step t, the geodesic distance to the closest goal is given by d(t), the angle between agent's ' heading direction and the ray from agent to closest goal is given by ((t), and did collide(t) indicates if the action the agent took at time t resulted in a collision at time The training reward is given by:
Reward:
RFindx (f) = a add(( - 1) - d(t)] + B1(d(t) S D Detosel 1) - 0()) + - (did_collide(t))
with a = 3 = 1, = 0.3 and Delose = 3.
F.3.2 GazeAtObject
The GazeAtObject skill starts near the object, and provides some final refinement steps until the agent close enough to call a grasp action, i.e. is in arm's 's length of the object and the object is centered and visible. The agent needs to move closer to the object and then adjust its head tilt until the candidate object is close and centered. makes predictions move and rotate the head, well as to center the object and make sure it's within arm's length, that the discrete grasping policy can execute.
The GazeAtObject skill is supposed to start off from locations and help reach a location within arm's 's length of candidate object. This is trained by first initialising the agents close to candidate start receptacles. The agent is then tasked reach close to the agent and adjust its head tilt such that the candidate object is close and centered in the agent's camera view. We provide details on the training setup.
Input observations: Ground truth semantic segmentation of candidates objects, head depth sensor, joint sensor giving all head and arm joint states, sensor indicating if the agent is holding any object, clip embedding for the target object name.
The robot again starts in "navigation mode," with its arm retracted, with the gripper facing downwards, and with the head/camera facing the base, base at an angle of 5 degrees of the center object and on one of the "viewpoint" locations pre-computed during episode generation. The object will therefore be assumed to be visible.
Initial state:
Actions: This policy predicts base translation and rotation waypoints, camera tilt, well discrete "grasp" action.
Success condition:
The center pixel on the camera should correspond to a valid candidate object and the agent's base should within 0.8m from the object.
Reward: We train the gaze-policy mainly with a dense reward based distance to goal. Specifically, assuming the distance of the end-effector the closest candidate goal at time is d(t) (in metres), the agent receives reward proportional to d(l 1) d()). Further, when the agent reaches with 0.8m, we provide an additional reward for incentivizing the agent to look towards the object.
Let O(t) denote the angle (in radians) between the ray from agent's 's camera to the object and camera's normal. Then the reward is given as:
RGaze (t) = ald(t - d(t) (d(() S ((())(((t))
with a = 2, 3 and = 0.8 in our case.
The agent receives an additional positive reward of 2 once the episode succeeds and receives negative reward of -0.5 for centering its camera towards wrong object.
30

---

F.3.3 PlaceObject
Finally, the robot must move its arm in order to place the object free spot in the world. In this case, it starts at viewpoint near goal_receptacle. It must move up to the object and open its gripper in order to place the object on this surface.
Input observations: Ground truth segmentation of goal receptacles, head depth sensor, joint sensor, sensor indicating if the agent is holding any object, CLIPP [14] embedding for the name of object being held.
Initial configuration: Arm retracted, with gripper down and holding onto an object, head facing the base. The agent is spawned on a viewpoint with its base facing the object with an error of at most 15 degrees.
Actions: Base translation and rotation waypoints, all arm joints (arm extension, lift, gripper yaw, pitch and roll), manipulation mode action that be invoked only once in an episode to turn the agent's head towards the arm and rotate the base left by 90 degrees. The agent is not allowed to move its base while in manipulation mode.
Success condition: The episode succeeds if the agent releases the object and the object stays on the receptacle for 50 timesteps.
The agent receives a positive sparse reward of 5 when it releases the object and the object comes in contact with a target receptacle. Additionaly, we provide a positive reward of 1 for each step the object stays in contact with the target receptacle. receives a negative reward of the agent releases the object but the object does not come in contact with the receptacle.
Reward:
F.4 Concept Fusion
In the main paper, we introduced two key approaches based on end-to-end reinforcement learning and heuristic baseline. Both methods are dependent on the detection results generated by a readily available open vocabulary object detector [27]. Notably, these 2D detection models do not leverage information from prior time steps to inform their detection decisions.
In order to address these limitations. we explored the application of ConceptFusion [15], an open- scene representation technique. ConceptFusion hamnessss foundation models like CLIP [14), DINO [109], and others to construct 3D maps from multiple images. For our experimentation, employed the open-source implementation of ConceptFusion, which utilizes the Segment Anything Model (SAM) [110] for object segmentation in RGB images and CLIP for feature extraction from each segmentation mask. It's important to note that our experiments were conducted a simulated environment, obviating the need for GradSLAM [111], as we had access to ground truth depth maps and pose information to support our map construction efforts.
During our initial experimentation, we observed that ConceptFusion demanded significant com- putational resources and memory, with processing times reaching up to 5 seconds per frame for map construction. Remarkably, it's worth noting that the authors of ConceptFusion have recently published a new paper titled "ConceptGraphss: Open-Vocabulte Vocabulary 3D Scene Graphs for Perception and Planning," [1 12] which addresses some of the computational challenges we encountered. However, we leave the exploration of ConceptGraphs as a potential avenue for future research.
G Additional Analysis
Here, provide some additional analysis of the different skills we trained to complete the Open- Vocabulary Mobile Manipulation task.
31

---

Episode start
Find object
Find receptacle
Place object
Pick box from stand and place it chair.
Pick a toy from table and place on a stool.
Pick a multiport from a stool and place it on table.
We show multiple executions of the Open- Vocabulary Mobile Manipulation task in variety of simulated environments.
Figure 18:
Nav.
Manip.
Perception
FindObj
Gaze
FindRec
Place
Total
Heuristic
Heuristic
Ground Truth
291.8
65.5
8.4
360.5
Heuristic
RL
Ground Truth
293.5
19.4
64.3
84.4
438.7
RL
Heuristic
Ground Truth
295.1
105.0
7.0
401.6
RL
RL
Ground Truth
302.4
25.7
112.8
45.9
455.2
Heuristic
Heuristic
DETIC [27]
335.0
29.5
6.7
361.8
Heuristic
DETIC [27]
330.1
152.0
27.5
68.2
556.5
RL
Heuristic
DETIC [27]
509.5
153.3
7.1
610.4
RI.
DETIC [27]
539.1
101.3
124.4
33.7
634.7
Table 5: The number of steps that the agent takes performing each of the skills for different baselines.
Note that here we only consider the cases where the skill terminates. The last column gives the total number of steps the agent takes on average for executing the four skills.
G.1 Number of steps taken in each stage by different baselines
Table 5 shows the number of steps taken by each skill in our baseline. With DETIC perception, we observed that the RL skills explored less efficiently than our simple heuristic-bas -based planner; this translates to far fewer steps taken successful episodes, although because RI. exploration essentially "gives up" if object isn't nearby, it can take lots of steps many situations. In the real world, saw similar behavior sometimes, the RL policies would not explore enough to be able to find goal at all.
32

---

Nav.
Manip.
Perception
FindObj
Gaze
Pick
FindRec
Place
Place terminates
Heuristic
Heuristic
Ground Truth
100.0
65.1
65.1
62.1
62.1
Hcuristic
RL
Ground Truth
100.0
65.6
64.3
64.3
61.3
52.2
RL
Hcuristic
Ground Truth
100.0
76.3
76.2
66.8
66.8
RL
RL
Ground Truth
100.0
77.0
74.5
74.5
65.1
60.6
Heuristic
Heuristic
DETIC [27]
100.0
34.7
34.7
31.1
31.1
Heuristic
RL
DETIC [27]
100.0
33.9
27.2
27.2
24.4
17.6
RL
Heuristic
DETIC [27]
100.0
32.9
32.7
24.2
24.2
RL
RL
DETIC [27]
100.0
34.7
24.9
24.8
18.1
15.3
Table 6: We report the percentage of times each skill gets invoked for each of the different baselines.
The last column gives the percentage of times the agent finishes executing all skills.
FindObj Success.
PickObj Success.
FindRec Success UC,1
Overall Success
Manip.
Perception
SC,U1
SCCU1
Total
SC,U1
SC,U1
UCCUI
Heuristic
Heuristic
Ground Truth
50.9
53.2
54.1
46.4
47.2
48.5
27.5
30.0
31.5
4.1
5.2
Heuristic
RI.
Ground Truth
54.9
58.2
56.5
48.6
55.1
51.5
39.0
37.3
42.3
14.5
12.0
13.2
RL
Heuristic
Ground Truth
67.1
64.1
65.4
55.0
51.0
54.8
44.8
44,8
43.7
6.2
7.8
7,3
Ground
14.8
Heuristic
Heuristic
DETIC [27]
22.2
21.1
28.7
12.5
10.5
15.2
3.2
3,3
5.3
0.9
0.7
0,4
Houristic
RL
DETIC [27]
22.4
22.2
29.4
11.9
11.8
13.2
5.1
3.5
5.8
0.3
1.4
RL
Heuristic
DETIC [27]
18.7
23.0
21.9
9.9
11.8
11.5
5.3
6.0
0.0
RL
RI.
DETIC (27)
21.5
20.7
21.7
10.9
11.0
10.2
6.9
6.2
6.2
1.0
0.7
Table 7: Performance breakdown by seen and unseen categories, and compared to overall performance.
In our baselines, relied heavily pretrained object detector for generalization, a dramatic difference in performance between seen and unseen objects.
don't see
Next, observe that the Gaze and Place policies, which trained with ground truth perception, take significantly longer terminate with DETIC perception.
Finally, in Table 6, we look at the percentage of times the agent attempts each of the different skills. We find that the RL trained FindObj skill terminates more often than the heuristic FindObj skill and episodes terminate less frequently with DETIC perception when compared GT perception.
G.2 Performance on Seen vs. Unseen Object Categories
Table 7 shows results broken down by seen vs. unseen instances, and seen vs. unseen categories. Specifically we look at these two pools of objects from the validation set:
SC,UI: Seen category, unseen instance. An object of class that appeared in the training data (e.g., ''cup"), but not specific "cup" that appeared in the training data.
UC,UI:
Unseen instance of unseen category; an object of type that did appear in the training data at all.
In general, because we are relying on DETIC and not training our own semantic perception for this baseline, do not see large difference between the two categories of object.
G.2.1 Example DETIC predictions
In Table 5, we observe that the Gaze policy takes significantly longer time to terminate with DETIC [27] perception. The gaze policy (see Fig. 19) tries to center the agent on the object of interest by allowing the agent to move its base and camera tilt. For this, it relies on DETIC's ability detect novel objects. Now, we visualize DETIC segmentations of agent's egocentric observations by placing agent the points where the Gaze skill is expected start: the object's viewpoints. We observe that while DETIC succeeds in few cases, it fails at consistently detecting the objects in the egocentric frame.
33

---

Initially: object not at center of frame Gaze skill centers the agent object
Figure 19: RL. Gaze skill in action: The agent is allowed to move its base and change its camera tilt to get closer to objectand bring objectat the center of its camera frame
Human
Commercially
Manipulation DOF
Approximate Cost
Name
Mobile
Sized
Safe
Available
Boston Dynamics Spot
$200,000
Franka Emika Panda
*
7
$30,000
Locobot
x
x
5
$5,000
Fetch
V
X
7
$100,000
Hello Robot Stretch
$19,000
Stretch with DexWrist
$25,000
Table 8: Notes on platform selection. We chose the Stretch with WexWrist as good compromise between manipulation, navigation, and cost, while being human-safe and approximately human-sized.
H Hardware Setup
Here, will discuss choices related to the real-world hardware setup in extra detail along with information about the tools that we use for the visualization the robot. This appendix contains notes on how to set up the robotics stack in the real world, useful tools that we contribute, and some best practices for development. Setting up mobile robots is hard, and one of the main goals of the HomeRobot project is to make both easy and somewhat affordable for researchers.
H.1 Hardware Choice
We describe some options for commercially available robotics hardware Tab. 8. While the Franka Emika Panda is not mobile robot, include here because it's very commonly used platform in both industrial research labs and at universities, making its price fair comparison point for what is reasonable.
34

---

Observation
Groundtruth
DETIC
Observation
Groundtruth
DETIC
book couch
book bench
credit card toilet
toilet
can opener cabinet
table
dumbbell counter
backpack table chair
table chair
canister table
toilet
couch
Figure 20: Visualization of groundtruth and DETIC [27] segmentation masks for agent's egocentric RGB observations. Note that we use a DETIC vocabulary consisting of the fixed list of receptacle categories and target objectname. We observed that DETIC often fails to accurately detect all the objects present in the given frame.
H.2
Robot Setup
One challenge with low-cost mobile robots how we can run GPU- and compute-intensin models to evaluate modern AI methods on them. The Stretch, like many similar robots, does have onboard GPU, and will always have limited compute than is available similar workstation.
As described in Sec. 4, we address this with a simple network configuration shown in Fig. 21. There are three components:
1. The desktop running code our case, the eval_ episode, py script from HomeRobot which connects to remote mobile manipulator.
2. The dedicated router an off-the-shelf consumer router, such Netgear Nighthawk router. This should ideally be dedicated for your robot and desktop setup ensure good performance.
3. The mobile robot itself: Stretch with Dex Wrist, described above.
After the robot is configured, then you just need to run one script, a ROS launch file, as described in the HomeRobot startup instructions, which can be done over SSH. Then, with properly configured robot and router, you can visualize information on the desktop side, showing the robot's position, map from SLAM, and cameras. On the robot side, the only necessary command is:
roslaunch home_robot hw Seartup_stretchhheeeee
launch
35

---

Desktop
Observatins Boat cuol
All model code
(perception & RL policies)
High-level planning code
Heuristic policies
Visualizations (e.g. RVIZ)
Mobile Manipulator
Router
Runs SLAM for robot pose Open-source low-
Internet connection
Wired desktop
Wireless to robot
level control stack
Figure 21: Network setup diagram for HomeRobot. We can run visualizations on a GPU-enabled workstation while running only the necessary code on robot for low-level control and SLAM.
Checking network performance. We describe the visualization tools available briefly in the next section, but to check that the setup working properly, you can start rviz and wave your hand in front of the robot - you should see minimal latency when waving a hand in front of the camera.
Timing between the robot and the remote workstation. We use ROS [1 113 layer, and to implement low-level control on the robot. This also provides network communication. However, due to potential latency between the robot and the desktop, we also need to make sure that observations are up to date.
communications
We set up the robot block after executing most navigation motions, in order make this process simpler until there an up date image observation from the robot side. This means that timing between the robot and the workstation is extremely important: if we do not have up-to-date timing, we might have SLAM poses and depth measurements that do not match, which will lead to worse performance.
We solved this by having a clock on the robot side publish its time over ROS, and configuring all systems use this ROS master clock instead of system time. This prevents the user from having to worry about Linux time synchronization protocols like NTP when setting up the robot for the first time.
36

---

H.3 Visualizing The Robot
Figure 22: Exploring real-world apartment during testing. The robot uses Detic [27] to perceive the world and update a 2D map (center) which captures where it's seen relevant classes, and which obstacles exist; detections aren't It always reliable, especially given large and changing vocabulary of objects that we care about. the HomeRobot stack, we provide a variety of tools for visualizing and implementing policies, including integration of RVIZ (right).
We use RVIZ, a part of ROS, to visualize results and progress. Fig. 22 shows three different outputs from our system: on the far left, an image from the test environment being processed by Detic; in the center, top-down map generated by the navigation planner described in Sec. E.2; and on the right, an image from RVIZ with the point cloud from the robot's head camera registered against the 2D lidar map created by Hector SLAM.
One advantage of the HomoRobot stack is that it is designed to work with existing debugging tools especially ROS [113]. ROS a widely-used framework for robotics software development that comes with lot of online resources, official support from Hello Robot, and a rich and thriving open-source community with wide industry backing-
H.4 Using The Stretch: Navigation vs. Position Mode
We leave API documentation to the HomeRobot code base, but want to note one other complexity when using the robot. Stretch's 's manipulator arm is pointed to the right of its direction of motion, which means that it cannot both look where it is going and manipulate objects at once. This allows the robot to be lower cost and fit the human profile more information on the robot's design is available in other work [22].
However, it's something important to consider when trying control Stretch to perform various tasks. We use Stretch in one of two modes:
the robot's camera is facing forward; we use reactive low-level control for navigation; the robot can rotate in place, roll backward, and will reactively track goals sent from the desktop.
Navigation mode:
Manipulation mode: the robot's camera is facing towards its arm; we do not use reactive low-level control for navigation and do not rotate the base. Instead, we treat the robot's base as an extra, lateral degree of freedom for manipulation.
This is especially relevant when grasping or placing; it means that, for our heuristic policies, the robot transitions into manipulation mode after moving close enough the goal, and may track slightly to the left the right, in order act if it had a full 6dof manipulator.
All in all, these changes make the low-cost robot more capable and easier to use for a variety of tasks [12, 24, 25].
37


Reviews:
[{"review_id": "YjsY1YR-O_E", "reviewer": "Reviewer_Jmt1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents the Open-Vocabulary Mobile Manipulation (OVMM) benchmark. The authors define OVMM as the task of picking an arbitrary object in an environment and placing it at an arbitrary goal location, where the start and goal locations can be spread out over large multi-room areas and the objects do not belong to a predefined set. The purpose of OVMM is to measure progress in the field toward general purpose home robot assistants. The paper also presents the HomeRobot software stack for conducting OVMM experiments in simulation and real world environments. Reinforcement learning and heuristic approaches for OVMM are implemented in HomeRobot and compared in experiments.", "summary_of_recommendation": "**Post-Rebuttal Update**: I am upgrading to Weak Accept due to the authors' satisfactory rebuttal.\n\nI recommend a Weak Reject due to the list of weaknesses identified above. While the OVMM problem setting is interesting, the paper's contributions over prior work are not clear to me. I may reconsider if the authors are able to address my concerns in the revision period, or the other reviewers have very different assessments of the work.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please address the concerns raised in the weaknesses section above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose and study an important problem setting. I agree that OVMM is a useful measure of how the field has progressed toward its longstanding goal of general-purpose home robot assistants. Running the benchmark yearly to assess progress as the authors suggest can be very valuable to the community and of high impact, as it can highlight which components of OVMM require the most attention. The paper has a thorough appendix and useful videos for intuition. Table I is a helpful comparison to prior benchmarks and OVMM has an impressive amount of object instances compared to prior work.\n\nHowever, the paper has several weaknesses listed below. \n- The paper claims that OVMM is the “first reproducible mobile-manipulation benchmark for the real world,” and that existing efforts for reproducibility in robotics fall short due to lack of standardized hardware across labs. However, it is not clear how the proposed benchmark constitutes reproducible robotics. Existing initiatives include low-cost open-source hardware, large offline datasets, and shared remote hardware testbeds such as the Real Robot Challenge (https://arxiv.org/abs/2109.10957). The authors suggest the reproducibility is a result of the open-source software, low-cost robot ($25,000), and shared API for simulation and real world experiments, but (1) prior works have these components and (2) this does not solve the reproducibility issue in robotics: even if everyone adopted this hardware, it would not be possible to standardize the environment (lighting, friction, etc.). Furthermore, the fact that OVMM supports arbitrary objects *reduces* standardization rather than the suggested opposite. I would suggest either reducing the emphasis on reproducibility or providing further justification why OVMM is fundamentally different from previous attempts in this regard.\n- The demonstration video showcases capabilities that do not appear to be state-of-the-art. The system is extremely slow even with the 4x speedup applied to the video, and the task is a simple pick-and-place action coupled with navigation. While HomeRobot may have a continuous action space it does not appear to be continuous control, as the robot strings together a sequence of short movements rather than a continuous one. Demos from projects such as SayCan (https://say-can.github.io/) seem to be far more proficient at the same task, with much more flexible natural language queries. Pick-and-place is also a significant limitation that should be added to the limitations section; most home tasks cannot be accomplished with a single pick-and-place (e.g., cooking, cleaning, laundry).\n- HomeRobot code is not included in the supplement and is said to be released at the time of final submission, despite being a major claimed contribution of the paper. The existence of the robotics stack is a major differentiator in Table I, but it is not described with much detail in the main text.\n- In general the use of the terms HomeRobot and OVMM seem to change throughout the text and overlap at times. For example, HomeRobot is defined in sentence 1 of the abstract as an affordable compliant robot for the home, and then later defined differently in the introduction as a software stack. The benchmark is sometimes called OVMM, HomeRobot OVMM, HomeRobot+OVMM. The contributions are also not the most clear. It would be ideal to clarify terminology throughout.\n\nAlso, there is existing work in scene graphs for multi-room environments that may be useful to include in related work, for example Hierarchical Mechanical Search (https://arxiv.org/abs/2012.04060) and SayPlan (https://arxiv.org/abs/2307.06135; looks like this was released after the CoRL submission deadline).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "XWV8mSlG5TR", "reviewer": "Reviewer_CNKn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, a benchmark for the Open-Vocabulary Mobile Manipulation (OVMM) task is established. The benchmark consists of both a simulation component and a real-world component to implement the task of picking an object to a target location. This work takes a step forward to the standardization of some particular robotic tasks.", "summary_of_recommendation": "I like the idea to have a large scale benchmark to standardize current robotic tasks both in the simulation and real-world environment. However, the task setup in the proposed benchmark is a little simple and does not well align with real-world environment. \n\nThanks the authors for the rebuttal. The rebuttal explains some of my concerns, and now I am inclined to accept this paper.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "As stated in the Weaknesses above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- I agree that it is the time that to have a large scale benchmark to standardize current robotic tasks both in the simulation and real-world environment. And the motivation of this work is quite clear and solid. \n- The paper is organized and the definition of the OVMM task is clearly illustrated. The accompanied video and appendix provides some more implementation details about this work. \n\nWeaknesses:\n- The OVMM task is defined as \"picking any object in any unseen environment, and placing it in a commanded location\". Although the definition of the task is clear, the setup of this task demonstrated in the paper is actually a little simple. Only the seen and unseen objects are considered. However, in real-word environment, it is likely that there might be more complicated circumstances such as obstacles, impossible to be directly picked up, etc. I have some concerns about the adaptation ability of the proposed benchmark to more complex tasks. Also the benchmark does not support multiple agents. \n- An heuristic and RL approaches are provided as baselines. And the ground truth and DETIC segmentation results are also used respectively. I wonder if there are some other stronger RL methods or 3D semantic information could also be included in the benchmark. With only two baselines on a single task, this benchmark seems a little weak and not promising enough.  \n- For the Sim2Real process, is there any sim2real training mechanism or directly transfer the trained model in the simulation to the real-world environment.  \n- Some minor problems:\n  - In Fig.1, it is hard for me to distinguish the real-world from the simulation environment as indicated in the Line 57\n  - Table 1 only lists some comparison aspects among different works, while no summarization or conclusion is given. It will be useful to have an analysis on these aspects.  Especially, I don't quite get the idea of what the robotics stack mean in the table.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "TmkmvtKJoj", "reviewer": "Reviewer_uxxU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the problem of Open-Vocabulary Mobile Manipulation OVMM). It proposes a definition for the key-task, provides an extensive benchmark for both simulation and robot hardware, and additionally includes both learned and heuristic baselines for evaluating how challenging the task is. Overall, the definition of the open-vocabulary mobile manipulation task does not feel particularly substantial since visual navigation and object retrieval has been well-studied within the robotics field recently with the advent of foundation models. The benchmark and simulation environment are welcomed, but the limiting assumptions prevent this benchmark from being as general as it first reads in the abstract. I weakly recommend this paper be accepted.", "summary_of_recommendation": "Overall, I recommend this paper be weakly accepted. The simulator and benchmark are the most substantial contribution, but without access to the robot hardware component (which many labs may not have), the value of HomeRobot simulator over ManiSkill-2 for example feels more minor. The paper is well written and clear, although the paper writing presents a grand vision for OVMM that the limiting assumptions prevents this paper from fully realizing. Since the main contribution would be useful for the community, I recommend this paper be accepted.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- What does “partial availability” mean for the manipulation column?\n- What do the numbers in table 3 represent, task success rate (label that in caption if so). Do success rate numbers of a task like Pick depend on how often previous actions (FindObj) were successful? \n- How is OMVMMAgent defined?\n- In limitations, why are full natural language queries out-of-scope? What is meant by that? \n- There are very little details on the Heuristic approach in section 4, what heuristics are used?\n- In table 4, it seems the results for FindObj are significantly better than the simulation results, why is that the case, it seems very unintuitive.\n\n[1] Chen, Boyuan, et al. \"Open-vocabulary queryable scene representations for real world planning.\" 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- Provides a benchmark and tools for evaluating open-vocabulary mobile manipulation in both simulation and on robot hardware (Hello Robot Stretch), which is timely and valuable to the community. The open source nature of HomeRobot library is also appreciated.\n- The comparisons of the new benchmark against existing one (Table 1) is a useful review and provides context for what gaps are existing in robot simulators. \n- The careful experimental design to test generalization across seen/unseen categories/instances is appreciated and a nice addition.\n\nWeaknesses:\n- Having the start receptacle always be available “to help agents know where to look for the object” seems like a pretty strong assumption, and seems counter to the OVMM vision since the goal is to handle unseen environments.\n- Having all possible receptacle categories be seen during training seems like a strong assumption and seems like it may contribute to making this significantly simpler for navigation.\n- Table 2 caption text is too close to the actual body text which makes it hard to read\n- While the reviewer recognizes resolving all modeling simplifications of a simulator is not reasonable, the last of physical simulation of grasping makes training for the full OVMM in simulation hard, especially when cluttered scenes heavily impact affordances. \n- The lack of motion planners is an important and standard component of a robotics stack that this is missing.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "MS0wLsMBHB", "reviewer": "Reviewer_kzRx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work contributes a benchmark and corresponding open-source software stack for the problem of open-vocabulary mobile manipulation using a relatively low-cost platform (Hello Robot Stretch) in simulation and on hardware. For benchmarking, the authors provide 60 hand-curated home scenes with  a large and diverse number of annotated target objects for manipulation. Furthermore, the paper provides two baseline solutions for the problem of target object transport (i.e., find object - pick object - locate receptacle - place object) based on planning and reinforcement learning respectively. These baselines achieve around 20% of overall success rate both in simulation as well as on hardware. ", "summary_of_recommendation": "This paper shows no real theoretical contribution, but is valuable for practitioners. As the main topic is the provided software stack and benchmarking framework, this works' value will be determined by the provided software quality and uptake of the community which is hard to judge upfront. I do think that the presented work is technically sound and that a significant effort was put into it. As the topic of mobile manipulation for household robotics is of high practical interest, I do see real potential and would recommend the work for publication subject to the clarifications mentioned above. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Clarify contributions w.r.t. sim-to-real: without physics engine, I don't see how interaction policies could have a chance to transfer to hardware\n- Maybe the title should reflect better the actual focus of the paper (benchmarking, software stack)\n- Typo in line 210: should read \"off-the-shelf policy\"\n- Tables 3 & 4 use different units (Table 3 uses percent, whereas Table 4 uses ratios) - that should be unified.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I like this paper. It's well written and related work is put in adequate context. Fair and comparable benchmarking is an important topic in robotics in general, and I applaude the authors' efforts to make a contribution in this direction. Large multi-modal models opened exciting possibilities for interfacing with robotic manipulation systems, so the presented line of work is of high practical relevance.  \n\nI do think that the authors ought to sharpen their contributions somewhat. The claimed sim-to-real transfer I find dubious, as their current simulation environment is basically limited to perception. However, in manipulation the most difficult (and critical for realism) simulation aspect is interaction/contact. Currently, the authors use a simple action (I assume rigid attachment to the end-effector) in place of physical interaction simulation. Therefore, I don't see how the proposed approach would stand up to existing solutions (e.g., the cited ManiSkill) which do provide a physics engine and thus ought to have a much better chance of sim-to-real transfer of interaction policies.  ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ### Summary, Strengths and Weaknesses
This paper introduces the OVMM benchmark, which tackles the task of selecting any object from an unfamiliar setting and accurately placing it at a designated location. The benchmark encompasses both simulation and real-world elements, facilitating the assessment of diverse strategies. Reviewers agree that the mobile manipulation benchmark focusing on large-scale open-vocabulary conditions is important. The paper's structure is mostly clear and comprehensible. 

The reviewers initially raised a few concerns on the definition of OVMM and limited physical interaction simulation, which is crucial in robotics. It is also regrettable that the authors failed to acknowledge prior contributions like RoboCup@Home, which has been organized to benchmark language-driven mobile manipulation using open and standard platforms (simulation, Toyota HSR, Pepper) since 2007.

The reviewers agree in their recommendation to accept the paper because most of the concerns have been addressed. I agree with their consensus.
Decision: Accept (Poster)