OK-Robot:
What Really Matters in Integrating Open-Knowledge
Models for Robotics
Peiqi Liu*1

Yaswanth Orru*1

Nur Muhammad Mahi Shafiullah†1

Chris Paxton2

Lerrel Pinto†1

New York University1 , AI at Meta2
https://ok-robot.github.io“OK Robot, move the Takis on the desk to the nightstand”

1

1.3.
VoxelMap
Drop

NavigationPlan
Primitive(“Takison the desk”)
Drop(<RGBD>, “nightstand”)

VoxelMap +2. 1.
Lang-SAM
NavigationPlan
AnyGrasp(“Takis on the desk”)

2

3

Grasp(<RGBD>, “Takis”)

VoxelMap
1.1.VoxelMap

NavigationPlan
NavigationPlan(“Takis(“Takis onon thethe desk”)
desk”)

1

2

1

3

2
2

3

“Move the soda can tothe box”

13“Move the purple
shampoo to the red bag”

“Move the white meds
box to the trash bin”

Fig. 1: OK-Robot is an Open Knowledge robotic system, which integrates a variety of learned models trained on publicly available data, topick and drop objects in real-world environments. Using Open Knowledge models such as CLIP, Lang-SAM, AnyGrasp, and OWL-ViT,
OK-Robot achieves a 58.5% success rate across 10 unseen, cluttered home environments, and 82.4% on cleaner, decluttered environments.

VLMs and robotics primitives, while being flexible enough toincorporate newer models as they are developed by the VLM
and robotics community.
We present OK-Robot, an Open Knowledge Robot that
integrates state-of-the-art VLMs with powerful robotics primitives for navigation and grasping to enable pick-and-drop.
Here, Open Knowledge refers to learned models trained onlarge, publicly available datasets. When placed in a new homeenvironment, OK-Robot is seeded with a scan taken from an
iPhone. Given this scan, dense vision-language representations
are computed using LangSam [24] and CLIP [9] and stored
in a semantic memory. Then, given a language-query for an
object that has to be picked, language representations of thequery is matched with semantic memory. After this, navigationand picking primitives are applied sequentially to move to thedesired object and pick it up. A similar process can be carriedout for dropping the object.
To study OK-Robot, we tested it in 10 real world homeenvironments. Through our experiments, we found that on anever seen, natural home environment, a zero-shot deploymentof our system achieves 58.5% success on average. However,I. I NTRODUCTION
this success rate is largely dependant on the “naturalness” ofCreating a general-purpose robot has been a longstanding the environment, as we show that with improving the queries,
dream of the robotics community. With the increase in data- decluttering the space, and excluding objects that are clearly
driven approaches and large robot models, impressive progress adversarial (too large, too translucent, too slippery), this successis being made [1–4]. However, current systems are brittle, rate reaches about 82.4%. Overall, through our experiments,
closed, and fail when encountering unseen scenarios. Even we make the following observations:the largest robotics models can often only be deployed in • Pre-trained VLMs are highly effective for openpreviously seen environments [5, 6]. The brittleness of these
vocabulary navigation: Current open-vocabulary visionsystems is further exacerbated in settings where little robotic
language models such as CLIP [9] or OWL-ViT [8] offer
data is available, such as in unstructured home environments.
strong performance in identifing arbitrary objects in the real
The poor generalization of robotic systems lies in stark
world, and enable navigating to them in a zero-shot manner
contrast to large vision models [7–10], which show capabilities(see Section II-A.)of semantic understanding [11–13], detection [7, 8], and • Pre-trained grasping models can be directly applied toconnecting visual representations to language [9, 10, 14]mobile manipulation: Similar to VLMs, special purpose
At the same time, base robotic skills for navigation [15],robot models pre-trained on large amounts of data can be
grasping [16–19], and rearrangement [20, 21] are fairly mature.
applied out of the box to approach open-vocabulary grasping
Hence, it is perplexing that robotic systems that combine
in homes. These robot models do not require any additionalmodern vision models with robot-specific primitives perform
training or fine-tuning (see Section II-B.)
so poorly. To highlight the difficulty of this problem, the • How components are combined is crucial: Given the
recent NeurIPS 2023 challenge for open-vocabulary mobile
pretrained models, we find that they can be combined withmanipulation (OVMM) [22] registered a success rate of 33%
no training using a simple state-machine model. We also
for the winning solution [23].
find that using heuristics to counteract the robot’s physical
So what makes open-vocabulary robotics so hard? Unfortulimitations can lead to a better success rate in the real world
nately, there isn’t a single challenge that makes this problem(see Section II-D.)
hard. Instead, inaccuracies in different components compound • Several challenges still remain: While, given the imand together results in overall drop. For example, the quality
mense challenge of operating zero-shot in arbitrary homes,of open-vocabulary retrievals of objects in homes is dependent
OK-Robot improves upon prior work, by analyzing theon the quality of query strings, navigation targets determinedfailure modes we find that there are significant improvementsfrom VLMs may not be reachable to the robot, and the choicethat can be made on the VLMs, robot models, and robotof different grasping models may lead to large differences inmorphology, that will directly increase performance of opengrasping performance. Hence, making progress on this problem
knowledge manipulation agents (see Section III-D).
requires a careful and nuanced framework that both integrates
To encourage and support future work in open-knowledge
robotics,
we will share the code and modules for OK-Robot,* Denotes equal contribution and † denotes equal advising.
Correspondence to: mahi@cs.nyu.edu
and are committed to supporting reproduction of our results.
Abstract— Remarkable progress has been made in recent years
in the fields of vision, language, and robotics. We now havevision models capable of recognizing objects based on language
queries, navigation systems that can effectively control mobile
systems, and grasping models that can handle a wide range ofobjects. Despite these advancements, general-purpose applicationsof robotics still lag behind, even though they rely on thesefundamental capabilities of recognition, navigation, and grasping.
In this paper, we adopt a systems-first approach to develop a newOpen Knowledge-based robotics framework called OK-Robot. By
combining Vision-Language Models (VLMs) for object detection,navigation primitives for movement, and grasping primitives
for object manipulation, OK-Robot offers a integrated solution
for pick-and-drop operations without requiring any training. To
evaluate its performance, we run OK-Robot in 10 real-worldhome environments. The results demonstrate that OK-Robotachieves a 58.5% success rate in open-ended pick-and-drop tasks,
representing a new state-of-the-art in Open Vocabulary Mobile
Manipulation (OVMM) with nearly 1.8× the performance ofprior work. On cleaner, uncluttered environments, OK-Robot’s
performance increases to 82%. However, the most important
insight gained from OK-Robot is the critical role of nuanced
details when combining Open Knowledge systems like VLMs withrobotic modules.

More information along with robot videos are available on our
project website: https://ok-robot.github.io.
II. T ECHNICAL C OMPONENTS AND M ETHOD
Our method, on a high level, solves the problem describedby the query: “Pick up A (from B) and drop it on/in C”,
where A is an object and B and C are places in a real-world
environment such as homes. The system we introduce is acombination of three primary subsystems combined on a Hello
Robot: Stretch. Namely, these are the open-vocabulary object
navigation module, the open-vocabulary RGB-D grasping
module, and the dropping heuristic. In this section, we describe
each of these components in more details.

collected by the camera, giving us a point cloud where eachpoint has an associated semantic vector coming from CLIP.
Then, we voxelize the point cloud to a 5 cm resolution and foreach voxel, calculate the detector-confidence weighted averagefor the CLIP embeddings that belong to that voxel. This voxel
map builds the base of our object memory module. Note thatthe representation created this way remains static after the
first scan, and cannot be adapted during the robot’s operation.
This inability to dynamically create a map is discussed in our
limitations section (Section V).

Querying the memory module: Our semantic object memory
gives us a static world representation represented as possibly
non-empty voxels in the world, and a semantic CLIP vector
associated with each voxel. Given a language query, we convert
A. Open-home, open-vocabulary object navigation
it to a semantic vector using the CLIP language encoder. Then,
The first component of our method is an open-home, open- we find the top voxel where the dot product between the
vocabulary object navigation model that we use to map a home encoded vector and the voxel’s semantic representation isand subsequently navigate to any object of interest designated maximized. Since each voxel is associated with a real location
by a natural language query.
in the home, this lets us find the location where a queried
Scanning the home: For open vocabulary object navigation, object is most likely to be found, similar to Figure 2(a).
When necessary, we implement “A on B” as “A near B”. We
we follow the approach from CLIP-Fields [27] and assume
a pre-mapping phase where the home is “scanned” manually do so by selecting top-10 points for query A and top-50 points
using an iPhone. This manual scan simply consists of taking for query B. Then, we calculate the 10 × 50 pairwise euclidean
a video of the home using the Record3D app on the iPhone, distances and pick the A-point associated with the shortest (A,B) distance. Note that, during the object navigation phase, we
which results in a sequence of posed RGB-D images.
Alternatively, this could be done automatically using frontier- use this query only to navigate to the object approximately,
based exploration [15, 25, 26], but for speed and simplicity we and not for manipulation.
This approach gives us two advantages: our map can be as
prefer the manual approach [26, 27]. We take this approachsince the frontier-based approaches tend to be slow and lower resolution than those in prior work [26, 27, 30], andcumbering, especially for a novel space, while our “scan” take we can deal with small movements in object’s location after
less than one minute for each room. Once collected, the RGB-D building the map.
images, along with the camera pose and positions, are exported Navigating to objects in the real world: Once our navigation
to our library for map-building.
model gives us a 3D location coordinate in the real world,
To ensure our semantic memory contains both the objects of we use that as a navigation target for our robot to initializeinterest as well as the navigable surface and any obstacles, the our manipulation phase. In previous works [15, 27, 31], therecording must capture the floor surface alongside the objects navigation objective was to go and look at an object, which
and receptacles in the environment.
can be done while staying at a safe distance from the object
Detecting objects: On each frame of the scan, we run an open- itself. In contrast, our navigation module must place the robot
vocabulary object detector. Unlike previous works which used at an arms length so that the robot can manipulate the target
Detic [7], we chose OWL-ViT [8] as the object detector since object afterwards. Thus, our navigation method has to balance
we found it to perform better in preliminary queries. We apply the following objectives:the detector on every frame, and extract each of the object
1) the robot needs to be close enough to the object tobounding box, CLIP-embedding, detector confidence, and passmanipulate it,them onto the object memory module of our navigation module.
2) the robot needs some space to move its gripper, so there
Building on top of previous work [27], we further refine
needs to be a small but non-negligible space between thethe bounding boxes into object masks with Segment Anything
robot and the objectm and(SAM) [28]. Note that, in many cases, open-vocabulary object
3) the robot needs to avoid collision during manipulation,
detectors still require a set of natural language object queries
and thus needs to keep its distance from all obstacles.
that they try to detect. We supply a large set of such object We use three different navigation score functions, each associqueries, derived from the original Scannet200 labels [29], such ated with one of our previous concerns, and evaluate them onthat the detector captures most common objects in the scene. each point of the space to find the best position to place the
Object-centric semantic memory: We use an object-centric robot.
memory similar to Clip-Fields [27] and OVMM [25] that we
Let a random point be ⃗x, the closest obstacle point as ⃗xobs ,call the VoxelMap. The object masks are back-projected in and the target object as ⃗xo . Then, we can define the following
real-world coordinates using the depth image and the pose three functions s1 , s2 , s3 to capture our three criterion. Then,

PlantBrown bag

Red body
spray
Pink powder
bottleUmbrella
Brown hat

Orangelaundry bag

Brown hat
Start point

Bed(a) Open-vocabulary object localization
using VoxelMap(b) Open-vocabulary navigation planning
using VoxelMap and heuristics weighted A*

Fig. 2: Open-vocabulary, open knowledge object localization and navigation in the real-world. We use the VoxelMap [25] for localizing
objects with natural language queries, and use an A* algorithm similar to USANet [26] for path planning.

their weighted sum s to find the ideal navigation point x⃗∗ inour space that minimizes s(⃗x), and the direction is towards
the vector from x⃗∗ to x⃗o .
s1 (⃗x) = ||⃗x − ⃗xo ||s2 (⃗x) = 40 − min(||⃗x − ⃗xo ||, 40)(1/||⃗x − ⃗xobs ||, if ||⃗x − ⃗xobs ||0 ≤ 30
s3 (⃗x) =0,
otherwises(⃗x) = s1 (⃗x) + 8s2 (⃗x) + 8s3 (⃗x)
To navigate to this target point safely from any other point inspace, we follow a similar approach to [26, 32] by building
an obstacle map from our previously captured posed RGB-D
images. We build a 2D, 10cm×10cm grid of obstacles overwhich we navigate using the A* algorithm. To convert ourvoxel map to an obstacle map, we first set a floor and ceilingheight. Presence of occupied voxels in between them implies
the grid cell is occupied, while presence of neither ceiling nor
floor voxels mean that the grid cell is unexplored. We markboth occupied or unexplored cells as not navigable. Aroundeach occupied point, we mark any point within a 20 cm radius
as also non-navigable to account for the robot’s radius and aturn radius. In our A* algorithm, we use the s3 function as aheuristic on the node costs to navigate further away from any
obstacles, which makes our generated paths similar to ideal
Voronoi paths [33] in our experiments.
B. Open-vocabulary grasping in the real world
Unlike open-vocabulary navigation, for grasping, our methodneeds to physically interact with arbitrary objects in the realworld, which makes this part significantly more difficult. As
a result, we opt for using a pre-trained grasping model togenerate grasp poses in the real world, and filter that withlanguage-conditioning using a modern VLM.

Grasp perception: Once the robot reaches the object using thenavigation method outlined in Section II-A, we use a pre-trained
grasping model or heuristic to generate a grasp for the robot.
We point the robot’s RGB-D head camera towards the object’s
location in space, as given to us by our semantic memory
module, and capture an RGB-D image from it (Figure 3, column
1). We backproject and convert the depth image to a pointcloudas necessary. Then, we pass this information to our grasp
generation module. The grasp generation module that we use
in our work is AnyGrasp [19], which generates collision freegrasps with a parallel jaw gripper in a scene given a single
RGB image and a pointcloud.
AnyGrasp provides us with possible grasps in the scene(Figure 3 column 2) with grasp point, width, height, depth,
and a “graspness score”, which indicates uncalibrated model
confidence in each grasp. However, such modules generally
generate all possible grasps in a scene, which we need to filter
using the language query.
Filtering grasps using language queries: Once we get all
proposed grasps from AnyGrasp, we filter the grasps using
LangSam [24]. We use LangSam [24] to segment the captured
image and get the desired object’s mask with the language
query (Figure 3 column 3). Then, we project all the proposed
grasp points onto the image and find the grasps that fall into
the object mask (Figure 3 column 4).
We pick the best grasp using a heuristic, where if the grasp
score is S and the angle between the grasp normal and floor
4
normal is θ, then the new heuristic score is S − (θ /10). This
heuristic prioritizes grasps with the highest graspness score but
also a horizontally flat proposed grasp. We prefer horizontal
grasps because they are robust to small calibration errors on
the robot, while vertical grasps need to be quite point-accurate
to be successful. Being robust to hand-eye calibration errors is
a desired property as we transport the robot to different homes

Robot view

AnyGrasp proposals

LangSam mask

Grasp filtering

Final grasp

Fig. 3: Open-vocabulary grasping in the real world. From left to right, we show the (a) robot POV image, (b) all suggested grasps from
AnyGrasp [19], (c) object mask given label from LangSam [24], (d) grasp points filtered by the mask, and (e) grasp chosen for execution.

over the course of our experiments.

LangSam [24] similar to Section II-B using the drop language
Grasp execution: Once we identify the best grasp (Figure 3 query. Then, we align that segmented point cloud such thatcolumn 5), we use a simple pre-grasp approach [34] to grasp X-axis is aligned with the way the robot is facing, Y-axis−
p is the grasp point is to its left and right, and the Z-axis of the point cloud is
our intended object. Let’s assume that →→−and a is the approach vector given by our grasping model. aligned with the floor normal. We call this aligned pointcloudPa . Finally, we normalize the point cloud so that the robot’s XThen, our robot gripper follows the following trajectory:
and Y- coordinate is (0, 0), and the floor plane is at z = 0. On
−
−
−
−
−
−
−
⟨→
p − 0.2→a, →p − 0.08→a, →p − 0.04→a, →p⟩
the aligned, segmented point cloud, we consider the X- andY-coordinates
for each point, and find the respective medians
Put simply, our method approaches the object from a pre-grasp
on
each
axis
that
we call xm and ym . Finally, we find a drop
position in a line with progressively smaller motions. Moving
heightusingz= 0.2 + max{z | (x, y, z) ∈ Pa ; 0 ≤ x ≤max
slower as we approach the object is important since the robot
x
;
|y
−
y
|
<
0.1}
on the segmented, aligned pointcloud.
mm
can knock over light objects otherwise. Once we reach the
We
add
a
small
bufferof 0.2 to the height to avoid collisions
predicted grasp point, we close the gripper in a close loop
between
the
robot
and
the
drop location. Finally, we move thefashion to make sure we can get a solid grip on the object
robot
gripper
above
the
drop
point, and open the gripper towithout crushing it. Finally, after grasping the object, we liftdroptheobject.
While
thisheuristic
sometimes fails to placeup the robot arm, retract it fully, and rotate the wrist to haveanobjectonaclutteredsurface,inourexperiments it performs
the object tucked over the body. This behavior maintains thewellonaverage.
robot footprint while ensuring the object is held securely by
the robot and doesn’t fall while navigating to the drop location.
D. Deployment in homes
C. Dropping heuristic
Once we have our navigation, pick, and drop primitive in
After picking up an object, we find and navigatte to the place, we combine them directly to create our robot method
location to drop it using the same methods as described in that can be applied in any novel home directly. For a new
Section II-A. Unlike in HomeRobot’s baseline implementa- home environment, we can “scan” the room in under a minute.
tion [25], which assumes that the drop-off location is a flat Then, it takes less than five minutes to process that into oursurface, we extend our heuristic to also cover concave objects VoxelMap. For our ablations, it takes about 50 minutes to train
such as sink, bins, boxes, and bags. First, we segment the the necessary implicit semantic fields/SDF models such as
point cloud P captured by the robot’s head camera using CLIP-Fields or USA-Net if we are using them. Once that isdone, the robot can be immediately placed at the base and start objects sourced from each home. As a result, each of the success
operating. From arriving into a completely novel environment and the failure of the robot tells us something interesting about
to start operating autonomously in it, our system takes under applying open-knowledge models in robotics, which is what
10 minutes on average to complete the first pick-and-drop task. we analyze over the next sections.
In Appendix C, we provide the details of all our home
State machine model: The transition between different modexperiments
and results from the same, and in Appendix B weules happens automatically, in a predefined fashion, once ashowasubsetof the objects OK-Robot operated on. Snippets
user specifies the object to pick and where to drop it. Sinceofourexperiments
are in Figure 1, and full videos can be seen
we do not implement error detection or correction, our stateonourproject
website.
machine model is a simple linear chain of steps leading fromnavigating to object, to grasping, to navigating to goal, and to
dropping the object at the goal to finish the task.

B. Ablations over system components

Apart from the navigation and manipulation strategies that
Protocol for home experiments: To run our experiment in a we used in the home experiments, we also evaluated a number
novel home, we first move the robot to a previously unobserved of alternative semantic memory module and open vocabularyroom. There, we record the scene and create our VoxelMap. navigation modules. We compared them by evaluating them in
Concurrently, we arbitrarily pick between 10-20 objects in three different environment setups in our lab.
each scene that can fit in the robot gripper. These are objects
Alternate semantic navigation strategies: We evaluate the“found” in the scene, and are not ones selected beforehand.
following semantic memory modules:
We come up with a language query for each chosen object
using GPT-4V [35] to keep the queries consistent and free of • VoxelMap [25]: VoxelMap converts every detected object
to a semantic vector and stores such info into an associatedexperimenter bias. The effect of different queries for the samevoxel. Occupied voxels serve as an obstacle map.
object on OK-Robot is discussed in Section III-D. Then, we
query our navigation module to filter out all the navigation • CLIP-Fields [27]: CLIP-Fields converts a sequence ofposed RGB-D images to a semantic vector field by using
failures; i.e. objects whose location could not be found by our
open-label object detectors and semantic language embedsemantic memory module. Then, we execute pick-and-drop on
ding models. The result associates each point in the spaceremaining objects sequentially, without resets between trials.
with two semantic vectors, one generated via a VLM [9],and another generated via a language model [36], which isIII. E XPERIMENTS
then embedded into a neural field [37].
We evaluate our method in two set of experiments. On• USA-Net [26]: USA-Net generates multi-scale CLIP feathe first set of experiments, we evaluate between multiple
tures and embeds them in a neural field that also doubles
alternatives for each of our navigation and manipulation
as a signed distance field. As a result, a single model canmodules. These experiments give us insights about which
support both object retrieval and navigation.
modules to use and evaluate in a home environment as apart of our method. On the next set of experiments, we took We compare them in the same three environments with a fixed
our robots to 10 homes and ran 171 pick-and-drop experiments set of queries, the results of which are shown in Figure 5.
to empirically evaluate how our method performs in completely Alternate grasping strategies: Similarly, we compare multiple
novel homes, and to understand the failure modes of our system. grasping strategies to find out the best grasping strategy for
Through these experiments, we look to answer a series our method.
of questions regarding the capabilities and limits of current • AnyGrasp [19]: AnyGrasp is a single view RGB-D based
Open Knowledge robotic systems, as embodied by OK-Robot.
grasping model. It is trained on the GraspNet dataset which
Namely, we ask the following:
contains 1B grasp labels.
1) How well can such a system tackle the challenge of pick • Open Graspness [19]: Since the AnyGrasp model is free
but not open source, we use an open licensed baselineand drop in arbitrary homes?
trained on the same dataset.
2) How well do alternate primitives for navigation andgrasping compare to the recipe presented here for building • Contact-GraspNet [16]: We use Contact-GraspNet as aprior work baseline, which is trained on the Acronym [38]an Open Knowledge robotic system?
dataset. One limitation of Contact-GraspNet is that it was3) How well can our current systems handle unique chaltrained on a fixed camera view for a tabletop setting. As alenges that make homes particularly difficult, such as
result, in our application with a moving camera and arbitrary
clutter, ambiguity, and affordance challenges?
locations, it failed to give us meaningful grasps.
4) What are the failure modes of such a system and its• Top-down grasp [25]: As a heuristic based baseline, we
individual components in real home environments?
compare with the top-down heuristic grasp provided in the
A. List of home experiments
HomeRobot project.
Over the 10 home environment, OK-Robot achieved a In Figure 5, we see their comparative performance in three
58.5% success rates in completing full pick-and-drops. Notably, lab environments. For semantic memory modules, we see that
this is a zero-shot algorithm, and the success rate is over novel VoxelMap, used in OK-Robot and described in Sec. II-A,

Fig. 4: All the success and failure cases in our home experiments, aggregated over all three cleaning phases, and broken down by mode of
failure. From left to right, we show the application of the three components of OK-Robot, and show a breakdown of the long-tail failuremodes of each of the components.

Success
Navigation failure

Semantic memory module
Cleanup level

VoxelMap
Clip fields
USA NetGrasping module
AnyGrasp
AnyGrasp
Open Source
Top down

58

none

20

40

60

80

100

15
71

low

25
12

82

high
0

0

Manipulation failure
Placing failure

20

40

16
4

60

Percentage of trials

13

80

100

Fig. 6: Failure modes of our method in novel homes, broken downby the failures of the three modules and the cleanup levels.

Fig. 5: Ablation experiment using different semantic memory andgrasping modules, with the bars showing average performance andthe error bars showing standard deviation over the environments.

C. Impact of clutter, object ambiguity, and affordance

outperforms other semantic memory modules by a small margin.
It also has much lower variance compared to the alternatives,
meaning it is more reliable. As for grasping modules, AnyGrasp
clearly outperforms other grasping methods, performing almost
50% better in a relative scale over the next best candidate, topdown grasp. However, the fact that a heuristic-based algorithm,
top-down grasp from HomeRobot [25] beats the open-source
AnyGrasp baseline and Contact-GraspNet shows that building
a truly general-purpose grasping model remains difficult.

What makes home environments especially difficult compared to lab experiments is the presence of physical clutter,language-to-object mapping ambiguity, and hard-to-reach positions. To gain a clear understanding of how such factors playinto our experiments, we go through two “clean-up” processes
in each environment. During the clean-up, we pick a subset ofobjects that are free from ambiguity from the previous rounds,
clean the clutter around objects, and generally relocated themin an accessible locations. We go through two of such clean-up
rounds at each environment, which gives us insights about
the performance gap caused by the natural difficulties of ahome-like environment.

Fig. 7: Samples of failed or ambiguous language queries into oursemantic memory module. Since the memory module depends on
pretrained large vision language model, its performance shows
susceptibility to particular “incantations” similar to current LLMs.

We show a complete analysis of the tasks listed section III-A
which failed in various stages in Figure 6. As we can see fromthis breakdown, as we clean up the environment and remove
the ambiguous objects, the navigation accuracy goes up, andthe total error rate goes down from 15% to 12% and finally
all the way down to 4%. Similarly, as we clean up cluttersfrom the environment, we find that the manipulation accuracy
also improves and the error rates decrease from 25% to 16%
and finally 13%. Finally, since the drop-module is agnosticof the label ambiguity or manipulation difficulty arising fromclutter, the failure rate of the dropping primitive stays roughly
constant through the three phases of cleanup.

Object is transparent, so
pointcloud is imperfect,
so grasp is imperfect

Top-down grasp on tallcounter is unreachable

Diagonal grasp on acylindrical object isunstableFine grasps on small
objects are vulnerable tocalibration errors

Grasps on round objects
are unstable when not
perfectly diametrical

Grasps on flat objects
collide with env if not
perfectly executed

Fig. 8: Samples of failures of our manipulation module. Most failures
stem from using only a single RGB-D view to generate the grasp andthe limiting form-factor of a large two-fingered parallel jaw gripper.

reasons our OK-Robot can fail is when a natural language
query given by the user doesn’t retrieve the intended object
from the semantic memory. In Figure 7 we show how somequeries may fail while semantically very similar but slightly
modified wording of the same query might succeed.
Generally, this has been the case for scenes where there
are multiple visually or semantically similar objects, as shownin the figure. There are other cases where some queries may
pass while other very similar queries may fail. An interactive
system that gets confirmation from the user as it retrieves anobject from memory would avoid such issues.

While our method can show zero-shot generalization incompletely new environments, we probe OK-Robot to better
understand its failure modes. Primarily, we elaborate on how
our model performed in novel homes, what were the biggest
challenges, and discuss potential solutions to them.
We first show a coarse-level breakdown of the failures, only
considering the three high level modules of our method in
Figure 6. We see that generally, the leading cause of failure is
our manipulation failure, which intuitively is the most difficult
as well. However, at a closer look, we notice a long tail of
failure causes, which is presented in figure 4.
We see that the leading three cause of failures are failingto retrieve the right object to navigate to from the semantic
memory (9.3%), getting a difficult pose from the manipulation
module (8.0%), and hardware difficulties (7.5%). In this section,
we go over the analysis of the failure modes presented in
Figure 4 and discuss the most frequent cases.

Grasping module limitations: One potential failure mode ofour system is that our manipulation is performed by executing
the outputs of a pre-trained model-generated grasps that are
predicted based on a single RGB-D image, with a model thatwasn’t designed for the Hello Robot: Stretch gripper.
As a result, sometimes such grasps are unreliable or
unrealistic, as shown in Figure 8. There are cases where theproposed grasp is infeasible given the robot joint limits, oris simply too far from the robot body. Development of better
heuristics will let us sample better grasps for a given object.
In some other cases, the model generates a good grasp pose,but as the robot is executing the grasping primitive, it collides
with some minor environment obstacle. Since we do not plan
the grasp trajectory, and instead try to apply the same grasp
trajectory in every case, some such failures are inevitable.
Better grasping models that generates a grasp trajectory as
well as a pose may solve such issues. Finally, our grasping
module struggles with flat objects categorically, like chocolatebars and books, since it’s difficult to grasp them off a surface
with a two-fingered gripper.

Natural language queries for objects: One of the primary

Robot hardware limitations: While our robot of choice, aD. Understanding the performance of OK-RobotHello Robot: Stretch, is able to pick-and-drop a number ofobjects, there are certain hardware limitations that determines
what the robot can and cannot manipulate. For example, therobot has a 1 kg (2 lbs) payload limit when the arm is fully
extended, and as such our method is unable to move objects
like a full dish soap container. Similarly, objects that are far
from navigable floor space, such as in the middle of a bed, oron high places, is difficult for the robot to reach because of thereach limits of the arm. Finally, in some situations, the robot
hardware or the RealSense camera can become miscalibrated
over time, especially during continuous testing in homes. This
miscalibration can lead to error since the manipulation module
requires hand-eye coordination in the robot.
IV. R ELATED W ORKS
A. Vision-Language models for robotic navigation
Early applications of pre-trained open-knowledge models
has been in open-vocabulary navigation. Navigating to various
objects is an important task which has been looked at in a wide
range of previous works [25, 31, 39], as well as in the context of
longer pick-and-place tasks [40, 41]. However, these methods
have generally been applied to relatively small numbers ofobjects [42]. Recently, Objaverse [43] has shown navigation
to thousands of object types, for example, but much of thiswork has been restricted to simulated or highly controlledenvironments.
The early work addressing this problem builds upon representations derived from pre-trained vision language models, suchas SemAbs [44], CLIP-Fields [27], VLMaps [45], NLMapSayCan [46], and later, ConceptFusion [47] and LERF [30].
Most of these models show object localization in pre-mapped
scenes, while CLIP-Fields, VLMaps, and NLMap-SayCanshow integration with real robots for indoor navigation tasks.
USA-Nets [26] extends this task to include an affordancemodel, navigating with open-vocabulary queries while doing
object avoidance. ViNT [48] proposes a foundation model for
robotic navigation which can be applied to vision-language
navigation problems. More recently, GOAT [31] was proposed
as a modular system for “going to anything” and navigating to
any object in any environment. ConceptGraphs [49] proposed
an open scene representation capable of handling complex
queries using LLMs and creating a scene graph.
B. Pretrained robot manipulation models
While humans can frequently look at objects and immediately
know how to grasp it, such grasping knowledge is not easily
accessible to robots. Over the years, there has been many
works that has focused on creating such a general robot grasp
generation model [1, 50–55] for arbitrary objects and potentiallycluttered scenes via learning methods. Our work focuses onmore recent iterations of such methods [16, 19] that are trainedon large grasping datasets [18, 38]. While these models only
perform one task, namely grasping, they predict grasps acrossa large object surface and thus enable downstream complex,
long-horizon manipulation tasks [20, 21, 56].

More recently, there is a set of general-purpose manipulation
models moving beyond just grasping [57–61]. Some of theseworks perform general language-conditioned manipulation
tasks, but are largely limited to a small set of scenes andobjects. HACMan [62] demonstrates a larger range of object
manipulation capabilities, focused on pushing and prodding. In
the future, such models could expand the reach of our system.
C. Open vocabulary robot systems
Many recent works have worked on language-enabledtasks for complex robot systems. Some examples include
language conditioned policy learning [57, 63–65], learning goalconditioned value functions [3, 66], and using large language
models to generate code [67–69]. However, a fundamental
difference remains between systems which aim to operate
on arbitrary objects in an open-vocab manner, and systems
where one can specify one among a limited number of goals
or options using language. Consequently, Open-Vocabulary
Mobile Manipulation has been proposed as a key challenge forrobotic manipulation [25]. There has previously been efforts tobuild such a system [70, 71]. However, unlike such previous
work, we try to build everything on an open platform and ensure
our method can work without having to re-train anything for anovel home. Recently, UniTeam [23] won the 2023 HomeRobot
OVMM Challenge [22] with a modular system doing pickand-place to arbitrary objects, with a zero-shot generalization
requirement similar to ours.
In parallel, recently, there have been a number of papers
doing open-vocabulary manipulation using GPT or especially
GPT4 [35]. GPT4V can be included in robot task planning
frameworks and used to execute long-horizon robot tasks,
including ones from human demonstrations [72]. ConceptGraphs [49] is a good recent example, showing complex
object search, planning, and pick-and-place capabilities to openvocabulary objects. SayPlan [73] also shows how these can use
used together with a scene graph to handle very large, complex
environments, and multi-step tasks; this work is complementary
to ours, as it doesn’t handle how to implement pick and place.
V. L IMITATIONS , O PEN P ROBLEMS AND
R EQUEST FOR R ESEARCH
While our method shows significant success in completely
novel home environments, it also shows many places wheresuch methods can improve. In this section, we discuss a few
of such potential improvement in the future.
A. Live semantic memory and obstacle maps
All the current semantic memory modules and obstacle mapbuilders build a static representation of the world, without
a good way of keeping it up-to-date as the world changes.
However, homes are dynamic environments, with many smallchanges over the day every day. Future research that can build
a dynamic semantic memory and obstacle map would unlock
potential for continuous application of such pick-and-drop
methods in a novel home out of the box.

B. Grasp plans instead of proposals
Currently, the grasping module proposes generic grasps
without taking the robot’s body and dynamics into account.
Similarly, given a grasp pose, often the open loop grasping
trajectory collides with environmental obstacles, which can beeasily improved by using a module to generate grasp plans
rather than grasp poses only.
C. Improving interactivity between robot and user
One of the major causes of failure in our method is innavigation: where the semantic query is ambiguous and theintended object is not retrieved from the semantic memory. Insuch ambiguous cases, interaction with the user would go along way to disambiguate the query and help the robot succeed
more often.
D. Detecting and recovering from failure
Currently, we observe a multiplicative error accumulation
between our modules: if any of our independent components
fail, the entire process fails. As a result, even if our modules
each perform independently at or above 80% success rate,
our final success rate can still be below 60%. However, with
better error detection and retrying algorithms, we can recoverfrom much more single-stage errors, and similarly improve ouroverall success rate [23].
E. Robustifying robot hardware
While Hello Robot - Stretch [74] is an affordable andportable platform on which we can implement such an openhome system for arbitrary homes, we also acknowledge thatwith robust hardware such methods may have vastly enhanced
capacity. Such robust hardware may enable us to reach highand low places, and pick up heavier objects. Finally, improved
robot odometry will enable us to execute much more finer
grasps than is possible today.
ACKNOWLEDGMENTS
NYU authors are supported by grants from Amazon, Honda,
and ONR award numbers N00014-21-1-2404 and N00014-211-2758. NMS is supported by the Apple Scholar in AI/ML
Fellowship. LP is supported by the Packard Fellowship. Our
utmost gratitude goes to our friends and colleagues who helped
us by hosting our experiments in their homes. Finally, we thankJay Vakil, Siddhant Haldar, Paula Pascual and Ulyana Piterbarg
for valuable feedback and conversations.
R EFERENCES[1] Lerrel Pinto and Abhinav Gupta. Supersizing selfsupervision: Learning to grasp from 50k tries and 700
robot hours. ICRA, 2016.
[2] Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz,
and Deirdre Quillen. Learning hand-eye coordination for
robotic grasping with deep learning and large-scale datacollection. The International journal of robotics research,
37(4-5):421–436, 2018.
[3] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen
Chebotar, Omar Cortes, Byron David, Chelsea Finn,

Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog,
et al. Do as I can, not as I say: Grounding language
in robotic affordances. Conference on Robot Learning(CoRL), 2022.
[4] Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja
Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, andLerrel Pinto. On bringing robots home, 2023.
[5] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana
Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine
Hsu, et al. Rt-1: Robotics transformer for real-worldcontrol at scale. arXiv preprint arXiv:2212.06817, 2022.
[6] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen
Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding,
Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2:
Vision-language-action models transfer web knowledge torobotic control. arXiv preprint arXiv:2307.15818, 2023.
[7] Xingyi Zhou, Rohit Girdhar, Armand Joulin, PhilippKrähenbühl, and Ishan Misra. Detecting twenty-thousandclasses using image-level supervision. In European
Conference on Computer Vision, pages 350–368. Springer,
2022.
[8] Matthias Minderer, Alexey Gritsenko, Austin Stone,
Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy,
Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani,
Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf,
and Neil Houlsby. Simple open-vocabulary object detection with vision transformers. In European Conference
on Computer Vision, pages 728–755. Springer, 2022.
[9] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen
Krueger, and Ilya Sutskever. Learning transferable
visual models from natural language supervision. In
International Conference on Machine Learning (ICML),
volume 139, pages 8748–8763, 2021.
[10] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, andRoozbeh Mottaghi. Ok-vqa: A visual question answering
benchmark requiring external knowledge. In Proceedingsof the IEEE/cvf conference on computer vision and patternrecognition, pages 3195–3204, 2019.
[11] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur
Mensch, Katie Millican, Malcolm Reynolds, Roman Ring,
Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong,
Sina Samangooei, Marianne Monteiro, Jacob Menick,Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh,
Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira,
Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.
Flamingo: a visual language model for few-shot learning,2022.
[12] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao
Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su,
Jun Zhu, et al. Grounding DINO: Marrying DINO withgrounded pre-training for open-set object detection. arXiv
preprint arXiv:2303.05499, 2023.

[13] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae
Lee. Visual instruction tuning, 2023.
[14] Alec Radford, Jeffrey Wu, Rewon Child, David Luan,
Dario Amodei, and Ilya Sutskever. Language models areunsupervised multitask learners. OpenAI Blog, 2019.
[15] Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot. Navigating to objects in the real world. Science Robotics, 8(79):eadf6991,
2023.
[16] Martin Sundermeyer, Arsalan Mousavian, Rudolph
Triebel, and Dieter Fox. Contact-graspnet: Efficient 6dof grasp generation in cluttered scenes. In 2021 IEEE
International Conference on Robotics and Automation(ICRA), pages 13438–13444. IEEE, 2021.
[17] Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael
Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea,
and Ken Goldberg. Dex-net 2.0: Deep learning to planrobust grasps with synthetic point clouds and analytic
grasp metrics. arXiv preprint arXiv:1703.09312, 2017.
[18] Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu
Lu. Graspnet-1billion: a large-scale benchmark forgeneral object grasping. In Proceedings of the IEEE/CVF
conference on computer vision and pattern recognition,
pages 11444–11453, 2020.
[19] Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao
Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen
Xie, and Cewu Lu. Anygrasp: Robust and efficient
grasp perception in spatial and temporal domains. IEEE
Transactions on Robotics, 2023.
[20] Ankit Goyal, Arsalan Mousavian, Chris Paxton, Yu-Wei
Chao, Brian Okorn, Jia Deng, and Dieter Fox. Ifor:
Iterative flow minimization for robotic object rearrangement. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pages 14787–
14797, 2022.
[21] Weiyu Liu, Tucker Hermans, Sonia Chernova, and Chris
Paxton. Structdiffusion: Object-centric diffusion forsemantic rearrangement of novel objects. arXiv preprint
arXiv:2211.04604, 2022.
[22] Sriram Yenamandra, Arun Ramachandran, Mukul Khanna,
Karmesh Yadav, Devendra Singh Chaplot, Gunjan Chhablani, Alexander Clegg, Theophile Gervet, Vidhi Jain,
Ruslan Partsey, Ram Ramrakhya, Andrew Szot, TsungYen Yang, Aaron Edsinger, Charlie Kemp, Binit Shah,
Zsolt Kira, Dhruv Batra, Roozbeh Mottaghi, Yonatan Bisk,
and Chris Paxton. The homerobot open vocab mobile
manipulation challenge. In Thirty-seventh Conference
on Neural Information Processing Systems: Competition
Track, 2023.
[23] Andrew Melnik, Michael Büttner, Leon Harz, Lyon
Brown, Gora Chand Nandi, Arjun PS, Gaurav Kumar
Yadav, Rahul Kala, and Robert Haschke. Uniteam: Open
vocabulary mobile manipulation challenge. arXiv preprint
arXiv:2312.08611, 2023.
[24] Luca Medeiros. Lang segment anything. https://github.
com/luca-medeiros/lang-segment-anything, 2023.

[25] Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet,
Tsung-Yen Yang, Vidhi Jain, Alexander William Clegg,
John Turner, et al. Homerobot: Open-vocabulary mobile
manipulation. arXiv preprint arXiv:2306.11565, 2023.
[26] Benjamin Bolte, Austin Wang, Jimmy Yang, Mustafa
Mukadam, Mrinal Kalakrishnan, and Chris Paxton. Usanet: Unified semantic and affordance representations for
robot memory. arXiv preprint arXiv:2304.12164, 2023.
[27] Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel
Pinto, Soumith Chintala, and Arthur Szlam. Clip-fields:
Weakly supervised semantic fields for robotic memory.
arXiv preprint arXiv:2210.05663, 2022.
[28] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi
Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer
Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar,
and Ross Girshick. Segment anything. In ICCV, pages
4015–4026, October 2023.
[29] David Rozenberszki, Or Litany, and Angela Dai.
Language-grounded indoor 3d semantic segmentation in
the wild, 2022.
[30] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo
Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages
19729–19739, 2023.
[31] Matthew Chang, Theophile Gervet, Mukul Khanna, Sriram Yenamandra, Dhruv Shah, So Yeon Min, Kavit Shah,
Chris Paxton, Saurabh Gupta, Dhruv Batra, et al. Goat:
Go to any thing. arXiv preprint arXiv:2311.06430, 2023.
[32] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram
Burgard. Audio visual language maps for robot navigation.
arXiv preprint arXiv:2303.07522, 2023.
[33] Santiago Garrido, Luis Moreno, Mohamed Abderrahim,
and Fernando Martin. Path planning for mobile robot
navigation using voronoi diagram and fast marching. In
2006 IEEE/RSJ International Conference on Intelligent
Robots and Systems, pages 2376–2381. IEEE, 2006.
[34] Sudeep Dasari, Abhinav Gupta, and Vikash Kumar.
Learning dexterous manipulation from exemplar object
trajectories and pre-grasps, 2023.
[35] OpenAI. GPT-4 technical report, 2023.
[36] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks, 2019.
[37] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik,
Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf:
Representing scenes as neural radiance fields for view
synthesis. European Conference on Computer Vision(ECCV), 65(1):99–106, 2020.
[38] Clemens Eppner, Arsalan Mousavian, and Dieter Fox.
Acronym: A large-scale grasp dataset based on simulation.
In 2021 IEEE International Conference on Robotics andAutomation (ICRA), pages 6222–6227. IEEE, 2021.
[39] Arsalan Mousavian, Alexander Toshev, Marek Fišer, Jana
Košecká, Ayzaan Wahid, and James Davidson. Visual
representations for semantic target driven navigation. In2019 International Conference on Robotics and Automation (ICRA), pages 8846–8852. IEEE, 5 2019.
[40] Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, andYoav Artzi. A persistent spatial semantic representation
for high-level natural language instruction execution. In
Conference on Robot Learning, pages 706–717. PMLR,
2022.
[41] So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. Film:
Following instructions in language with modular methods.
arXiv preprint arXiv:2110.07342, 2021.
[42] Matt Deitke, Dhruv Batra, Yonatan Bisk, Tommaso
Campari, Angel X Chang, Devendra Singh Chaplot,
Changan Chen, Claudia Pérez D’Arpino, Kiana Ehsani,
Ali Farhadi, et al. Retrospectives on the embodied ai
workshop. arXiv preprint arXiv:2210.06849, 2022.
[43] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs,
Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana
Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse:
A universe of annotated 3d objects. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 13142–13153, 2023.
[44] Huy Ha and Shuran Song. Semantic abstraction: Openworld 3d scene understanding from 2d vision-language
models, 2022.
[45] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram
Burgard. Visual language maps for robot navigation. In
2023 IEEE International Conference on Robotics andAutomation (ICRA), pages 10608–10615. IEEE, 2023.
[46] Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao,
Keerthana Gopalakrishnan, Michael S. Ryoo, Austin
Stone, and Daniel Kappler. Open-vocabulary queryable
scene representations for real world planning. In arXiv
preprint arXiv:2209.09874, 2022.
[47] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala,
Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh
Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari,
et al. Conceptfusion: Open-set multimodal 3d mapping.
arXiv preprint arXiv:2302.07241, 2023.
[48] Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, and Sergey Levine.
ViNT: A Foundation Model for Visual Navigation. In 7th
Annual Conference on Robot Learning (CoRL), 2023.
[49] Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama
Chellappa, et al. Conceptgraphs: Open-vocabulary 3dscene graphs for perception and planning. arXiv preprint
arXiv:2309.16650, 2023.
[50] Abhinav Gupta, Adithyavairavan Murali, Dhiraj Prakashchand Gandhi, and Lerrel Pinto. Robot learning inhomes: Improving generalization and reducing dataset
bias. Advances in Neural Information Processing Systems,
31:9094–9104, 2018.
[51] Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael
Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea,

and Ken Goldberg. Dex-Net 2.0: Deep learning to planrobust grasps with synthetic point clouds and analytic
grasp metrics. In Robotics: Science and Systems (RSS),2017.
[52] Jeffrey Mahler, Matthew Matl, Xinyu Liu, Albert Li,
David Gealy, and Ken Goldberg. Dex-net 3.0: Computing
robust robot vacuum suction grasp targets in point clouds
using a new analytic model and deep learning, 2018.
[53] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian
Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen,
Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke,
et al. QT-Opt: Scalable deep reinforcement learningfor vision-based robotic manipulation. arXiv preprint
arXiv:1806.10293, 2018.
[54] Yuzhe Qin, Rui Chen, Hao Zhu, Meng Song, Jing Xu,
and Hao Su. S4g: Amodal single-view single-shot se(3)grasp detection in cluttered scenes, 2019.
[55] Arsalan Mousavian, Clemens Eppner, and Dieter Fox.
6-dof graspnet: Variational grasp generation for object
manipulation, 2019.
[56] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu,
J. Tremblay, and D. Fox. Progprompt: Generating
situated robot task plans using large language models.
In 2023 IEEE International Conference on Robotics andAutomation (ICRA), page 11523, 2023.
[57] Mohit Shridhar, Lucas Manuelli, and Dieter Fox.
Perceiver-Actor: A multi-task transformer for robotic
manipulation. In CoRL, pages 785–799. PMLR, 2023.
[58] Priyam Parashar, Jay Vakil, Sam Powers, and Chris
Paxton. Spatial-language attention policies for efficientrobot learning. arXiv preprint arXiv:2304.11235, 2023.
[59] Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty
Altanzaya, and Lerrel Pinto. Behavior transformers:
Cloning k modes with one stone. Advances in neural
information processing systems, 35:22955–22968, 2022.
[60] Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi
Shafiullah, and Lerrel Pinto. From play to policy:
Conditional behavior generation from uncurated robot
data, 2022.
[61] Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, andKaterina Fragkiadaki. Act3d: 3d feature field transformers
for multi-task robotic manipulation. In Conference on
Robot Learning, pages 3949–3965. PMLR, 2023.
[62] Wenxuan Zhou, Bowen Jiang, Fan Yang, Chris Paxton,
and David Held. Learning hybrid actor-critic maps
for 6d non-prehensile manipulation. arXiv preprint
arXiv:2305.03942, 2023.
[63] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLIPort:
What and where pathways for robotic manipulation. In
CoRL, pages 894–906. PMLR, 2022.
[64] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar,
Jonathan Tompson, Sergey Levine, and Pierre Sermanet.
Learning latent plans from play. In CoRL, pages 1113–
1132. PMLR, 2020.
[65] Corey Lynch and Pierre Sermanet. Language conditioned
imitation learning over unstructured data. Robotics:

Science and Systems, 2021.
[66] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu
Li, Jiajun Wu, and Li Fei-Fei. VoxPoser: Composable
3D value maps for robotic manipulation with language
models. In CoRL, 2023.
[67] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol
Hausman, Brian Ichter, Pete Florence, and Andy Zeng.
Code as Policies: Language model programs for embodied
control. In icra, pages 9493–9500. IEEE, 2023.
[68] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar,
Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large
language models. arXiv preprint arXiv: Arxiv-2305.16291,
2023.
[69] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit
Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse
Thomason, and Animesh Garg. ProgPrompt: Generating
situated robot task plans using large language models. In
ICRA, pages 11523–11530. IEEE, 2023.
[70] Naoki Yokoyama, Alex Clegg, Joanne Truong, Eric
Undersander, Tsung-Yen Yang, Sergio Arnaud, Sehoon
Ha, Dhruv Batra, and Akshara Rai. ASC: Adaptive
skill coordination for robotic mobile manipulation. arXiv
preprint arXiv:2304.00410, 2023.
[71] Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean
Kirmani, Brianna Zitkovich, Fei Xia, Chelsea Finn, andKarol Hausman. Open-world object manipulation using
pre-trained vision-language model. In arXiv preprint,
2023.
[72] Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi,
Jun Takamatsu, and Katsushi Ikeuchi. Gpt-4v(ision)
for robotics: Multimodal task planning from humandemonstration. arXiv preprint arXiv:2311.12015, 2023.
[73] Krishan Rana, Jesse Haviland, Sourav Garg, Jad AbouChakra, Ian Reid, and Niko Suenderhauf. Sayplan:
Grounding large language models using 3d scenegraphs for scalable task planning. arXiv preprint
arXiv:2307.06135, 2023.
[74] Charles C Kemp, Aaron Edsinger, Henry M Clever, andBlaine Matulevich. The design of stretch: A compact,
lightweight mobile manipulator for indoor human environments. In 2022 International Conference on Robotics
and Automation (ICRA), pages 3150–3157. IEEE, 2022.

A PPENDIX AS CANNET 200 TEXT QUERIES’poster’, ’bed’, ’cart’, ’light switch’,’backpack’, ’power strip’, ’baseball’,’mustard’, ’bathroom vanity’, ’water
To detect objects in a given home environment pitcher’, ’closet’, ’couch’, ’beverage’,using OWL-ViT, we use the Scannet200 labels. The ’toy’, ’salt’, ’plant’, ’pillow’, ’broom’,
full label set is here: [’shower head’, ’spray’, ’pepper’, ’muffins’, ’multivitamin’,’inhaler’, ’guitar case’, ’plunger’,’towel’, ’storage bin’, ’nightstand’,’range hood’, ’toilet paper dispenser’,’radiator’, ’telephone’, ’pillar’, ’tissue’adapter’, ’soy sauce’, ’pipe’, ’bottle’,
box’, ’vent’, ’hair dryer’, ’ledge’,’door’, ’scale’, ’paper towel’, ’paper’mirror’, ’sign’, ’plate’, ’tripod’,
towel roll’, ’stove’, ’mailbox’,’chair’, ’kitchen cabinet’, ’column’,’scissors’, ’tape’, ’bathroom stall’,’water cooler’, ’plastic bag’, ’umbrella’,’chopsticks’, ’case of water bottles’,’doorframe’, ’paper’, ’laundry hamper’,’hand sanitizer’, ’laptop’, ’alcohol’food’, ’jacket’, ’closet door’, ’computer
disinfection’, ’keyboard’, ’coffee
tower’, ’stairs’, ’keyboard piano’,maker’, ’light’, ’toaster’, ’stuffed’person’, ’table’, ’machine’, ’projector
animal’, ’divider’, ’clothes dryer’,screen’, ’shoe’].
’toilet seat cover dispenser’, ’file
A PPENDIX B
cabinet’, ’curtain’, ’ironing board’,
S AMPLE OBJECTS FROM OUR TRIALS’fire extinguisher’, ’fruit’, ’object’,During our experiments, we tried to sample objects that’blinds’, ’container’, ’bag’, ’oven’,canplausibly be manipulated by the Hello Robot: Stretch’body wash’, ’bucket’, ’cd case’, ’tv’,
gripperfrom the home environments. As a result, OK-Robot’tray’, ’bowl’, ’cabinet’, ’speaker’,encountereda large variety of objects with different shapes’crate’, ’projector’, ’book’, ’school
andvisualfeatures.
A subsample of such objects are presented
bag’, ’laundry detergent’, ’mattress’,inthe
Figures
9,10.
’bathtub’, ’clothes’, ’candle’, ’basket’,’glass’, ’face wash’, ’notebook’, ’purse’,
A PPENDIX C’shower’, ’power outlet’, ’trash bin’,
L IST OF HOME EXPERIMENTS’paper bag’, ’water dispenser’, ’package’,A full list of experiments in homes can be found in Table I.
’bulletin board’, ’printer’, ’windowsill’,’disinfecting wipes’, ’bookshelf’,’recycling bin’, ’headphones’, ’dresser’,’mouse’, ’shower gel’, ’dustpan’, ’cup’,’storage organizer’, ’vacuum cleaner’,’fireplace’, ’dish rack’, ’coffee kettle’,’fire alarm’, ’plants’, ’rag’, ’can’,’piano’, ’bathroom cabinet’, ’shelf’,’cushion’, ’monitor’, ’fan’, ’tube’,’box’, ’blackboard’, ’ball’, ’bicycle’,’guitar’, ’trash can’, ’hand sanitizers’,’paper towel dispenser’, ’whiteboard’,’bin’, ’potted plant’, ’tennis’,’soap dish’, ’structure’, ’calendar’,’dumbbell’, ’fish oil’, ’paper cutter’,’ottoman’, ’stool’, ’hand wash’, ’lamp’,’toaster oven’, ’music stand’, ’waterbottle’, ’clock’, ’charger’, ’picture’,’bascketball’, ’sink’, ’microwave’,’screwdriver’, ’kitchen counter’, ’rack’,’apple’, ’washing machine’, ’suitcase’,’ladder’, ’ping pong ball’, ’window’,’dishwasher’, ’storage container’,’toilet paper holder’, ’coat rack’, ’soap
dispenser’, ’refrigerator’, ’banana’,’counter’, ’toilet paper’, ’mug’, ’markerpen’, ’hat’, ’aerosol’, ’luggage’,Arm smartphone holder

Gray toy dragon

Toy plant

White shirt

Tangled earphones

Playing cards

Blue gloves

Toy cactus

Toy grapes

Medicine bottles

Grey ragBlue hair oil bottle

Blue pretzel pack

Toothpaste

White pretzel

Blue body wash

Fig. 9: Sample objects on our home experiments, sampled from each home environment, which OK-Robot was able to pick and drop
successfully.

Purple strap

Yellow ginger paste packet

Blue bag

Steel wool

Translucent grey cup

Black face wash

Gold wrapped chocolate

Black head bandBlue eyeglass case

Flu!y headbands

Yogurt drinks

Lotion pumpBlue hair gel tube

Brown trail mix bag

White Apple bag

Small hand sanitizer

Fig. 10: Sample objects on our home experiments, sampled from each home environment, which OK-Robot failed to pick up successfully.

TABLE I: A list of all tasks in the home enviroments, along with their categories and success rates out of 10 trials.

Pick object

Place location

ResultHome 1silver cup
blue eye glass caseprinted paper cup, coffee cup [white table]
small red and white medication
power adapter
wrapped paper
blue body wash
blue air spray
black face wash
yellow face wash
body spraysmall hand sanitizer
blue inhaler device(window)
inhaler box(window)
multivitamin container
red towel
white shirt

silver cup
blue eye glass caseprinted paper cup, coffee cup [white table]
small red and white medication
power adapter
blue body wash
blue air sprayyellow face wash
small hand sanitizer
blue inhaler device(window)
inhaler box(window)
red towel
white shirt

silver cup
printed paper cup, coffee cup [white table]
blue body wash
blue air sprayyellow face wash
small hand sanitizer
inhaler box(window)
white shirt

fanta cantennis ball
black head band [bed]purple shampoo bottle
toothpaste

Cleanup level: none
white table
chair
Chair
Grey Bedstudy table
white table
chair

white table
dust binwhite cloth bin (air conditioner)
white cloth bin (air conditioner)
Cleanup level: low
white table
dust binChairwhite table
white table
white table
study table
dust binwhite cloth bin(air conditioner)
white cloth bin(air conditioner)
Cleanup level: highwhite table
dust binwhite table
white table

dust binwhite cloth bin(air conditioner)
Home 2
Cleanup level: Nonedust binsmall red shopping bag
white racksmall red shopping bag

Success
Success
Manipulation failure
Success
Success
Navigation failure
Success
Success
Manipulation failure
Success
Navigation failure
Manipulation failure
Success
Success
Navigation failure
Success
Success

Success
Navigation failure
Success
Success
Navigation failure
Success
Success
Success
Success
Manipulation failure
Success
Success
Success

Success
Success
Success
Success
Manipulation failure
Manipulation failure
Success
Success

Success
Success
Manipulation failure
Success
Success
Continued on the next page

Pick object
orange packaging
green hair cream jar [white rack]green detergent pack [white rack]
blue moisturizer [white rack]green plastic coverstorage container
blue hair oil bottle
blue pretzels packblue hair gel tube
red bottle [white rack]
blue bottle [air conditioner]
walletfanta cantennis ball
black head band [bed]purple shampoo bottle
toothpasteorange packaging
green detergent pack [white rack]
blue moisturizer [white rack]
blue hair oil bottle
blue pretzels packwalletfanta canpurple shampoo bottle
orange packaging
blue moisturizer [white rack]
blue hair oil bottle
blue hair gel tube
red bottle [white rack]
blue bottle [air conditioner]apple
ice cream
green lime juice bottle
yellow packet
red packet
orange cancooking oil bottle
pasta sauce
orange box [stove]green bowl
washing gloves
small oregano bottle
yellow noodles packet [stove]blue dish wash bottle
scrubberdressing salad bottle

Place location
dust binwhite table

white rack
white rack
brown desk
white cloth bin(air conditioner)

Cleanup level: low
black trash can
red target bag
red target bag
red target bag
red target bag
black trash canwhite rack
white rack

Cleanup level: highblack trash cansmall red shopping bag
black trash canwhite rack
dust bintarget bag
white cloth bin(air conditioner)
Home 3
Cleanup level: none
white plate
white and green bag
red basket

card board box

sinkgreen bag [card board box]
red basket
red basket
card board box

ResultSuccess
Navigation failure
Success
Navigation failure
Navigation failure
Manipulation failure
Success
Success
Manipulation failure
Success
Success
Manipulation failure

Success
Success
Success
Success
Success
Success
Manipulation failure
Navigation failure
Success
Success
Manipulation failure

Success
Success
Success
Success
Manipulation failure
Success
Placing failure
Success

Success
Success
Success
Manipulation failure
Manipulation failure
Success
Manipulation failure
Manipulation failure
Manipulation failure
Success
Success
Success
Success
Success
Navigation failure
Navigation failure
Continued on the next page

Pick object

apple
ice cream
green lime juice bottle
yellow packet
red packet
orange cancooking oil bottle
green bowl
washing gloves
small oregano bottle
yellow noodles packet [stove]blue dish wash bottle

apple
ice cream
green lime juice bottle
orange cancooking oil bottle
washing gloves
small oregano bottle
yellow noodles packet [stove]blue dish wash bottle

pepsi
birdieblack hat
owl like wood carving
red inhaler
black sesame seeds
bananaloose-leaf herbal tea jar
red pencil sharpener
fast-food French fries container
milk
socks[bed]purple gloves
target bag
muffintissue paper
grey eyeglass box

pepsi
birdieowl like wood carving
red inhaler
black sesame seeds
loose-leaf herbal tea jar
fast-food French fries container

Place location

ResultCleanup level: low
white plate
red basket
red basket
green bagSuccess
Success
Success
Success
Manipulation failure
Success
Success
Manipulation failure
Success
Success
Manipulation failure
Success

card board box
marble surface [red basket]
sink
red basket
card board box
Cleanup level: highwhite plate
red basket
red basket
card board box

Success
Success
Success
Success
Manipulation failure
Success
Success
Success
Successsink
red basket
red basket
card board box
Home 4
Cleanup level: none
black chair
cloth bin
bed

black chair
blue shopping bag [metal drying rack]
plastic storage drawer unit

cloth bingrey bed
table

Cleanup level: lowbasketwhite drawer
plastic storage drawer unit
bed
table
chair

Success
Success
Navigation failure
Success
Manipulation failure
Manipulation failure
Manipulation failure
Success
Navigation failure
Placing failure
Success
Navigation failure
Manipulation failure
Success
Success
Success
Manipulation failure

Success
Success
Navigation failure
Success
Success
Success
Success
Continued on the next page

Pick object
milk
purple gloves
target bag
muffintissue paper
grey eyeglass box

pepsi
birdiered inhaler
black sesame seeds
bananaloose-leaf herbal tea jar
milk
purple gloves
target bag
muffinPlace location
chairbasketbaskettable

Cleanup level: highbasketbed
plastic storage drawer unit
desk

chair
basketbasketbed

ResultSuccess
Success
Placing failure
Success
Manipulation failure
Navigation failure

Success
Success
Success
Success
Manipulation failure
Manipulation failure
Success
Success
Success
Success

Home 5
Cleanup level: none
tiger balm topical ointment
pink shampoo
aveeno sunscreen protective lotion
small yellow nozzle spray
black hair care spraygreen hand sanitizer
white hand sanitizer
white bowl [ketchup]
blue bowl
blue sponge
ketchupwhite salt
black pepper
blue bottle
purple light bulb box
white plastic bag
rag

trader joes shapping bag
trader joes shapping bag

black sofa chair
trader joes shapping bag

black drawer
trader joes shopping bag
bed
white rack

Navigation failure
Success
Success
Manipulation failure
Manipulation failure
Manipulation failure
Navigation failure
Success
Manipulation failure
Success
Manipulation failure
Manipulation failure
Success
Navigation failure
Success
Success
Success

Cleanup level: lowpink shampoo
aveeno sunscreen protective lotion
small yellow nozzle spraywhite bowl [ketchup]
blue sponge
ketchupwhite salt
black pepper
blue bottle
purple light bulb box
ragpink shampoo

black sofa chairbed
trader joes shopping bag
trader joes shopping bag
black sofa chair
white rack
Cleanup level: hightrader joes shopping bag

Navigation failure
Manipulation failure
Manipulation failure
Success
Success
Success
Success
Navigation failure
Success
Manipulation failure
Success

Success
Continued on the next page

Pick object
green hand sanitizer
white bowl [ketchup]
blue sponge
ketchupwhite salt
purple light bulb box
rag

Place location
black sofa chair
bed
black drawer
white drawer
trader joes shopping bag
black sofa chair

ResultSuccess
Manipulation failure
Success
Success
Success
Success
Success

Home 6
Cleanup level: nonetranslucent grey cup
green mouth spray box
green eyeglass container
blue bagblack burn ointment box
white vitamin bottle
McDonald’s paper bag
purple medicine packaging
grey rag
sparkling water can [sink]gold wrapped chocolatelemon tea carton
metallic golden beverage canred bottle
tea milk bottle
nyu water bottle [sink]white hand wash

stove
chair

stove
chair
sink
countertop
table
table
table
table

Manipulation failure
Success
Success
Manipulation failure
Navigation failure
Navigation failure
Success
Success
Success
Success
Manipulation failure
Success
Success
Success
Navigation failure
Success
Navigation failure

Cleanup level: lowtranslucent grey cup
green mouth spray box
blue bagblack burn ointment box
McDonald’s paper bag
grey rag
sparkling water can [sink]lemon tea carton
metallic golden beverage canred bottle
nyu water bottle [sink]white hand wash

blue bagblack burn ointment box
grey rag
sparkling water can [sink]lemon tea carton
metallic golden beverage canred bottle
nyu water bottle [sink]white hand wash

brown box
brown box
sink
chair
stove
brown box
table
sink
Cleanup level: highbrown box
sink
chair
table
stove

Navigation failure
Manipulation failure
Success
Success
Navigation failure
Success
Success
Success
Navigation failure
Success
Success
Success

Success
Manipulation failure
Success
Success
Success
Success
Navigation failure
Manipulation failure
Manipulation failure

Home 7
Continued on the next page

Pick object
blue plastic bag rollgreen bagtoy cactus
toy vanbrown medical bandage
power adapter
red herbal tea
apple juice box
paper towel
toy bearyellow ball
black pants
purple water bottle
blue eyeglass casebrown toy monkey
blue hardware box [table]
green zandu balm container

green bagtoy cactus
toy vanbrown medical bandage
red herbal tea
apple juice box
paper towel
toy bearpurple water bottle
blue eyeglass casegreen zandu balm container

green bagtoy cactus
toy vanred herbal tea
apple juice box
paper towel
toy bearyellow ball
purple water bottle
green zandu balm container

cyan air sprayblue gloves
blue peanut butter
nutellagreen baggreen bandage box
green detergent
black ‘red pepper sauce’

Place location
Cleanup level: nonebasket[window]
desk
chair
chair
brown cardboard box
brown cardboard box
blue cardboard box
bed blanket
bed blanket
basket[window]
deskblue cardboard box
blue cardboard box
Cleanup level: lowbasketbasketchair
brown box
brown box
basketdesk
desk
blue cardboard box
Cleanup level: highstool [window]
table
white basket
brown cardboard box
brown cardboard box
blue cardboard box
white basket
bed
black tote bagblue cardboard box
Home 8
Cleanup level: nonebrown shelf [sink]kitchen sinkblack stovetablebrown shelf [sink]trash cankitchen sink

ResultNavigation failure
Success
Success
Success
Success
Navigation failure
Success
Success
Success
Success
Success
Success
Success
Manipulation failure
Navigation failure
Success
Success

Success
Success
Success
Manipulation failure
Success
Success
Success
Success
Success
Manipulation failure
Success

Success
Success
Success
Success
Success
Success
Success
Success
Success
Success

Success
Success
Success
Success
Success
Success
Success
Manipulation failure
Continued on the next page

Pick object
red bag
black bagred spray [brown shelf]
steel wool
white aerosol
white pretzel
purple crisp
plastic bowl
playing card

cyan air apray
blue gloves
blue peanut butter
green baggreen bandage box
green detergent
red bag
black bagwhite aerosol
white pretzel
purple crisp
plastic bowl
playing card

cyan air apray
blue gloves
blue peanut butter
green baggreen bandage box
green detergent
black bagwhite aerosol
purple crisp
playing card

toy grapes
purple strap
red foggy body sprayarm smartphone holder
medicine bottle
yogurt beverage
blue shaving cream canblue cup
purple tape
black shoe brush
fluffy headband
black water bottle
yellow eyeglass casepaper cup
lotion pumpPlace location
chair
chair
kitchen countertop
trash canblack stove
kitchen countertop
microwave
Cleanup level: low
chair
sink
brown shelf
brown shopping bag
microwave
chair
trash canblack stove
kitchen countertop
microwave
Cleanup level: highbrown shelf [sink]stove
black stovebrown shelf [sink]
microwave
chair
table
chair
microwave
Home 9Cleanup level: none
black laundry bag

bed

table

brown shopping bag
black chair

ResultSuccess
Success
Success
Manipulation failure
Success
Success
Success
Manipulation failure
Success

Success
Success
Navigation failure
Success
Success
Success
Manipulation failure
Success
Success
Success
Success
Manipulation failure
Success

Success
Success
Success
Success
Success
Manipulation failure
Success
Success
Success
Success

Success
Manipulation failure
Manipulation failure
Success
Manipulation failure
Navigation failure
Navigation failure
Success
Manipulation failure
Navigation failure
Manipulation failure
Placing failure
Success
Manipulation failure
Manipulation failure
Continued on the next page

Pick object
nasal sprayplastic bag

Place location
trash basket

ResultManipulation failure
Success

Cleanup level: low
toy grapes
red foggy body sprayarm smartphone holder
yogurt beverage
blue shaving cream canblue cup
black shoe brush
fluffy headband
black water bottle
nasal sprayplastic bag

red foggy body sprayarm smartphone holder
yogurt beverage
blue shaving cream canblue cup
black water bottle
nasal sprayplastic bag

grey toy dragon
purple body spray
hand sanitizer
toy plant
brown trail mix bag
hanging blue shirt
white apple bag
white and pink powder bottle
cough syrup bottle
tangled ear phones
red deodrant stick[table]
black body sprayhair treatment medicine bottle
green tea package
portable speaker [green tea package]
wooden workout gripper
brown box
blue bulb adapter
game controller

grey toy dragon
purple body spray
hand sanitizer
toy plant
brown trail mix bag

brown paper bag
brown paper bag
desk
black bagblack chairfolded chair
trash basket
Cleanup level: highbrown paper bag
desk
black bagblack chair
white bed
folded chair
trash basket
Home 10
Cleanup level: none
bed
shelf
bed [shelf]
cloth bin
table
shelf
office chair
chair
chair
chair
office chair

office chair
office chair
Cleanup level: loworange bag
table
bed

Manipulation failure
Success
Success
Success
Success
Success
Manipulation failure
Navigation failure
Success
Navigation failure
Success

Success
Manipulation failure
Success
Success
Success
Success
Success
Success

Success
Manipulation failure
Success
Success
Manipulation failure
Success
Manipulation failure
Success
Success
Success
Success
Success
Manipulation failure
Success
Success
Navigation failure
Navigation failure
Success
Success

Success
Success
Navigation failure
Success
Manipulation failure
Continued on the next page

Pick object
white and pink powder bottle
cough syrup bottle
red deodrant stick[table]
black body spraygreen tea packagebrown box
blue bulb adapter

purple body spray
toy plant
white and pink powder bottle
cough syrup bottle
red deodrant stick[table]
black body spraygreen tea package
blue bulb adapter

Place location
black chair [bed]shelf [bed]bed [rack]
rack [bed]orange bag
black chair [bed]Cleanup level: highorange bag
bed
shelf [bed]black chair
table
shelf

ResultSuccess
Success
Success
Placing failure
Success
Success
Manipulation failure

Success
Success
Navigation failure
Success
Navigation failure
Success
Success
Success