Open-World Object Manipulation using Pre-Trained Vision-Language Models
Authors: Austin Stone
Ted Xiao
Yao Lu
Keerthana Gopalakrishnan
Kuang-Huei Lee
Quan Vuong
Paul Wohlhart
Sean Kirmani
Brianna Zitkovich
Fei Xia
Chelsea Finn
Karol Hausman
Venue: CoRL 2023 Poster (2023)

Abstract: For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary, e.g. ``can you get me the pink stuffed whale?'' to their sensory observations and actions. This brings up a notably difficult challenge for robots: while robot learning approaches allow robots to learn many different behaviors from first-hand experience, it is impractical for robots to have first-hand experiences that span all of this semantic information. We would like a robot's policy to be able to perceive and pick up the pink stuffed whale, even if it has never seen any data interacting with a stuffed whale before. Fortunately, static data on the internet has vast semantic information, and this information is captured in pre-trained vision-language models. In this paper, we study whether we can interface robot policies with these pre-trained models, with the aim of allowing robots to complete instructions involving object categories that the robot has never seen first-hand. We develop a simple approach, which we call Manipulation of Open-World Objects (MOO), which leverages a pre-trained vision-language model to extract object-identifying information from the language command and image, and conditions the robot policy on the current image, the instruction, and the extracted object information. In a variety of experiments on a real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments. In addition, we show how MOO generalizes to other, non-language-based input modalities to specify the object of interest such as finger pointing, and how it can be further extended to enable open-world navigation and manipulation. The project’s website and evaluation videos can be found at https://robot-moo.github.io/.

Open-World Object Manipulation using Pre-Trained Vision-Language Models
Austin Stone", Ted Xiao", Yao Lu", Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, Chelsea Finn, Karol Hausman Robotics at Google
Abstract: For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary, e.g. "can you get me the pink stuffed whale?" to their sensory observations and actions. This brings up a notably difficult challenge for robots: while robot learning approaches allow robots to learn many different behaviors from first-hand experience, it is imprac- tical for robots to have first-hand experiences that span all of this semantic infor- mation. We would like robot's policy to be able to perceive and pick up the pink stuffed whale, even if it has never seen any data interacting with a stuffed whale before. Fortunately, static data on the internet has vast semantic information, and this information is captured in pre-trained vision-language models. In this paper, we study whether we can interface robot policies with these pre-trained models, with the aim of allowing robots to complete instructions involving object cate- gories that the robot has never seen first-hand. We develop simple approach, which we call Manipulation of Open-World World Objects (MOO), which leverages a pre-trained lssoonlanguage model to extract object-identifying information from the language command and image, and conditions the robot policy on the current image, the instruction, and the extracted object information. In a variety of exper- iments on real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments. In addition, we show how MOO generalizes to other, non- lee-lannuge-bbaeee input modalities to specify the object of interest such as finger pointing, and how it can be further extended to enable open-world navigation and manipulation. The project's website and evalu- ation videos can be found at https: https: h/tps:://rooottmeeee gi thub.io/
1 Introduction
For a robot to be able to follow instructions from humans, it must cope with the vast variety of language vocabulary, much of which may refer to objects that the robot has never interacted with first-hand. For For example, consider the scenario where a robot has never seen or interacted with a plush animal from its own camera, and it is asked, "can you get me the pink stuffed whale?" How can the robot complete the task? While the robot has never interacted with this object category before, the internet and other data sources cover much wider set of objects and object attributes than the robot has encountered in its own first-hand experience. In this paper, we study whether robots can tap into the rich semantic knowledge captured in such static datasets, in combination with the robot's own experience, to be able to complete manipulation tasks involving novel object categories.
Computer vision models can capture the rich semantic information contained in static datasets. Indeed, composing modules for perception, planning, and control in robotics pipelines is a long- standing approach [1, 2, 3). allowing robots to perform tasks with a wide set of objects [4]. How- ever, these pipelines are notably brittle, since the success of latter motor control modules relies on precise object localization. On the other hand, several prior works have trained neural network policies with pre- trained image representations [5, 6, 7, 8] and pre-trained language instruction em- beddings [9, 10, 11, 12]. While this form of vanilla pre-trate pre-training cralization, it does not provide a mechanism for robots to ground and manipulate novel semantic concepts, such as unseen object categories referenced in the language instruction. This leads to crossroads - some approaches can conceivably generalize to many object categories but rely on fragile pipelines; others are less brittle but cannot generalize to new semantic object categories.
improve efficiency and gen-
Indicates equal contribution. Please direct correspondence to tedxiao@google.com
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

Prompt+ Object Name
VLM
Object Detection with VLM
Train 106 objects, demos
Generakization Novel Obyects
Instruction Beyond Langunge
Overview of MOO. We train language-conditionas policy conditioned on object locations from frozen VLM. The policy is trained on demonstrations spanning set 106 objects using VLM-based object- centric representations, enabling generalization to novel objects, locations produced from new modalities.
Figure I:
To allow robots to generalize to new semantic concepts, we specifically choose to leverage open- vocabulary pre-trained vision-language models (VLMs), rather than models pre-trained one modality alone. Such models capture the rich information contained in diverse static datasets, while grounding the linguistic concepts into a perceptual representation that can be connected to the robot's observations. Crucially, rather than using the pre-trained model for precise state estimation in its en- tirety (akin to pipelined approaches), we only use the VLM to identify the relevant objects in the image by coarsely localizing them, while allowing an end-to-end trained policy to use this informa- tion along with the original observation to perform the task. More specifically, our system receives language instruction and uses a VLM to identify the 2D image coordinates of objects in the instruc- tion. Along with the image and the instruction, the 2D coordinates of the objects are fed into our manipulation policy allowing it to ground the natural language to objects and know which objects to act upon without seeing any demonstrations with those objects. The VLM is frozen throughout all of our training, and the policy is trained with the real VLM detector in the loop to prevent the brittleness that can plague prior pipelined approaches.
The main contribution of this paper is flexible approach for open-world object manipulation that interfaces policy learning with pre-trained vision-language models. An overview is given in Fig. 1. The pre-trained models are trained on massive static image and language data that far exceeds the robot's own experience. The robot's policy is trained perform skills using demonstration data covering more modest yet still physically diverse set of 106 training objects. The composition of the pre-trained vision-language model and the control policy leads to an overarching language- conditioned policy that can complete commands that refer to novel semantic categories.
We study the performance of our method across 1, 472 evaluations on real robotic manipulator, where we find that our approach is significantly more successful than recent robot learning methods. Beyond verbal object descriptions, we also find that the trained policy can be easily combined with other means of communicating intent, e.g., pointing at an object and inferring the object description using VLM, showing generic image the object of interest, or using simple GUI. Finally, our experiments further show that our method can be integrated with an open-vocabulary object navigation model called Clip-on-We Wheels (CoW), to complete mobile manipulation tasks involving novel objects. Throughout this paper, we refer to our approach as Manipulation of Open-vocabulary Objects (MOO) and the integrated mobile manipulation system as CoW-MOO.
2 Related Work
Leveraging Pre- Trained Models in Robotic Learning. Using off-the-shelf vision, speech, or lan- guage models is a long-standing approach in robotics [13, 14, 10]. Modern pre-trained vision and language models have improved substantially over older models, and have played an increasing role in robotics research. A large body of prior work has trained policies top of frozen or fine-tuned visual representations [5, 15, 6, 16, 17, 18, 19, 7, 8, 20, 21]. while other works have leveraged pre- trained language models [22, 23, 9, 10, 11, 24, 25, 12]. Unlike these prior works, we aim to leverage vision- lssiun-language models that ground language in visual observations. Our use vision-language models enables generalization novel semantic object categories, which cannot be achieved by using vision models or language models individually.
Generalization in Robotic Learning. A number of recent works have studied how robots can com- plete novel language instructions [26, 22, 23, 9, 10, 11, 27, 28, 24], typically focusing on instructions with novel combinations of words, i.e. compositional generalization, or instructions with novel ways

---

to describe previously-seen objects and behaviors. Our work focuses on how robots can complete instructions with entirely new words that refer to objects that were not seen in the robot's demonstra- tion dataset. Other research has studied how robot behaviors like grasping and pick-and-plaa can be applied to unseen objects [29, 30, 31,32, 32, 33, 34, 35, 36, 37], foeusing on generalization to visual or physical attributes. Our experimental settings require visual and physical object generalization but also require semantic object generalization. That is, unlike these prior works, the robot must be able to ground description of previously-unseen object category.
Vision-Language Models for Robotic Manipulation. Two closest related works to our approach are CLIPort [38] and PerAct [12] that use the CLIP vision- langunla model as a backbone of their policy. Both of these approaches have demonstrated impressive level of generalization to unseen semantic categories and attributes. Inspired by these works, we aim to expand them to more general manipulation settings by: i) removing the need for depth cameras or camera calibration, ii) expand- ing and demonstrating that the hereby introduced representation works with other modalities such as pointing to the object of interest, iii) moving beyond 2D manipulation tasks, e.g. demonstrating the approach on tasks such as reorienting objects upright well mobile manipulation tasks.
Open-World Object Detection in Computer Vision. Historically, object-detection methods have been restricted to a fixed category map covering a limited set of objects [39, 40, 41, 42]. These methods work well for the object categories on which they are trained, but have no knowledge of objects outside their limited vocabulary. Recently, a new wave of object detectors have emerged that aim to go beyond simple ddoeedvocabullry tasks by replacing the fixed one-hot category prediction with a shared
langugg embedding space that can be used to answer open-vocabulay object queries [43, 44, 45, 46). Typically these methods rely on internet-scale data in the form of pairs of image and associated descriptive text to learn the grounding of natural language to objects. Various methods have been employed to then extract object localization information in the form of bound- ing boxes and segmentation masks. In our work, we use the OWL-ViT detector due to it's strong performance in the wild and publicly available implementation [43].
3 Manipulation of Open- World Objects (MOO)
The key goal of MOO is to develop a policy that can leverage the visually-grounded semantic in- formation captured by pre-trained vision-language models for generalization to object types not in the policy training set. More specifically, we aim to use the VLM to localize objects described in a given instruction, while allowing the policy complete the task using both the instruction and the object localization information from the VLM. MOO accomplishes this in two stages. First, the current observation and the words in the instruction corresponding to object(s) are passed to the VLM to localize the objects. Then, the object localization information and the instruction sans object information are passed the policy, along with the original observation, to predict actions.
The key design choice of MOO lies in how to represent object information encoded in VLMs and how to feed that information the instruction-conditionns policy. In the remainder of this section, we first overview the set-up, then describe the design of these crucial aspects of the method, and finally provide an overview of the model architecture and the training procedure as well describe practical implementation details that allows us to deploy MOO on real robots.
3.1 Problem Set-Up
Formally, we assume that the robot, with image observations o E O and actions a E A, is pro- vided with set of expert demonstrations Drobat collected via teleoperation. Each demonstration corresponds to a sequence of observation-actioon pairs {(o), a,)) collected over a time horizon T. and is annotated with structured language instruction e for the task being performed in the demonstration. To help facilitate object generalization, we assume that these language instructions are structured as combination of template and list of object descriptions within that template. For example, for the instruction & = "move yellow banana near cup.". the template is "move X near y." and the object descriptions are X = "yellow banana" " and Y ='cup." " Inspired by RT-1 [24], in this work, we focus on five different types of skills defining the templates: "pick X." "move X near Y "knock X over," "place X uprighi," " and "place X into Y.".
All of the objects seen in the demonstrations are drawn from a set Suobats and our objective is to complete new structured language instructions with a seen template but novel objects that are not in
3

---

Instruction
Encoded Instruction
Feer (apppiee
Adton
Current Image and Recent
Owi-VIT
Initial Image
Efficinn
Tokon Learner
Transformar
Mack
Figure 2: MOO architecture: We extract object location (represented as the center the bounding box) on the first frame of an episode. The segmentation mask is concatenated channel-wise to the input image for cach frame. We remove the language embedding for everything except the task so that the object specific information is only provided through the object instance mask.
(repregated
Siohut, which also have novel object descriptions. In aiming complete this goal, our approach will leverage imitation learning and vision-language models, which we briefly review the Appendix.
3.2 Representing Object Information
To utilize the object knowledge encoded in the VLMs, we need to pick a representation that can be easily transferred to the text-conditioned control policy. We start by identifying the instruction template (represented by verb v) and object X (or list of objects X, ..... from the instruction L. Equipped with an object description X, we query a VLM to produce bounding box of the object of interest with the prompt = "An image of an X", and use the resulting detection (if any) as conditioning of our policy. To reduce the reliance of the exact segmentation of the object dimensions, we select a single pixel that is at the center of the predicted bounding box as the object representation, In the case of one object description, we use a single-channel object mask with the value set to 1.0 at the pixel of the object's predicted location. In the case of two object descriptions, we set the pixel value of the first to be 1.0 and the second to be 0.5.
This design has two main advantages: first, it is generic representation that works with objects of any size as long as they are visible, and second, it is compatible with large selection of vision meth- ods such as bounding boxes or segmentation masks as these can be easily transformed into single, object-centric pixel location. We ablate other object representation choices in the experiments.
Importantly, this approach can handle object descriptions that not previously demonstration data, as long as it is sufficiently represented in the static large-scale training data of the VLM. For any unseen objects, "pick stuffed toy whale." " Once the object description is translated into a pixel location by the VLM, the robot's policy trained on demonstration data only needs to be capable of interpreting the mask location and how to physically manipulate the novel object's 's shape, rather than needing to also ground the semantic object description.
in the robot's
simply include description in the task command, e.g.,
3.3 Architecture and Training of MOO
We present the model architecture and information flow of MOO in Fig. 2. As described above, we extract the object descriptions from the language instruction and together with the initial image feed them into the VLM to output object locations in the image. This information is then represented as an object mask with dots the center of the objects of interest.
Once we obtain the mask, we append it channel-wise to the current image together with the recent image history, which is passed into the RT-1 policy architecture [24]. We use language model to encode the previously extracted verb v part of the language instruction an embedding space of an LLM. The images are processed by an EfficientNet [47] conditioned on the text embedding via FiLM [48]. This is followed by a Token Learner [49] to compute a small set of tokens, and finally Transformer [50] to attend over these tokens and produce discretized action tokens. We refer the reader to the RT-1 paper for details regarding the later part of the architecture. The action space corresponds to the 7-DoF delta end-effector pose of the arm (including x, y. z, roll, pitch, yaw and gripper opening). The entire policy network is trained end-to-end using the imitation learning objective and we specify the details of the objective in the Appendix (Equation 1). Importantly, VLM used to detect the objects is frozen during training, so that it does not overfit to the objects in the robot demonstration data. The policy trained with this frozen VLM in the loop, so that the policy can learn to be robust to errors made by the VLM given other information in the image.

---

3.4 Practical Implementation
To detect objects in robot images, practice, we find that it is capable of detecting most elearly visible objects without any fine-tuning, given a descriptive natural language phrase. The interface to the detector requires natural language phrase describing what to detect (e.g.. "An image of a small blue elephant toy.") along with an image to run the detection on. The output from the model is a score map indicating which locations most likely to correspond to the natural language description and their associated bounding boxes. We select universal score threshold filter detections. To detect our objects, prompt-engineering using descriptive phrases including the color, size, and shape of objects, though most of our prompts worked well on the first attempt. We share the prompts in the Appendix.
the Owl-ViT open- vocabulary object detector [43]. In
rely some
To make the inference time practical on real robots, we extract the object information via VLM only in the first episode frame. By doing so, most of the heavy computation is executed only once at the beginning and we can perform teml-time control for the entire episode. Since the information is appended to the current observation, we rely the policy find the corresponding object in the current image if the object has moved since the first timestep.
3.5 Training Data
We start with the demonstra- tion data used by RT-1 [24] cov- ering 16 unique objects. De- spite the use of the VLM for semantic generalization, we pect that the policy will re- quire more physical object di- versity to generalize to novel ob- jects. Therefore, we expand the dataset with additional diverse "pick" data across diverse objects, for total of 106 objects, shown in Fig- ure 3. We choose to only expand the set of objects for the picking skill, since it is the fastest skill to execute and therefore allows for the greatest amount of diverse data collection within limited budget of demonstrator time. Our additional set of 90 diverse objects only appear in "pick" episodes. All other tasks, such as "move near" or "place into",, must be learned from the original 16 objects in the RT-1 dataset. Detailed statisties are Appendix Figure 9.
Objects (all skills)
90 Diverse Objects ("plck" skill only)
Novel Objects
of 90
(unseen during training)
Figure 3: (Left) RT-1 objects that account for R 70% of training data covering all skills. (Middle) Diverse training objects that appear only in "pick" demonstrations. (Right) Unseen objects used only for evaluation.
4 Experiments
Our experiments answer the following questions: 1) Does MOO generalize across objects for differ- ent skills including unseen objects? 2) Does MOO generalize beyond new objects - Is MOO robust distractors, backgrounds and environments? 3) Can the intermediate representation used support nin-lingci modalities to specify the task? 4) Does the object generalization performance scale with the (a) number of training episodes, (b) number of unique objects in the training episodes and (c) size of the model? 5) Can MOO be used for open- open-world navigation and manipulation?
4.1 Experimental Setup
Seen and unseen objects. The training data collected with teleoperation table-top environ- ments across a set of 106 different object types. We evaluate performance on a subset with 49 objects "seen" in training and report the performance "seen". We hold out 47 objects not present in train- ing and report performance on these as "unseen". These 47 held-out objects are comprised of 22 objects of the categories seen training and 25 objects of unseen categories. These objects are listed in Appendix Table 1. Note that previous works often focus on unseen combinations of previously seen commands and objects (e.g. "pick an apple" even though the training data contains "move an apple into a bowl" and "pick a bowl"); we adopt a more strict definition of unseen objects, where our unseen objects were not seen in the robot's training demonstration data at any point for any task, therefore making our unseen performance a zero-shot object generalization problem. Further- more, we report results across different environments that introduce novel textures, backgrounds, locations, and additional open-world objects not present in the training data.
5

---

MOO (our data)
MOO (original RT-1 data)
RT-1 (our data)
VIMA-Iike (our data)
1,00
0.75
0.50
0.25
0.00
Pick Skill
Non-pick Skills
Pick Skill
Non-pick Skills
Pick Skill
Non-pick Skills
Unseen Objects, Seen Categories (22)
Seen Objects (49)
Unseen Categories (25)
Figure 4: Main Results. While baseline methods perform competitively on in-distribution combinations objects and skills seen during training. they fail to generalize to novel objects. MOO substantially improves generalization to novel objects, especially those in unseen categories and for the "pick" skill.
Evaluation details. We evaluate on of tabletop tasks involving manipulating diverse set of objects. We use mobile manipulators with degree-off of-eee-oofooo arm and two-fingered gripper (Appendix Figure 10). Our experiments evaluate the percent of successfully completed manipulation commands which include five skills: "pick", , "move near", "knock,"" "place upright," and "place into" across a set of evaluation episodes (definition and success criteria follow RT-1 [24] and are described in the Appendix). To study object specificity and robustness, for all evaluation episodes, we include between two to four distractor objects the scene which the robot should not interact with. For each evaluation episode, table. There is no consistent placement of the target object relative to distractors. We repeat this process 21 times and report the performance. We present the experimental setup in Appendix Figure 10.
randomly scatter the evaluation object(s) and the distractor objects onto
We compare two prior methods; RT-1 [24] and modified version of VIMA [25], referred as "VIMA-like". VIMA-like preserves the cross-attention mechanism, but uses the mask image as the prompt token and the current image as state token. These modifications are necessary because VIMA uses Transporter- based action space and not applicable to our task, i.e., our robot arm moves in 6D and has a gripper that can open and close continuously. These two baselines correspond to common alternatives where computer vision data used as a pre-training mechanism (as in RT-I) or object-centric information is fed to the network through cross attention (as in VIMA-like).
Baselines.
4.2 Experimental Results
Generalization to Novel Objects. We investigate the question: Does MOO generalize across objects for different manipulation skills includ- ing objects never seen at training time? Experiments are presented in Figure and example trajectories shown in Appendix Figure 12. Rel- the pick
MOO (our data)
MOO (original RT-1 data)
RT-1 (our data)
VIMA-IIke (our data)
1.00
0.75
0.50
ative the baselines tasks, MOO exhibits substantial im- provement over the seen object per- formance as well as the unseen ob- jects, which in both cases reaches ~ 50% improvement. MOO can cor- rectly utilize a VLM find novel ob- jects and incorporate that information more effectively than the VIMA-like baseline. When comparing the performance on seen objects for the skills other than pick, we observe a slightly worse performance than for the pick tasks. This understandable since the "Seen objects" for "Non-pick skills" have only been seen during the pick episodes as shown in Appendix Figure learned object generalization across the skills so
0.25
0.00
Open-Worid Objects
Challenging Textures
Environments
Figure 5: MOO is able generalize to new objects, textures, and environments with greater success than prior methods. Visualiza- tions are shown in Figure 6.
This demonstrates MOO's ability to transfer that the objects that have only been picked can now be also used for other skills. In addition, MOO exhibit generalization unseen objects (i.e. unseen any previous tasks, including pick) that is at the same level as for unseen objects for the pick skill, and 50% better than baseline.

---

(a)
(b) Challenging Textures
(c) Open-World Objects
New Environments
Figure 6: study the robustness of MOO, we evaluate on (a) new environments, (b) challenging texture backgrounds which are visually similar to unseen objects in the scene, and (c) additional open-world objects.
Input
Mask
Rollouts
Text Instruction
(a)
"pick yellow highlighter
Point with Finger
(b)
Target Imago Upload
(c)
Click GUI
(d)
Figure 7: We explore using various input modalitics to generate the single-ping pixel object representations used by
MOO. (a) shows the standard mask generation process using OWL-ViT with a text instruction. (b) shows using VLM to generate text caption, then fed to OWL-ViT. (c) shows an uploaded image to prompt OWL-VVT. (d) shows user providing ground-truth mask via GUI.
Robustness Beyond New Objects. To further test the robustness MOO, we analyze novel evalua- tion settings with significantly increased difficulty and visual variation, which are shown in Figure 6. To reduce the number of real robot evaluations, we focus this comparison on the picking skill. The results presented in Figure 5. Across these challenging evaluation scenes, MOO is significantly more robust compared to ViMA-like [25] and RT-1 [24]. This indicates that the use of VLMs in MOO not only improves generalization to new objects that the robot has not interacted with, but also significantly improves generalization to new backgrounds and environments.
Input Modality Experiments. To answer our third question, we perform a number of qualitative experiments testing different input modalities (detailed description in the Appendix). We find that MOO is able to generalize to masks generated from a variety of upstream input modalities, even under scenarios outside the training distribution including scenes with duplicate objects and clutter.
As the first qualitative example, Figure 7(b) illustrates that VLM such as PaI.I [51] can infer what object a human is pointing at, allowing OWL-ViT to generate an accurate mask of the object of in- terest. Secondly, OWL-ViT can also use visual query features instead of textual features to generate mask, enabling images of target objects to act as conditioning for MOO, shown in Figure 7(c). This modality is useful in cases where text-based mask generation due to ambiguity in natural lan- guage, or when target images are found in other scene contexts. We explore both the setting where target images are sourced from similar scenes or from diverse internet images. Finally, we show that MOO can interpret masks directly provided by humans via GUI, as shown in Figure 7(d). This is useful in cases where both text-based and image-based based mask generation is difficult, such with duplicate or cluttered objects. MOO is robust to how upstream input masks were generated, and our preliminary results suggest interesting future avenues in the space of human-rofe interaction.
7

---

CoW: "find the pepsi"
MOO: "pick up the pepsi"
Figure 8: We present CoW-MOO, system that combines an open- -en-bocabulary object navigation by CoW [52]
with open-world manipulation by MOO. Full videos are shown on the project's website.
MOO Ablations. We conduct a number of ablations to assess the impact of the size and diversity of our dataset and the scale (in terms of number of parameters) of our model. In Appendix Table we vary both the number of unique objects in the training set (reducing it from 106 to 53 to 16 unique training objects) and the number of total training episodes (reducing by half from 59051 training episodes to 29525) while keeping all objects in the dataset. We aim to vary these two axes independently to determine the impact of the overall size of the dataset vs its object diversity on the final results. Interestingly, we find that seen object performance is not affected by reducing object diversity, but generalization to unseen objects is very sensitive object diversity.
Additionally, we investigate the impact of scaling model size. We train two smaller versions of MOO where scale down the total number of layers and the layer width by constant factor. The version of MOO that we use in our main experiments has 111M parameters, which, for the purpose of this ablation, we then reduce by order of magnitude down to 10.2M and then by 5X again down to 2.37M. Comparing different sizes of the model, we find significant drop offs in both "seen" (from 98% to 54% and 39% respectively) and "unseen" object performance (from 79% to 50% and 13%; see Appendix Figure 11 for graph of the results). We also note that we could not make MOO larger than 1 111M parameters without increasing the latency on robot an unacceptable level, but we expect continued performance gains with bigger models if latency requirements can be relaxed.
Open- Woond Navigation and Manipulation. Finally, we consider how such a system can be inte- grated with open-vocabut vocabulary object-based navigation. Coincidentally, there is an open-vocabulary object navigation algorithm called Clip on Wheels (CoW) [52]; we implement a variant of CoW and combine it with MOO, which we refer to as CoW-MOO. CoW handles open-vocabulary navigation to object of interest, upon which MOO continues with manipulating the target object. This com- bination enables a truly open-world task execution, where the robot is able to first find an object it has never interacted with, and then successfully manipulate it to accomplish the task. We show example qualitative experiments in Figure and in the video of this system on the project's 's w
5 Conclusion and Limitations
In this paper we presented MOO, an approach for leveraging the rich semantic knowledge captured by vision-language models in robotic manipulation policies. We conduct 1,472 real world evalua- tions to show that MOO allows robots to generalize to novel instructions involving novel objects, enables greater robustness to visually challenging table textures and new environments, is amenable to multiple input modalities, and can be combined with open-vocabulary semantic navigation.
Despite the promising results, MOO has multiple important limitations. First, the object mask rep- resentation used by MOO may struggle in visually ambiguous cases, such as where objects are overlapping or occluded. Second, we expect the generalization of the policy still be limited by the motion diversity of training data. For example, we expect that the robot may struggle to grasp novel objects with drastically different shapes or sizes than those seen in the training demonstration data, even with successful object localization. Third, instructions are currently expected to conform to a set of templates from which target objects and verbs can be easily separated. We expect this limita- tion could be lifted by leveraging an LLM to extract relevant properties from freeform instructions. Finally, MOO cannot currently handle complex object descriptions involving spatial relations, such as "the small object to the left of the plate." Fortunately. we expect performance on tasks such as these to improve significantly as vision-language models continue advance moving forward.
hnttps//roboo-moot github.
8

---

References
[1] N. NILLSON. Shakey the robot. SRI International, Technical Note, 323, 1984.
[2] S. Thrun, M. Montemerlo, H. Dahlkamp, D. Stavens, A. Aron, J. Diebel, Fong. J. Gale, M. Halpenny, G. Hoffmann, et al. Stanley: The robot that the darpa grand challenge. Journal of Field Robotics, 23(9):661-69,, 2006.
[3] P. Karkus, X. Ma, D. Hsu, L. P. Kaelbling, W. S. Lee, and T. Lozano-Pérez. Differentiable algorithm networks for composable robot learning. Robotics: Science and Systems (RSS), 2019.
[4] A. Curtis, X. Fang. I.. P. Kaelbling, T. Lozano-Pérez, and C. R. Garrett. Long-horizon ma- nipulation of unknown objects via task and motion planning with estimated affordances. In International Conference on Robotics and Automation (ICRA), pages 1940-1946. IEEE, 2022.
[5] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-137,, 2016.
[6] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and Funkhouser. Learning synergies between pushing and grasping with supe-super deep reinforcement learning. In 2018 IEFF/RSJ
International Conference on Intelligent Robots and Systems (IROS), pages 4238- 4245. IEEF, 2018
[7] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. International Conference on Machine Learning (ICML), 2022.
[8] Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. Conference on Robot Learning (CoRL), 2022.
[9] S. Nair, E. Mitchell, K. Chen, B. Ichter, S. Savarese, and C. Finn. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning (CoRL), pages 1303-1315, 2021.
[10] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. BC-Z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning (CoRL), pages 991-1002, 2021.
[11] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. Conference on Robot Learning (CoRL), 2022.
[12] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: multi-task transformer for robotic manipulation. Conference Robot Learning (CoRL., 2022.
[13] S. Teller, M. R. Walter, M. Antone, A. Correa, R. Davis, L. Fletcher, E. Frazzoli, J. Glass, How, A. S. Huang, et al. voice-commandable robotic forklift working alongside hu- mans in minimally-preparce on Robotics and Automation, pages 526-533. IEEE, 2010.
outdoor environments. In 2010 IEEE International Conference
[14] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. PoseCNN: A convolutional neural network for 6d object pose estimation in cluttered scenes. Roboties: Science and Systems (RSS), 2018.
[15] S. Kumra and C. Kanan. Robotic grasp detection using deep convolutional neural networks. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 769-776. IEEE, 2017.
[16] B. Zhou, P. Krähenbühl, and V. Koltun. Does computer vision matter for action? Science Robotics, 4(30), 2019.
[17] L. Yen-Chen, A. Zeng, S. Song. P. Isola, and T.-Y. Lin. Learning to see before learning to act: Visual pre-training for manipulation. IEEE International Conference on Robotics and Automation (ICRA), pages 7286-7293. IEFF, 2020.

---

[18] B. Chen, A. Sax, G. Lewis, I. Armeni, S. Savarese, A. Zamir, J. Malik, and L. Pinto. Ro- bust policies via mid-level visual representations: An experimental study in manipulation and navigation. Conference on Robot Learning (CoRI.), 2020.
[19] R. Shah and V. Kumar. Rrl: Resnet as representation for reinforcement learning. International Conference Machine Learning (ICMI.), 2021.
[20] Radosavovic, Xiao, S. James, Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. Conference Robot Learning (CoRL), 2022.
[21] Y.J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. VIP: Towards univer- sal visual reward and representation via value-implicit pre-training. International Conference on Learning Representations (ICLR), 2023.
[22] F. Hill, S. Mokra, N. Wong, and T. Harley. Human catructioo-followiing with deep reinforce- ment learning via transfer-learning from text. arXiv preprint hit::0005003882 2020.
[23] C. Lynch and P. Sermanet. Grounding language in play. Robotics: Science and Systems (RSS), 2021.
[24] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. RT-1: Robotics transformer for real-world control at scale. Robotics:
Science and Systems (RSS), 2023.
[25] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. VIMA: General robot manipulation with multimodal prompts. International Conference on Machine Learning (ICML), 2023.
[26] S. Stepputtis, Campbell, M. Phielipp. Lee, C. Baral, and H. Amor. Language- conditioned imitation learning for robot manipulation tasks. Neural Information Processing Systems (NeurIPS), 2020.
[27] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 1105-1 2022.
[28] H. Liu, L. Lee, K. Lee, and P. Abbeel. Instruction-following agents with jointly pre-trained vision- lasounlaanguage models. arXiv preprint arXiv:2210.1344 2022.
[29] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In JEEE international conference robotics and automation (ICRA), 2016.
[30] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. Robotics: Science and Systems (RSS), 2017.
[31] S. Levine, Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4-5), 2018
[32] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In IEEE International Conference on Robotics and Automation (ICRA), 2017.
[33] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673, 2018.
[34] S. Dasari, Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, Singh, Levine, and
C. Finn. Robonet: Large-scale multi-robot learning. In Conference on Robot Learning, 2019.
[35] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, Abbeel, and I.. Pinto. Visual imitation made casy. In Conference Robot Learning (CoRL), 2020.
10

---

[36] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao. D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach, R. Julian, C. Finn, al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. International Conference on Machine Learning (ICML), 2021.
[37] B.-H. Wu, S. Nair, R. Martin-Martin, L. Fei-Fei, and C. Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[38] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on Robot Learning, 2022.
[39] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 580-587, 2014.
[40] Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real- time object detection. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
[41] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cmn. In IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
[42] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2999-3007, 2017.
[43] M. Minderer, A. A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen, X. Wang. X. Zhai, T. Kipf, and N. Houlsby. Simple open-vocabulary object detection with vision transformers. European Conference on Computer Vision (ECCV), 2022.
[44] X. Gu, T.YY Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations (ICLR), 2022.
[45] A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pages 1780-1790, October 2021.
[46] Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, and J. Gao. Regionclip: Region-based language-image pretraining. In IEEE/CVF Conference on Computer Vision and Paltern Recognition (CVPR), pages 16793-16803, June 2022.
[47] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (ICML), pages 6105-6114, 2019.
[48] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[49] M. S. Ryoo, A. Piergiovanni, Arna, M. Dehghani, and A. Angelova. Tokenlearner: What can 8 learned tokens do for images and videos? Neural Information Processing Systems (NeurIPS), 2021.
[50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A. N. Gomez, E. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[51] X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, et al. Pali: A jointly-scaled multilingual language-image model. International Conference on Learning Representations (ICLR), 2023.
[52] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. Computer Vision and Pattern Recognition ( (CVPR), 2023.
[53] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.

---

Appendix
Imitation Learning and RT-1
imitation learning setup. The goal of language- conditioned imitation learning is to learn a policy (a o, where is robot action that should be applied given the current observation o and task instruction l. To learn a language- segguage-onddtiooned policy T, we build top of RT-1 [24], recent roboties transformer-basee model that achieves high lev- els of performance across a wide variety of manipulation tasks. RT-1 uses behavioral cloning [53], which optimizes æ by minimizing the negative log-likelihood of action a given the image ob- servations seen so far in the trajectory and the language instruction, using demonstration dataset containing N demonstrations:
MOO builds upon a language-conditonn
(
Σ
Σ
log ((a /e(0) 3)=))
t=
Vision-Language Models
In recent years, there has been a growing interest in developing models that can detect objects in images based on natural language queries. These models, known vision-language models (VLMs), are enabling detectors to identify wide range of objects based on natural language queries. Typically the queries are tokenized and embedded in sigh-dimensional space by pre-trained language encoder, and the image is processed by separate network to extract image features into the same embedding space as the text features. The language and image representations are then combined to make predictions of the bounding boxes and segmentation masks. Given natural language query, ạ. and an image observation on which to run detection, o, these models aim to produce set of embeddings for the image Si(o) with shape (height, width, feature dim) and an embedding of the language query Si(q) with shape feature dim such that logits fo(o) fr(44 gives logit score map and is maximized at regions in which correspond to the queries in q. Each image embedding location within f.(o) is also associated with a predicted bounding box or mask indicating the spatial extent of that object corresponding to f(()). In this work, we use the Owl-ViT detector [43], which we discuss further in Sec. 3.4.
Datasets
We collect teleoperated demonstration data that focuses increasing object diversity for the most efficient skill to collect data for, the picking task. This dataset of 13,239 episodes was collected with similar procedure to [24], with expert utilizing Oculus Virtual Reality controllers for teleoperation. Detailed dataset statistics are in Figure and Table 1.
12

---

Included
Included Evaluation
Object
Training
Seen Object
Unseen Object,
Unseen Category
Seen Category
red grapefruit can coke zero can pineapple spindrift lemon spindrift can love kombucha original pepper can fruit gummies instant oatmeal pack brie cheese cup coffee mixing stick
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
white sparkling diet pepper can lemon sparkling water can black pen orange plastic bottle blue pen coffee cup sleeve regular 7up can small salmon plate diet coke can lemonade plastic bottle original redbull can numi tea bag popcorn chip bag cereal scoop blackberry hint water green cookies bag watermelon hint water spoon coffee cup lid green pear coffee cup iced tea can ito en green tea pink lunch box chocolate caramel candy small beige plate large yellow spatula
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
large hot pink plate
yes
red bowl green bowl
yes
yes
orange spatula
yes
large blue plate
yes
large baby pink plate
yes
small purple plate
yes
small blue spatula
yes
small green plate
yes
table tennis paddle
yes
green brush
yes
rubiks cube
yes
gray suction toy toy ball with holes
yes
yes
large tennis ball gray microfiber cloth toy boat train
yes
yes
yes
13

---

teal and pink toy car dna chew toy slinky toy raspberry baby teether small purple spatula milano dark chocolate badminton shuttlecock chain link toy orange cup head massager square cheese boiled egg blue cup chew toy fish toy egg separator blue microfiber cloth yellow pear small orange rolling pin
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
wrist watch
yes
yes
pretzel chip bag disinfectant wipes
yes
yes
yes
pickle snack octopus toy catnip toy
yes
yes
yes
yes
yes
orange
yes
yes
7up can
yes
yes
apple coke can
yes
yes
yes
yes
swedish fish bag large green rolling pin place green can upright
yes
yes
yes
yes
yes
yes
black sunglasses blue chip bag
yes
yes
yes
yes
pepsi can
yes
yes
pink shoe
yes
yes
blue plastic bottle
yes
yes
green
yes
yes
orange can
yes
water bottle
yes
yes
redbull can
yes
yes
green jalapeno chip bag
yes
yes
rxbar chocolate
yes
rxbar blueberry brown chip bag
yes
yes
yes
yes
green rice chip bag
yes
yes
sponge chocolate peanut candy
yes
yes
yes
yes
banana
yes
yes
oreo cheese stick
yes
yes
yes
yes
yellow bowl large green plate white coat hanger
yes
yes
yes
yes
yes
yes
green microfiber cloth small blending bottle floral shoe dog rope toy red cup
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
14

---

fork disinfectant pump blue balloon bird ornament red plastic shovel whisk
yes
yes
yes
yes
yes
yes
yes
yes
yes
baby toy brown dinosaur toy pikmi pops confetti toy white marker holder white toilet scrub
yes
yes
yes
yes
yes
pink stapler green dolphin toy purple eggplant
yes
yes
yes
small green dinosaur toy green blocks navy toy gun gray pouch
yes
yes
yes
yes
small red motorcycle toy bike pedal
yes
yes
с clamp
yes
burgundy paint brush
yes
transparent hint water bottle shiny steel coffee grinder holder
yes
yes
transparent plastic cup
yes
shiny steel mug
yes
shiny steel scooper
yes
shiny pink steel bowl
yes
light pink sunglasses pink marker
yes
yes
cold brew can
yes
ginger lemon kombucha
yes
green cup green sprite can large orange plate pineapple hint water small blue plate small hot pink plate black small duck small purple bowl purple toy boat black chip bag teal sea salt chip bag blue sea salt chip bag salt seaweed snack
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
gray sponge
yes
red velvet snack bar red bell pepper
yes
yes
blue toy boat train green tennis ball
yes
yes
Table List of objects used training evaluation. There are types objects in evaluation: objects which were seen in training. 22 unseen objects of categories which were seen in training, 25 unseen objects of unseen categories.
15

---

Pick Task
Pick Task, Selected for "Seen Object" Evaluation
Non-Pick Tasks
RT- Data
Number Training Episodes
3000
2000
1000
Newly added diverse Pick Task data
"Unseen Category"
Evaluation
"Unseen Objectt
Seen Category' Evaluation
cod can tlue dannti botti geot can polnium chis
7vg youen dny how
can
Arg
BPG msxat adwys
Are unspre deu
Longgng any
eaedd aelee peuzee
aon ununs wood
docce Aunopeee
Degy
AeD priggesa protu
upo dhi, merdia
uad anya
uet cuete
Ace urnesoup unntug
onuoe seeit moue
sveyytill apdlnd
Oreal peips Alaua
evett abunso atlem
maad seed geess
endiapoo
adiu dop
angynee Aad
eedind
wed you adtvyy
2on4 ungesseeee
Dupunds uoust
coneeytuune sufts
riquuryy umuure aetutt
Ausepites
adtves
snead plmpas wrgs
Object
Figure 9: Distribution of training objects across the training dataset. We augmented RT-1 data (on the left) with a large number of diverse pick episodes (in the middle) in order to demonstrate strong generalization to unseen objects (on the right). Blue and yellow bars represent "pick" episodes and red bars represent other tasks like "move near" or "knock." Yellow bars portray the objects randomly selected for "Seen Object" evaluations. Objects for "Unseen Category" and "Unseen Object, Seen Category" evaluations are shown the right.
Experiments
We show a visualization of our 7-DoF manipulation robot in Figure 10.
Skills. Our experiments evaluate the percent of successfully completed manipulation commands which include five skills: "pick", "move near", "knock," "place upright," and "place into" across set of evaluation episodes. The definition of the tasks follows RT-1 [24]: For "pick", success is defined as (1) grasping the specified object and (2) lifting the object at least inches from the table top. For "move near", success is defined as (1) grasping the specified object and (2) placing it within inches of the specified target object. For "knock", success is defined as placing the specified object from an "upright" position onto its side. "Place upright" tasks are the inverse "knokk" and involve placing an object from its side into an upright position. Finally, "place into" tasks involve placing apple into a bowl.
one object into another, such
Robustness evaluation details. We evaluate the robustness of MOO on a variety of visually chal- lenging scenarios with drastically different furniture and backgrounds, as shown in Figure 6; the results are reported in Figure 5. The first set of these difficult evaluation scenes introduces six evalu- ations across five additional open-world objects that correspond to various household items that have not been seen any point during training. The second set of difficult scenes introduces 14 evalu- ations across two patterned tablecloths; these tablecloth textures are significantly more challenging than the plain gray counter-tops seen in the training demonstration dataset. Finally, the last set of difficult scenes include 14 evaluations across three new environments in natural kitchen and office spaces that were never present training. These new scenes simultancously change the counter-top materials, backgrounds, lighting conditions, and distractor items.
Input modality demonstration details. We explore the ability of MOO to incorporate object- centric mask representations that are generated via different processes than the one used during training. During training, an OWL-VIT generates mask visual representations from textual prompts, as described in Section 3.2. We study whether MOO can successfully accomplish manipulation tasks given (1) a mask generated from a text caption from a generative VLM, (2) a mask generated from an image query instead of text query, or (3) mask directly provided by a human via GUT. For each of these cases, we implement different procedures for generating the object mask representation, which are then fed to the frozen MOO policy.
16

---

Robot Observation
(512, 640) Monocular RGB
7 degree-oo-freedomm arm, 2 finger gripper
5
Randomized distractor objects
Evaluation command: "Pick orange rolling pin"
skill
object
Figure 10: Image of our robot hardware and evaluation setting.
Mask Sensitivity Study We originally noted the qualitative observation that MOO seemed to ex- hibit robustness to imperfect object-centric mask localization. We ran an additional experiment of evaluating MOO policy with masks that contain artificially added localization noise. We study four cases where we ablate the mask while keeping the starting scene the same: (1) the baseline when the mask is at the centroid of the object, (2) when the mask is still on the object of interest but not at the centroid, (3) when the mask is off of the object entirely but still within roughly 5em, and (4) when the mask is far and more than 5em from the object of interest. We add the artificial noise manually by starting with the centroid mask and then manually ablating the masks. We run total of 20 trials across 5 tasks involving one seen object (green rolling pin), unseen object in seen category (cold brew can), and three unseen object categories (egg plant, shiny sunglasses, transparent bottle). We find that performance degrades more noise is added: case (1) achieves 5/5 successes, case (2) achieves 4/5 successes, case (3) achieves 3/5 successes, and case (4) achieves 3/5 successes. Qualitatively,
observe that the policy sometimes initially reaches for the inaccurate mask location, and is sometimes able to recover and re-approach the correct target object. In one notable example, the policy faced an off-center mask on the lower part of the clear bottle (a visually challenging object, since no transparent objects were in the training set) and grasped the clear bottle in the upper section, which is the correct strategy for re-orienting the bottle upright. Additionally, a few examples showed that the policy was able to retry after failures caused by inaccurate masks; this suggest that the policy does not just memorize going to the location of the mask, but instead also pays attention to semantics. We provide the table of quantitative results in Table 2.
Training data ablation. We ablate the amount of data used to train MOO, and find that both data diversity and data scale are important, as shown in Table 3.
Prompts used
We use the following prompts to OWL-ViT detect our objects. All prompts were prefixed with the phrase "An image of a.
17

---

Mask Ablation Amount
Pick Skill
Upright Skill
Skill
Knock Skill
Green Rolling Pin Figeplant Shiny Glasses
Clear Bottle
Cold Brew Can
buseline centroid mask off-ceuter on-object mask off-object thun off object more than Sem mask
1
1
1
0
0
Table 2:
We evaluate MOO policy for 20 trials with masks that contain artificially added localization noise. Notably, we find that MOO is able to recover from misspecified masks which may not be centered on the target object, for both objects and novel objects altogether.
Dataset Filtering
Objects Episodes per Object Seen objects
Unseen objects
100%
100%
98
79
100%
92
18%
100%
88
19
100%
50%
46
38
100%:
10%
23
Table 3:
Performance of MOO in pereentage of success relative to the amount of data used for training. Both data scale and data diversity are important.
7up can - "white can soda"
banana - "banana"
black pen - "black pen"
blue chip bag - "blue bag of chips"
blue pen "blue pen"
brown chip bag - "brown bag chips"
cereal scoop - "cereal scoop"
chocolate peanut candy - "bag of candy snack" coffee cup + "coffec cup"
coke can + "red can of soda" coke zero can -> "can of soda"
disinfectant pump + "hottle" fork + "fork"
green "green aluminum can"
green cookies bag - "green snack food bag"
green jalapeno chip bag + "green bag of chips" green sprite can + "green soda can" knife + "knife"
orange can + "orange aluminum can" orange plastic bottle - "orange bottle" oreo -> "cookie snack food bag" pepsi can -> "blue soda can" popcorn chip bag "bag of chips" pretzel chip bag + "bag of chips" red grapefruit can + "red aluminum can" redbull can - "skinny silver can of soda"
rxbar blueherry "small blue rectangular snack food bar" spoon "spoon"
swedish fish bag - "bag candy snack food"
water bottle - "clear plastic waterbottle with white cap" white sparkling can - "aluminum can"
blue plastic bottle + "clear plastic waterbottle with white cap" dict pepper can "can of soda"
disinfectant wipes + "ycllow and blue pack" green rice chip bag -> "green bag of chips" orange "round orange fruit" paper bowl "round bowl"
rxbar chocolate + "small black rectangular snack food bar" sponge - "scrub sponge"
blackberry hint water - "clear plastic bottle with white cap" pineapple hint water "clear plastic bottle with white cap"
18

---

Seen
Unseen
(111M)
100.00%
75.00%
(10.2M)
50.00%
(2.37M)
25.00%
0.00%
10
50
100
Parameters Count (Millions)
Figure 11: Pick success vs. model size. We see continuous improvements on both seen and unseen objects increase the number of parameters of model architecture while keeping the data set size fixed. In comparison main model, we scaled down layer widths and depth by the same constant multiplier. We expect more performance gains larger model capacity, yet are currently unable to scale further due to real time inference constraints on our robot.
watermelon hint water -+ "clear plastic bottle with white cap" regular 7up can - "can of soda" lemonade plastic bottle "clear plastic bottle with white cap" diet coke + "silver of soda"
yellow pear + "yellow pear"
green pear - "green pear"
instant oatmeal pack + "flat brown pack of instant oatmeal"
coffee mixing stick - "small thin flat wooden popsicle stick" coffee cup lid "round disposable coffee cup lid"
coffee cup sleeve -> "brown disposable coffee cup sleeve" numi tea bag - "small flat packet of tea"
fruit gummies - "small blue bag of snacks" chocolate caramel candy + "small navy bag of candy" original redbull can + "can of energy drink with dark blue label" cold brew can - "blue and black can"
ginger lemon kombucha + "yellow and tan aluminum can with brown writing" large orange plate - "circular orange plate"
small blue plate "circular blue plate"
love kombucha - "white and orange can soda"
original pepper can - "dark red can of soda"
ito green tea "light green can of soda" ieed tea can "black can of soda" cheese stick "yellow cheese stick in wrapper"
bric cheese cup + "small white cheese cup with wrapper" pineapple spindrift can - "white and cyan can of soda" lemon spindrift can - "white and brown can of soda" lemon sparkling water can "yellow of soda" milano dark chocolate "white pack snacks" square cheese + "small orange rectangle packet" boiled - "small white in plastic wrapper" pickle snack - "small black and green snack bag"
19

---

mowe pard brew danjnew (grwen
pick pick watch
move bird omament near MTIS
pick Email orange roling pri
Figure 12: Example images of policy detecting and grasping objects not seen during training time. The object detections colored in correspondence the text above the image, and the images ordered left right across time.
red cup - "plastic red cup" blue cup "plastic blue cup"
orange cup "plastic orange cup"
green cup + "plastic green cup"
head massager - "metal head massager with many wires" chew toy - "blue and yellow toy with orange polka dots"
wrist watch - "wrist watch"
small orange rolling pin "small orange rolling pin with wooden handles"
large green rolling pin "large green rolling pin with wooden handles"
rubiks cube - "rubiks cube"
blue microfiber cloth - "blue cloth"
gray microfiber cloth + "gray cloth" green microfiber cloth
"green cloth"
small blending bottle - "small turqoise and brown bottle"
large tennis ball - "large tennis ball"
table tennis paddle - "table tennis paddle"
octopus toy - "purple toy octopus"
pink shoe - "pink shoe"
floral shoe + "red and blue shoe" whisk + "whisk"
orange spatula "orange spatula"
20

---

small blue spatula -> "small blue spatula"
large yellow spatula -
"large yellow spatula"
egg separator - "large pink cooking spoon" green brush + "green brush"
small purple spatula + "small purple spatula"
badminton shuttlecock- + "shuttlecock"
black sunglasses "black sunglasses"
toy ball with holes - "toy ball with holes" red plastic shovel - "red plastic shovel" bird ornament - "colorful ornament with blue and yellow confetti"
blue balloon > "blue balloon animal"
catnip toy - "small dark blue plastic cross toy"
raspherry baby teether + "red and green baby pacifier"
slinky toy + "gray metallic cylinder
dna chew toy - "big orange spring"
gray suction toy "gray suction toy"
teal and pink toy car -> "teal and pink toy car" two pound purple dumbbell - "purple dumbbell" one pound pink dumbbell + "pink dumbbell" three pound brown dumbbell + "brown "umbbell" dog rope toy "white pink and gray rope with knot" toy - "fish"
chain link toy + "skinny green rectangular toy" toy boat train - "plastic toy hoat" white coat hanger - "white coat hanger"
21


Reviews:
[{"review_id": "uIL5L2lxZb", "reviewer": "Reviewer_ZXLg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes to integrate an object detection VLM into robotics policies by creating an extra channel for object detections, thus allowing the policy to generalise to completely unseen objects at test time. This approach is validated through large-scale real world experiments and a set of ablations. Its generalization capabilities are tested not only for novel objects, but also for novel environment conditions.", "summary_of_recommendation": "This is a strong paper which explores a very promising direction: integrating web-scale VLMs into robotics policies. The paper achieves this in an elegant way by giving the VLM output as an extra channel of input to the policy, without overly restricting the policy. The experiments are thorough, with impressive results. The paper could perhaps be improved a little further by strengthening the arguments in favour of some of the specific design decisions, but overall the method is sound and this is a valuable contribution to the robot learning community.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please see the main review for questions for rebuttal.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths:\n1. Incorporating VLMs trained on captioned Internet data into robotics methods makes sense and is a valuable research direction, since more and more of this Internet data is being produced by millions of users, whereas robotics data is not growing at the same pace yet. The paper accomplishes this integration in an elegant way, providing this VLM information as an extra channel to the policy model but not overly restricting the policy.\n2. Experiments with other user input modalities such as pointing are useful, since this is quite a practical way for humans to communicate with each other and with robots.\n3. The design choices in the architecture are well-made to make the method immediately practical but also quite general in the long term. One small example: even though current instructions are focused on 1-2 objects, it is possible to add more objects to the mask by simply changing the pixel values in that channel, without needing to change the number of channels in the model. Another example: running the VLM only on the first frame, for computational scalability.\n4. The experiments are thorough and constitute a strong investigation into different levels of generalisation, with especially impressive performance on new environments, which is promising for bringing robots into real situations.\n5. The ablations are valuable and provide interesting insights, for example comparing the number of training objects vs number of training episodes and in which cases one matters more than the other.\n\n### Weaknesses:\n\n1. In the Introduction, the proposed method is framed in contrast to pipeline approaches, which are described as being more “brittle”. The reason given is that the latter control stages depend on accurate object detections in the earlier stages. This is intuitively true, but perhaps the paper could also benefit from experimental evidence for this claim, i.e. a direct experimental comparison against a pipeline approach to show that this new proposed method is less brittle. On Line 182, it is claimed that the MOO policy can be “robust to errors made by the VLM”, which is very interesting: some qualitative examples of this would be helpful to see and discuss.\n2. The design decision of removing object names from the instruction after the objects are detected and before passing the instruction to the rest of the policy does have its benefits, because it forces the policy to learn to interpret the object detections in the mask channel. However, it also has its downsides. For example, it is not so obvious how the policy can learn to correct for mistakes of the VLM if the names of the objects it should interact with are removed from the instruction. Therefore, either an ablation on this design decision or a stronger justification for removing these names may be helpful here.\n3. The idea of keeping only one-pixel detections from VLMs as inputs to the policy may have a slight tension with the argument made in the introduction. VLMs trained on static data currently have much more training data than robotics models, and so will likely do a better job of not only detecting the object centre but also detecting a good bounding box, and maybe even a segmentation. By keeping only a single pixel from the VLM output, the rest of this difficult work (i.e. finding the shape of the object) is shifted to the robotics policy, which has much less training data to learn from. Would it be possible to elaborate further on why discarding the bounding boxes from the VLM is beneficial here? Or if another model like Segment Anything were to be used as an additional channel, would the authors expect this to help or hurt performance? This is maybe more consistent with the argument made in the Introduction, which is that we want to shift the burden of open-set object detection away from the robotics policy and onto the web-scale VLMs as much as possible.\n\n### Minor issues:\n\n1. Minor referencing issue: should Table 1, when referenced on Line 302, be referenced as Appendix Table 1?\n\n### Questions for rebuttal:\n\n1. In Line 317, it is reported that the model cannot be made too large due to latency. For future work, would it make sense to have a larger high-level semantic reasoning model running at a lower frequency, and then also a smaller but higher-frequency low-level control model?\n2. In the appendix, it is mentioned that prompts to OWL-ViT are prefixed with “An image of a”. This sounds reasonable, given that these are the captions that CLIP expects, and OWL-ViT uses CLIP. In the original CLIP paper, when performing zero-shot ImageNet classification, the authors report significant performance gains from using not just one but a set of many templates such as “A photo of”, “A blurry close-up of”, etc, and then aggregating scores across these templates for improved robustness. Would the authors consider trying out this technique for this robotics data? It may help with robustness, and lessen the need for the prompt re-mapping listed in the appendix.\n3. How would this work when there are multiple apples in the scene, and the instruction is something like “move the apples next to each other”? In theory, OWL-ViT should give multiple good detections, and the mask channel should contain the 1.0 pixel in multiple places. Would the policy work in these cases, when the output of OWL-ViT has multiple instances of the same object category?\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "fxLv9ufw3e", "reviewer": "Reviewer_VQYg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a simple modification to the RT-1 architecture, that extends the input to also take a mask indicating a pixel(s) that roughly lies at the centre of a target object(s) and change the input language instruction from a full sentence specifying what task to complete (e.g. \"pick banana\") to a verb associated with the target skill (e.g. \"pick\"). The authors also extend the training dataset of RT-1 to contain additional demonstrations of the \"pick\" skill for 90 novel objects and train the modified RT-1 architecture on the original RT-1 data plus their newly collected data. The authors then study the ability of their proposed model to generalise five different skills to unseen object categories, showing superior performance compared to two considered baselines, and investigate the robustness of their policy to new environments and challenging textures.", "summary_of_recommendation": "Although the proposed modifications are simple and appear to work quite well for generalising to novel objects, the experiments section is unclear, making it difficult to access on what objects and how the proposed method was evaluated. And most importantly, it is difficult to assess the statistical significance of the presented results. Moreover, despite the authors acknowledging that \"the robot may struggle to grasp novel objects with drastically different shapes or sizes than those seen in the training demonstration data, \" this limitation could be at least partially addressed by incorporating depth data into the framework.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- In Section 3, you highlight that \"The key design choice of MOO lies in how to represent object information encoded in VLMs and how to feed that information to the instruction-conditioned policy.\" Have you maybe tried representing this information as a segmentation mask, for example, by combining an open vocabulary object detection model with a segmentation model, or as a bounding box? And do you have any insights on how this affects the performance? Given that this representation is a key design choice, I wonder if the choice of a one-hot pixel representation was derived from experiments or if it was the only representation attempted.\n- In Section 4.1, you wrote, \"We hold out 47 objects not present in training and report performance on these as \"unseen\"\". However, Figure 9 shows only 13 objects \"Selected for \"unseen\" evaluation\". Moreover, in the evaluation details, it is written that the experimental process is repeated 21 times to report the performance. Can you clarify how many unseen objects the methods were tested on and which are these objects? \n- The evaluation details could be clearer as it is unclear how many evaluations were conducted for each of the experiments, whether every unseen object was considered from the ones shown in Figure 3, or whether they were sampled from all the objects independently for each method. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed extension to the RT-1 architecture appears to be simple yet effective. Moreover, specifying the target skill with just the verb while indicating the target object on which the skill should be executed is quite appealing and intuitive and is the right step in the direction of developing/learning general skills that can operate on unseen object categories. \n\nMy main concern is that the proposed architecture does not depend on depth information. Although the authors see this as an advantage compared to prior work, depth is fundamental to manipulating previously unseen objects without prior knowledge. To see why this is the case, consider attempting to execute a task \"pick cube\" where the cube is a previously unseen object. Furthermore, consider a setting where the gripper is directly above the cube and all the robot has to do is move the gripper downwards and close it to pick up the cube. Given an RGB image of the gripper above the cube, without prior knowledge, there is no way to infer whether the gripper is vertically aligned with the cube, and whether the policy should move the gripper forwards, backwards or downwards. This is because either:\n\n- The gripper is already aligned.\n- The cube is quite small, and the gripper has to be moved closer to the robot to align vertically with the cube.\n- The cube is quite large, and the gripper has to move further away from the robot to align vertically with the cube.\n\nOf course, some of this ambiguity is partially solved with the camera's perspective. However, depth is fundamental to manipulating previously unseen objects in general; otherwise, this becomes an ill-defined problem. \n\nNow, in the experiments, it is shown that the proposed policy can successfully pick many previously unseen objects. However, I believe this is because the objects considered are similar to those used during training. Although the authors explicitly wrote, \"...we adopt a more strict definition of unseen objects, where our unseen object categories were not seen in the robot's training demonstration data at any point for any task\", closely examining Figure 3 reveals that many of the object categories present in the novel objects can also be seen in the training objects. For instance, there is a green rolling pin in the novel objects and an orange rolling pin in the training objects. Cans and bottles can be seen in the novel and training objects. The same holds for bath ducks, bottles and food packaged in foil bags (e.g. bags of crisps). Hence, it is likely that the policy can only manipulate the considered \"novel\" objects without depth information because it has seen training data for similar objects with similar dimensions. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Syn6PfKxM5O", "reviewer": "Reviewer_6xeG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present MOO - a method that harnesses a pre-trained vision-language model that informs a transformer-based robot policy and allows it to better generalise to manipulating novel objects never seen during training. They present extensive evaluation of the method including variations of it which go beyond language-based inputs.", "summary_of_recommendation": "I recommend weak accept, happy to change to accept post-rebuttal, depending on discussion and answers.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* lines 227-228 - we train on 106 objects and we pick 49 out of them to be the _seen_ ones, is that the case? If yes, why 49, how were they chosen? Also, in Figure 4, left, 49 are the seen objects only for Pick, correct, for the other skills it is still 16? Or it was 16 only during training, as no demonstrations with novel objects were collected for skills apart from pick.\n* why do we need a template for the language? Doesn't the VLM model support more complex sentences?\n* the data collection for the policy training is not discussed in the paper anywhere\n* Figure 5 mentions Open-world objects. Can you elaborate what are those and how are they different from the unseen ones? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-structured and presented. Main points are communicated well, with supporting visualisations which aid the reader greatly.\n\nStrengths:\n* The presented method is easy-to-understand and, as the authors mention quite easy to incorporate with any downstream policy architecture or upstream vision-language model (the 1-pixel segmentation masks concatenated to the input images channel-wise).\n* extensive evaluation + the method is shown to work on a physical robot.\n\nWeaknesses:\n* the actual technical contribution is rather small and incremental - both the pre-trained VLM model and the policy architecture are taken off-the-shelf. What is changed is the input to the architecture with the 1x1 pixel masks (which have their drawbacks as discussed in the paper)\n* the method depends on bboxes for the coordinates - what if the center of a bbox does not \"fall on\" the object - think of a crecend-shaped object where the center if its bbox is not on it. This is partly addressed in the limitations but would have been good to see how robust is the method to \"moving\" the 1x1 mask around - e.g. by simply adding noise to the actual coordinates. How much would that hurt performance? Is it sufficient for the 1x1 mask to be close to the object, assuming no clutter, or it has to be *on* it?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "HueDWE0xPM", "reviewer": "Reviewer_VUMx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed an approach to robot mobile manipulation that leverages rich semantic knowledge captured by large-scale and visual language models.\nThe proposed method, called Manipulation of Open-World Objects (MOO), is an approach that obtains robot policies from human linguistic commands.\nIn addition, it showed higher performance for novel objects and novel instructions compared to existing state-of-the-art models such as RT-1.\n", "summary_of_recommendation": "The recommendation is based on the excitement surrounding open-world objects and open vocabulary in the era of LLM, along with the high quality of the paper and research. However, it is suggested to make some improvements in additional writing, argument organization, and formatting.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "There are several questions about the architecture of the proposed method.\nOne is why such a structure was necessary.\nThe proposed method seems to be a combination of several architectures, including Owl-ViT, FiLM, and Transformer.\nHowever, the description of why this coupling method or combination was chosen is unclear.\nAre these components substitutable in other similar models? Or is there a fundamental structure or function hidden somewhere?\nThere is no detailed description of the above, and it remains unclear.\n\n\nWhat is the definition of the term Open-World Object Manipulation?\nI couldn't find a description of the definition. It is unclear in what sense it is Open-World.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This research is a valuable combination of state-of-the-art VLM models and robotics control.\nIt is exciting to deal with open-world objects and open vocabulary in the age of LLM.\nHowever, the fact that LLM can be used to support open vocabulary has probably already been demonstrated with RT-1 and others.\nTherefore, the use of LLM to deal with open vocabulary is not a big surprise.\n\nThe quality of the paper and research is high.\nHowever, the content is compressed and lacks technical details.\n\nThis paper appears to be an exhaustive evaluation with a wealth of experiments and evaluations.\nThe targeted tasks, data sets, instructions, and situations do not differ significantly from RT-1. It can be said that this is an incremental study.\n\nThe main text of this paper is based on the Appendix for details.\nYou will not know some of the details unless you read the Appendix.\nI recommend that if they are mentioned in the main text, they be included in the main text, not the Appendix.\n\nThe paper seems to lack attention to detail.\nFor example, the citation format is disjointed and not uniform. Some information does not seem to be correctly described.\nAuthors are expected to review the descriptions for all citations.\nIn addition, there are too many ArXiv preprints for citations. (Almost half!) Preprints are not peer-reviewed, which is also a concern for the credibility of the paper. Therefore, preprint citations should be avoided whenever possible.\n\nIn the Introduction, I have the impression that the description of pre-trained VLMs trained on vision and language data (not robot observation data) and the description of learning and manipulation of robot policies are mixed up.\nEach has been studied separately.\nIt is important that these separate things are integrated in this study.\nHowever, it may be a good idea first to introduce a discussion of the earlier individual research areas in isolation.\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of open-world object manipulation using pre-trained vision language models. The architecture leveraged Owl-VIT for providing grounded object manipulation targets from language and a scene image. The extracted object masks are then provided to the RT-1 policy architecture. Experiments are demonstrated on an indoor mobile manipulation system with novel unseen objects. 

The authors are requested to take into consideration the detailed feedback provided by reviewers. Specifically, (i) improving the technical details of the system (ii) commenting on to what extend the results are tied to the specific VLM and policy architecture used and (iii) robustness to perceptual errors/noise.
Decision: Accept (Poster)