Towards Open-World Grasping with Large Vision-Language Models
Authors: Georgios Tziafas
Hamidreza Kasaei
Venue: CoRL 2024 (2024)
Keywords: Foundation Models for Robotics
Open-World Grasping
Open-Ended23 Visual Grounding
Robot Planning
TL;DR: A novel system formulation for open-world grasping, combining large LLMs as GPT-4v with segmentation and grasp synthesis models for open-ended grounding, planning and grasp ranking.

Abstract: The ability to grasp objects in-the-wild from open-ended language instructions constitutes a fundamental challenge in robotics.
An open-world grasping system should be able to combine high-level contextual with low-level physical-geometric reasoning in order to be applicable in arbitrary scenarios.
Recent works exploit the web-scale knowledge inherent in large language models (LLMs) to plan and reason in robotic context, but rely on external vision and action models to ground such knowledge into the environment and parameterize actuation.
This setup suffers from two major bottlenecks: a) the LLM's reasoning capacity is constrained by the quality of visual grounding, and b) LLMs do not contain low-level spatial understanding of the world, which is essential for grasping in contact-rich scenarios.
In this work we demonstrate that modern vision-language models (VLMs) are capable of tackling such limitations, as they are implicitly grounded and can jointly reason about semantics and geometry. 
We propose \texttt{OWG}, an open-world grasping pipeline that combines VLMs with segmentation and grasp synthesis models to unlock grounded world understanding in three stages: open-ended referring segmentation, grounded grasp planning and grasp ranking via contact reasoning, all of which can be applied zero-shot via suitable visual prompting mechanisms.
We conduct extensive evaluation in cluttered indoor scene datasets to showcase \texttt{OWG}'s robustness in grounding from open-ended language, as well as open-world robotic grasping experiments in both simulation and hardware that demonstrate superior performance compared to previous supervised and zero-shot LLM-based methods.

Towards Open-World Grasping with Large Vision-Language Models
Georgios Tziafas Department of Artificial Intelligence University of Groningen the Netherlands g g.t.ta att.ziaafas@rug n1
Hamidreza Kasaei Department of Artificial Intelligence University of Groningen the Netherlands hamidreza kasaei n1
Abstract: The ability to grasp objects in-the-wild from open-ended language instructions constitutes fundamental challenge robotics. An open-wone grasping system should be able combine high-level contextual with low-level physical-geometric reasoning in order to be applicable in arbitrary scenarios. Re- cent works exploit the web-scale knowledge inherent in large language models (LLMs) to plan and reason in robotic context, but rely on external vision and action models to ground such knowledge into the environment and parameter- ize actuation. This setup suffers from two major bottlenecks: a) the LLM's rea- soning capacity constrained by the quality of visual grounding, and b) LLMs do contain low-level spatial understanding of the world, which essential for grasping in contact-rich scenarios. In this work we demonstrate that mod- ern vision-language models (VLMs) are capable of tackling such limitations, they are implicitly grounded and can jointly reason about semantics and geome- try. We propose OWG, an open-world grasping pipeline that combines VLMs with segmentation and grasp synthesis models to unlock grounded world understand- ing in three stages: open-ended referring segmentation, grounded grasp planning and grasp ranking via contact reasoning, all of which can be applied zero-shot via suitable visual prompting mechanisms. We conduct extensive evaluation in cluttered indoor scene datasets to showcase OWG's robustness in grounding from open-en eeeeeended language, as well as open- open-world robotic grasping experiments in both simulation and hardware that demonstrate superior performance compared to pre- vious supervised and zero-shot LLM-based methods. Project material available at https://tzziffas
wwt.tgg.i.t......iir..o http:://rzaafasggiibbb//WCCC
Keywords: Foundation Models for Robotics, Open- Visual Grounding, Robot Planning
Open-World World Grasping, Open Ended
Introduction
somcthing with?
Following grasping instructions from free-form natural language in open-ended environments is multi-faceted problem, posing several challenges to robot agents. Consider the example of Fig. I: The robot has to decipher the semantics of the user instruction (i.e., "what would child want play with?"), recognize the appearing objects and ground the target (i.e., the white toy), reason about the feasibility of the grasp to generate ap- propriate plan (i.e., first remove the blocking juice box), and finally select a suitable grasp based on
Lage Multimodsl Model (GPT-4.)
Segronsstivn
Grrap
trooncnng
Crayg Mossin
Gruup Lanking ni Hasuengg
Figure 1: Challenges of open-workd grasping tack- led with VLMs. The overall pipeline combines VLMs with segmentation and grasp synthesis mod- els to ground open-ended language instructions plan and reason about how grasp the desired object.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

the object geometry and potential collisions. It becomes clear that to deal with the full scope of open-world grasping, agents should integrate high-level semantic with low-level physical-geometric reasoning, while doing so in a generalizable fashion.
In recent years, Large Language Models (LLMs) [1, 2, 3, 4, 5], have emerged as new paradigm in robotics and embodied AI, due their emergent general knowledge, commonsense reasoning and semantic understanding of the world [6, 7, 8, 9, 10]. This has led to a multitude of LLM-based approaches for zero-shot robotic task planning [11, 12, 13. 14, 15], navigation [16. 17, 18, 19] and manipulation [20, 21, 22, 23, 24, 25], where the L.M decomposes a high-level language instruction into a sequence of steps, therefore tackling complex, long-horizon tasks by composing primitive skills. However, a notorious limitation of LLMs is their lack of world grounding - they cannot di- rectly reason about the agent and environment physical state [26], and lack deep knowledge when it comes to low-level, physical properties, such as object shapes, precise 3D geometry, contact physics and embodiment constraints [27]. Even when equipped with external visual modules for perceiving the world, the amount of information accessed by the LLM is bottlenecked by the visual model's interface (e.g. open-vocabulary detectors [28, 29, 30] cannot reason about object relations such as contacts). Recently, Large Vision-Language Models (LVLMs) integrate visual understanding and language generation into a unified stream, allowing direct incorporation of perceptual information into the semantic knowledge acquired from language [31, 32, 33, 34]. Preliminary explorations with LVLMs [35] have illustrated two intriguing phenomena, namely: by combining LVLMs with segmentation models and constructing suitable visual prompts. LVLMs can unleash extraor- dinary open-ended visual grounding capabilities [26], and b) effective prompting strategies like chain-of-thought [36] and in-context examples [I] seem to also emerge in LVLMs. Motivated by these results, we perform an in-depth study of the potential contributions of LVLMs in open-ended robotic grasping. this paper, we propose Open World Grusper (OWG): an integrated approach that is applicable zero-shot for grasping in open-ended environments, object catalogs and language instructions. OWG combines LVLMs with segmentation [37] and grasp synthesis models [38], which supplement the LVLM's semantic knowledge with low-level dense spatial inference. OWG decomposes the task in three stages: open-ended referring segmentation, where the target object is grounded from open-ended language, (ii) grounded grasp planning, where the agent reasons about the feasibility of grasping the target and proposes a next action, and (iii) grasp ranking, where the LVLM ranks grasp proposals generated from the grasp synthesizer based on potential contacts.
In summary, our contributions threefold: a) propose novel algorithm for grasping from open-ended language using LVLMs, b) we conduct extensive comparisons and ablation studies in real cluttered indoor scenes data [39, 40], where we show that our prompting strategies enable LVLMs to ground arbitrary natural language queries, such as open-vocabulary object descriptions, referring expressions and user- afforddfosss, while outperforming previous zero-shot vision-language models by a significant margin, and c) we integrate OWG with a robot framework and conduct experiments both in simulation and in the real world, where we illustrate that LVLMs can advance the performance of zero-shot approaches in the open-world setup
2 Related Works
Visual Prompting for Visiun-Language Models Several works investigate how to bypass fine- tuning VLMs, instead relying on overlaying visual/semantic information to the input frame, a prac- tise commonly referred to visual prompting. Colorful prompting tuning (CPT) is the first work that paints image regions with different colors and uses masked language models to "fill the blanks" [41]. Other methods try to use CLIP [42] by measuring the similarity between visual prompt and a set of text concepts. RedCircle [43] draws a red circle on an image, forcing CLIP to focus on a specific region. FGVP [44] further enhances the prompt by specifically segmenting and highlight- ing target objects. Recent works explore visual prompting strategies for L.LLMs such as GPT-4v, by drawing arrows and pointers [35] or highlighting object regions and overlaying numeric IDs [26]. In the same vein, in this work we prompt GPT-4v reason about visual context while being grounded to specific spatial elements of the image, such objects, regions and grasps.

---

LLMs/LVLMs in Robotics Recent efforts use LLMs as an initialization for vision-language-aatinn models [45, 46), fine-tuned in robot demonstration data with auxiliary VQA tasks [46, 45, 47J. Such end-to-end approaches require prohibitive resources to reproduce, while still struggling to general- ize out-of-distribution, due to the lack large-scale demonstration datasets. Alternatively, modular approaches invest on the current capabilitiess of LLMs to decompose language instructions into a sequence of high-level robot skills [48, 11, 12, 22, 14], low-level Python programs composing external vision and action models APIs [13, 23, 21, 22, 25, 49]. Such approaches mostly focus on the task planning problem, showcasing that the world knowledge built in L.LMs enables zero- shot task decomposition, but require external modules [28, 29, 30, 42] to ground plan steps to the environment and reason about the scene. Recent works study the potential of LVLMs for inherently grounded task planning [27, 50, 51]. In [50], the authors use GPT-4v to map videos human per- forming tasks into symbolic plans, but do not consider for downstream applications. VILA [27] feeds observation images with text prompts to LVLM to plan without relying on external detec- tors. However, produced plans expressed entirely in language and assume already obtained skill library to execute the plans. MOKA [51] proposes a keypoint-basee visual prompting scheme to parameterize low-level motions, but still relies on external vision models to perform grounding, and does not consider referring expressions and clutter. In our work, we use visual marker prompt- ing to leverage LVLMs for the full stack of the open- open-world grasping pipeline, including grounding referring expressions, task planning and low-level motion parameterization via grasp ranking.
comantics-informer Grasping Most research on grasping assumes golden grounding, i.e., the target object is already segmented from the input scene. Instead, they focus on proposing 4- DoF grasps from RGB-D views [52, 53, 54, 38, 55, 56, 57], or 6-DoF poses from 3D data [58, 59, 60, 61, 62, 63, 64). Recently, several works study manguage-guidee grasping in an end- to-end fashion, where a language model encodes the user instruction to provide conditioning for grasping [65, 66, 39J. However, related methods typically train language-conditioned graspers that struggle generalize outside the training distribution. Another similar line of works is that of task-oriented grasping [67, 68], where recent Musdddddd methods [69] exploit the vast knowledge of LLMs to provide additional semantic context for selecting task-oriented grasps, but do not con- sider the grounding problem, clutter or referring expressions. Further, none of the above approaches consider the planning aspect, typically providing open-loop graspers that do not incorporate envi- romment feedback. In this work, we leverage LVLMs to orchestrate a pipeline for language-guided grasping clutter, exploiting it's multimodal nature to jointly ground, reason and plan.
Method
3.1 Prerequisites and Problem Statement
Large Vision-Languaae Models VLMs receive a set of RGB images of size H X W: II.M. I E R XWxx and a sequence of text tokens T, and generate a text sequence y of length L: y W1.L = {wi,..., Wr.} from a fixed token vocabulary w, W, such that: y = F(ILM, T). pair input X' (I:M: : T) is referred as the prompt, with the text component T typically being a user instruction or question that primes the VLM for specific task.
The images-text
Grasp Representations We represent grasp via an end-effector gripper pose G, with G E Bd for 4-DoF and G E 5 for 6-DoF grasping. Such representation contains a 3D position and either a yaw rotation or a full SO(3) orientation for 4-DoF and 6-DoF respectively. 4-DoF grasps assume that the approach vector is calibrated with the camera extrinsics, and hence can be directly drawn as rectan- gles in the 2D image plane (see bottom of Fig. 2), which happens to be favorable representation for VLMS, as grasp candidates can be interpreted as part of the input image prompt. A motion primitive is invoked to move the arm to the desired gripper pose G, e.g. via mnverse-kinematies solvers.
'More sophisticated motion planning algorithms, e.g. with integrated obstacle avoidance, can be utilized orthogonal to approach.
3

---

for a soup, potestillly
Feet like tomalo toright
Target object
Curment obeetvadion
CoT: marker the Plan
JERRREE U pick
Large
Multimodal Model (GPT-4v)
Region of Inteeest
Grasp Markers
S(-)
G
LB
4%
World execution
Cel:: marker avoid Ranked graups [5]4 1.
Segmentation
Updar absercation
Given a user instruction and an observation, OWG first invokes a segmentation model to recover pixel-level masks, and overlays them with numeric IDs visual markers in new image. Then the VLM subsequently activates three stages: (i) grounding the target object from the language expression in the marked image. (ii) planning on whether should grasp the target or remove surrounding object, and (iii) invoking a grasp synthesis model to generate grasps and ranking them according to the object's shape and neighbouring information. The best grasp pose (highlighted here in pink not part of the prompt) is executed and the observation is updated for a new run, until the target object is grasped. Best viewed in color and zoom.
Figure 2: Overview of OWG:
Problem Statement Given an RGB-D observation I, E HxWx3 D: E R.H and an open- ended language query T, which conveys an instruction to grasp a target object, the goal of OWG is to provide policy T(a, I, Dr. T). Assuming E {1,..., .N} the N objects that appear in the scene and n² the target object, then at each time step t, the policy outputs a pose for grasping an object: a₂ (n),, G((n) = G(n,I,, Dz), t= 1,...,, where the last step T always maps to grasping the target object: Ør = Gr(n*). We refer to the function G as the grasp generation function, which corresponds to a pretrained grasp synthesis network from RGB-D views [38] We note that our policy outputs directly the actual gripper pose G(n), and the object-centric abstraction is used implicitly (details in next sections).
We wish to highlight that in most grasp synthesis pipelines [38, 55, 53, 57, 56), it's always 7 = 1 and a1 = G1 (n²), which corresponds to an open-loop policy attempting to grasp the object of interest once. Our formulation for > allows the VLM to close the loop by re-running after each step, which enables visual feedback for planning and recovery from failures external disturbances.
3.2 Pipeline Overview
OWG combines VLMs with pretrained 2D instance segmentation and grasp synthesis models. Seg- mentation methods like SAM [37] and its variants [70, 71] have demonstrated impressive zero-shot performane. Similarly, view-based grasp synthesis networks [56, 55, 38, 53, 57] have also shown to be transferable to unseen content, as they are trained without assumptions of objectness or semantics in their training objectives. The zero-shot capabilities of these models for low-level dense spatial tasks is complementary to the high-level semantic reasoning capabilities of VLMs, while both use images as the underlying representation, hence offering a very attractive coupling for tackling the open-world grasping problem. The overall pipeline can decomposed in three subsequent stages: (i) open-ended referring segmentation, (ii) grounded grasp planning, and (iii) grasp generation and ranking. A schematic of OWG is shown in Fig. 2 and described formally in Algorithm 1. Prompt implementation details can be found in Appendix A.
Open-ended referring segmentation In this stage, the target object of interest must be segmented from the input RGB image I, given the instruction T. To enable this, we first run our segmentation model S I2 # - {0, 1) XxxW and then draw the N generated masks Mi.N S(I)) with
additional visual markers in new frame I This step aims to exploit the VLM's s OCR capabilities
'Other plont [59] or voxel-based [62] methods for 3D grasp generation can be utilized orthogonal to approach, which uses single RGB-D view.

---

Open- Referring Segmentation
Grounded Graep Planning
Grasp Ranking Rensoingg
User I velenng
medking tife
Prouee: Aomosi
Prospti- Iros ki kined grop ills catud Fire but mond
cortiies rail anguu lew d
bale
Iopeed yeb sumbord le of otgans man palac. rotmin thee
Giriin otpa macor biu with ra pxport
and to soui bodes neg which wor mas Rics sete dats of caaa on pouelal allish the tuke suar bitan of inago
GrrS-4 195,
elgens are dioty mosting
and (8) Dhed gaog pome
od conter ar
acosa is. Thaw oten mas rinrd the cus te wop
thety 141 1s), of and 11) The soe hesl wond hone the hos faveale grng pons de sn andii jec
faliiarrii
bectes tremeeer Hyrtaal
bike manddd wetegge.
Mer
My Ealleon (4 S.n i.A 2.4)
Figure 3: Example GPT-4v responses (from left to right): a) Open-ended referring segmentation, i.e., ground- ing, b) Grounded grasp planning, and e) Grasp ranking contact reasoning. We omit parts of the prompt and response for brievity. Full prompts Appendix and more example responses Appendix E.
and link each segment in the frame with a unique ID that the VLM can use to refer to it. After augmenting the image with visual markers, we pass the prompt < I, I("h T > to the VLM. We refer to this VLM generation as Firroudd such that: n* = Fground (I,, I1" T) where n² the target object and Mn its segmentation mask. We note that T can contain free-form natural language referring to target object, such as open object descriptions, object relations, affordances etc.
Grounded grasp planning This stage attempts to leverage VLM's vi- sual reasoning capabilities in order to produce a plan that maximizes the chances that the target object n* graspable. If the target object is blocked by neighboring objects, the agent should remove them first by picking them an placing them in free tabletop space. Similar to [27], we construct a text prompt that describes these two options (i.c., remove neigh- bor or pick target) as primitive actions for the VLM to compose plans from. We provide the marked image II" gether with the target object n² (from the previous grounding stage) to de- termine a plan: Pı.T = Fillan (I,m, n²), grasp the object with marker ID n E [1....,N] N]. As motivated carlier, in order to close the loop, we take the target of the first step of the plan ñ = pı and move to the grasping stage of our pipeline.
Algorithm 1: Open-World World Grasper (OWG)
Require: Initial observation (I1, D1), language instruc- tion T. segmentor S(.), grasp generator G(-), VLMs Faround __plan Frrnk
Ensure: n² * n
t-1
while nı" # do
Generate segmentation masks Mi.N with S(,))
Draw visual markers from MI.N in a new frame I," Object of interest Next object to grasp b Grasp generation
n² + Feround (I,, I," T)
D-Fplan (I." ".m.n" .nn)) _n
G1:K +- G(I,, De, Ma
Crop Rol and draw grasps Cp from L, Ma. G1:K
G'I:K &- Frank (car)
> Grasp ranking
Execute grasp g'1
6 --1+1
Update observation Ie, D.
end while
Pr E (I,...,N), Each pr corresponds to the decision to
Grasp generation and ranking After determining the current object to grasp n, invoke grasp synthesis model G to generate grasp proposals. To that end, we element-wise multiply the mask Ma with the RGB-D observation, thus isolating only object n² in the input frames: Z, I, @ Ma, D, = Dj @ Ma. The grasp synthesis network outputs pixel-level quality, angle and width masks which can be directly transformed to 4-DoF grasps 91.K = G D.) [56, 55, 38], where K the total number of grasp proposals. Then, we crop a small region of interest Ca around the bounding box of the segment in the frame I, from its mask M₁. We draw the grasp proposals G1:K as 2D grasp rectangles within the cropped image Ca and annotate each with numeric ID marker, similar to the grounding prompt. We refer to the marked cropped frame c/a Then, we prompt the VLM to rank the drawn grasp proposals: Ji, = Fronk (ck) where the prompt instructs the VLM to rank based on each grasp's potential contacts with neighboring objects. Finally, the grasp ranked best by the VLM Gi is selected and sent to our motion primitive for robot execution.
4 Experiments
In this section, compare the open-ended grounding capabilities of OWG previous zero-shot methods in indoor cluttered scenes (Sec. 4.1). Then, we demonstrate its potential for open-world
5

---

grasping both in simulation and in hardware (Sec. 4.2). Finally, we investigate the effect of several components of our methodology via ablation studies (Sec. 4.3).
4.1 Open-Ended Grounding in Cluttered Scenes
In order to evaluate the open-ended po- tential of OWG for
Method
Found. Model
Name
Attribute
Spatial Relation
Vieal Relation
Semantic Relation
Affordance
Multi-
hop
Arg.
[72]
RedCCnnd (49)
44.8-164
13.3
43,01933
grounding. we cre- ate a small subset of OCID-VLG test split [39], which we man- ually annotate for broad range of grasping instructions. As we strive for zero-shot usage in open scenes, we mostly experiment with previous visual prompting techniques for large-scale VLMs, such as CLIP [43, 44,44,,75 72]. as well as the recent Set-of-Mark prompting methodology for GPT-4v [26], which con- stitutes the basis of our method. We also include comparisons with open-source grruudd- LVLM QWEN-VL-2 [31]. Please see Appendix C for details on the test dataset, baseline imple- mentations and more comparative ablations and qualitative results.
PGVP [44]
QWEN--VL-2(1)
QWEN
64.3
60.9
52.4
44.0
47.1
11.9
40.1
46.1-159
ScM (26)
GPT-4: [75]
54.8
42.3
54.5
57.9
53.9
62.5
45.5
53.159.4
OWG (Ous)
GPT-4:
[23]
85.7
88.8
75.8
73.7
76.9
938
79,2
80.859.4
Table I: Zero-shot referring segmentation mloU(%% results per language instruction type for cluttered indoor scenes from OCID [40].
We observe that both CLIP-based visual prompting techniques and open-source LVLMs are decent in object-based but fail to relate objects from the visual prompts. Even GPT-4v-based SoM prompt- ing method is not directly capable of handling cluttered tabletop scenes from depth cameras, as is evident by the 53.1% averaged mIoU across all query types. Overall, our OWG- WWG-grounder achieves an averaged mIoU score of 80.8%, which corresponds to a 27.7% delta from the second best ap- proach. Importantly, OWG excels at semantic and affordance- queries, something which is essential in human- robot interaction applications but is missing from modern vision-languae mod- els. We identify two basic failure modes: a) the LVLM confused the target description with another object, e.g. due to same appearance or semantics, and b) the LVLM reasons correctly about the object and where it is roughly located, but chooses a wrong numeric ID to refer to it.
4.2 Open-World Grasping Robot Experiments
Fick the Hor
Pio Bc Eashliga hiceed be Maccmmal
In this section we wish to evaluate the full stack of OWG, incl. grounding, grasp planning and grasp ranking via contact reasoning, in scenarios that emu- late open-world grasping challenges. To that end, we conduct experiments in both simulation and in hard- ware, where in each trial we randomly place 5-15 objects in tabletop and instruct the robot to grasp an object of interest. We conduct trials in two scenar- ios, namely: a) isolated, where all objects are scat- tered across the tabletop, b) cluttered, where objects are tightly packed together leading to occlusions and rich contacts. We highlight that object-related query trials contain distractor objects that share the same category with the target object.
Grapp dz kii
Baselines We compare with two basclines, namely: a) CROG [39], an end-to-end referring grasp syn- thesis model trained in OCID [40] scenes, and b) SayCan-IM [12], an L.M--based zero-shot plan- ning method that actualizes embodied reasoning via chaining external modules for segmentation, grounding and grasp synthesis, while reasoning with LLM chain-of thain-of-thoughts [74]. Our choice of baselines aims at showing the advantages of using an
Figure 4: Open-ended language-guided
grasping trials in Gazebo (top) and real robot (bottom), in isolated (left column) and clut- tered (right column) scenes.

---

LVLM-based method vs. both implicit andto-enn approaches, as well as modular approaches that rely solely on LLMs to reason, with visual processing coming through external tools. See details in baseline implementations in Appendix B.
Implementation Our robot setup consists of two URSe arms with Robotiq 2F-140 parallel jaw grippers and an ASUS Xtion depth cam-
Sctup
CROG [39]
SayCan-IN [12]
OWG (Oans)
seen
sussen
500n
wnzeon
5e07
unseen
Simulation (x50)
era. We conduct 50 trials per scenario in the Gazebo simulator [75], using 30 unique object models. For real robot experiments, we conduct trials per scemario having the initial scenes as similar as possible between baselines. In both SayCan-IM and our method, Mask-RCNN [76] is utilized for 2D instance segmentation while
-Iaolated
600
78.0
-Chattrred
38.0
22.0
48.0
360
62.0
06.0
Real-Weeld (X6)
-Itolated
50.0
18.6
06.6
33.3
83.3
66.6
-Cluttered
16.6
16.6
16.6
50.0
50.0
Table 2: Averaged success rates (%) over simulated and real-world grasping trials. The x represents num- ber of trials per cell.
GR CenvNee [38] pretrained in Jacquard [52] is used as the grasp synthesis module. Our robotic setup is illustrated in Fig. 4, while more details can be found in Appendix B. To investigate general- ization performance, all method are evaluated in both scenarios, in two splits: (i) seen, where target objects and queries are present in the method's training data or in-context prompts, and (ii) unseen, where the instruction refers to objects that do not appear in CROG's training data or SayCan-IM's in-context prompts. Averaged success rate per scenario is reported, where trial is considered suc- cessful if the robot grasps the object and places it in pre-defined container position.
Results We observe that the supervised method CROG struggles when used at un- con-
seen data, in both scenarios. trary, both SayCan-IM and OWG demon- strate immunity to seen/unseen objects, il- lustrating the strong zero-shot capabilities of LLM-based approaches, which can nat- urally generalize the concepts of object cateporiesattiioutssrlllaiossss from language. SayCan-IM is limited by the external vision models and hence struggles in cluttered scenes, where its detector sometimes fails to perceive the target object, resulting in lower final success rates compared to OWG, especially in the real- world experiments. OWG consistently outperforms both baselines both in simulation and in the real robot, with an 15% and ~ 35% improved averaged success rate respectively. In Fig. 5, we illustrate the decomposition of failures across grounding and grasping in our baselines for 25 Gazebo trials per scenario, where we automatically test for the target object's grounding results alongside success rate. We observe that OWG consistently reduces the error rates in both grasping and grasping compared the baselines in all scenarios and test splits. We believe that these results encouraging for the future of LVLMs in robot grasping.
Figure 5: Distribution of failures across grounding and grasping in Gazebo grasping trials for isolated (left) and cluttered (right). OWG improves performance across both modes in both setups and test splits.
4.3
Ablation Studies
In out ablations we wish to answer the following questions: a) What is the bottleneck introduced by the segmentation model in the open-ended grounding performance?, b) What are the contributions of all the different visual prompt elements considered in our work?, and e) What is the contribution of the LVLM-based grasp planning and ranking in robot grasping experiments? The grounding ablations for the first two questions are organized in Table 3, while for the latter in Table 4.
Instance segmentation bottleneck We compare the averaged mloU of our OWG grounder in a subset of our OCID-VIGG evaluation data for three different segmentation methods and ground-truth masks. We employ: a) SAM [37], b) the RPN module of the spen-vocabulary detector ViLD [28], and c) the RGB-D two-stage instance segmentation method UOIS [77]. where we also provide the depth data as part of the input. ViLD-RPNN and UOIS both achieve a bit above 70%, which is
7

---

~ 15% delta from ground- truth masks, while SAM offers the best baseline with 10.8% delta from ground-truth. Implementation details and related visualizations in Appendix C.
Visual prompt components Visual prompt design choices have shown significantly affect the perfor-
Method
mloU (%)
OWG (w/ Ground-Triee Mask)
86.6
mance of LVLMs. We ablate all components of our
-wo reference
23.2
grounding prompt and observe the contribution of each
-Wo number overlay
54.6
-wo high-res
61.3
one via its averaged mloU in the same subset as above
-w/o self-consistency
70.9
(see details in Appendix A.2). The most important prompt component is the reference image, provided alongside the marked image. Due to the high clutter of our test scenes, simply highlighting marks and label IDs in a single frame, as in SoM [26] hinders the recogni- tion capabilities of the LVLM, with a mloU drop from 86.6% to 23.2%. Further decluttering the marked image also helps, with overlaying the numeric IDs, using high-resolution images and highlighting the in- side of each region mask being decreasingly important. Surprisingly, also marking bounding boxes leads to a 12% mloU drop compared to avoiding them, possibly due to occlusions caused by lots of boxes in cluttered areas. Finally, self-consistency and chain-of-thought prompting components that were added also improve LVLM's grounding performance by 16 and 10% respectively, by ensembling multiple responses and enforcing step-by-step reasoning.
-w/ box
74.6
-w/o CoT prompt
77.6
-w/o mask fill
81.1
SAM [37] ViLD-RPN [78] UOIS [77]
75.8
72.9
71.1
Table 3: Grounding ablation studies.
Grasp-Related Ablations We quantify the contribution of our grasp planning and ranking stages in the open- world grasping pipeline, by replicating trials as in the pre-
Method
Isolated
Cluttered
OWG
84.0
68.0
-w/o planning
S0.0
46.0
vious section and potentially skipping one or both of these
-wfo grasp ranking
82.0
60.0
stages. As we see in Table 4, the effect of these compo- nents is not so apparent in isolated scenes, as objects are not obstructed by surroundings and hence most proposed grasps are feasible. The effect becomes more prominent in the cluttered scenario, where the lack of grasp planning leads to a success rate decrease of 22%. This is because without grasp planning the agent attempts to grasp the target immediately, which almost always leads to a collision that makes the grasp fail. Grasp ranking less essential, a lot of contact-related information is existent in the grasp quality predictions of our grasp synthesis network. However, it still provides important boost in final success rate (8% increase). When skipping both stages, the agent's performance drops drastically in cluttered scenes, as it is unable to recover from grasp failures, and hence always fails when the first attempted grasp was not successful.
-w/o both
80.0
42.0
Table 4: Averaged success rates (%) over 50 simulated grasping trials per scenario.
5 Conclusion, Limitations & Future Work
In this paper we introduce OWG, novel system formulation for tackling open-world grasping. Our focus is on combining LVLMs with segmentation and grasp synthesis models, and visually prompt the LVLM to ground, plan and reason about the scene and the object grasps. Our works sets a foundation for enabling robots to ground open-ended language input and close-the-loop for effective grasp planning and contact reasoning, leading to significant improvements over previous zero-shot approaches, as demonstrated by empirical evaluations, ablation studies and robot experiments.
Limitations First, as OWG is a modular approach, it suffers from error cascading effects intro- duced by the segmentor and grasp synthesis models. However, improvements in these areas mean direct improvement to the OWG pipeline. Second, we currently use 4-DoF grasps to communicate them visually to GPT-4v, which constrains grasping to single view. In the future we would like to integrate 6-DoF grasp detectors and explore new prompting schemes to aggregate and rank grasp information visually. Third, our results suggest that LVLMs still struggle to ground complex object relationships. More sophisticated prompting schemes beyond marker overlaying, or instruct-tuning in grasp-related data, might be a future direction for dealing with this limitation.
8

---

References
[1] B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, and et. al. Lan- guage models few-shot leamners ArXiv, abs/2005.14165, 2020. URL https: / /appi. bemanticscholes (oraraarroogg/ooooo::::121111
[2] OpenAl. Gpt-4 technical report, 2023.
[3] H. Touvron, Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, Hambro, Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 2023, URL https: //app..aaaa//////.
[4] H. Touvron, Martin, Stone, Albert, Almahairi, and Y. Llama 2: Open foundation and fine-tuned chat models. ArXiv, (bbs/2307.09888 2023. URL https: //api.
semant cschol laro org/ ( CCoppssDD:2595500
[5] A. Chowdhery, Narang. J. Devlin, M. Bosma, G. Mishra, and A. R. et. al. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 14:240:1-240:111 2022. URL
https://apt semanticscholl wwtpr////oooooo:::oo...m (DtrpuuID:247911990 247951931
[6] W. Gurnee and M. Tegmark. Language models represent space and time. ArXiv, abs/2310.02207, 2023. URL https: sewant Crr.aa/uusICCrr
263608756.
[7] Z. Jiang, F. Xu, Araki, and G. Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2019, URL https: / //pppppp..////// semant icscholar. org/ :
[8] F. Petroni, T. Rocktäschel, Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel. Lan- guage models as knowledge bases? ArXiv, 11011999900666 2019. URL attps://apit semanticscholas wwt.sschhh....c.oooooueeeeeeeeee www.ooorpggIDia/oo..o.om 20000pp9ssI:22225555511
[9] C. H. Song. J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su. LIm-planner: Few- shot grounded planning for embodied agents with large language models. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1986-2997, 2022. URL https: //api. semant org/ Corpus 254408960.
[10] Y. Ding, X. Zhang, S. Amiri, Cao, H. Yang, C. Esselink, and Zhang. Robot task planning and situation handling in open worlds. ArXiv, abs/2210.01287,, 2022. URL https: //aii. semanticschen
[11] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, and C. F. et. al. Do as can, not as say: Grounding language robotic affordances. In Conference on Robot Learning, 2022. URL https://aat api. wwwanticschollie... wttp/orppssD//.CC.
9097
[12] W. Huang, F. Xia, T. Xiao, H. Chan, Liang. R. Florence, A. Zeng. J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, S. Levine, K. Hausman, B. Ichter. Inner monologue: Embodied reasoning through planning with language models. Conference on Robot Learning, 2022. URL https: //api. semanticscholat wwtp:orrgg///rrr////oooo Corpus httppp////ppp.eeeaniicscchhhohio
250451569.
[13] Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. 2023 IEEE International Conference Robotics and Automation (ICRA), pages 11523-11530,
2022. URL https:://// UtL. wew.nttccccooaar.rorg//ooooooo

---

[14] W. Huang, F. Xia, D. Shah, D. Driess, A. Zeng, Y. Lu, P. R. Florence, I. Mordatch, S. Levine, K. Hausman, and B. Ichter. Grounded decoding: Guiding text generation with grounded models for robot control. ArXiv, bbbs/233..00855, 2023. URL https://api. semanti cechoticccllarr.. wtg/CorpussDDC :
[15] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: from natural language instructions to feasible plans. Autonomous Robots, 47:1345 1365, 2023. URL https:
semanteeantttii iesaanttcsccoles. :
[16] B. Yu, Kasaci, and M. Cao. L3mvn: Leveraging large language models for visual target nav- igation. 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3554-3560, 2023. URL https: / wwwanticscoolaatteoeoooeccooccee httpps//paa...ssmnnccccoo.....e CrrpuupssI 258079021.
[17] G. Zhou, Y. Hong, and Q. Wu. Navgpt: Explicit reasoning in vision-and-language navi- gation with large language models. ArXiv, abs/2305.16986, 2023. URL https: / /api. org semanticschol htt.://chooaaa.ooooooooaaaa::0a.
[18] A. Rajvanshi, K. Sikka, X. Lin, B. Lee, H.-P. Chiu, and A. Velasquez. Saynav: Ground- ing large language models for dynamic planning to navigation new environments. ArXiv, 055/2309.04077. 2023. URL https: //ppi/app...p semanticscholar www.aaroooroooooorrrr...
261 661660608.
[19] B. Lin, Y. Zhu, Z. Chen, X. Liang. J. zhuo Liu, and X. Liang. Adapt: Vision-language navigation with modality-aligned action prompts. 2022 IEEE/CVE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 15375-15385, 2022. URL https: //api. www.nticschollee.eeoeet
oro.oorg/ooo..oo. CCrp///rrsss::: 249209579.
[20] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. H. Vuong, P Wohlhart, B. Zitkovich, F. Xia, C. Finn, and K. Hausman. worldworld object manipulation using pre-trained vision-language models. ArXiv, Abs/2303.000903 2023. URL https://api: semanticscholas org :ttp:/Cooo::::2288BU0S
[21] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. R. Florence, and A. Zeng- Code as policies: Language model programs for embodied control. 2023 IEEE Interna- tional Conference on Roboties and Automation (ICRA), pages 1493-95000 2022. URL https: //api. www.mmtttiii insaoolaccccooarr wwwpoolaarrrgggroooooeeoo :rgg/Corpuss:::2555553 252355542
[22] A. Zeng. A. Wong, S. Welker, K. Choromanski, Tombari, A. Purohit, M. S. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, and P. R. Florence. Socratic models: Composing zero- shot multimodal reasoning with language. ArXiv, abs/2204.0055 000988 2022. URL https: //api. www.nttcsmcnooooooe.oeeee
27922522.
[23] Huang, Jiang, H.-W. Dong, Qiao, Gao, and H. Li. Instruct2act: Mapping multi- modality instructions robotic actions with large language model. ArXiv, abs/2305.11176, 2023. URL http:p/s//p....sstt..ccccooooo org/ :ttr:/oorpss::::::55: :5:276622222...226
[24] S. Vemprala, R. Bonatti, A. F. C. Bucker, and A. Kapoor. Chatgpt for robotics: De- sign principles and model abilities. ArXiv, abs/2306. 17582, 2023. URL https: / //api. semantio cscholar org Sttgg//oooust:25555
622.
[25] W. Huang, C. Wang, R. Zhang, Y. Li, Wu, and I.. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. ArXiv, 0002307777773.. 2023. URL https: //api....// semant orrp:oo.ooo/oooo :
[26] Yang, H. Zhang, Li. X. Zou, yue Li, and Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. ArXiv, abs/2310.11441, 2023. URL https://
api api. www.smennniccholaat...... org (D: :tt.::aaaaaao///out:I:I::::/
10

---

[27] Y. Hu, F. Lin, T. Zhang, L. Yi, and Y. Gao. Look before you leap: Unveiling the power of gpt-4v in rohotic vision-language planning. ArXiv, abs/2311. 178442 2023. URL https:
[28] X. Gu, T.-Y Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and lan- guage knowledge distillation. International Conference on Learning Representations, 2021. URL https:://///..c /taps://apii.. semant insiaoaacccchhar..... wwt. http//ooouD:::3341111111
[29] M. Minderer, A. A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Amnb, M. Dehghani, Z. Shen, X. Wang. X. Zhai, Kipf, and N. Houlsby.
Simple open- open-vocabulary object detection with vision transformers. ArXiv, abs/2205.06230,
wwwanticschooee..e..eee.esscccee 2022. URL http:s/./ap....smmcccccoooooo... wttp:cchho....ooooooroo.aaaaaaa.
[30] A. Kamath, M. Singh, LeCun, Misra, G. Synnaeve, and Carion. Mdetr mod- ulated detection for end-to-end multi-modal understanding. 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pages 1760-1770, 2021. URL https://api.
semanticscholie org Dtt.aaaa...o//ooooaaa0:23333330
[31] Bai, S. Bai, S. Yang. Wang, Tan, P. Wang. Lin, Zhou, and Zhou. Qwen-vl: A frontier large lasion-language model with versatile abilities. ArXiv, abs/2308.1299t. 2023. URL https: /api. semanticscholoo..c.
Pttp/CoorpuIID:23387...
[32] W. Dai, J. Li, D. Li, A. M. H. Tiong, Zhao, W. Wang, B. A. Li, P. Fung, and S. C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305.0633 066500, 2023. URL https: //tp:/appppp semanti comaanicscholar.... wttph/oolll...rooooosstt.... 258615266.
[33] H. Liu, C. Li, Q. Wu, and J. Lee. Visual instruction tuning. ArXiv, abs/2304.08485, 2023. URL https::/aapp wtt.ppisseaatnscchooeeeooo http.//occooss..:251117777777771
[34] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-languag understanding with advanced large language models. ArXiv, hbds/334.10592, 2023. URL.
https: //:::/appp semanticschen wttp:/aaarrrrrooooooeeeeee Ittp://ppppiiii:: : 2558999130.
[35] Z. Yang, L. Li, K. Lin, J. Wang, C.C. Lin, Z. Liu, and L. Wang. The dawn of Imms:
Preliminary explorations with gpt-4v(ision). ArXiv, abs/2309, 17421, 2023. URL https:
// //api. www.ssemmnnicchhooooo
org/ org/Corrps 22:33310991.
[36] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero- shot reasoners. ArXiv, 0bs/2205.119916, 2022. URL https: /api. wemanticschots org/Corpus 2490:17:99944
[37] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, Dollár, and R. B. Girshick. Segment anything. 2023 IEEF/CVF International Conference on Computer Vision (ICCV), pages 3992-4003, 2023. URL https:
// //api. wew.iiismmtticccchhoooee
wotg/oorrsssi............ 257952310.
[38] S. Kumra, S. Joshi, and F. Sahin. Antipodal robotic grasping using generative residual con- volutional neural network. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (/ROS), pages 9626 9626-9633, 2019. URL https: //aptp::://p..../ www.nticschaaolo.....
bTg/CorpuusDD:2225888
[39] G. Tziafas, XU, A. Goel, M. Kasaei, Li, and H. Kasaei. LLaggaag--uuides robot grasping: Clip-based referring grasp synthesis clutter. In Tan, M. Toussaint, and K. Darvish, editors, Proceedings of The 7th Conference Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 1450-1466. PMLR, 06-09 Nov 2023.
11

---

[40] M. Suchi, T. Patten, and M. Vineze. Easylabel: A semi-automan automatic pixel-wise object annota- tion tool for creating robotic rgh-d datasets. 2019 International Conference on Robotics and Automation (ICRA), pages 6678-6684, 2019.
[41] Y. Yao, A. Zhang, Z. Zhang, Z. Liu, T. seng Chua, and M. Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. ArXiv, abs/2109.11797, 2021. URL https:/ appi. semant i cocco war.chooarrooo wwwprr/////ro//i/ 237 237633382. 635382.
[42] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. CoRR, 0bs/2103.000200 2021. URL https: http://xxxvvv///// abs/ 2013.11...0 11.03.000 00020.
[43] A. Shtedritski, C. Rupprecht, and A. Vedaldi. What does clip know about red circle? vi- sual prompt engineering for vlms. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1953-11963, 2023. URL https: aet.eatm
atg/CorpusI::2581088
[44] I.. Yang, Y. Wang, X. Li, X. Wang, and J. Yang. Fine-grained visual prompting, ArXiv, abs/2306.04356, 2023. URL https: /pas:///ppii cemamticsccoola..rrg//oooo
259096008.
[45] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, and C. F, et. al. Rt-1: Robotics transformer for real-world control at scale. ArXiv, abs/2212.06817, 2022. //a /app..
URL https:
semant /emant www.olaw..rrgeee/e/eae
ID: 25:
[46] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, K. Choromanski, T. Ding. D. Driess, C. Finn, P. R. Florence, and C. F. et. al. Rt-2: Vision-languuag-actiion models transfer web knowledge to robotic control. ArXiv, abs/2307.15818, 2023. URL https: /api. semant ietnniccsn icscholar.
: 22029:::26222339
[47] Y. Mu, Q. Zhang, M. Hu, W. Wang, M. Ding, J. Jin, B. Wang, J. Dai, Y. Qiao, and P. Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. ArXiv, abs/2305.15021, 2023. URL https://apl remanticscholi
Corpusss:
258865718.
[48] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. ArXiv, abs/2201.07207, 2022. URL https: https///app..eeannieeeee.e. semant ietiantiscchhlls www..org//eo wttppaar..rr//ouu..::::.. Co.pusID::2400522222 246035276.
[49] Jin, D. Li, Y. J. Shi, P. Hao, F. Sun, J. Zhang, and B. Fang. Robotgpt: Robot manipulation learning from chatgpt. IEEE Robotics and Automation Letters, 9:2543-2550, 2023. URL
/ api. semanticschot httrpps///p...emmmnccccoooooo htt.:/cccho.oo..oooooorrrpsaaaia
org Corp//rrrpDsuceeee 26:
[50] N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi. Gpt-4v(ision) for roboties:
Multimodal task planning from human demonstration. ArXiv, abs/2311.1201s 2023. URL
https:/
apa.................. semanticscho www.ccccclll....ooooooooooo hww.cchhhrrrrr......oooooo.oo.oe ort.aar..r///oopusst//// 26529950:6
[51] F. Liu, K. Fang. P. Abbeel, and S. Levine. Moka: Open-world robotic manipulation through mark-based visual prompting. Robotics: Science and Systems XX, 2024. URL https:// //
api. www.stiischoseeeieeee. ceschoclccrrccc Btgg/CorpusID:::
[52] A. Depierre, E. Dellandréa, and I.. Chen. Jacquard: A large scale dataset for robotic grasp detection. 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS),
pages 3511-3516, 2018.
[53] Ainetter and F. Fraundorfer. End-to-end trainable deep neural network for robotic grasp de- tection and semantic segmentation from rgb. 2021 IEEE International Conference Robotics and Automation (ICRA), pages 13452-13458, 2021.
12

---

[54] Y. Jiang, S. Moseson, and A. Saxena. Efficient grasping from rgbd images: Learning using new rectangle representation. 2011 IEEE International Conference on Robotics and Automa- fion, pages 3304-3311 2011.
[55] D. Morrison, P. Corke, and J. Leitner. Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach. ArXiv, abs/1804.05111 2018. URL atttps://apt
semant i ceattiicchhaar...o org. www.aaa...r//ooootttt... : 4891707.
[56] S. Kumra, S. Joshi, and F. Sahin. Gr-convnet v2: A real-time multi-grasp detection net- work for robotic grasping. Sensors (Basel, Switzerland), 22, 2022. URL https::/appl semant i csscholar org/ CorpusID:2517005
[57] Y. Xu, M. M. Kasaei, S. H. M. Kasaci, and Z. Li. Instance-wise grasp synthesis for robotic grasping. ArXiv, abs/2302.07822, 2023.
[58] Mahler, J. Liang. Niyaz, M. Laskey, Doan, X. Liu, A. Ojea, and Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. ArXiv, abs/1 hess/703309312. 2017. URL https: semant www.saniiccchocr..oooooo CorpusID: 61.66138899 38957.
[59] H. Fang, C. Wang, M. Gou, and C. Lu. Graspnet-lbillion: A large-scale benchmark for gen- eral object grasping. 2020 IEEE/CVF Conference Computer Vision and Pattern Recog- nition (CVPR), pages 11441-11450, 2020. URL https: sem.iiiimmmtttcccoolloo...
oxc/CorpusI::2195449
[60] C. Eppner, A. Mousavian, and D. Fox. ACRONYM: A large-scale grasp dataset based on simulation. In Under Review at ICRA 2021, 2020.
[61] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for ob- ject manipulation. 2019 IEEF/CVF International Conference on Computer Vision (ICCV), pages 2901-2910, 2019. URL https:://///.. app. www.ntiisccooi.........o.......e orgp:arrrrrrrroreeeeeee CrrpCorp
166228416.
[62] M. Breyer, Chung, L. Ott, R. Siegwart, and Nieto. Volumetric grasping network: Real-time 6 dof grasp detection in clutter. In Conference on Robot Learning, 2021. URL
org/ https: / api. semanticschooe I : 230435660.
[63] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox. 6-dof grasping for target-driven object manipulation in clutter. 2020 IEEE International Conference on Roboties and Automa- tion (ICRA), pages 6232 6238, 2019. URL https: /ttp::/aappiioe stappiisemmaticsccoollooo org/
Corpus 0088109961
[64] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact gratact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. 202/ IEEE International Conference on Robotics and Au- tomation (ICRA), pages 13438-13444, 2021. URL https: /ppi..p..../// semant icsccolar.
org/Corpus ID: 232380275.
[65] V. D. An, M. N. Vu, Huang. N. Nguyen, H. Le, D. Vo, and A. Nguyen. Language-driven grasp detection. ArXiv, abs/2406.0990 09909, 2024. URL httptt api semanticscoolar.
org/Corpus TD: 2705219299 942.
[66] Y. Lu. Fan, B. Deng. Liu, Y. Li, and S. Wang. VI-grasp: 6-dof interactive grasp policy for language-oriented objects in cluttered indoor scenes. 2023 IEFF/RS International Conference Intelligent Robots and Systems (IROS), pages 976-983, 2023. URL https:
//api. semant ccscholar......cooomocccooa arg/CorpusII: 25533351477
[67] Ardón, É. Pairet, P. Petrick, S. Ramamoorthy, and K. Lohan. Learning grasp affor- dance reasoning through semantic relations. IEEE Robotics and Automation Letters, 4:4571- 4578, 2019. URL https:://// / http:////app.ssmanniiieeeeo www.nticscholla.t..ooo. org (ttp/CoooCSII:::545553)
13

---

[68] A. Murali, W. Liu, K. Marino, S. Chernova, and A. K. Gupta. Same object, different grasps: Data and semantic knowledge for task-oriented grasping. In Conference on Robot Learning, 2020. URL https::/tsat. //api. http://a...smmiiicccici.. www.nticscholeee.e..ee...occccce .rg. /ot///orrr::22333300
[69] C. Tang, D. Huang, W. Ge, W. Liu, and H. Zhang. Graspgpt: Leveraging semantic knowledge from large language model for task-oriented grasping. IEEE Robotics and Automation Let- lers, 8:7551-7558, 2023. URL https::///// 260154903.
semanticscoolaas Corpg///r
[70] X. Zou, J. Yang. H. Zhang, F. Li. L. Li, Gao, and Y. Lec. Segment everything everywhere all at once. ArXiv, abs/2304.06718, 2023. URL https: api wew.eesssattncooo....o.o org herg/orrussDD : 258108410.
[71] Li, H. Zhang, P. Sun, X. Zou, S. Liu, J. Yang, C. Li, L. Zhang, and Gao. Semantic-sam: Segment and recognize anything any granularity. arXiv preprint arXiv:2307.00767, 2023.
[72] Subramanian, W. Merrill, T. Darrell, M. Gardner, S. Singh, and A. Rohrbach. Re- clip: A strong zero-shot baseline for referring expression comprehension. In Anmual Meeting of the Association for Computational Linguistics, 2022. URL https://aat
semanticschole wwt.org///ooooo :ttp://oopssI::24111555
[73] Gpt-Av(ision) system card. CozpusID:26321800
2023. URL https:/ /api. semantiiccchllar..rrr
[74] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing
reasoning and acting in language models. ArXiv, abs/2210.03629, 2022. URL https:// api www.seeeesssseeeeeeeeeeeee iascooaas org. ottg//orpuss:::2227625553
[75] N. P. Koenig and A. Howard. Design and use paradigms for gazebo, open-source multi- robot simulator. 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Ca. No.04CH37566), 3:2149-2154 vol.3, 2004.
[76] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick. Mask r-cnn. 2017. URL https://app. semanticscholas wtt.s/chhh.aaa..o/coooossaaaa 5444555877
[77] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. Unseen object instance segmentation for robotic environments. IEEE Transactions on Robotics. 37:1343-1359, 2020. URL https://api.
semanticschen arr org
[78] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and lan- guage knowledge distillation. International Conference on Learning Representations, 2021. URL https: /api.. semant semanticscholar org/ wttp://ooppsS1::22184411110
14

---

Towards Open-World Grasping with Large Vision-Language Models
Supplementary Material
A
LVLM prompts
A.1 Prompts
Prompts for the three cases considered in this work, namely: open-ended referring segmentation. grounded grasp planning, and grasp ranking can be found below.
Open-ended referring segmentation:
Takes observation image, a marked copy with highlighted instance masks and label IDs and an open-ended language query referring to particular object instance, and outputs the label ID of the referred object. The LVLM is encouraged to provide tfains-of-thoughee reasoning in cases where the input query contains complex expressions that involve multiple object and spatial relations.
Grounded grasp planning: grasp grasp_paaaaniiiiieeeaaeaaaaaaaee planning.txt
Takes the marked image and the label ID of the target object to be grasped, and outputs plan ensure the target object will be graspable. The plan consists of remove actions for blocking objects and final pick action for the target.
Grasp ranking: grasp grasp_rankinge ranking.txt
Takes cropped bounding box image around the next object to be picked, marked with grasp proposals from a 4-DoF grasp synthesis model and a set of label IDs, and outputs sorted list of grasp IDs, from most to least confidence for a successful grasp. The LVLMs is encouraged to reason about the object shape and their neighbouring objects before producing final ranking.
A.2 Visual Prompt Design
In the following, we summarize the key visual prompting elements that were used for prompting the LVLM in the context of OWG:
The most common failure mode of visual marker prompting with GPT-4v is that it sometimes struggles to discriminate which ID corresponds to what segment. Especially in cluttered scenes, label IDs might severely overlap within small frame regions. Several techniques can assist in making the markers more clear to the LVLM: a) we adopt the algorithm of [1] for overlaying numeric IDs within the frame with minimal overlap, b) we paint both the internal of each segment's mask and its ID with the same unique color (colors are chosen to visually distinguishable), and c) increase the resolution of the marked image and the size layout of the markers.
Clarity of visual markers
Reference Image If not highlighting the internal of each segment, GPT-4v sometimes refers to regions with wrong IDs, especially in highly cluttered scenes. But if the masks are highlighted with high opacity, then the appearance of the object becomes less visible and GPT-4v struggles to recognize it. We propose technique to ameliorate this by passing both the original (reference) and the marked image and constructing a text prompt that explains that the latter corresponds to annotated segments of the first.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Chain-of-thoughte Chain-of- Ohain-of-Thought (CoT) prompting is a rell-established methodology for guid- ing LLMs to perform multi-step reasoning and reduce hallucinations [2]. We find that LVLMSs share similar properties and prompting them to reason about their final answer before producing it can robustify the response quality. For grounding, we ask GPT-4v to decompose the input instruction in steps and refer to all intermediate referenced objects. For grasp planning, we ask it to explicitly mention all object IDs that are blocking the target object, before producing a plan. For grasp ranking, we decompose the prompt in three steps: (i) identify the category of the target object and provide general description of what constitutes a good grasp for it given its shape, (ii) list the grasp IDs that will most likely lead to contact with neighboring objects, and (iii) rank the grasp IDs based on the previous two steps.
Self-consistency Even with zero temperature, we observe that the outputs of GPT-4v are not always reproducible. We find that sometimes GPT-4v might produce different responses different runs, even with exactly the same prompt. In attempt to reduce the effect of this phenomenon and robustify LVLM outputs, we use the self-consistency method developed for LLMs [3]. In particular, we ask GPT-4v to provide multiple responses, parse each one separately and then perform majority voting to determine the most consistent output.
B
Robot experiments
B.1 Setups
Our object catalog for seen/unseen trials is shown in Fig. 1. In Gazebo, isolated scenarios are generated by ensuring all spawned objects have a fixed 3D distance, while cluttered scenarios we ensure contact between the target object and neighbouring objects, by first spawning the target and then sampling different poses for other object models around it. In real-robot experiments, we manually setup the scenes while making sure to replicate the setup close possible for fair comparisons between baselines. In all trial scenes that contain distractor objects, the user instruction refers to some property that disambiguates the target instance from other objects of the same category, using names, attributes and spatial relations. We also conduct experiments without distractors for affordance-based queries, which require semantic reasoning to be correctly grounded.
For real robot experiments, we use the default torchvision implementation of Mask-RCNN, with the model weights provided by PyTorch Hub, fine- tuned in a few annotated scenes captures from our robot setup. For grasp synthesis, we generate a top- down orthographic projection of the scene, both for color and for depth (i.e. reverse depth- - heightmap). This is the input pass to the pretrained GR- ConvNet. In order to align regions from the 2D frame where Mask-RCNN provides segmentations and the orthograpic projection where our grasp syn- thesis model provides grasp poses, we use the Hun- garian matching algorithm to match the centers of outputs from both models, after projected to 3D and transformed world reference frame (robot base), using 3D euclidean distance as the cost function.
Figure 1: Seen (left column) and unseen (right column) object used in our robot experiments in Gazebo (top) and the real world (bottom).
B.2 Baseline Implementation
CROG CROG receives an single 448 x 448 RGB view and a natural language query, and provides both an instance segmentation mask for the target object, as well as a set of 4-DoF grasp proposals, assuming that the gripper approaches the object aligned with the perspective of the camera. We the checkpoint provided by the original paper, trained in the multiple split of OCID-VLG dataset, which contains 90k scene- grayp data from around 1,000 unique scenes from 31
2

---

Name
Attribute
Spatial Rel.
Visual Rel.
Sem. Rel.
Multi-hop
Affordance
Total
42
26
33
19
13
24
16
173
Table 1: Number of samples in grounding evaluation dataset.
object categories. The model uses CLIP's pretrained ResNet-50 visual and BERT text encoders, but fine-tunes them end-to-end in OCID scenes for joint grounding and grasp synthesis tasks.
SayCan-IM Our SayCan-IM baseline follows the implementation publicly released by the SayCan work [4], which can be found in this HTTP URL. In particular, the pipeline uses the ViLD [5] open-vocab object detector to turn the input observation image into list of object names and then lets the LLM generate sequence of pick-and place actions to perform in order to solve the task given by the user. We made the following modifications the above baseline:
1. In the original implementation, the robot only has access the output plan is confined to only selecting what objects to pick and where to place them (based on the detected object list from ViLD). In our implementation, we also provide legsualggoondiii
saick_and and_place skill, and
tool, which lets the LLM invoke CLIP [6] to rank a list of candidate objects with a given text description and select the most similar one. This is to allow the LLM reason about attribute concepts besides object category (e.g. "gel the blue mug" ").
2. Besides the names the appearing objects, we also provide their bounding box coordinates, as detected by ViLD, in x1y1x2y2 format in the prompt. This was introduced in order to enable the LLM also reason about the locations of objects and resolve spatial relation queries, as well reason about the feasibility of grasping objects by checking whether their bounding boxes overlap.
3. We replace the pick_and_ple__eeeeeaaaapppppppp_ and_place primitive skill with two distinct skills: remove and pick. The first skill corresponds removing blocking object in order to clear the path for grasping the target. The second skill corresponds to picking the target object that the user requested. Both skills use GR-ConvNet [7] under-the-hood to sample grasp proposals, select the one with higher predicted grasp quality, and use an IK solver to control the robot arm.
4. We used the observe-reason-ae act prompting style first introduced by Inner Monologue [8] and later improved by ReAct [9]. Unlike the vanilla implementation, which simply produces a plan of steps without feedback, with this technique we let the LLM plan one step at a time, and integrate feedback from the environment (e.g. CLIP outputs, grasp failures etc.) before planning again.
The system prompt and in-context examples used in SayCan-IM baseline are shown in Fig. 2. As we mention in main paper, for the real robot experiments, replace ViLD-RPN with Mask-RCNN [10] for instance segmentation, and use CLIP with prompts for all object used in experiments to recognize the categories and provide the object list state to the LLM.
C Offline grounding experiments
C.1 OCID Dataset Details
We manually annotate 173 images from OCID dataset with the following query types: a) name (open- vecabulayyy object descriptions), b) attribute, c) spatial relations, d) visual relations, e) semantic relations, f), multi-hop reasoning, and g) user-affordances. The number annotations per query type given Table We make sure to include unique test scenes from the dataset and include images with heavy clutter. The target of each scene within a query type unique, and make sure include images with distractor objects (of the same category the target) for all query types that require relational reasoning (all except name and affordance).
3

---

Regarding our custom FGVP-CLIP baseline (FGVP"), we present analytical comparisons and ablation in the following subsection.
C.2 Baselines Implementation and Ablations
We utilize the provided demo applications for the end-to-end methods (SEEM, PolyFormer) to conduct grounding experiments manually. For CLIP-based baselines, we re- re-implement all methods from the corresponding papers (ReCLIP, RedCirele, FGVP). - We use the ViT-B visual encoder to extract features from image segments and the default BERT text encoder to represent the input query. CLIP-based baselines compute the cosine similarity between segment and text features rank them and select the most similar segment the final result via the argmax operator. fround-trutte masks used for all CLIP-based baselines, similar GPT-4y ones. We would like highlight that in the original papers, the aforementioned baselines potential post-processing steps to enhance the grounding capabilities of CLIP. In particular, ReCLIP uses syntactic parsing to extract entity and relation words/phrases from the input query, as well as spatial relation resolution heuristics (e.g. "left', 'on' etc. designed specifically for the RefCOCO dataset) to process the relations analytically and combine CLIP predictions only for the entities. RedCirele and FGVP additionally utilize "subtraction" " post-pre post-psocsssing step, where they further subtract from the similarity values the average in set of mined hard-negative queries (again selected for specific dataset). We believe that such steps constitute domain- aware hand-crafted efforts, which even though helpful, do not represent the challenges of open-ended generalization, which is the primary focus of this work. As result, we do not consider such post-processing steps in our baseline implementation.
Comparisons with end-to-end approaches The need for manual annotations exhaust all possible language query inputs, as well as the need for manual testing via online demo applications for the considered specialist end-to-end methods (SEEM, PolyFormer) restrained from conducting experiments in large-scale. Instead, images. Results are given in Table 3. Results follow similar patterns to the larger test set of the main paper. Specialist models (SEEM, PolyFormer) struggle with even simple name queries, scoring below 15% on average. This is potentially due to the high discrepancy between the training distribution of RefCOCO and Visual Genome and our test data, as well as the lack of relational and affordance- language in these datasets. GPT-4v-based methods still compare favourably CLIP-based baselines, even in the SoM setting where single marked image is used. Overall, OWG-grounder achieves averaged mloU score of 70.4%. which is almost x2 from the previous approach.
originally conducted experiments in smaller subset of 52
CLIP Visual Prompt A Ablations To further analyze the performance CLIP-based baselines, we conduct ablation studies where we use specific elements of each method. particular, we study: a) effect of using multi-templates for the text prompt. where we average text embeddings from multiple versions of the query, using templates from the original paper, b) averaging similarity scores from the visual prompt and crops of each segment, originally proposed in ReCLIP, e) different visual prompt schemes, like drawing a boundary (rectangle or ellipse as in RedCircle), converting
Crop White-Back.
Blur-Rev Gray-Rev
Multi Temp.
Rect.
Ellipse
Mask
mloU
18.3
x
31.1
X
34.8
AMNAKA
33.7
X
24.6
26.3
X
34.9
X
AXXKA
X
x
41.5
EXXXA
X
XXX
x
43.0
X
51.8
X
X
X
X
X
51.2
Table 2: Component ablation studies for CLIP-based visual prompting. Results in %.

---

Found. Model
Spatial Relation
Visual Relation
Semantic Relation
Multi-
Method
Name
Attribute
Affordance
Avg.
hop
PolyFormer
20.9
13.3
2.6
0.8
3.1
6.7
8.3
8.0
SEEM
23.3
10.1
4.6
10.5
10.2
7.9
17.5
12.1
ReCLIP
CLIP
36.9
40.0
12.7
14.2
20.1
23.0
34,0
25.9
RedCirele
CLIP
33.3
21.1
19.7
15.4
18.8
24.0
47.4
25.7
FDVP
CLIP
25.1
19.0
23.7
25.2
12.3
22.5
22.8
21.6
SoM
GPT-4v
40.1
25.0
23.3
40.3
42.5
60.0
21.2
36.1
OWG (Ours)
GPT-4v
45.7
55.4
78.8
90.3
59.4
70.4
Table 3: Segmentation - -mloU(%) results in different language input types for cluttered indoor scenes from OCID.
to grayscale or blurring the rest of the frame (as proposed in FGVP), as well as a prompt that we discover ourselves works good, using a white background for the rest of the frame. We note that in paper's results the element combinations used are the following:
ReCLIP: rectangle prompt, multi-templates, blur-reverse + crop,
RedCircle: ellipse prompt, multi-templates, gray-reverse + blur-reverse,
FGVP: mask prompt, multi-templates, gray-reverse + blur-reverse
Ablation results are shown in Table Our findings are the following: 1) drawing rectangle prompt outperforms ellipse and mask (object contours) in itself, but ensembling rectangles and masks gives the best result, 2) using multiple text templates outperforms single-template only when ensembling multiple visual inputs, c) the most effective component is our method of replacing the rest of the frame with white background, compared to grayscale and reverse operators of FGVP, while ensembling all together gives the best performance. We call our custom FGVP baseline FGVP" We present analytical results per query type for CL.IP-based baselines versus GPT-4v methods, as the original paper, for our extended evaluation dataset in Table ??. CGVV''rerreeen the best configuration of CLIP-based visual prompting found by our ablation experiments. Results follow similar patterns to the smaller subset of the main paper, with significant performance boost for CI.IP-based baselines. However, GPT-4v-based methods still compare favourably to CLIP-based baselines, even in the SoM setting where single marked image is used. Our OWG visual prompt scheme dramatically outperforms all baselines, with a margin of 27.7% from SoM and 29.0% from the best found CLIP visual prompt methodology, showcasing its superiority in cluttered indoor scenes context as in OCID.
C.3 Instance Segmentation Ablations
We the checkpoints provided by the authors for UOIS [11] unseen object instance segmentation, as well the ViRL-RPN checkpoint and hyper-parans from the implementation in this HTTP URL. For SAM, we use the ViT-L variant of the released SAM [12] checkpoints, and search for optimal haper-parameters for automatic mask generator, resulting in the following configuration: pred_ serdd_iu_thresstee wwwethhheee... 92, stability_scom setpttt_ssorretrreeiiee threeh=0..0. 95. We ap- ply non-maximum suppression with an iou_ threshold=O... and remove nested masks, i.e. masks that completely inside other masks of higher score threshold. This step aids in keeping only object-level SAM predictions and decreasing the over-segmeenttion behavior that default SAM provided in our first implementation. In turn, this leads to less cluttered visual markers for our OWG grounding module. Example instance segmentation masks for the different methods are illustrated in Fig. 5.
D GPT-4y Example Responses
In Figs. 6, 7, we provide example responses for grounding different types of language queries in OCID scenes. We observed that GPT-4y, augmented with marked image prompting, can ground not just relact-related queries but also complex referring expressions that require reasoning about space, visual attributes, semantics and user-affordances. Interestingly, we find that GPT-4y responds
5

---

to queries that require symbolic reasoning concepts such as counting and negation, which are notoriously hard to emerge specialist grounding models. In Fig. 9, we provide some example responses corresponding to failure cases. Main failure modes include: a) grounding a distractor instead the desired object, b) not finding the object of interest at all, c) providing a correct reasoning and identifying the target in the raw image, but providing wrong ID of an irrelevant object.
References
[1] Yang, H. Zhang, F. Li, X. Zou, yue Li, and J. Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. ArXiv, abs/2310.11441, 2023. URL https: //api.
semanticscholar Org/CorpusID: 266149987.
[2] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero- shot reasoners. ArXiv, abs/2205. 2022. URL battps://agy semanticscholar. org/
CorpusID:2490110
[3] X. Wang. J. Wei, D. Schuurmans, Q. Le, E. H. hsin Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. ArXiv, abs/2203.11171, 2022, URL
https://api. http:///aappssmmiiiscccieee semanticscholar cecellaarrrrg//eeeee :r.//borrpus:::227555.5.. 247595263.
[4] M. Ahn, A. Brohan, N. Brown, Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho. J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng. Do as can, not as say: Grounding language robotic affordances, 2022. URL https:/ /ttp///aaxvvvv//a org/ /204 01691
[5] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open- object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2021. URL https://api. semanti cscholarr wwwo/oog/ooo..ooooooo het//CrruusD:D:: 238744187.
[6] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00022 2021. URL ptsttss://rxiie org/ /bs//
[7] S. Kumra, S. Joshi, and Sahin. Gr-convnet v2: real-time multi gratsp detection net- work for robotic grasping. Sensors (Basel, Switzerland), 22, 2022. URL https://api. semanticscholar.
[8] W. Huang, F. Xia, T. Xiao, Chan, J. Liang. P. R. Florence, A. Zeng. J. Tompson, I. Mordatch, Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, Levine, Hausman, and Ichter. Inner monologue: Embodied reasoning through planning with language models. In Con- ference Robot Learning, 2022. URL https://appl semanticscholar. semnartticcchol.....ooro/oopooo 250451569.
[9] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. ArXiv, abs/2210.03629, 2022. httpttts: semanticscholar, org Irg/CorpusID: 252762395.
[10] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick. Mask r-cnn. 2017. URL https://api. semanticechol org/CorpusID: :D::61557133 54465873.
[11] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. Unseen object instance segmentation for robotic environments. IEEE Transactions Robotics, 37:1343-1350 2020. URL https://api:
semanticscholar. wrtp:ooooo/// 220044228.

---

[12] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.- Lo, P. Dollár, and R. B. Girshick. Segment anything. 2023 IEEF/CVF International Conference on Computer Vision (ICCV), pages 3992-4003, 2023. URL https: //api. II 257952310.

---

highly
plowsing ep odeer so the Soooe other objecss onnces all the Mocking paccing
plawing excuediityy right you
tual the liged ptruct Ths on ctlp on rouines duat tar nou boundig boace (dl you. 4.2. dlaed jot, B. bn 02 By 22 weetaa
dlan elljec is ciple
sl c2 and me
Aemientii your las siep plan tooda bo "akes".
Consile dellocing talk coloo owe
sieaid prowddigg uy dis of odjer tine rasa discapati to he goes ded- ing, vieaa poootingg (M- "Ma and gooe iy The vieall goearding tial dholld to seod desorale wi tf alyacc
de apeil
colject ctec upu pempaess
The meuere chocks
lieditg slyace sove thon
innr lovallr taga
grepulie
pick caljer IDo (og picc (I The yirk not dheilihe so qlál the sagen olject and give it sorte user
Vee ant alannd ane penvical
ovataal bostis to the kermi thac tao viills Wasn coningg a plus, spthu the stject ID plactadare with specific sbjon italancc akng
Uic cakkeing cuampio aas zacke bll poudduld interct with th crntrl robut and underiant visit ropuesic fermal aboald lik.
nee appicce
Ute lipert "Ihy sowld boo. reed cooe light.
Gleecakee:
Vooas Naidee hyyy (Wrr 8-11 mikk [4]
107 laveet Yand bog. TolyTeyyh [3 65, ois 6.7% 3 ki]
'oxaa Yoad uad wiyl sdy? (W.. duti, 16 74g
hand Mucmee Ntylazy2 (AFT, 844, BAl)
1.05)
117 hand axdt Juty (12y (3.4% 065, 2:46,4 (47)
<Rebir twx pir The cueticl call nood bers, rad coor light iaduriia the they soald like ts use the cae sct ara light sooaa with ar light bull,, lanng a Back coliiie the earler airg fer easter herwozs stid 22-3.4 fasitgg (TA yontii lor marior herwoas 211122 y2-005 Basitight ETS, incianu that Jandd
pudiig fuilkyy (T) could lead Salann la doolld mesov baglor naller [X]
Cleun You in unapudyyc (0)
akaba, thougt faked to grupp the muaker (S) saoudd spas
Cleun sakee You mouoidatl nomoved out
aloba, dhought Stocilt the arge (Blehiit (70 a andd debi. iiteaad pick gin BC
Clearcatonn Yoe euccenttally pickad
Rkobo, dhoght comphoat the aak.
Abl and gloud
All of
trpst "jst oun. Bulio nod fis gnay keybwen??
hund: bax. Sileldde (Bur,
244)
Yurter stylchy
6.7, 844
107 kaeea thdlighr shylxdgh (6:6, 35, 0--9 E.5)
band boc. Niabb322 [124, 0,25,
107 kaee byyheae 43ylkcy22 [8-14 0.5 6-32, 0.67)
band Nampaa/ 31p1x222 (A75,
107 kvva Blewe Ni5000,7 96-6 6.3 1.5 o-1)
manc loda cnd. S1ab222: Bet 0US. bX1]
107 12, unen baad moedl Ntx <5,7 1017 6:7, (77,336
107 15, manc Mupler bby 2222
2.54
107 15, woen Bod Nayysdy? 12.5 6-1, 333. a30
107 in soaa
20 (ouct 8:33 w71,
"Reker toeple Tue soe sentee "g the carn filles ean to the gnay bryhwrat 1 chockt 1s Hdetth the poy keyloast To ideeriy the coe ttae ebjars 3) fil the use xhose was is the
Say sest j/cck-t thaod can Fales There ass keyfoord mewe the scoce (19) (4)) should faa iothe poa inokee sinad granding toe to ohich oue of
ARohe usave toal pransing (A.)
Obearattoo: Keyfoaddd (0) logyy
(II)
Keartyy whichioe
bychad.
koybard (b] cxntiaato and ist tt you co go so oonier weakldte catetuzz,, 20-1000
bas
inil1 bi-y--i si yo-i cemer would be pull-i
Canaal cand
34 1-----6
345 (e-6.551)
"herkker hie carall has that dlavk do koyboandd (5) coraal hes (4) Aller eegecinng he lovaken mveeiaacs of cercel bos 4) ouppect io voher ife spporr taal thic is owalay wit
Morter (2lc
oulled yi a2-0 yo-i city whichh enstagsin with canal box coced sate 0ct-6.34 ylouii a2-d you
Koybaard [f] moeeiiaas and pot on your 56, ad 22-e which ase courapping with cnoal hed [4] contiaanec (et-n b6, slia id-6-df yo-noss. eoceee ayping wit excal
commimato 24, qi aditt s2-40e 42-4 yo-400y) This tod dinetly plikiing cseal hos (1) would lea rallaa, au sheakd Sind menove the naighhoeiiigg (lgen. 193 [91 (17)
(1)) posttrmioease and di yii wow.ooyyyeeeetiiiii
Spanga ceasdaaae oct-i 1555
abia
waick
box [4] ceddinane (01-6.24,
unike removell [i]
Cllcanaatoon You
secocdd wlo
*Rleer thrnght The sarker (2) ofe way twil procoed with menasing the kayhoard [4].
Altbe unikee renovett) [e]
"Bleer "Rehurthuught thright: Billd to
kephoard wit dooldd apais
Altbe unikee venovett) [e]
"ileer thught 1 billdd to goog keppowrd wit dooldd epala
KRleeer maniow [d]
nilekrr mRehurthuught The sarki [3)edd the buyloadd sun the way t'd prooted as mennding coda (10).
KRleerr actice merioe [18)
Ritbe thrggtt thecmark keybward and wod ore (10) thow goooor ranurinn the spenge
"Bleber acticn mosion [17)
natbe thight: fiidd
spwege tn sieekl rpiix.
"Rleber sctloc monove [17)
picted
hlube thught the thormak koylwanlte,, (el, sedeven (10) tie sponge (1)) inceet of way siwe bee loged lyoval (4)) ansilnle bibeekgro aiic
Kileeer action pick (f)
(4)
ntube thight tneooemponni makk
Mieer acticntoooo dass
bbe groode
tigite
Ads of cpaoii
cptoeit Jolon!
8
Figure 2: System prompt and in-context examples for the LLM the SayCan-IM baseline. The input field task given by the user, obs contains the scene state or foedback from actions that is updated in each episode step, and episode_ apiiadeeiistory the previous observation--houuh--acc cycles in the current episode.
user_ user_inppal input corresponds to the

---

RGB
PolyFormer
CLIP-FGVP
GPT-4v OWG
letters
marker the yellow fruit
Iwant to something bealthy
Figure 3: Example output segmentation masks of different grounding methods for OCID scenes.

---

OCID Segmentation Mask comparisons
Grund-Buuth
Ground-yuth
SAN
XIU--PM
UKS
REB
Grund Tuth
SAN
VIDD 8P4
UD15
RCB
Grund-ruth
3-8PN
UOIS
Ground-butth
Figure 4: Comparison different segmentation models for seven example cluttered indoor scenes from OCID dataset
10

---

OCID Segmentation Mask comparisons
Grund-Buuth
Ground-yuth
SAN
XIU--PM
UKS
REB
Grund Tuth
SAN
VIDD 8P4
UD15
RCB
Grund-ruth
3-8PN
UOIS
Ground-butth
Figure 5: Comparison different segmentation models for seven example cluttered indoor scenes from OCID dataset

---

Object Brand Name
Object Attribute: Maturity
User: Feb package
User: non ripe pear
GPT-4v:
abject
deseription package,"
the ripe pear: will GPT-4v: two pears vouble on floor. desemmine
pears
image slantlly any packages cnuld potentially natch the descrittion. Upon exsmining the new image. unike package with the
image
ripe, I will look for visual cues such is color Typically, nen-ripe peir may have greener and timmer uppeorance comppred to ripe end, which would be yollow and possilly vollas looking. the image, spoors gremeer
tex "Fat" on This package located lowands the left side of the image standing upsigtt and appears to bea box of some wei. Nost, will look the unigue pazkage
pear the loft
marked
package image corresponds
secms moce yellow likely non-ripe pear, ripe. narked intage to fird the ID corroqonding the greemer paur In the mus kedi image, the greener pour on the left has the ID [4]. Therefore, this is the ID that curosponds the hro My
right the
package
be
the marked image with the ID [12] oa it. The package same position both images, confinning that this is the correct object.
My final answer in: [12).
Object Content Type
Object Attribute: Colors
User: mondites
great
GPT-4s: To find the object manches descrigtion "soodles." 'I fint
GPT-4v: To find the "green and red com flakes," will fist look the raw Bakes, green
examine
image idutill asy
modlles. Upas
image idantiy box peedoninanll
inspection, see a pookage with image that arpenes to repraemnt nootles
the left side of the table. Nost, will lock at the markel image find the some
rol elements, which munches deseription "green
flakes" This hos on the right sdde the image, nost to smother curcal bas with blue and yellow color scheme. New, will switch to the marked image to find the nol flakes highligheed the 19].
package and determine its unique ID. The package with. the moalls color associated with the My (5).
Highilghted
with the
My
Object Functionality
Object Attribute: Shape
User: rod soft thing the heir T-4:
User:
thing haie" I will Billow these steps: I. Ideatify any red objects the row image,
abjoci mutches doscription
GPT-4x: To find the object that matches description "cubo," will finst
look the raw image to identify my ebjoets that lave cobic shape. the mow be cube towards the right side the and
Determine which of these red objects could deseribed "sott" and used for the lair. the typically
image, there object that
there ral object with while ihat
slightly above
nusted
be
drying
sbject
equal-lengh edges square Now, imngge to the same abject with cornoponding ID. object with cubic slupe is the some location is highlighted with color and las the ID number next it. My final snower is
image, draped other itenss, the marked image to find the sunte object and its corresonndng ID. The red towel in the marked image has been highligheed with miqqe color and is labeled with the ID aumber 4.
center
look
final
Figure 6: Example GPT--vv responses (1/3): object-related queries
12

---

Relations: Relative Location wart Object
Relations: Absolute Location
GPT-4v: fnnd the "Sood box under the choco banana." 1 will firt ióently thocs-bmmen raw luage. clocc-latatio apgeas an image of banuna and chocolats un louate towarch the betumn right the Now. sill look for food bex that posiiiondd under the san image, is likely the "Fod box" mottioned la the descrigtion To End the comsquaainng
toodiusst product
"lefmos: toochpante product"" will Identify worhpeste products in the mv image. Looking the new image. can two ites that resemale upaste
GPII-v::
dissally cademnali
other lying, dlown right. Now, swiich the marked image find Lse coreepodiing IDs for these wollimmte spright, has
image, mne standlnng spright
will switch the murked
libdldd and look the Disecily undernotti cacsa-
under the
products. letiesst toothpase product which innthpacie protus hnghtligheed and neat [10].
choco harara bax,
banana bux with ID B. there anoteer box highiighed with difieeent color This food
labe
the choco
Finul [10].
My unswer ic:
Relations: Same Attribuee
Relations: Superlative Attribute
User: mug of same color the keyboard
Lleet: thitkkeet muricr
GPI--vv; to find the mug the same color the keybeard, will linz
'thckkes ken," will demtify markers the imauge. Theme markers visible: our with bikr and mer with cap appears thicker than the one with th groen cap. Now, will at the marked image to find ihr corrsqueling the thicker meker with the blle In the marked image, the thickeer
GPIT-v
identiy lhe color the keyboard the new imagee buybuard appears
white, Neat, will look mug thet is also white. ta the image, there are bayleadd.
green cap.
theue which matches color of keybounl. Nuw, will locate ID. In the markedd lmage, the white mag lighlighted with uniqu color lass mumber white mag has the [11] My final amowe ic [1]].
mags the
foe let
mug
the markee image to tind its
cop highlighted ID mumher nrse in The fie ker murker with the Nur cop assox associane laned with ID
Finsl anower [R]
Multi-Hop Reasoning: Counting
Multi-Hop Reasoning: Negation
User: teunth quariiall myyet from the
spintical ubject tuy" in ram image, will fist iieniity all the spherical ob and then count from the top the image
GPT-4v.
User: sharsgoo that is not standing upright 1-Av: in find the shimpon thar sinnching
first look the raw image identify any objects thet resemible shumpoo bottles the naw image, these two bottles lying
inage,
several spherical object which appear to frais and a ball. Couning these aijucii cum the the fire splanical chject founth yet orange
the wonal
ubice One standing uprigt, and the other marked image find dumppoo boull that is uat standing uprigls. The boxile lying on siile has a unique color and nameric ID next This ID [3].
the bull, Now, I vill look at the marked image fine the ID conesponding to the fourth spheiiei oiject from the which the
orangss
the marked imag, lis Upan
color examining the marked imaege I can we that the fourth sphaiical object Emomthe ID number My ansorri [1:].
ocange highligheed
My
the
Figure Example GPT-4v responses (2/3): relational reasoning queries
13

---

Semantic Reasoning: Utility
Semantic Reasoning: Material
User: sumething put ceneol In
User: something g crganically
in, looking lool similar comsiner that woold be approprint for holding conal. In the now the left side the table, bordd New, cenal
GPT-4v::
crange which grows orgarical y. Now. will look the markes image to find the some onagge and correpooding The unique volur and manerk next to May tinal arswer is: [3).
GPE-Av
imags,
Image.
inuge. It's placed
marked highlighted with
look or the markerd imagy in find the same borel and in commsqueelngg In. The same location the next
unique color and
nameric ID nect
My tinal answer is: [10).
User Affordance: State
User Affordance: Function
User: my hands are wet
User: I need a toy retease some stress
object wet. need idenetly an obsert that could be associated with wer hands, in the image, the
GPT-4v:
description "My
object commonly used for stess releff, such as stress ball or fidget tny; the ra imagr.
GFT-4v
toy
stress,
candiiare
abject issmbles
tovel dlthh that would use to dry their hands. Looking at the mw image. can see pink movlllkk nbyert oa the left safe of the mage. This Now. look at the marked image tind the same towei-ike object and curesponding ID. The pink towel-lie like objest the marked has die iD highlighted the towe--lke abjert is [14).
potentially be a toy used for stress reltef. located towarss the right side the mayg, near the evnne, and is ned with whith spors. Nev, will lonk at the sbject
aljici
that
dr
markeed linage Fins marctes the description and location in the raw image is highisghted with a unigue eoler and has the ID mumber parters highlighoed assoctated In' mimber 171.
connsquading
cijject that
la the marked the hull
objects
My final answer to: [10].
My ftnal
Figure 8: Example GPT-4v responses (3/3): semantics-related queries
14

---

User: canned food
laying on
image, will for object that resembles cas or container typically used for preserving food. the saw image, can saz box what appoins to be coscil on the right side, bui that conned food, However, there object that clerty resembles vaditional metal
GP
"canned
laying look sr the new image to iduntify the hox that resembiles corn flakes hos and positioned on its side. In the zaw image, there is box with what appears bottom right, and indeed loying imaae
food, there image.
flakes design marked image Next, will
description the marked image. Therefore, there no label to mutch description "armel food! these images.
that matches description highhligttd with color and las the mumber 2 next to it. My final answer is: [2).
unique
My final answerr is: these lobel ID for "cumal food" the peovided
withy
ge
Useen stapler nest the marker
User: blue and black marker
GPT--s:: To find the stapler standing the marker, will fins identify abjects
GPT-->: To find the "blue and black marker." fint look the image,
locate comespending black marked image. the raw image. can see a suppler the
image
mutch description, marker that appears a blue cup and black body, which locsted lowards the botton right of the ubll. Now. will swatch the marked image to rind the same marker with the corresponding ID.
image Keenify any the can
are severa
keyboard, and immediately the right of the supler is marker. New. will switch the marked image tind the DDs that correspond to these obsects. In stspler
the blue
black
marked
stapler and the murker is right nest to with LD (6). My final in (?)
(7),
amage unique
number
sdintified in the now image, and it has the mumber [12] next My final anserr is: (12).
Figure 9: Example of GPT-4v response failures.
15


Reviews:
[{"review_id": "sGnZByg265", "reviewer": "Reviewer_HEJP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a framework that utilises VLMs, more specifically LVLMs, to tackle the problem of \"robotic grasping\" in a setting where task instructions are open-ended language phrases. The paper argues that while LLMs are quite good at task-level abstract reasoning, they are weak at spatial and geometric reasoning, due to lack of direct access to visual information. Frameworks that propose to use VLMs such as CLiP or SAM in conjunction with LLMs, often suffer from bottlenecks presented by VLMs. LVLMs, on the other hand, are inherently trained on both vision and language data, and are hence very good at combining abstract task-level reasoning with spatial reasoning. Specifically, the framework consists of three stages, namely: \"open-ended referring segmentation\", \"Grounded grasp planning\", and \"Grasp ranking via contact reasoning\". In stage 1, building on prior work, the given input image is semantically grounded through labelling each object with a unque number overlayed on it. In stage 2, both the initial image, the grounded image, and the input text phrase are provided to the LVLM to produce a task plan as a sequence of objects that need to be removed or picked, in order to accomplish picking up the goal object. In stage 3, the first object proposed in the task plan is picked and removed/placed. These stages are repeated until the task is achieved.   Experimental study is provided to compare their method of prompting with prior works, showing an improved performance on the task of Open-Ended Grounding. They then compare their method with VLM-based baseline and an LLM based baseline.   Ablation study is shown to verify the importance of each component in their proposed framework.", "summary_of_recommendation": "Overall, this is good work and is among the first set of papers to tightly integrate LVLMs with grasping for zero-shot open-ended tasks. However, the experimental sections need improved presentation, such as fixing missing references to the tables and organizing the content for smoother conceptual flow.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I do not have any questions after thoroughly reviewing the paper, supplementary material, the video, and some of the related works. However, the paper's main material alone is likely to raise certain ambiguities and questions. I have shared my remarks on how these issues may be reduced in the 'Review' section above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LISDeLzkqD", "reviewer": "Reviewer_Adjp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The work presents a simple yet novel framework which enables zero-shot open-world grasping in a three-stage setting. The first stage involves open-world object referral using open-ended language queries, enabling the identification of target objects even with relational expressions to the target. The second stage focuses on planning a sequence of actions to clear obstructing objects, ensuring a clear path for grasping the target. In the third stage, the system ranks and selects the most promising grasp from multiple proposals. The framework's effectiveness is demonstrated both quantitatively and qualitatively, particularly in cluttered environments and semantic language queries beyond the object name, outperforming previous supervised and LLM-based methods in simulation and real-world experiments.", "summary_of_recommendation": "The work presents promising advancements in open-world grasping for robotics as demonstrated by its substaintial improvements over supervised and LLM based methods while avoiding the retraining overhead. However, to strengthen the paper, improvements in presentation and a more comprehensive inclusion of relevant literature are recommended. I was a bit torn on the recommendation and I am relying on the authors answers to maintain it or even improve it.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "### SayCan-IM Baseline\n\n  - What specific role does the large language model (LLM) play in the SayCan-IM baseline? Is its function limited to planning according to a predefined objective (distance to the target object center) and invoking CLIP, or does it have broader applications within the baseline?\n  - Given the context of open-ended language grounding, do the authors see the potential for a more nuanced use of pure LLMs to provide? how might this offer a fairer comparison against vision-language models (VLMs)? My main concern is that there is overlapping functionality between the instance segmentation model and the VLM that can incur extra computational costs and using pure LLMs efficiently might circumvent the functional redundancy.  \n\n### Real-World Experiment Reproducibility \n- **Replication Guarantees:** The setting of the real-world experiments appears difficult to replicate. Is there a mechanism, maybe the readily available instance segmentation mechanism that overlays the current scene masks and the prespecified ones and ensures consistency across all methods tested? This would help validate the experimental results and conclusions drawn.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OyqBoCtKSg", "reviewer": "Reviewer_hpr3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents Open World Grasper (OWG), a system that leverages modern vision-language models (VLMs) to enable robotic grasping in open-world scenarios based on open-ended language instructions. The system addresses two major limitations of previous approaches: the need for better visual grounding and the inclusion of low-level spatial understanding, which are critical for handling contact-rich environments. OWG combines VLMs with segmentation and grasp synthesis models to perform three main tasks: open-ended referring segmentation, grounded grasp planning, and grasp ranking through contact reasoning. This pipeline allows for zero-shot application, utilizing suitable visual prompting mechanisms to integrate high-level semantic reasoning with low-level physical-geometric insights. Extensive evaluations demonstrate OWG's robustness and superior performance compared to prior methods in both simulated and real-world robotic grasping experiments.", "summary_of_recommendation": "This paper doesn't have unique contributions and is very similar to lots of recent papers in this field, thus I would like to reject this paper.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "See above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LldvFWtU9n", "reviewer": "Reviewer_PGH6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method that integrates vision language models (e.g., GPT-4o) into the open-world robot grasping pipeline. Through the use of VLM, the grasping agent is capable of handling arbitrary user instructions and also effective under cluttered grasping scenarios. Specifically, the authors propose to use set-of-marks prompting and chain-of-thought prompting to generate grasping plans, especially in clutter scenarios where some objects might need to be removed first before the robot can grasp the target object. The authors also propose to use VLM to rank grasping proposals for better grasping qualities.", "summary_of_recommendation": "This paper provides some meaningful contributions on integrating VLM into the open-world grasping pipeline, but suffers from some notable weaknesses in methodology, experiments and writing. Edit: Most of them are addressed by authors' rebuttal, so I raised my score.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Please see \"Weaknesses and Questions\" in the review and address the raised issues in methodology, experiments, and writing.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Update:
The authors responded well to reviewer feedback. While there are still some lingering concerns about the paper novelty, I find the paper's execution and results convincing, and it does show a novel approach that leverages VLMs to synthesize low-level manipulation actions. As such I recommend accepting the paper.

Original:
While reviewers commend the zero-shot capabilities of the proposed method, they are concerned about the thoroughness of the technical explanations and experimental design. I ask the authors to carefully address all reviewer feedback during the rebuttal period.
Decision: Accept