Towards Open-World Grasping with Large Vision-Language Models

Authors: georgios  tziafas
hamidreza  kasaei
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:52:55

Abstract: The ability to grasp objects in-the-wild from open-ended language instructions constitutes a fundamental challenge in robotics.
An open-world grasping system should be able to combine high-level contextual with low-level physical-geometric reasoning in order to be applicable in arbitrary scenarios.
Recent works exploit the web-scale knowledge inherent in large language models (LLMs) to plan and reason in robotic context, but rely on external vision and action models to ground such knowledge into the environment and parameterize actuation.
This setup suffers from two major bottlenecks: a) the LLM's reasoning capacity is constrained by the quality of visual grounding, and b) LLMs do not contain low-level spatial understanding of the world, which is essential for grasping in contact-rich scenarios.
In this work we demonstrate that modern vision-language models (VLMs) are capable of tackling such limitations, as they are implicitly grounded and can jointly reason about semantics and geometry. 
We propose \texttt{OWG}, an open-world grasping pipeline that combines VLMs with segmentation and grasp synthesis models to unlock grounded world understanding in three stages: open-ended referring segmentation, grounded grasp planning and grasp ranking via contact reasoning, all of which can be applied zero-shot via suitable visual prompting mechanisms.
We conduct extensive evaluation in cluttered indoor scene datasets to showcase \texttt{OWG}'s robustness in grounding from open-ended language, as well as open-world robotic grasping experiments in both simulation and hardware that demonstrate superior performance compared to previous supervised and zero-shot LLM-based methods.

Main paper: 
 related works: 
 Visual Prompting for Vision-Language Models Several works investigate how to bypass finetuning VLMs, instead relying on overlaying visual/semantic information to the input frame, a practise commonly referred to as visual prompting. Colorful prompting tuning (CPT) is the first work that paints image regions with different colors and uses masked language models to "fill the blanks" [41] . Other methods try to use CLIP [42] by measuring the similarity between a visual prompt and a set of text concepts. RedCircle [43] draws a red circle on an image, forcing CLIP to focus on a specific region. FGVP [44] further enhances the prompt by specifically segmenting and highlighting target objects. Recent works explore visual prompting strategies for LVLMs such as GPT-4v, by drawing arrows and pointers [35] or highlighting object regions and overlaying numeric IDs [26] . In the same vein, in this work we prompt GPT-4v to reason about visual context while being grounded to specific spatial elements of the image, such as objects, regions and grasps.LLMs/LVLMs in Robotics Recent efforts use LLMs as an initialization for vision-language-action models [45, 46] , fine-tuned in robot demonstration data with auxiliary VQA tasks [46, 45, 47] . Such end-to-end approaches require prohibitive resources to reproduce, while still struggling to generalize out-of-distribution, due to the lack of large-scale demonstration datasets. Alternatively, modular approaches invest on the current capabilities of LLMs to decompose language instructions into a sequence of high-level robot skills [48, 11, 12, 22, 14] , or low-level Python programs composing external vision and action models as APIs [13, 23, 21, 22, 25, 49] . Such approaches mostly focus on the task planning problem, showcasing that the world knowledge built in LLMs enables zeroshot task decomposition, but require external modules [28, 29, 30, 42] to ground plan steps to the environment and reason about the scene. Recent works study the potential of LVLMs for inherently grounded task planning [27, 50, 51] . In [50] , the authors use GPT-4v to map videos of human performing tasks into symbolic plans, but do not consider it for downstream applications. VILA [27] feeds observation images with text prompts to an LVLM to plan without relying on external detectors. However, produced plans are expressed entirely in language and assume an already obtained skill library to execute the plans. MOKA [51] proposes a keypoint-based visual prompting scheme to parameterize low-level motions, but still relies on external vision models to perform grounding, and does not consider referring expressions and clutter.. In our work, we use visual marker prompting to leverage LVLMs for the full stack of the open-world grasping pipeline, including grounding referring expressions, task planning and low-level motion parameterization via grasp ranking.Semantics-informed Grasping Most research on grasping assumes golden grounding, i.e., the target object is already segmented from the input scene. Instead, they focus on proposing 4-DoF grasps from RGB-D views [52, 53, 54, 38, 55, 56, 57] , or 6-DoF poses from 3D data [58, 59, 60, 61, 62, 63, 64] . Recently, several works study language-guided grasping in an endto-end fashion, where a language model encodes the user instruction to provide conditioning for grasping [65, 66, 39] . However, related methods typically train language-conditioned graspers that struggle to generalize outside the training distribution. Another similar line of works is that of task-oriented grasping [67, 68] , where recent LLM-based methods [69] exploit the vast knowledge of LLMs to provide additional semantic context for selecting task-oriented grasps, but do not consider the grounding problem, clutter or referring expressions. Further, none of the above approaches consider the planning aspect, typically providing open-loop graspers that do not incorporate environment feedback. In this work, we leverage LVLMs to orchestrate a pipeline for language-guided grasping in clutter, exploiting it's multimodal nature to jointly ground, reason and plan. 
prerequisites and problem statement: 
 Large Vision-Language Models VLMs receive a set of RGB images of size H × W : I 1:M , I ∈ R H×W ×3 and a sequence of text tokens T , and generate a text sequence Y of length L: Y . = w 1:L = {w 1 , . . . , w L } from a fixed token vocabulary w i ∈ W, such that: Y = F(I 1:M , T ). The images-text pair input X = ⟨I 1:M , T ⟩ is referred to as the prompt, with the text component T typically being a user instruction or question that primes the VLM for a specific task.Grasp Representations We represent a grasp via an end-effector gripper pose G, with G ∈ R 4 for 4-DoF and G ∈ R 6 for 6-DoF grasping. Such representation contains a 3D position and either a yaw rotation or a full SO(3) orientation for 4-DoF and 6-DoF respectively. 4-DoF grasps assume that the approach vector is calibrated with the camera extrinsics, and hence can be directly drawn as rectangles in the 2D image plane (see bottom of Fig. 2 ), which happens to be a favorable representation for VLMs, as grasp candidates can be interpreted as part of the input image prompt. A motion primitive is invoked to move the arm to the desired gripper pose G, e.g. via inverse-kinematics solvers. CoT: Looking for a soup, potentially a canned item. Target object: [6] CoT: The marker [1] is blocking the can [6] . Plan: Problem Statement Given an RGB-D observation1. remove [1] 2. pick [6]I t ∈ R H×W ×3 , D t ∈ RH×W and an openended language query T , which conveys an instruction to grasp a target object, the goal of OWG is to provide a policy π(a t | I t , D t , T ). Assuming n ∈ {1, . . . , N } the N objects that appear in the scene and n * the target object, then at each time step t, the policy outputs a pose for grasping an object:a t = G t (n), G t (n) = G(n, I t , D t ), t = 1, . . . , T, where the last step T always maps to grasping the target object: a T = G T (n * ). We refer to the function G as the grasp generation function, which corresponds to a pretrained grasp synthesis network from RGB-D views [38] 2 We note that our policy π outputs directly the actual gripper pose G = G(n), and the object-centric abstraction n is used implicitly (details in next sections).We wish to highlight that in most grasp synthesis pipelines [38, 55, 53, 57, 56] , it's always T = 1 and a 1 = G 1 (n * ), which corresponds to an open-loop policy attempting to grasp the object of interest once. Our formulation for T > 1 allows the VLM to close the loop by re-running after each step, which enables visual feedback for planning and recovery from failures / external disturbances.OWG combines VLMs with pretrained 2D instance segmentation and grasp synthesis models. Segmentation methods like SAM [37] and its variants [70, 71] have demonstrated impressive zero-shot performane. Similarly, view-based grasp synthesis networks [56, 55, 38, 53, 57] have also shown to be transferable to unseen content, as they are trained without assumptions of objectness or semantics in their training objectives. The zero-shot capabilities of these models for low-level dense spatial tasks is complementary to the high-level semantic reasoning capabilities of VLMs, while both use images as the underlying representation, hence offering a very attractive coupling for tackling the open-world grasping problem. The overall pipeline can be decomposed in three subsequent stages:(i) open-ended referring segmentation, (ii) grounded grasp planning, and (iii) grasp generation and ranking. A schematic of OWG is shown in Fig. 2 and described formally in Algorithm 1. Prompt implementation details can be found in Appendix A.Open-ended referring segmentation In this stage, the target object of interest must be segmented from the input RGB image I t given the instruction T . To enable this, we first run our segmentation model S : R H×W ×3 → {0, 1} H×W and then draw the N generated masks M 1:N = S(I t ) with additional visual markers in a new frame I m t . This step aims to exploit the VLM's OCR capabilitiesUser: I broke my mug, can you get me something to fix it? GPT-4v: To find the object that best matches the description: "I broke my mug, can you get me something to fix it?", I will look for an item that could be used to repair a broken mug, which would most likely be some form of adhesive. In the first image, there are several items on the table, but the one that stands out as the potential adhesive is the tube near the bottom of the image. This tube could be a glue or a similar substance that could be used to fix a broken mug. In the second image, the tube is marked with the color blue and has the ID number 3 inside a black rectangle. My final answer is: To grasp [9] , it appears that objects [17] and [6] are directly breaking access to it. These object must be removed first to clear the way for a successful grasp of [9] . My final answer is: 1. remove [17] 2. remove [6] 3. pick [9] RPrompt: [...] Provide a list of the grasp IDs sorted from best to worst.The object of interest outlined with a red line appears to be a banana [...]. The grasp poses closest to surrounding object are [1] , [2] and [9] [...]. Good grasp poses are those whose centers are inside the body of the banana and not close to surrounding objects. These would likely be: [4] , [5] , [6] and [7] . The next best would be those that are inside the body but closer to the edges, which might be [3] and [8] . The least favorable grasp poses are the closest to surrounding objects or at the very ends of the banana, which are [1] , [2] and [9] . My final answer is: [4, 5, 6, 7, 3, 8, 1, 2, 9] and link each segment in the frame with a unique ID that the VLM can use to refer to it. After augmenting the image with visual markers, we pass the prompt < I t , I m t , T > to the VLM. We refer to this VLM generation as F ground , such that: n * = F ground (I t , I m t , T ) where n * the target object and M n * its segmentation mask. We note that T can contain free-form natural language referring to a target object, such as open object descriptions, object relations, affordances etc. Algorithm 1: Open-World Grasper (OWG)Require: Initial observation (I 1 , D 1 ), language instruc- tion T , segmentor S(•), grasp generator G(•), VLMs F ground , F plan , F rank Ensure: n * ̸ = ñ t ← 1 while n * ̸ = ñ do Generate segmentation masks M 1:N with S(I t ) Draw visual markers from M 1:N in a new frame I m t n * ← F ground (I t , I m t , T ) ▷ Object of interest ñ ← F plan (I m t , n * )[0] ▷ Next object to grasp G 1:K ← G(I t , D t , M ñ)▷ Grasp generation Crop RoI and draw grasps c ñ′ fromI t , M ñ, G 1:K G ′ 1:K ← F rank (c ñ′ ) ▷ Grasp ranking Execute grasp G ′ 1 t ← t + 1 ▷ Update observation I t , D t end whileGrounded grasp planning This stage attempts to leverage VLM's visual reasoning capabilities in order to produce a plan that maximizes the chances that the target object n * is graspable. If the target object is blocked by neighboring objects, the agent should remove them first by picking them an placing them in free tabletop space. Similar to [27] , we construct a text prompt that describes these two options (i.e., remove neighbor or pick target) as primitive actions for the VLM to compose plans from. We provide the marked image I m t together with the target object n * (from the previous grounding stage) to determine a plan: p 1:T = F plan (I m t , n * ), p τ ∈ {1, . . . , N }. Each p τ corresponds to the decision to grasp the object with marker ID n ∈ {1, . . . , N }. As motivated earlier, in order to close the loop, we take the target of the first step of the plan ñ = p 1 and move to the grasping stage of our pipeline.Grasp generation and ranking After determining the current object to grasp ñ, we invoke our grasp synthesis model G to generate grasp proposals. To that end, we element-wise multiply the mask M ñ with the RGB-D observation, thus isolating only object n * in the input frames:Ĩt = I t ⊙ M ñ, Dt = D t ⊙ M ñ.The grasp synthesis network outputs pixel-level quality, angle and width masks which can be directly transformed to 4-DoF grasps G 1:K = G( Ĩt , Dt ) [56, 55, 38] , where K the total number of grasp proposals. Then, we crop a small region of interest c ñ around the bounding box of the segment in the frame I t , from its mask M ñ. We draw the grasp proposals G 1:K as 2D grasp rectangles within the cropped image c ñ and annotate each one with a numeric ID marker, similar to the grounding prompt. We refer to the marked cropped frame as c ′ ñ. Then, we prompt the VLM to rank the drawn grasp proposals:G ′ 1:K = F rank (c ′ ñ)where the prompt instructs the VLM to rank based on each grasp's potential contacts with neighboring objects. Finally, the grasp ranked best by the VLM G ′ 1 is selected and sent to our motion primitive for robot execution. 
experiments: 
 In Table 1 : Zero-shot referring segmentation -mIoU(%) results per language instruction type for cluttered indoor scenes from OCID [40] .In order to evaluate the open-ended potential of OWG for grounding, we create a small subset of OCID-VLG test split [39] , which we manually annotate for a broad range of grasping instructions. As we strive for zero-shot usage in open scenes, we mostly experiment with previous visual prompting techniques for large-scale VLMs, such as CLIP [43, 44, 72] , as well as the recent Set-of-Mark prompting methodology for GPT-4v [26] , which constitutes the basis of our method. We also include comparisons with open-source visually-grounded LVLM QWEN-VL-2 [31] . Please see Appendix C for details on the test dataset, baseline implementations and more comparative ablations and qualitative results.We observe that both CLIP-based visual prompting techniques and open-source LVLMs are decent in object-based but fail to relate objects from the visual prompts. Even GPT-4v-based SoM prompting method is not directly capable of handling cluttered tabletop scenes from depth cameras, as is evident by the 53.1% averaged mIoU across all query types. Overall, our OWG-grounder achieves an averaged mIoU score of 80.8%, which corresponds to a 27.7% delta from the second best approach. Importantly, OWG excels at semantic and affordance-based queries, something which is essential in human-robot interaction applications but is missing from modern vision-language models. We identify two basic failure modes: a) the LVLM confused the target description with another object, e.g. due to same appearance or semantics, and b) the LVLM reasons correctly about the object and where it is roughly located, but chooses a wrong numeric ID to refer to it. In this section we wish to evaluate the full stack of OWG, incl. grounding, grasp planning and grasp ranking via contact reasoning, in scenarios that emulate open-world grasping challenges. To that end, we conduct experiments in both simulation and in hardware, where in each trial we randomly place 5-15 objects in a tabletop and instruct the robot to grasp an object of interest. We conduct trials in two scenarios, namely: a) isolated, where all objects are scattered across the tabletop, b) cluttered, where objects are tightly packed together leading to occlusions and rich contacts. We highlight that object-related query trials contain distractor objects that share the same category with the target object.Baselines We compare with two baselines, namely: a) CROG [39] , an end-to-end referring grasp synthesis model trained in OCID [40] scenes, and b) SayCan-IM [12] , an LLM-based zero-shot planning method that actualizes embodied reasoning via chaining external modules for segmentation, grounding and grasp synthesis, while reasoning with LLM chain-of-thoughts [74] . Our choice of baselines aims at showing the advantages of using an Implementation Our robot setup consists of two UR5e arms with Robotiq 2F-140 parallel jaw grippers and an ASUS Xtion depth camera. We conduct 50 trials per scenario in the Gazebo simulator [75] , using 30 unique object models. For real robot experiments, we conduct 6 trials per scenario having the initial scenes as similar as possible between baselines. In both SayCan-IM and our method, Mask-RCNN [76] is utilized for 2D instance segmentation while GR-ConvNet [38] pretrained in Jacquard [52] is used as the grasp synthesis module. Our robotic setup is illustrated in Fig. 4 , while more details can be found in Appendix B. To investigate generalization performance, all method are evaluated in both scenarios, in two splits: (i) seen, where target objects and queries are present in the method's training data or in-context prompts, and (ii) unseen, where the instruction refers to objects that do not appear in CROG's training data or SayCan-IM's in-context prompts. Averaged success rate per scenario is reported, where a trial is considered successful if the robot grasps the object and places it in a pre-defined container position. We observe that the supervised method CROG struggles when used at unseen data, in both scenarios. In contrary, both SayCan-IM and OWG demonstrate immunity to seen/unseen objects, illustrating the strong zero-shot capabilities of LLM-based approaches, which can naturally generalize the concepts of object categories/attributes/relations from language. SayCan-IM is limited by the external vision models and hence struggles in cluttered scenes, where its detector sometimes fails to perceive the target object, resulting in lower final success rates compared to OWG, especially in the realworld experiments. OWG consistently outperforms both baselines both in simulation and in the real robot, with an ∼ 15% and ∼ 35% improved averaged success rate respectively. In Fig. 5 , we illustrate the decomposition of failures across grounding and grasping in our baselines for 25 Gazebo trials per scenario, where we automatically test for the target object's grounding results alongside success rate. We observe that OWG consistently reduces the error rates in both grasping and grasping compared to the baselines in all scenarios and test splits. We believe that these results are encouraging for the future of LVLMs in robot grasping.In out ablations we wish to answer the following questions: a) What is the bottleneck introduced by the segmentation model in the open-ended grounding performance?, b) What are the contributions of all the different visual prompt elements considered in our work?, and c) What is the contribution of the LVLM-based grasp planning and ranking in robot grasping experiments? The grounding ablations for the first two questions are organized in Table 3 , while for the latter in Table 4 .We compare the averaged mIoU of our OWG grounder in a subset of our OCID-VLG evaluation data for three different segmentation methods and ground-truth masks. We employ: a) SAM [37] , b) the RPN module of the open-vocabulary detector ViLD [28] , and c) the RGB-D two-stage instance segmentation method UOIS [77] , where we also provide the depth data as part of the input. ViLD-RPN and UOIS both achieve a bit above 70%, which is a ∼ 15% delta from ground-truth masks, while SAM offers the best baseline with a 10.8% delta from ground-truth. Implementation details and related visualizations in Appendix C. We quantify the contribution of our grasp planning and ranking stages in the openworld grasping pipeline, by replicating trials as in the previous section and potentially skipping one or both of these stages. As we see in Table 4 , the effect of these components is not so apparent in isolated scenes, as objects are not obstructed by surroundings and hence most proposed grasps are feasible. The effect becomes more prominent in the cluttered scenario, where the lack of grasp planning leads to a success rate decrease of 22%. This is because without grasp planning the agent attempts to grasp the target immediately, which almost always leads to a collision that makes the grasp fail. Grasp ranking is less essential, as a lot of contact-related information is existent in the grasp quality predictions of our grasp synthesis network. However, it still provides an important boost in final success rate (8% increase). When skipping both stages, the agent's performance drops drastically in cluttered scenes, as it is unable to recover from grasp failures, and hence always fails when the first attempted grasp was not successful. 
conclusion, limitations & future work: 
 In this paper we introduce OWG, a novel system formulation for tackling open-world grasping. Our focus is on combining LVLMs with segmentation and grasp synthesis models, and visually prompt the LVLM to ground, plan and reason about the scene and the object grasps. Our works sets a foundation for enabling robots to ground open-ended language input and close-the-loop for effective grasp planning and contact reasoning, leading to significant improvements over previous zero-shot approaches, as demonstrated by empirical evaluations, ablation studies and robot experiments.Limitations First, as OWG is a modular approach, it suffers from error cascading effects introduced by the segmentor and grasp synthesis models. However, improvements in these areas mean direct improvement to the OWG pipeline. Second, we currently use 4-DoF grasps to communicate them visually to GPT-4v, which constrains grasping to single view. In the future we would like to integrate 6-DoF grasp detectors and explore new prompting schemes to aggregate and rank grasp information visually. Third, our results suggest that LVLMs still struggle to ground complex object relationships. More sophisticated prompting schemes beyond marker overlaying, or instruct-tuning in grasp-related data, might be a future direction for dealing with this limitation. 
: 
 Following grasping instructions from free-form natural language in open-ended environments is a multi-faceted problem, posing several challenges to robot agents. Consider the example of Fig. 1: The robot has to decipher the semantics of the user instruction (i.e., "what would a child want to play with?"), recognize the appearing objects and ground the target (i.e., the white toy), reason about the feasibility of the grasp to generate an appropriate plan (i.e., first remove the blocking juice box), and finally select a suitable grasp based on the object geometry and potential collisions. It becomes clear that to deal with the full scope of open-world grasping, agents should integrate high-level semantic with low-level physical-geometric reasoning, while doing so in a generalizable fashion.In recent years, Large Language Models (LLMs) [1, 2, 3, 4, 5] , have emerged as a new paradigm in robotics and embodied AI, due to their emergent general knowledge, commonsense reasoning and semantic understanding of the world [6, 7, 8, 9, 10] . This has led to a multitude of LLM-based approaches for zero-shot robotic task planning [11, 12, 13, 14, 15] , navigation [16, 17, 18, 19] and manipulation [20, 21, 22, 23, 24, 25] , where the LLM decomposes a high-level language instruction into a sequence of steps, therefore tackling complex, long-horizon tasks by composing primitive skills. However, a notorious limitation of LLMs is their lack of world grounding -they cannot directly reason about the agent and environment physical state [26] , and lack deep knowledge when it comes to low-level, physical properties, such as object shapes, precise 3D geometry, contact physics and embodiment constraints [27] . Even when equipped with external visual modules for perceiving the world, the amount of information accessed by the LLM is bottlenecked by the visual model's interface (e.g. open-vocabulary detectors [28, 29, 30] cannot reason about object relations such as contacts). Recently, Large Vision-Language Models (LVLMs) integrate visual understanding and language generation into a unified stream, allowing direct incorporation of perceptual information into the semantic knowledge acquired from language [31, 32, 33, 34] . Preliminary explorations with LVLMs [35] have illustrated two intriguing phenomena, namely: a) by combining LVLMs with segmentation models and constructing suitable visual prompts, LVLMs can unleash extraordinary open-ended visual grounding capabilities [26] , and b) effective prompting strategies like chain-of-thought [36] and in-context examples [1] seem to also emerge in LVLMs. Motivated by these results, we perform an in-depth study of the potential contributions of LVLMs in open-ended robotic grasping. In this paper, we propose Open World Grasper (OWG): an integrated approach that is applicable zero-shot for grasping in open-ended environments, object catalogs and language instructions. OWG combines LVLMs with segmentation [37] and grasp synthesis models [38] , which supplement the LVLM's semantic knowledge with low-level dense spatial inference. OWG decomposes the task in three stages: open-ended referring segmentation, where the target object is grounded from open-ended language, (ii) grounded grasp planning, where the agent reasons about the feasibility of grasping the target and proposes a next action, and (iii) grasp ranking, where the LVLM ranks grasp proposals generated from the grasp synthesizer based on potential contacts. In summary, our contributions are threefold: a) we propose a novel algorithm for grasping from open-ended language using LVLMs, b) we conduct extensive comparisons and ablation studies in real cluttered indoor scenes data [39, 40] , where we show that our prompting strategies enable LVLMs to ground arbitrary natural language queries, such as open-vocabulary object descriptions, referring expressions and user-affordances, while outperforming previous zero-shot vision-language models by a significant margin, and c) we integrate OWG with a robot framework and conduct experiments both in simulation and in the real world, where we illustrate that LVLMs can advance the performance of zero-shot approaches in the open-world setup. 
 
 Appendix: A LVLM promptsPrompts for the three use cases considered in this work, namely: open-ended referring segmentation, grounded grasp planning, and grasp ranking can be found below.• Open-ended referring segmentation: referring_segmentation.txt Takes an observation image, a marked copy with highlighted instance masks and label IDs and an open-ended language query referring to a particular object instance, and outputs the label ID of the referred object. The LVLM is encouraged to provide chains-of-thought reasoning in cases where the input query contains complex expressions that involve multiple object and spatial relations.• Grounded grasp planning: grasp_planning.txt Takes the marked image and the label ID of the target object to be grasped, and outputs a plan to ensure the target object will be graspable. The plan consists of remove actions for blocking objects and a final pick action for the target.• Grasp ranking: grasp_ranking.txt Takes a cropped bounding box image around the next object to be picked, marked with grasp proposals from a 4-DoF grasp synthesis model and a set of label IDs, and outputs a sorted list of grasp IDs, from most to least confidence for a successful grasp. The LVLMs is encouraged to reason about the object shape and their neighbouring objects before producing a final ranking.In the following, we summarize the key visual prompting elements that were used for prompting the LVLM in the context of OWG:Clarity of visual markers The most common failure mode of visual marker prompting with GPT-4v is that it sometimes struggles to discriminate which ID corresponds to what segment. Especially in cluttered scenes, label IDs might severely overlap within small frame regions. Several techniques can assist in making the markers more clear to the LVLM: a) we adopt the algorithm of [1] for overlaying numeric IDs within the frame with minimal overlap, b) we paint both the internal of each segment's mask and its ID with the same unique color (colors are chosen to be visually distinguishable), and c) increase the resolution of the marked image and the size layout of the markers.Reference Image If not highlighting the internal of each segment, GPT-4v sometimes refers to regions with wrong IDs, especially in highly cluttered scenes. But if the masks are highlighted with high opacity, then the appearance of the object becomes less visible and GPT-4v struggles to recognize it. We propose a technique to ameliorate this is by passing both the original (reference) and the marked image and constructing a text prompt that explains that the latter corresponds to annotated segments of the first.8th Conference on Robot Learning (CoRL 2024), Munich, Germany.Chain-of-thoughts Chain-of-Thought (CoT) prompting is a well-established methodology for guiding LLMs to perform multi-step reasoning and reduce hallucinations [2] . We find that LVLMs share similar properties and prompting them to reason about their final answer before producing it can robustify the response quality. For grounding, we ask GPT-4v to decompose the input instruction in steps and refer to all intermediate referenced objects. For grasp planning, we ask it to explicitly mention all object IDs that are blocking the target object, before producing a plan. For grasp ranking, we decompose the prompt in three steps: (i) identify the category of the target object and provide a general description of what constitutes a good grasp for it given its shape, (ii) list the grasp IDs that will most likely lead to contact with neighboring objects, and (iii) rank the grasp IDs based on the previous two steps.Self-consistency Even with zero temperature, we observe that the outputs of GPT-4v are not always reproducible. We find that sometimes GPT-4v might produce different responses at different runs, even with exactly the same prompt. In an attempt to reduce the effect of this phenomenon and robustify LVLM outputs, we use the self-consistency method developed for LLMs [3] . In particular, we ask GPT-4v to provide multiple responses, parse each one separately and then perform majority voting to determine the most consistent output.Our object catalog for seen/unseen trials is shown in Fig. 1 . In Gazebo, isolated scenarios are generated by ensuring all spawned objects have a fixed 3D distance, while in cluttered scenarios we ensure contact between the target object and neighbouring objects, by first spawning the target and then sampling different poses for other object models around it. In real-robot experiments, we manually setup the scenes while making sure to replicate the setup as close as possible for fair comparisons between baselines. In all trial scenes that contain distractor objects, the user instruction refers to some property that disambiguates the target instance from other objects of the same category, using names, attributes and spatial relations. We also conduct experiments without distractors for affordance-based queries, which require semantic reasoning to be correctly grounded. For real robot experiments, we use the default torchvision implementation of Mask-RCNN, with the model weights provided by PyTorch Hub, finetuned in a few annotated scenes captures from our robot setup. For grasp synthesis, we generate a topdown orthographic projection of the scene, both for color and for depth (i.e. reverse depth -heightmap). This is the input we pass to the pretrained GR-ConvNet. In order to align regions from the 2D frame where Mask-RCNN provides segmentations and the orthograpic projection where our grasp synthesis model provides grasp poses, we use the Hungarian matching algorithm to match the centers of outputs from both models, after projected to 3D and transformed to a world reference frame (robot base), using 3D euclidean distance as the cost function.CROG CROG receives an single 448 × 448 RGB view and a natural language query, and provides both an instance segmentation mask for the target object, as well as a set of 4-DoF grasp proposals, assuming that the gripper approaches the object aligned with the perspective of the camera. We use the checkpoint provided by the original paper, trained in the multiple split of OCID-VLG dataset, which contains 90k scene-query-grasp data from around 1,000 unique scenes from 31 SayCan-IM Our SayCan-IM baseline follows the implementation publicly released by the SayCan work [4] , which can be found in this HTTP URL. In particular, the pipeline uses the ViLD [5] open-vocab object detector to turn the input observation image into a list of object names and then lets the LLM generate a sequence of pick-and-place actions to perform in order to solve the task given by the user. We made the following modifications to the above baseline:1. In the original implementation, the robot only has access to a pick_and_place skill, and the output plan is confined to only selecting what objects to pick and where to place them (based on the detected object list from ViLD). In our implementation, we also provide a visual_grounding tool, which lets the LLM invoke CLIP [6] to rank a list of candidate objects with a given text description and select the most similar one. This is to allow the LLM reason about attribute concepts besides object category (e.g. "get the blue mug").2. Besides the names of the appearing objects, we also provide their bounding box coordinates, as detected by ViLD, in x1y1x2y2 format in the prompt. This was introduced in order to enable the LLM to also reason about the locations of objects and resolve spatial relation queries, as well as reason about the feasibility of grasping objects by checking whether their bounding boxes overlap.3. We replace the pick_and_place primitive skill with two distinct skills: remove and pick.The first skill corresponds to removing a blocking object in order to clear the path for grasping the target. The second skill corresponds to picking the target object that the user requested. Both skills use GR-ConvNet [7] under-the-hood to sample grasp proposals, select the one with higher predicted grasp quality, and use an IK solver to control the robot arm.4. We used the observe-reason-act prompting style first introduced by Inner Monologue [8] and later improved by ReAct [9] . Unlike the vanilla implementation, which simply produces a plan of steps without feedback, with this technique we let the LLM plan one step at a time, and integrate feedback from the environment (e.g. CLIP outputs, grasp failures etc.) before planning again.The system prompt and in-context examples used in our SayCan-IM baseline are shown in Fig. 2 . As we mention in our main paper, for the real robot experiments, we replace ViLD-RPN with a Mask-RCNN [10] for instance segmentation, and use CLIP with prompts for all object used in experiments to recognize the categories and provide the object list state to the LLM.We manually annotate 173 images from OCID dataset with the following query types: a) name (open-vocabulary object descriptions), b) attribute, c) spatial relations, d) visual relations, e) semantic relations, f), multi-hop reasoning, and g) user-affordances. The number of annotations per query type given in Table 1 . We make sure to include unique test scenes from the dataset and include images with heavy clutter. The target of each scene within a query type is unique, and we make sure to include images with distractor objects (of the same category as the target) for all query types that require relational reasoning (all except name and affordance).Regarding our custom FGVP-CLIP baseline (FGVP * ), we present analytical comparisons and ablation in the following subsection.We utilize the provided demo applications for the end-to-end methods (SEEM, PolyFormer) to conduct grounding experiments manually. For CLIP-based baselines, we re-implement all methods from the corresponding papers (ReCLIP, RedCircle, FGVP) . We use the ViT-B visual encoder to extract features from image segments and the default BERT text encoder to represent the input query. CLIP-based baselines compute the cosine similarity between segment and text features to rank them and select the most similar segment as the final result via the argmax operator. Ground-truth masks are used for all CLIP-based baselines, similar to GPT-4v ones. We would like to highlight that in the original papers, the aforementioned baselines use potential post-processing steps to enhance the grounding capabilities of CLIP. In particular, ReCLIP uses syntactic parsing to extract entity and relation words/phrases from the input query, as well as spatial relation resolution heuristics (e.g. 'left', 'on' etc. -designed specifically for the RefCOCO dataset) to process the relations analytically and combine CLIP predictions only for the entities. RedCircle and FGVP additionally utilize a "subtraction" post-processing step, where they further subtract from the similarity values the average in a set of mined hard-negative queries (again selected for a specific dataset). We believe that such steps constitute domain-aware hand-crafted efforts, which even though helpful, do not represent the challenges of open-ended generalization, which is the primary focus of this work. As a result, we do not consider such post-processing steps in our baseline implementation.Comparisons with end-to-end approaches The need for manual annotations to exhaust all possible language query inputs, as well as the need for manual testing via online demo applications for the considered specialist end-to-end methods (SEEM, PolyFormer) restrained us from conducting experiments in large-scale. Instead, we originally conducted experiments in a smaller subset of 52 images. Results are given in Table 3 . Results follow similar patterns to the larger test set of the main paper. Specialist models (SEEM, PolyFormer) struggle with even simple name queries, scoring below 15% on average. This is potentially due to the high discrepancy between the training distribution of RefCOCO and Visual Genome and our test data, as well as the lack of relational and affordance-based language in these datasets. GPT-4v-based methods still compare favourably to CLIP-based baselines, even in the SoM setting where single marked image is used. Overall, our OWG-grounder achieves an averaged mIoU score of 70.4%, which is almost ×2 from the previous approach.To further analyze the performance of CLIP-based baselines, we conduct ablation studies where we use specific elements of each method. In particular, we study: a) effect of using multi-templates for the text prompt, where we average text embeddings from multiple versions of the query, using templates from the original paper, b) averaging similarity scores from the visual prompt and crops of each segment, as originally proposed in ReCLIP, c) different visual prompt schemes, like drawing a boundary (rectangle or ellipse -as in RedCircle), converting to grayscale or blurring the rest of the frame (as proposed in FGVP), as well as a prompt that we discover ourselves works good, using a white background for the rest of the frame. We note that in our paper's results the element combinations we used are the following: ReCLIP: rectangle prompt, multi-templates, blur-reverse + crop, RedCircle: ellipse prompt, multi-templates, gray-reverse + blur-reverse, FGVP: mask prompt, multi-templates, gray-reverse + blur-reverse Ablation results are shown in Table 2 . Our findings are the following: 1) drawing a rectangle prompt outperforms ellipse and mask (object contours) in itself, but ensembling rectangles and masks gives the best result, 2) using multiple text templates outperforms single-template only when ensembling multiple visual inputs, c) the most effective component is our method of replacing the rest of the frame with white background, compared to grayscale and reverse operators of FGVP, while ensembling all together gives the best performance. We call our custom FGVP baseline FGVP * . We present analytical results per query type for CLIP-based baselines versus GPT-4v methods, as in the original paper, for our extended evaluation dataset in Table ??. FGVP * represents the best configuration of CLIP-based visual prompting as found by our ablation experiments. Results follow similar patterns to the smaller subset of the main paper, with a significant performance boost for CLIP-based baselines. However, GPT-4v-based methods still compare favourably to CLIP-based baselines, even in the SoM setting where single marked image is used. Our OWG visual prompt scheme dramatically outperforms all baselines, with a margin of 27.7% from SoM and 29.0% from the best found CLIP visual prompt methodology, showcasing its superiority in cluttered indoor scenes context as in OCID.We use the checkpoints provided by the authors for UOIS [11] unseen object instance segmentation, as well as the ViRL-RPN checkpoint and hyper-params from the implementation in this HTTP URL. For SAM, we use the ViT-L variant of the released SAM [12] checkpoints, and search for optimal hyper-parameters for automatic mask generator, resulting in the following configuration: points_per_side=24, pred_iou_thresh=0.92, stability_score_thresh=0.95. We apply non-maximum suppression with an iou_threshold=0.5 and remove nested masks, i.e. masks that are completely inside other masks of higher score threshold. This step aids in keeping only object-level SAM predictions and decreasing the over-segmentation behavior that default SAM provided in our first implementation. In turn, this leads to less cluttered visual markers for our OWG grounding module. Example instance segmentation masks for the different methods are illustrated in Fig. 5 .In Figs. 6, 7, 8, we provide example responses for grounding different types of language queries in OCID scenes. We observed that GPT-4v, augmented with marked image prompting, can ground not just object-related queries but also complex referring expressions that require reasoning about space, visual attributes, semantics and user-affordances. Interestingly, we find that GPT-4v responds to queries that require symbolic reasoning concepts such as counting and negation, which are notoriously hard to emerge in specialist grounding models. In Fig. 9 , we provide some example responses corresponding to failure cases. Main failure modes include: a) grounding a distractor instead of the desired object, b) not finding the object of interest at all, c) providing a correct reasoning and identifying the target in the raw image, but providing a wrong ID of an irrelevant object. 8th Conference on Robot Learning (CoRL 2024), Munich, Germany.More sophisticated motion planning algorithms, e.g. with integrated obstacle avoidance, can be utilized orthogonal to our approach.Other point-cloud[59] or voxel-based[62] methods for3D grasp generation can be utilized orthogonal to our approach, which uses single RGB-D view. 


Decision: Update:
The authors responded well to reviewer feedback. While there are still some lingering concerns about the paper novelty, I find the paper's execution and results convincing, and it does show a novel approach that leverages VLMs to synthesize low-level manipulation actions. As such I recommend accepting the paper.

Original:
While reviewers commend the zero-shot capabilities of the proposed method, they are concerned about the thoroughness of the technical explanations and experimental design. I ask the authors to carefully address all reviewer feedback during the rebuttal period.

Reviews: review: title: title: Experiments section needs minor revision
paper_summary: paper_summary: The authors propose a framework that utilises VLMs, more specifically LVLMs, to tackle the problem of "robotic grasping" in a setting where task instructions are open-ended language phrases. The paper argues that while LLMs are quite good at task-level abstract reasoning, they are weak at spatial and geometric reasoning, due to lack of direct access to visual information. Frameworks that propose to use VLMs such as CLiP or SAM in conjunction with LLMs, often suffer from bottlenecks presented by VLMs. LVLMs, on the other hand, are inherently trained on both vision and language data, and are hence very good at combining abstract task-level reasoning with spatial reasoning. Specifically, the framework consists of three stages, namely: "open-ended referring segmentation", "Grounded grasp planning", and "Grasp ranking via contact reasoning". In stage 1, building on prior work, the given input image is semantically grounded through labelling each object with a unque number overlayed on it. In stage 2, both the initial image, the grounded image, and the input text phrase are provided to the LVLM to produce a task plan as a sequence of objects that need to be removed or picked, in order to accomplish picking up the goal object. In stage 3, the first object proposed in the task plan is picked and removed/placed. These stages are repeated until the task is achieved.   Experimental study is provided to compare their method of prompting with prior works, showing an improved performance on the task of Open-Ended Grounding. They then compare their method with VLM-based baseline and an LLM based baseline.   Ablation study is shown to verify the importance of each component in their proposed framework.
main_review: main_review: # Clarity

Most sections of the paper are well written. Following remarks might help improve the paper's presentation further:
- Notation consistency: In abstract and related works, LVLMs and VLMs are treated differently. For example, CLIP is treated as VLM, while GPT-4v is treated as an LVLM. In the method section, the authors were supposed to use the term "LVLM" to continue this consistency. However, the term "VLM" has been used to indicate an "LVLM". 
- Line number 189: It must be $\hat{n}$ not $n^*$
- In the experiments section (section 4.1 and 4.2), main content did not refer to the tables present in the paper. For example, when starting the paragraph at line number 219, it would be nice to start with saying that the results for this section are plotted on table 1, and then continue discussing about various points that depend on this table. Same for section 4.2, where table 2 needs to be referenced. 
- In section 4.1, the term "OWG-Grounder" is mentioned for the first time. It would be good to introduce the term while discussing the method in section 3.2 "Open Ended Referring Segmentation" paragraph.
- Ambiguity in the term "scenario". Although the video clearly explains it, It would good to mention what a "scenario" is, at the beginning of the experiments section, at least once. 
- Section 4.2 "Results": In line number 288, "grasping and grasping", you may wanted to say "grounding and grasping"
- Providing a brief introduction to the OCID-VLG dataset in section 4.1 would be a good idea. 
- In section 4.1, line 226, the statement " We identify two basic failure modes: a) the LVLM confused the target description with another object, e.g. due to same appearance or semantics, and b) the LVLM reasons correctly about the object and where it is roughly located, but chooses a wrong numeric ID to refer to it." sounds speculative, as it is written as a standalone statement with no reference to specific experimental data or a citation to a prior work. If you are presenting it as a qualitative observation, it would be nice to add a phrase that it is a qualitative observation. 

# Quality
The quality of the paper is good. Abstract, Introduction, Related Works, and Method sections are well presented. Experiments section may need minor revision, in terms of table references, brief introduction to datasets, baselines and their relevance. 

# Originality
The presented work brings together known approaches, and presents improved prompting method to a different problem. Along the way, ablation studies shown in the paper help the community understand the implications of various components in the pipeline and how they affect the overall performance. 

# Relevance
This paper is quite relevant to grasping, and LVLM (Large Vision Language Models) space, showcasing:
- How to tightly integrate LVLM for grasping and bring in closed-loop framework for improved performance on cluttered scenes. 
- Some good insights from pure-vision and VLM papers and their application to robotics. 

# Limitations
Limitations section is well covered.
questions: questions: I do not have any questions after thoroughly reviewing the paper, supplementary material, the video, and some of the related works. However, the paper's main material alone is likely to raise certain ambiguities and questions. I have shared my remarks on how these issues may be reduced in the 'Review' section above.
limitations: limitations: 3
review_summary: review_summary: Overall, this is good work and is among the first set of papers to tightly integrate LVLMs with grasping for zero-shot open-ended tasks. However, the experimental sections need improved presentation, such as fixing missing references to the tables and organizing the content for smoother conceptual flow.
title: title: Reflection on Zero-shot OWG using Foundation Models
paper_summary: paper_summary: The work presents a simple yet novel framework which enables zero-shot open-world grasping in a three-stage setting. The first stage involves open-world object referral using open-ended language queries, enabling the identification of target objects even with relational expressions to the target. The second stage focuses on planning a sequence of actions to clear obstructing objects, ensuring a clear path for grasping the target. In the third stage, the system ranks and selects the most promising grasp from multiple proposals. The framework's effectiveness is demonstrated both quantitatively and qualitatively, particularly in cluttered environments and semantic language queries beyond the object name, outperforming previous supervised and LLM-based methods in simulation and real-world experiments.
main_review: main_review: ## Strengths

- **Zero-shot Performance:** The framework achieves impressive zero-shot capabilities in open-world grasping without requiring retraining or fine-tuning compared to prior methods whether supervised (CROG) or zero-shot (SayCan-IM).
- **Compositional Design:** The modular approach allows for better interpretability and analysis of errors at each stage.
- **Ablations, Benchmarking and Evaluation:** Evaluations are properly conducted, particularly evident in the diverse range of expressions used, such as attributes, spatial relations, and visual relations, to assess the initial stage in the framework for target referral. Additionally, the real-world experiments strengthen the position of the full framework especially in cluttered scenes compared to other methods.

- **Failure Recovery Mechanism:** The framework's grasp planner helps recover from failures and straightforwardly enhances the utility.

## Weaknesses

### Clarity Issues
- **Explanation of SayCan-IM Baseline:** The explanation of the SayCan-IM baseline, which contrasts the use of pure language models (LLMs) with external perception modules against the proposed framework's vision-language model (VLM) in open-world grasping, lacks clarity. The supplementary material could benefit from clearer explanations,  in-context prompts examples and additional references.
- **Appendix E Reference:** The paper references Appendix E for in-context examples, but this section does not appear in the supplementary material (Section A1.1).   
Could the authors clarify whether this is a misreference or if the content is missing?

 - **Self-Consistency:** The concept of self-consistency is mentioned in the main paper without a clear reference to an appropriate section in the appendix. Could the authors specify where this discussion can be found or provide a brief explanation?
 - **Duplicate Content:** There are two nearly identical paragraphs discussing in-context examples in the supplementary material (Section A1.1). Is this redundancy intentional, or should these sections be revised for clarity?
- **OWM 2D instance segmentation model:** The authors perform an ablation study on the OWG grounder performance on a curated test set from OCID dataset. The results show that the best 2D instance segmentation model is the UOIS (Xie et. al 2021), but still the main figure mentions SAM despite poor performance. For real-world experiments, authors are clear on the 2D instance segmentation model (Mask-RCNN) as stated "261 In both
SayCan-IM and our method, Mask-RCNN [65]
is utilized for 2D instance segmentation... " 
  * Is this the 2D instance segmentation model (Mask-RCNN) for the grounding experiments too?
  * Does this explain the discrepancy between the results in Table 1 and Table 3? I cannot see a matching mIOU performance between the two tables. More clarity on this issue is important. 

### Related Work and Comparison
- **Task-oriented grasping methods:** The scope of this work is closely related to the task-oriented grasping field (Tang et al., 2023). However, the authors did not explicitly position their work concerning similar studies. Mentioning such work can strengthen the paper and comparing it can consolidate it more. 
  * Could the authors elaborate on why there is no direct comparison with the Grasp-GPT approach?  
  * What were the considerations behind not including this work in the related work section? 


[1]: Tang, C., Huang, D., Ge, W., Liu, W., & Zhang, H. (2023). Graspgpt: Leveraging semantic knowledge from a large language model for task-oriented grasping. *IEEE Robotics and Automation Letters*.  
[2] Xie, C., Xiang, Y., Mousavian, A., & Fox, D. (2021). Unseen object instance segmentation for robotic environments. IEEE Transactions on Robotics, 37(5), 1343-1359.
questions: questions: ### SayCan-IM Baseline

  - What specific role does the large language model (LLM) play in the SayCan-IM baseline? Is its function limited to planning according to a predefined objective (distance to the target object center) and invoking CLIP, or does it have broader applications within the baseline?
  - Given the context of open-ended language grounding, do the authors see the potential for a more nuanced use of pure LLMs to provide? how might this offer a fairer comparison against vision-language models (VLMs)? My main concern is that there is overlapping functionality between the instance segmentation model and the VLM that can incur extra computational costs and using pure LLMs efficiently might circumvent the functional redundancy.  

### Real-World Experiment Reproducibility 
- **Replication Guarantees:** The setting of the real-world experiments appears difficult to replicate. Is there a mechanism, maybe the readily available instance segmentation mechanism that overlays the current scene masks and the prespecified ones and ensures consistency across all methods tested? This would help validate the experimental results and conclusions drawn.
limitations: limitations: 3
review_summary: review_summary: The work presents promising advancements in open-world grasping for robotics as demonstrated by its substaintial improvements over supervised and LLM based methods while avoiding the retraining overhead. However, to strengthen the paper, improvements in presentation and a more comprehensive inclusion of relevant literature are recommended. I was a bit torn on the recommendation and I am relying on the authors answers to maintain it or even improve it.
title: title: review
paper_summary: paper_summary: The paper presents Open World Grasper (OWG), a system that leverages modern vision-language models (VLMs) to enable robotic grasping in open-world scenarios based on open-ended language instructions. The system addresses two major limitations of previous approaches: the need for better visual grounding and the inclusion of low-level spatial understanding, which are critical for handling contact-rich environments. OWG combines VLMs with segmentation and grasp synthesis models to perform three main tasks: open-ended referring segmentation, grounded grasp planning, and grasp ranking through contact reasoning. This pipeline allows for zero-shot application, utilizing suitable visual prompting mechanisms to integrate high-level semantic reasoning with low-level physical-geometric insights. Extensive evaluations demonstrate OWG's robustness and superior performance compared to prior methods in both simulated and real-world robotic grasping experiments.
main_review: main_review: **Strengths:**
1. **Timely Topic:** The integration of vision-language models (VLMs) in robotics, particularly for grasping tasks, addresses a highly relevant and current area of research.
2. **Systematic Approach:** The paper is well-organized, detailing the OWG system's methodology, which includes segmentation, grasp planning, and ranking.
3. **Comprehensive Evaluation:** Extensive testing in both simulation and real-world settings provides a thorough evaluation of the system's effectiveness.

**Weaknesses:**
1. **Limited Novelty:** The approach closely follows existing methodologies and focuses on simple pick-and-place tasks in cluttered indoor scenes, a well-explored area. The paper lacks groundbreaking innovations.
2. **Scope of Application:** The focus on common scenarios does not sufficiently demonstrate the system's applicability to more complex or diverse environments.
3. **Clarity and Detail:** While generally clear, the paper could benefit from more detailed explanations of the visual prompting mechanisms and the specifics of the ablation studies.
4. **Discussion on Limitations:** There is an insufficient critical analysis of the system's limitations, particularly regarding its generalizability and potential shortcomings in more varied scenarios.

Overall, while the paper contributes to the understanding of using VLMs for robotic grasping, it may not meet the high standards expected at CoRL due to its limited novelty and the well-trodden nature of its experiments. Future work should focus on more innovative approaches and diverse applications.
questions: questions: See above
limitations: limitations: 3
review_summary: review_summary: This paper doesn't have unique contributions and is very similar to lots of recent papers in this field, thus I would like to reject this paper.
title: title: Review
paper_summary: paper_summary: This paper proposes a method that integrates vision language models (e.g., GPT-4o) into the open-world robot grasping pipeline. Through the use of VLM, the grasping agent is capable of handling arbitrary user instructions and also effective under cluttered grasping scenarios. Specifically, the authors propose to use set-of-marks prompting and chain-of-thought prompting to generate grasping plans, especially in clutter scenarios where some objects might need to be removed first before the robot can grasp the target object. The authors also propose to use VLM to rank grasping proposals for better grasping qualities.
main_review: main_review: ## Strengths

Using VLM is essential for robots to effectively perform open-world grasping under diverse cluttered scenes in the real-world that potentially require the robot to (1) plan and perform multiple actions (e.g., removing obstacles) before grasping the target object, and (2) handle arbitrary unstructured user instructions in daily life. The authors investigate and explore different design decisions that effectively integrate VLM into the open-world grasping pipeline, providing meaningful insights for the community.


## Weaknesses and Questions


### Methodology

- The method to use self-consistency prompting and to use high-resolution image input (Table 3) are not defined in the paper, even though they provide significant benefits to the proposed approach's object-grounding performance.
- Is using VLM for grasp ranking really helpful for performance, and how does it compare with using the grasp rankings from off-the-shelf grasping models? (Minor: With a stronger grasping model than the one used in the paper (e.g., Contact-GraspNet, Anygrasp), does it make a difference?)


### Experiments

- According to Table 2, the real-world evaluation is only conducted for 6 trials for both seen / unseen instruction settings. The results can be sensitive to the noise resulting from a low number of evaluation trials. Increasing the real-world evaluation to at least 12 trials per setting will make the results more compelling.
- For Table 4, authors only conducted a small number of trials (15) for simuation experiments, and the results are quite sensitive to noise. One should conduct a much larger number of simulation evaluation trials (e.g., 50 / 100), especially since simulation evaluations are more scalable than real evaluations.
- For Table 1, is the comparison with baselines fair, i.e., the baselines are also using ground-truth segmentation masks to calculate mIoU?
- The ablation experiment using SAM for instance segmentation reports surprising low numbers, which seems to be a mistake. One should use the lowest granularity mask in SAM or use the "return single-mask" mode in SAM.



### Writing

- The introduction and motivation of the work are weak, tedious, and do not reveal the true significance of VLMs in a robot grasping pipeline. The main significance of VLMs in robot grasping comes from their ability to handle complex cluttered scenarios that potentially require multi-step reasonings & robot actions, and to handle arbitrary, unstructured user instructions encountered in the daily world. In particular, previous works primarily focus on simple scenarios where the robot can directly grasp the target object upon seeing it. However, in more complex, cluttered settings typically encountered in the real world, some objects may obstruct the target object, and they need to be moved away before the robot can grasp the target object. In these scenarios, VLMs become essential for accomplishing the task, which is the focus of the paper. On the other hand, VLMs are less beneficial in simple grasping scenarios, where one can directly employ open-world detection-segmentation pipeline (e.g., GroundingDINO + SAM) and existing grasping models (e.g., AnyGrasp) to accomplish the task.
questions: questions: Please see "Weaknesses and Questions" in the review and address the raised issues in methodology, experiments, and writing.
limitations: limitations: 3
review_summary: review_summary: This paper provides some meaningful contributions on integrating VLM into the open-world grasping pipeline, but suffers from some notable weaknesses in methodology, experiments and writing. Edit: Most of them are addressed by authors' rebuttal, so I raised my score.