Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter
Authors: Georgios Tziafas
Yucheng XU
Arushi Goel
Mohammadreza Kasaei
Zhibin Li
Hamidreza Kasaei
Venue: CoRL 2023 Poster (2023)
Keywords: Language-Guided Robot Grasping
Referring Grasp Synthesis
Visual Grounding

Abstract: Robots operating in human-centric environments require the integration of visual grounding and grasping capabilities to effectively manipulate objects based on user instructions. This work focuses on the task of referring grasp synthesis, which predicts a grasp pose for an object referred through natural language in cluttered scenes. Existing approaches often employ multi-stage pipelines that first segment the referred object and then propose a suitable grasp, and are evaluated in private datasets or simulators that do not capture the complexity of natural indoor scenes. To address these limitations, we develop a challenging benchmark based on cluttered indoor scenes from OCID dataset, for which we generate referring expressions and connect them with 4-DoF grasp poses. Further, we propose a novel end-to-end model (CROG) that leverages the visual grounding capabilities of CLIP to learn grasp synthesis directly from image-text pairs. Our results show that vanilla integration of CLIP with pretrained models transfers poorly in our challenging benchmark, while CROG achieves significant improvements both in terms of grounding and grasping. Extensive robot experiments in both simulation and hardware demonstrate the effectiveness of our approach in challenging interactive object grasping scenarios that include clutter.

Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter
Georgios Tziafas'*
Yucheng Xu Arushi Goel Mohammadreza Kasaei²
Zhibin Li Hamidreza Kasaei
University of Groningen University of Edinburgh University College London (g.t. tziafas, h. kasaeei @rugg.l 2 (Yuchheng.XM Xu, A. Goel-1, m kasaej (eeraaeeeedddee. @ed. uc.uk alex. li@ucl. ac.uk
A Abstract: Robots operating in human-centric environments require the integra- tion of visual grounding and grasping capabilities to effectively manipulate objects based on user instructions. This work focuses on the task of referring grasp syn- thesis, which predicts a grasp pose for an object referred through natural language in cluttered scenes. Existing approaches often employ multi-stage pipelines that first segment the referred object and then propose suitable grasp, and are evalu- ated in simple datasets simulators that do not capture the complexity of natural indoor scenes. To address these limitations, we develop a challenging benchmark based on cluttered indoor scenes from OCID dataset, for which we generate refer- ring expressions and connect them with 4-DoF grasp poses. Further, we propose a novel end-to-end model (CROG) that leverages the visual grounding capabili- ties of CLIP learn grasp synthesis directly from image-text pairs. Our results show that vanilla integration of CLIP with pretrained models transfers poorly in our challenging benchmark, while CROG achieves significant improvements both in terms of grounding and grasping. Extensive robot experiments in both simu- lation and hardware demonstrate the effectiveness of our approach in challenging interactive object grasping scenarios that include clutter.
Keywords: Language- Guided Robot Grasping, Referring Grasp Synthesis, Visual Grounding
Introduction
Recent advancements in deep learning have paved the way for substantial breakthroughs in data-driven robotic grasp- ing. Several works have proposed to synthesize grasps from purely visual inputs [1, 2, 3, 4]. In parallel, there is emerg- ing work attempting to ground robotic perception [5, 6, 7] and action [8, 9] in natural language, aiming to enhance the ability of robots to interact with non-expert human users. In this work, propose to bridge these two avenues via the task of referring grasp synthesis, where the robot is able to grasp a targeted object of interest that is indicated verbally by a human user (see Fig. 1). We focus on investigating this task in natural indoor scenes which include ambiguity and clutter, and are more realistic.
Pass the white frant the Coke
Figure 1: 4-DoF referring grasp syn- thesis in clutter.
Most existing approaches study interactive grasping scenarios via multi-stage pipelines [10, 11, 12, 13, 14], where first the target object is localized from linguistic referring expression (i.e. visual
Equual contribution
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

grounding) and another module predicts a suitable grasp pose in a second step. The visual grounding models are trained either in benchmarks such as RefCOCO [15, 16, 17], which contain mostly outdoor scenes with few graspable objects, or custom datasets which limit the applicability of the learned model to fixed lab setups. Other robotics-related datasets collect language-annotated indoor scenes [5. 6. 7]. but are not directed towards grasping, or contain grasp annotations but lack language [18, 19]. Additionally, related datasets that study clutter do not explicitly study ambiguous objects in the scene [7, 20), i.e., objects of the same category appearing multiple times, and hence annotate only for such a category and a few attributes. They also mostly consider only pair-wise spatial relations between objects, which is not the case in free-form language (e.g. "leftmost bowl" is more natural than "bowl left from other bowls").
Alternatively, recent trend in language-basse based robot systems [21, 22] is to combine language mod- els [23] with pretrained vision-language foundation models such as CLIP [24] for zero-shot ground- ing, and CLIP-based end-to-end grasping policies trained via imitation learning [25, 26). Such ap- proaches achieve impressive results but evaluate mostly in simulators which are fairly distant from natural realistic scenes, making the sim-to-real transfer more challenging.
To tackle the above limitations, we establish a new challenging dataset, OCID-VLG, that studies end- to-end lision-laggaagegrrasiing in natural cluttered scenes. The dataset connects grasp annotations from the OCID- Grasp dataset [19] with referring expressions that include rich attribute vocabulary, model a broad range of relations, and explicitly consider ambiguity. Further, we propose an end-to- end model (CLIP-based Referring Object Grasping CROG), that extends CLIP's visual grounding with both pixel-level segmentation, as in [27], and grasp synthesis tasks, via a novel multi-task objective. Experimental evaluations show that the proposed model robust to referring expression complexity, and outperforms previous baselines that rely on vanilla integration of CLIP with the multi-stage approach. Extensive robot experiments demonstrate the effectiveness of the proposed model in challenging interactive grasping scenarios, in both simulation and real-world settings.
In summary, the main contributions of this work are: a) new challenging dataset for visual ground- ing and referring grasp synthesis in cluttered scenes, comprising approximately 90 000 language- mask-grasp annotations, an end-to-end learns a grasp policy by leveraging the powerful langgag alignment of CLIP, and demon- strate its performance merits compared to multi-stage baselines that utilize pretrained models, and, c) applying our proposed model in challenging interactive table cleaning scenarios through both simulation and real robot experiments.
sanguage-grasppen model, CROG, which efficiently
2 Related Works
Referring Expression Datasets Referring expressions are natural language descriptions that uniquely identify a target region in paired image, often by referring to object attributes and spatial relations. Several datasets [28] have been proposed in the past, with expressions and target bounding boxes / masks annotated manually [29, [0] or via a two-player game [31]. Most popular benchmarks include Plickr3k--Enttties [17] and the RefCOCO(/+/g) suite [15, 32], containing annotations for MSCOCO [33] scenes, collected via the refer-it game strategy [31]. Alternttvely. automatic re- ferring expression generation is pursued via the usage of symbolic scene graph annotations and synthetic language templates [34, 35, 36, 16]. Above benchmarks concern referring expressions for RGB images with generic content and are mostly for outdoor scenes. Recent works propose datasets with referring expressions for objects in indoor environments and RGB-D/ / 3D visual data [7, 5, 6], but do not consider clutter and are not connected with robot grasping. In our work, we adopt the automatic generation method of CLEVR- [36] to generate expressions for extracted scene graphs from OCID-Grasp dataset [19]. To the best of our knowledge, OCID-VLG is the first dataset that brings together referring expressions and grasp synthesis for cluttered indoor scenes.
Visual Grounding Visual grounding is formulated in literature through the tasks of referring ex- pression comprehension and referring image segmentation, depending on the type of localization required (box and mask respectively). Methods usually employ a two-stage detect-then rank ap- proach [37, 38, 39, 40], first generating object proposals and then ranking them according to their

---

correspondence to the expression. Single-stane methods [41, 42] alleviate the object proposal step by directly fusing vision-text features in joint space. Recently, the Transformer architecture has been employed for both task variants separately [3, 43, 44), or jointly [45, 46], showcasing strong cross- modal alignment capabilities compared to previous CNN-LSTM fusion techniques. The grounding task has been recently adapted for 3D data [5]. with similar two-stage methods fusing text features with point clouds [47, 48] or RGB-D views [49, 6]. Finally, transferring from large-scale vision- language pretraining [50, 51, 52, 24] is a common practice for usage in zero-shot [21, 22], or as an initialization for finetuning [25]. Similarly, in our work, we finetune the CLIP vision-language model [24] to further learn 4-DoF grasp synthesis in RGB.
Grasp Synthesis Grasp synthesis enables robots determine the optimal way to grasp objects by considering visual information. Current grasp synthesis methods can be roughly categorized into 4- DoF and 6-DoF [53], according the degrees of freedom (DoF) of the grasp configurations. 4-DoF grasp synthesis [54, 19, 55] defines grasps by a 3D position and top-down hand orientation (yaw), which is also commonly referred as "top-down grasp". 6-DoF grasp synthesis [18, 56, 57] defines grasp poses by 6D positions and orientations. Early works [1, 2] formulate 4-DoF grasp detection via decoding a set of grasp masks from the input RGB-D images and use camera calibration to transform the planar grasp into a gripper pose. Det-Seg [19] proposed a two-branch framework that generates semantic segmentation masks and uses them refine the predicted grasps. SSG [58] introduced an instance-wise 4-DoF grasp synthesis framework and showed its effectiveness and robustness in cluttered scenarios. In this work, we build on the idea of using the segmentation mask as extra signal for learning grasp synthesis, by making the masks object-specific via grounding them from referring expressions.
3 OCID-VLG Dataset
Visual grounding and grasp synthesis are mostly studied separately, and hence associated grounding datasets rarely involve cluttered indoor scenes [15, 17, 16] and lack grasp annotations [5, 6, 7], while grasp synthesis datasets lack language language-grounding [55, 54, 18, 19]. Our proposed dataset, aims to cover this gap, by providing a joint dataset for both grounding and grasp synthesis in scenes from OCID dataset [59]. The dataset consists of 763 indoor tabletop RGB-D scenes with high clutter, including 31 object categories from total of 58 unique instances. The OCID object catalog includes several object instances of the same category that vary in fine-grained details, granting it desirable domain for integration with language. We manually annotate the catalog with a rich variety of object-related concepts, as well as pair-wise and absolute spatial relations. For each scene, we provide 2D segmentation masks and bounding boxes (at both category and instance level), as well as a complete parse of the scene, providing all 2D/3D locations, category, attribute and relation information for each object a symbolic scene graph. We leverage the previous 75k hand-annotated 4-DoF grasp rectangles of OCID-Grasp [19] and connect each object in our scene graph with a set of grasp annotations. The grasp-annotated scene graphs are used to generate referring expressions with custom variant of the CLEVR data engine [35]. We heae--teetmmmkkboxx-grrree
OCID-VLGG
end up with 89, 639 unique and grasp synthesis tasks in an end-to-end fashion.
tuples, aimed to supervise both grounding
3.1 Referring Expression Generation
We first annotate a catalog of attribute and relation concepts that are used to refer to ambiguous objects in OCID-VLO scenes. For attributes, each object is annotated for its color, as well as with an instance-level description that refers to some object's property (e.g. brand, flavor, variety, maturity, function, texture, or shape). We note that not all objects are annotated for all mentioned concepts, but only for those that discriminate them from other objects of the same category. For spatial relations, we include both relative (e.g. "bowl left from mug") ") as well as absolute location (e.g. "lefimost bowl"" concepts. We adapt the relation resolution heuristics of [44] and use the relation set ( ("right", "rear right", "behind", "rear left", "left", "front left". , "front", "front right", "on" "on"), J. but augment with the absolute location set ("eeftmost, "rightmost", "furthest", "closest").
3

---

Datuset
Clutter
Vision
Ref.Expr.
Grasp Annot.
Num.Obj. Categories
Num. Scenes
Num. Expr.
Parses
Data
Aunot.
RefCOCO [15]
X
RGB
box.mask
X
19.9%
142.24 24
X
Flickr30k- Entities [17]
X
RGB
box
44.5k:
31.7k
158.94
X
CIEVVR-Rof+ [36]
X
RGB
hox,mask
60k
600k
Cops-ref [16]
X
RGB
box.mask
508
703
148.7k
ScanRefer [5]
X
3D
box
X
800
51.54
x
ReferI-RRGD [6]
X
RGB-D
box
X
7.6k
38. A16
Sun-Spot [7]
RGB-D
box
38
1.9k
7.9k
(OCID-Ref [20)
RGB,3D
box
58
2.3k
305.6k
x
Comell [55]
X
RGB-D
X
4-DoF
240
X
Jacquard [54]
RGB-D
X
4-DoFF
54k
GraspNet [18]
3D
6-DoFF
88
190
OCID-Grasp [19]
RGB-D
X
4-DoF
31
1.7k
OCID-VLG (ours)
RGB-D.3D
box.mask
4-DoFF
1.7k
89.66
Table 1:
Comparison of main features and statisties between existing 2D visual grounding and grasp
symthesis datasets with our OCID-VLG.
The complete list of the concept vocabulary and related statistics are provided in Appendix A.
After parsing scenes into scene graphs, we sample object and relation concepts generate referring expressions using the CLEVR data engine [35] and custom templates, that follow the structure:
[prefix] ([LOC]] [ATT,]) [OBJ,] ((that is)
[REL] the ([LOC2] [ATT2]) [OBJ2])
where [OBJ], [ATT], [REL], [LOC] denote an object concept (category or instance-level), an attribute (color), pair-wire relation and an absolute location respectively. The [prefix] is sampled from set of general robot directives, e.g. "Pick the". We construct template variations for 5 families, namely: name, (e.g. "chocolate corn flakes"). b) attribute, (e.g. "brown cereal box p)ckage"), c) relation, (e.g. "corn flakes behind the bowl"), d) location, (e.g. "closest cereal box") and e) mixed, (e.g. "cereal box to the rear left of the right apple"). ) for a total of 56 distinct sub-templates. Note that templates (b)-(e) are constrained to only sample target objects that are ambiguous in the scene, hence attribute or relation information are needed uniquely ground them.
3.2 Comparisons with Existing Datasets
OCID-VLG differs from existing datasets in several aspects and statistics, summarized in Table 1. Popular visual grounding datasets [15, 32, 17, 16] usually include few indoor scenes with cluttered content and provide only RGB data, limiting their applicability in the robotics domain. Robotics- related grounding datasets usually contain referring expressions for objects in room layouts [5, 6] (e.g. furniture), which are not directed towards grasping and do not consider clutter. Sun-Spot [7] contains tabletop cluttered scenes, but doesn't 't annotate segmentations and grasps and lacks absolute location annotations. Similarly, OCID-Ref [20] only provides boxes without segmentation and grasp annotations. We highlight that even though OCID-Ref could be used as a source of referring expres- sions for OCID-VLG,
chose to develop our own, OCID-Ref expressions lack rich object and relation vocabulary, lack absolute relations and inherit corrupted labels from OCID dataset. Grasp synthesis datasets are either in object-level and not consider clutter [55, 54], or include cluttered scenes but do not annotate referring expressions [18, 19J. Additionally, most mentioned datasets do not explicitly consider ambiguities in the scene. In OCID-VILS VLG, we use attributes and relations to refer to objects only in cases of ambiguity, aiming to prevent overfitting superficial object-relation correspondences that exist in the training data. Finally, as we use the CLEVR [35] engine to generate expressions, our dataset is further equipped with symbolic parses of both the visual and the language modalities, which could be potentially utilized for training models with additional supervision.
Method
This section discusses the proposed task, the implemented basclines and the details of our end-to-end model, CROG.

---

Problem Formulation. Referring grasp synthesis considers the problems of referring image segmentation and grasp synthesis in tandem. Given an RGB image 7 E RR x3 a depth image D E ##* and a natural language expression that refers to a unique object in the scene, the goal to predict both pixel-wise segmentation mask of the referred object M {0, 1] H as well as grasp configuration G (x". ","""0 (("))), where: (W, H)) the image resolution, (x ,y") the center of the optimal grasp in pixel coordinates, 0* the gripper's rotation in camera reference frame and l" the gripper width in pixel coordinates. As in [1, 2, 19, 58), G' is recovered from three masks: a grasp scalar quality mask Q E {0, 1) such that: (x",
argmax ( v),, an angle mask xt 2 #x and width mask L E {0. 1) H*W, such that: 8" O(x", ,y"), 1" L(x", .y)).
O E
{
T.
4.1
Multi-Stage Baselines
We design multi-stage baselines which integrate existing large-scale vision-language models with pretrained grasp synthesis models. To that end, we decompose the overall task into two stages, namely: a) a grounding function f(I,T) = M that segments the referred object from the RGB image, and b) grasp synthesis function g( D, .)) = G that isolates the object in the RGB-D inputs 1, D using the segmentation mask M produce grasp G.
Two-stage grounding with CLIP. The grounding function can be further decomposed into two steps, first using an off-the-shell detector [60] for object proposal generation faegm (I) = {M, } J and then ranking the segmented object proposals according to their similarity with the language description frank (M,, 7) = argmax,, ((,,, T), where 5() denotes a similarity metric between a segmented RGB object image M, and the language imput practise, we implement S via CLIP's [24] pretraining objective, i.e. computing cosine similarity between visual features from passing Mn to CLIP's visual encoder and sentence-wide wide embedding of T from CLIP's 's text encoder.
Mond-condd grasp synthesis. The grasp synthesis function is implemented via a pretrained network [I. 2] which receives the input pair 1, D and generates grasp G' via decoding the masks Q, O L. To isolate the desired object, given in mask M, we experimentally find that the best practise element multiply the mask with the RGB-D inputs before passing to the network.
4.1.1 Zero-shot baselines
First, existing powerful pretrained models are experimented to assess their zero-shot performance in our challenging setup, including two multi-stage variants that use GR-ConyNet [1] pretrained on the Jacquard [54] dataset grasp synthesis network, but differ in grounding follows:
SAM+CLIP. In this setup, we use the Segment Anything (SAM) [60] model for instance seg- mentation and CLIP [24] for ranking as explained above. Similar to [61], we find that passing both a cropped box and the mask of the object to CLIP's feature extractor and ensembling the final similarities boosts performance.
GLIP+SAM. For this variant, we use large pretrained visual grounding model, GLIP [62], for pre- dicting a bounding box around the relevant object of interest given the natural language command, and prompt the SAM [60] model to get a tight segmentation mask for the object of interest.
4.1.2 Supervised baselines with vanilla CLIP integration
Second, we implemented a vanilla integration of CLIP in grasp synthesis models pretrained in OCID-Grasp [19], which aims to explore whether a model using segmentation and grasping trained on OCID scenes can be extended to our setup without any language conditioning. Similar to the SAM+CLIP setup, CLIP used as ranker and the supervised model as both segmenter and grasper. We experiment with the two state-of-the-arr models in DCID-Grasp, Det-Seg [19] and SSG [58], which can both provide both segmentation and grasp predictions for input RGB-D scene.
4.2 CROG
We propose model for estimating both the segmentation mask M and the grasp masks Q, 2,0,1 T. in an end-to-end fashion (see the overview in Fig. 2). Our architecture is an extension of CRIS [27].
5

---

model originally proposed for adapting CLIP to do pixel-level segmentation. CRIS achieves this with four main components: a) unimodal encoders for image and text, b) a multi-modal FPN neck, c) cross-attention vision-language decoder, and d) projectors for text-to-pixel contrastive loss.
The unimodal encoders initialized from CLIP's visual and text encoders [24], but uti- lize visual feature maps from 2th-4th stages of CLIP's ResNet-50 visual encoder Fv2 E R x W #xC, Fes E I
# th xC', Fes E R x W xC2 and considers both the sentence-
Eirs
Fhi Meglind Per
Coickee
level F, E RC and the token embeddings Fe E RRRK where C and C" are the feature di- mensions and K the language sentence length. Visual feature maps F22, Fo3, Fu4 and the sentence embedding F, are fused in feature pyramid style [63] via the multi-modal neck, in order to generate pixel-wise multi multi-modal representations Fm E R NxC of the image-text pair, where N 16 16'
Figure 2: An overview of CROG.
W
The vision-language decoder uses a standard Transformer decoder [64] let the multi-modal fea- tures Fm cross-attend with all token embeddings F, and produce embeddings F. E RNXC' This process adaptively propagates semantic information from text to visual features. Finally, F. and Fs are projected into the same space, where contrastive alignment is performed via computing a binary cross entropy loss over the dot- dot-product of the projected embeddings, pushing them together in the regions of the ground truth segmentation mask.
To obtain a mask prediction, the projected features Fo, F. are dot-producted with sigmoid activation, reshaped to N 4 Y and upsampled to original image size. To adapt for grasp synthesis task, propose to further add three projectors for generating grasp masks Q, θ, L and supervise them with smooth L1- from the ground truth grasps, in parallel to the contrastive alignment loss of CRIS.
Experimental Results
This section evaluates our dataset using multi-stage baselines and compares them to our CROG model. Also, we conducted ablation studies to analyze the performance improvements and present the results of our robot experiments.
Implementation We initialize the vision and text encoders with the ResNet-50 and BERT weights from CLIP [24]. Input images are resized to 416 x 416, and texts BPPettokeizzed [64, 65]. The maximum length of input texts is set to 20. We train in multiple GPUs using the Adam optimizer with an initial learning rate 1e-4, that decays to 0.1 over 35 epochs.
Evaluation metrics
For grounding, we report referring image segmentation (RIS) [27, 66) metrics IoU and Precision@ X. IoU calculates averaged intersection over union for the predicted segmenta- tion and ground truth masks, while Precision @ X measures the percentage of predictions with loU higher than threshold X E {0.5, 0.6, 0.7, 0.8, 0.9). For referring grasp synthesis (RGS), Jacquard index J@M N [54, 19, 58) is presented, measuring the percentage of top-N grasp predictions that have an angle difference within 30 and higher than 0.25 IoU with the ground truth grasp rectangle.
5.1 OCID-VLG Results
The grounding and grasp synthesis results are reported on the test split of OCID-VLG, containing 17.7k samples from held-out scenes of OCID. The test set contains seen objects but in novel scene configurations, resulting in unseen referring expressions. Results for zero-shot and supervised base- lines and our CROG are in Table 2. Results show that baselines based on GR-ConvNet [1] pretrained on Jacquard [54], transfer poorly in OCID-VIL VLG, even with ground truth grounding (28.7% J J@1 1).
We find that the CRe-oonNett-based grasper tends to prefer edges, due to the top-down perspective of Jacquard images, which is not the case in OCID-VLG.

---

Zero-shot baselines score below 30% both tasks, as due to the CLIP-based ranking beehodology.false positives in grounding lead to incorrect grasping, gardless if the predicted grasp is correct for the mis- segmented object. Replac- ing large zero-shot mod- els with supervised meth- ods trained in OCID-Grasp (Det-Seg, SSG), offers
Method
RIS Pr@70
RGS
IoU
Pr
Pr@60
Pr@80
Pr@90
J@1
A@y
GT-Grounding!
28.7
70.2
GT Masks CLIP
35.0
35.0
35.0
35.0
35.0
35.0
11.9
26.8
CLIP
12.7
GLIP SAM
30.3
34.7
34.1
33.5
28.6
11.7
10.7
21.8
Det-eeg + CLIP
29.0
27.2
20.9
17.5
17.2
16.0
28.1
39.2
SSG + CLIP
33.6
35.6
35.6
35.5
35.5
32.8
33.5
34.7
CROG (ours)
81.1
96.9
94.8
87.2
64.1
16.4
77.2
87.7
Table 2: Comparison results in OCID-VLG test split. Baselines with use GR-ConyNet [1] pretrained Jacquard [54]. GT denotes the of ground truth data for providing an upper bound performance given perfect segmentation masks grounding.
marginal improvement in grounding (+3.6% in IoU), but significant in grasping (+23.2% in J@ I), while still low overall (39.2% J@Any). This indicates that even in presence of an OCIDD-pecific grasper, the ranking methodology of vanilla CLIP integration significantly limits the grounding per- formance.
The proposed CROG overcomes such limitations by fine-tuning grounding and grasp synthesis gether top of CLIP, and surpasses previous methods with large margin (+47.5% in IoU and +43.7% in J @@ 1), offering a much more competitive baseline for the proposed OCID-VLG dataset.
5.2 Ablation Studies
Ablation studies conducted to explore: a) the distribution of error according to referring expression type, and b) perfor- mance improvements of each main CROG component.
Referring Expression Type We first decompose the per- formance according to the type of the input expression and compare the analytical results of model with the best- performing baseline, SSG+CLIP (see Fig. 3). We observe that the CLIP baseline struggles with grounding spatial concepts such as relations and locations (less than 30% J@ I), due to the loss of spatial information introduced by the segment- then- rank pipeline. On the contrary, CROG is trained with dense pixel-text token alignment via the antsntnttenioon decoder, and capable of spatial grounding and robust across all types.
Pabriaag Pageeooe
Figure 3: Grasp synthesis ablations according to the type of input refer- ring expression.
Method
RIS JoU
RGS-JØ1
Effect of CROG components We ablate the three main char- acteristics of CROG: a) initializing from CLIP, b) combining contrastive with grasp mask decoding tasks in a single objec-
CROG
81.1
77.2
who CLIP init
73.9
71.0
contrastiv
73.4
who graup loss
79.3
tive, and e) dense text-pixel alignment with decoder. Results summarized in Table 3. All components are contributing to CROG's performance, where CLIP initialization is the most vital. Crucially, the removal of the contrastive grasp loss results in a decrease in CROG's grasping or grounding capability. This highlights the knowledge transfer between the two tasks that justifies our selection of a multi-task training objective.
who decoder
78.2
72.3
Table 3: CROG ablation study.
5.3 Robot Experiments
We conducted experiments with simulated and a real robot, where we want to evaluate the per- formance of our model the context of an interactive table cleaning task. Our setup consists of a dual-arm robot with two UR5e manipulators with parallel jaw grippers and Kinect sensor.
During each experiment, we randomly place 5-12 objects tabletop and provide language in- struction to the robot to pick target object and place it in predefined container position.

---

We place objects in two scenarios, namely: a) isolated, where objects are scattered across the workspace, and b) clut- tered, where closely pack objects together. We note that in each scene we include distractor objects of the same category as the queried object. Our setup and example trials are shown in Fig. 4, while more qualitative results are provided Appendix
Seou
Troit
Food Box
Food Can
Mug
Marker
Ceneal
Fiashlight
Overal
Loatted (Finas
30
Isolated Ground
10
8.0998)
5103163
(50063
6100%)
2 (0005))
(1915)
Iodhttd Saoce
(0.100))
(1999)
(2289)
08339)
(40%)
10005
(615)
Chraced Trarale)
50
Chaneral Grund (809))
(908))
(389)
1258
0009:
(00R)
3338
12236)
(000)
(421))
Table 4: Results of robot experiments Gazebo, where Ground.Ace de- notes the number of trials where the target object is segmented cor- rectly and
the number of successfully completed trials.
We conducted 50 trials per scenario in the Gazebo simulator [67], using object models from 7 categories of OCID-VLG, some as exact instances and others with different attributes. Object list, metrics, and recorded results are shown Ta- ble 4. The robot achieves grounding ac- curacy of 76% (38/50) and success rate of
Moel finklight
62% (31/50) in isolated and 60% (30/50), 42% (21/50) in cluttered scenes respec- tively, with grounding failures mostly for objects that are not similar in appear- ance to OCID-VIG categories. For real robot experiments, we initialize six unique scenes, three for isolated and three for cluttered scenarios, and provide grasping instructions for total of 34 trials. We highlight that this test is more challenging. as the object set used for experiments has no overlap with OCID-VLC instances. In isolated scenes, the grounding accuracy is 65% and the success rate 23.9%, while in cluttered it is 60% and 20.0% respectively. In both experiments, the model is able ground attribute concepts for unseen instances (e.g. "white and blue box") and disambiguate objects based on spatial relations. Grounding failures usually due to highly oc- cluded objects in the scene, especially if multiple distractor objects are present. Several failure cases in cluttered scenes are due to collisions during motion execution, nevertheless, the 4-DoF grasp is correctly predicted. Detailed experiments are shown in the supplementary video.
GUT greve WarrkT
clarre sponge
Figure 4: Interactive table cleaning trials in Gazebo (top) and real robot (bottom), in isolated (left column) and cluttered (right column) scenes.
6 Conclusion, Limitations and Future Work
This paper presents OCID-VLG, a new dataset for aanuaag-guidee 4-DoF grasp synthesis in clutter, offering the first benchmark that connects language instructions with grasping in an end-to-end fashion. Further, we propose CROG, a CLIP-based end-to-end model as a solution. Extensive experimental comparisons and ablation studies validated the effectiveness of CROG over previous methods, and set competitive baseline for our dataset. Overall, this research offers valuable insights into language-guider grasp synthesis and lays the foundation for future advancements in this field.
However, we found that CROG is limited when grounding concepts that lie outside the training distribution. We attribute this to the hiaeaiiiinn-inttuiingggg strategy, which trades off the zero-shot capacity of CLIP pretraining in favor of the dense finetuning tasks. Future work will explore methods efficiently learn the dense decoding tasks while maintaining better zero-shot grounding capability of CL.P. Finally, since CROG only considers RGB information, we would like to investigate whether further fusing depth data alongside RGB aids in grasp synthesis.

---

This work is supported by EU H2020 project "Enhancing Healthcare with Assistive Robotic Mobile Manipulation (HARMONY, 101017008)", as well as from Google Deep- Mind through the Research Scholar Program for the "Continual Robot Learning in Human-Centered Environments" project.
Acknowledgements:
References
[1] Kumra, Joshi, and F. Sahin. Antipodal robotic grasping using generative residual convo- lutional neural network. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9626-9633, 2019.
[2] D. Morrison, P. Corke, and J. Leitner. Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach. ArXiv, abs/1804.0517,, 2018.
[3] R. Chen, N. Gao, N. A. Vien, H. Ziesche, and G. Neumann. Meta-learning regrasping strategies for physical-agnostic objects. ArXiv, abs/2205. 2022.
[4] H. Kasaei and M. Kasaei. Mvgrasp: Real-time multi-view 3d object grasping in highly clut- tered environments. Robotics and Autonomous Systems, 160:104313, 2023.
[5] D. Z. Chen, X. Chang, and M. Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. ArXiv, abs/1912.08830, 2019.
[6] H. Liu, A. Lin, X. Han, L.. Yang, Y. Yu, and S. Cui. Refer-it-in-rghd: A bottom-up approach for 3d visual grounding in rgbd images. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 0028-6037, 2021.
[7] C. Mauceri, M. Palmer, and C. Heckman. Sun-spot: An rgb-d dataset with spatial referring ex- pressions. 2019 IEEE/CV International Conference on Computer Vision Workshop (ICCVW), pages 1883-1886, 2019.
[8] Luketina, N. Nardelli, G. Farquhar, N. Foerster, Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel. A survey of reinforcement learning informed by natural language. In International Joint Conference Artificial Intelligence, 2019.
[9] S. Stepputtis, J. Campbell, M. Phielipp. S. Lee, C. Baral, and H. B. Amor. Language- conditioned imitation learning for robot manipulation tasks. ArXiv, abs/2010. 12083, 2020.
[10] M. Shridhar and D. Hsu. Interactive visual grounding of referring expressions for human-robot interaction. ArXiv, abs/1806.0383. 1, 2018.
[11] D. Misra, J. Sung, K. Lee, and A. Saxena. Tell me dave: Context-sensitive grounding natural language to manipulation instructions. The International Journal of Robotics Research, 35:281 -300, 2014.
[12] J. Hatori, Y. Kikuchi, S. Kobayashi, K. Takahashi, Y. Tsuboi, Y. Unno, W. K. H. Ko, and J. Tan. Interactively picking real-world objects with unconstrained spoken language instructions. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3774-3781, 2017.
[13] Y. Chen, R. Xu, Y. Lin, and P. A. Vela. A joint network for grasp detection conditioned on natural language commands. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4576-4582, 2021.
[14] V. Blukis, R. A. Knepper, and Y. Artzi. Few-shot object grounding and mapping for natural language robot instruction following. In Conference on Robot Learning, 2020.
[15] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring expres- sions. ArXiv, abs/ 00202, 2016.

---

[16] Z. Chen, P. Wang. L. Ma, K.-Y. K. Wong, and Q. Wu. Cops-ref: A new dataset and task on compositional referring expression comprehension. 2020 /EEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10083-10092, 2020.
[17] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Computer Vision, 123:74-93, 2015.
[18] H. Fang, Wang, M. Gou, and C. Lu. Traaspee--billionn A large-scale benchmark for general object grasping. 2020 IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pages 11441-11450 2020.
[19] S. Ainetter and F. Fraundorfer. End-to-end trainable deep neural network for robotic grasp de- tection and semantic segmentation from rgb. 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 13452-13458, 2021.
[20] K.-J. Wang, Y.-H. Liu, H.-. Su, J.-W. Wang, Y.-S. Wang, W. H. Hsu, and W.-C. Chen. Ocid- ref: A 3d robotic dataset with embodied language for clutter scene grounding. In North Amer- ican Chapter of the Association for Compulational Linguistics, 2021.
[21] A. Zeng, A. S. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, Sind- hwani, J. Lee, Vanhoucke, and P. Florence. Socratic models: Composing zero-shot mul- timodal reasoning with language. ArXiv, abs/2204. 005988 2022.
[22] Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, R. Florence, and A. Zeng. Code policies: Language model programs for embodied control. ArXiv, abs/2209.07753, 2022.
[23] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. J. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. ArXiv, 141/2005..4165. 2020.
[24] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
[25] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. ArXiv, abs/2109.12098. 2021.
[26] C. Tang, D. Huang, L. Meng. W. Liu, and H. Zhang. Task-oriented grasp prediction with visual-language inputs. ArXiv, abs/2302. 14355, 2023.
[27] Z. Wang, Y. Lu, Q X. Tao, Y. Guo, M. Gong, and T. Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1686-116995, 2022.
[28) Y. Qiao, C. Deng, and Q. Wu. Referring expression comprehension: A survey of methods and datasets. IEEE Transactions on Multimedia, 23:4426-4440, 2020.
[29] H. de Vries, Strub, A. P. S. Chandar, O. Pietquin, H. Larochelle, and A. C. Courville, Guess- what?! visual object discovery through multi-modal dialogue. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4466-4475, 2016.
[30] Mao, Huang, A. Toshev, O.-M. Camburu, L. Yuille, and K. P. Murphy, Generation and comprehension of unambiguous object descriptions. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11-20, 2015.
10

---

[31] S. Kazemzadeh, V. Ordonez, M. andre Matten, and T. L. Berg. Referitgame: Referring to objects in photographs of natural scenes. Conference on Empirical Methods in Natural Language Processing, 2014.
[32] J. Mao, J. Huang, A. Toshev, O.-M. Camburu, A. L. Yuille, and K. P. Murphy, Generation and comprehension of unambiguous object descriptions. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11-20, 2015.
[33] T.Y.. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
[34] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:664-6766 2014.
[35] J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. B. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 1988-1997, 2016.
[36] R. Liu, C. Liu, Y. Bai, and A. L. Yuille, Clevr-reft: Diagnosing visual reasoning with refer- ring expressions. 2019 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4180-4189, 2019.
[37] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. CoRR, hetrg//bs/511..... 02283.
2015. URL
[38] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring expres- sions. CoRR, abs/ 00072, 2016. URL http: /ttp//aaxvvv///////iv//////a.avl orgooogg/aao abs/1608. 00272.
[39] A. Rohrbach, M. Rohrbach, R. Hu, T. Darrell, and B. Schiele. Grounding of textual phrases in images by reconstruction. CoRR, abs/1511.003755, 2015. URL http://arxivv org/abs/ 03745.
[40] R. Luo and G. Shakhnarovich. Comprehensioon-guidedd referring expressions. Abs//701.03439) 2017. URL http:://arxxi.oogabb////ttt 03439.
CoRR,
[41] Z. Yang, B. Gong, L. Wang, W. Huang, D. Yu, and J. Luo. A fast and accurate one-stage approach to visual grounding. CoRR, abs/1908.063544 2019. URL http://arxiv.nt abs/1908. 0604440988066......
06354.
[42] A. Sadhu, K. Chen, and R. Nevatia. Zero-shot grounding of objects from natural language queries. In Proceedings of the IEEE International Conference on Computer Vision, pages 2019.
[43] G. Feng, Z. Hu, L. Zhang, and H. Lu. Encoder fusion network with co-attention embedding for referring image segmentation. 202/ IEFF/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 15501-15510, 2021.
[44] S. Yang, G. Li, and Y. Yu. Cross-modal relationship inference for grounding referring pressions. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 4140-4149, 2019.
[45] M. Li and L. Sigal. Referring transformer: A one-step approach multi-task visual grounding. ArXiv, abs/2106.03000 2021.
[46] G. Luo, Zhou, X. Sun, L. Cao, C. Wu, C. Deng, and R. Multi-task collaborative network for joint referring expression comprehension and segmentation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10031-10040, 2020.
11

---

[47] P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. J. Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In European Conference on Computer Vision, 2020.
[48] L. Zhao, D. Cai, L. Sheng, and D. Xu. 3dvg-transte tra-ttransformer: Relation modeling for visual ground- ing on point clouds. 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pages 2908-2917, 2021.
[49] Huang, Y. Chen, Jia, and Wang. Multi-view transformer for 3d visual grounding. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15503- 15512. 2022.
[50] A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion. Mdetr modulated de- tection for end-to-end multi-modal understanding. 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pages 1760-1770, 2021.
[51] L. H. Li, P. Zhang, H. Zhang. J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang, K.-W. Chang, and J. Gao. Grounded language-image pre-training. 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 10955-10965, 2021.
[52] Li, R. R. Selvaraju, A. D. Gotmare, S. R. Joty, Xiong, and S. C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Neural Information Processing Systems, 2021.
[53] R. Newbury, M. Gu, L. Chumbley, A. Mousavian, Eppner, Leitner, Bohg, A. Morales, T. Asfour, D. Kragic, et al. Deep learning approaches to grasp synthesis: A review. arXiv preprint arXiv:220..025566 2022.
[54] A. Depierre, E. Dellandréa, and L. Chen. Jacquard: A large scale dataset for robotic grasp detection. 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3511-3516, 2018. 2018......08.
[55] Y. Jiang, S. Moseson, and A. Saxena. Efficient grasping from rghd images: Learning using a new rectangle representation. 2011 IEEE International Conference on Robotics and Automa- tion, pages
2011.
[56] C. Eppner, A. Mousavian, and Fox. Acronym: large-scale grasp dataset based on sim- ulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6222 6227. IEEE, 2021.
[57] J. Varley, C. DeChant, A. Richardson, J. Ruales, and P. Allen. Shape completion enabled robotic grasping. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 2442-24222 2447. IEEE, 2017.
[58] Y. Xu, M. M. Kasaei, S. H. M. Kasaei, and Z. Li. Instance grasp synthesis for robotic grasping. ArXiv, abs/2302.07824, 2023.
[59] M. Suchi, T. Patten, and M. Vincze. Easylabel: semi-automatic pixel-wise object annota- tion tool for creating robotic rgb-d datasets. 2019 International Conference on Robotics and Automation (ICRA), pages 6678-6684, 2019.
[60] A. Kirillov, E. Mintun, N. Ravi, Mao, Rolland, I., Gustafson, Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo. P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02643. 2023.
[61] S. Subramanian, W. Merrill, T. Darrell, M. Gardner, Singh, and A. Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. In Annual Meeting of the Association for Computational Linguistics, 2022.
12

---

[62] L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.- N. Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[63] T.Y.. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[64] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A. Gomez, t.. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[65] A. Radford, J. Wu, Child, D. Luan, D. Amodei, I. Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[66] H. Ding, C. Liu, Wang, and X. Jiang. Vision-language transformer and query generation for referring segmentation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 16301-16310, 2021.
[67] N. P. Koenig and A. Howard. Design and use paradigms for gazebo, an open-source multi- robot simulator. 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No. (4C3436335,6 3:2149-2154 vol.3, 2004.
13

---

A OCID-VLG Vocabulary
We visualize word cloud of the concept vocabulary of OCID-VLG in Fig. 5, while the full attribute concept catalog is given if Fig. 6. Besides common sub sub-phrases such "box", "food". "product", the wordcloud demonstrates that the most frequent concepts used to disambiguate objects are spa- tial predicates, both as pair-wise relations ("ftont", "right", etc.) and as absolute location (e.g. "leftmost" , "closest"). Certain object names (e.g. "kleenex", "tissues", "cereal") appear more frequently, as those are the objects that are most commonly ambiguous in OCID scenes, hence they spawn a lot of expressions referring to them. Finally, colors and brand names appear also frequently, as they are the most common discriminating attribute between objects of the same category.
tissues
nee
banana ia
cerea
orange sugats dats
app le cube
flashli nilbottle
sponge
ex
Light
produs
Bell
Pack
towe
1
rM
side
stapler
toothpast
te
nearest
corn
closest
kl eee
Feh
marke
bland
treactast
black
rightmos
bowl
beige
Vichy
ballo
comato
Food
lentil
S
shampoo
glue
keyboard
Choco
ba
Lemon
go
blue
ront
1
Topas
eef
e
f
pink
Lakes
pear
rediug
soda
potato
stick
brown
wh
dis
ite
tant
1
1g
cheess
soft
cup
yellow
Framppar
costneal
hand
rice
behind
Lovely
in tant
green
cubui
fur
thest
Aussle
noodle
pox
pa
dark Fanta ivo
Ka g
Apaghetti
e
mug
drink
Mega
rear
cof
Krappies
Time
lef
tmo S peach
Figure 5: Wordcloud OCID-VLG Vocabulary
The number of unique concepts per concept type, as well as the total number including paraphrases are presented in Table 5. Paraphrases include synonyms (e.g. "Coca-Cola", "Coke") ") as well as different phrasings of relations (e. "left of", "to the left side of").
Concept
Num.Unique
Num. Total
Category
30
55
Color
27
27
Instance
31
93
Relation
9
24
Location
Table 5: Number of concepts in OCID-VLG
Referring expressions might use instance-level names, attributes, relations, locations or combina- tions of all the above to disambiguate objects. We study the frequency of referring expressions on the OCID- VLG data splits in Table Most frequent type is name (which includes a lot of variety
Type
Train
Validation
Test
Name
20678
3014
5809
Attribute
2739
348
781
Relation
20501
2792
5769
Location
9306
1285
2672
Mixed
9997
1230
2718
Table 6: Number of referring expressions in OCID-VLG organized by type
in concepts such brand, flavor etc.) with pair-wise relations following closely. Spatial relations can always refer to the target uniquely by querying for relation to neighbouring object. Location
14

---

class
label
color
material
special
Apple
apple
red
organic
NaN
apple
applo 2
green
organic
NaN
bail1
blue
Maatic
NaN
ball
ball 2
yellow
Maztic
rugby ball
ball
baila
white
red and
platic
sprrs ball dots
polka ball ball with
bamara
barans_1
yellow
organic
NaN
pepper
@elppepee___
red
organic
NaN
binder
binder_
green
plastic
NaN
bowl
bowl 1
blue
ceramic
NaN
Topas box, Topas cereal
carmal box
oennal hrie
paper
Togass
Pack Mlega Fask com faakss
box
Pack
Choco Krisçieo box Choco Krisples cennal box Chhzn Krisples box, Choco Krispics
12 cannal box
conoal boxe
brown
paper
cereel_box
green
peper
Checos
Chocos
black
mug
migg.
mug
fachiight
Rashiight_
food_bag
big_?
trereparent
bag.heg
beg.pennne
food,
food,
and white
plactic
bag sprighttt penne bag, bog with posta.
rice bag Langkom rice
18
19
food,
foodi fnori_bag__ 4
white
plectic
hag chewer nowe bag big
with rice
Barlla
lond_eee box
food bow
Nus
paper
jngilato spaghent
banana
food box
food, box
yellow green
chocolate banena box box with choco banaaaa.
Figure 6: Full attribute catalog of OCID-VLG
and mixed follow at about half frequency, while color is last, as several objects OCID share color between different instances of the same category.
B Qualitative Results
We visualize predicted masks and grasp poses from the implemented baselines and the proposed CROG model in Fig. 7. We include two examples per referring expression type for scenes of OCID-VLG dataset. Zero-shot baselines based pretrained GR-ConvNet provide poor grasp proposals, while supervised baselines CL.PP (Det-Seg, SSG) are constrained by the ranking errors of CLIP. Due to segment-then-rann pipeline, spatial information about other objects lost when considering only the mask of single object. As a result, CLIP-based baselines struggles with grounding spatial relations. CROG is robust across referring expression types.
In Fig. 8, we visualize outputs of the CROG model during real robot experiments. The plots include predicted mask and grasp proposal, as well as the three decoded masks from CROG's grasp projec- tors (quality, angle and width masks). It should be noted that the corresponding input command is shown atop each image.
15

---

Input image
GT
SAM+
SAM+
SSG-CLIP
DeLS
CROG (Ours)
command:
Crab the Food can product
Text command:
Pass the bail 1 th spots
(a) Results in referring expressions by name.
image
SSG+CLIP
(Ours)
Text command:
Grasp the bive ball
Text command:
Grab the red soda drink
(b) Results referring expressions by altribute.
Input image
GT
SAM+
SAM+
SSG+CLIP
DeIS
CROG (Ours)
command:
the cereal package that rear right the bOX
Text command:
cereal box package
right of plaip mag
(c) Results in referring expressions by relation.
image
SAM+GLIP
SAM+GLIP
SSG+CLIP
DelSeg+CLIP
CROG
Text command:
Grasp the
nearest shampoo product
Text command:
Grasp the Neft ma marker
(d) Results referring expressions by location.
Input Image
GT
SAM+GLIP
SAM++LIP
SSG+CLIP
DetSeg+CLIP
CROG (Ours)
command:
Grasp the keyboard that the right the brown
Leenex
Text command:
Pick noodles the left of the white green
(e) Results in referring expressions by mix of concepts.
Figure 7: Qualitative results in OCID-VLG test scenes.
16

---

spange
Varea
weag Ligh
Vip vot
pick the closes shampco
gresp the on top or white
Figure 8: Qualitative results in real robot experiments.
17


Reviews:
[{"review_id": "cZBQsxXZn4", "reviewer": "Reviewer_upRx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the task of grasping objects from language referring expressions which can include name, intrinsic properties, and spatial relationships to other objects. It proposes a relabeling scheme for an existing dataset to augment it with language annotations including disambiguating intrinsic and extrinsic properties. It further proposes a method adapted from prior work for training a grasp and segmentation network on the dataset which finetunes a CLIP model for segmentation and grasp prediction. It presents comparisons against zero-shot baselines using off-the-shelf segmentation and language models. ", "summary_of_recommendation": "Overall I think the dataset and approach are valuable, so I weakly recommend accepting. However, I have some major concerns about the scalability of such a method, and about the evaluation protocol which I noted above.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Questions\n- Do held-out scenes in the OCID-VLG scenes contain seen objects? If objects in the test set were seen before but only in different configurations I feel this somewhat invalidates the point of a test set since it doesn't measure transfer to new objects, and should be made clearer in the text.\n- In the video included of \"ball on top of the box\", the referring expression is redundant since there is only one ball in the scene. The text states \"in each scene we include distractor objects of the same category as the queried object\". Is this a mistake?\n- Querying spatial relationships with 0-shot baselines is unfair since, as the authors note, the CLIP baselines have no access to spatial relationships given isolated object crops. It seems like this baseline is almost set up to fail, since including some rudimentary spatial post-processing would have been straightforward given a set of candidate objects, and ranking based on both CLIP similarity and spatial properties wouldn't be too challenging.\n- It would strengthen the experiment section to include tests on objects which are completely outside the training categories (not just instances) to investigate how much language understanding is retained on unseen categories through finetuning.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- I like the task of language gudied grasping, and appreciate the focus of the paper on referring expressions by spatial relations which is a notable lacking capability of many vision-language models\n- The dataset is a useful platform for conducting vision-language grasping research\n\nWeaknesses\n- One of the main motivations of using pretrained vision-language models and language is to capture their internet-scale pretraining and language understanding capabilities. Fine-tuning on a small set of objects with limited appearance largely throws away this internet-scale pretraining, as evidenced by failure cases being \"mostly for objects that are not similar in appearance to OCID-VLG categories\", as the authors note. Authors acknowledge this limitation and leave it to future work, however I feel it makes the results less interesting given their limited scalability. \n- The performance on grasping and grounding is not very high (around 20% and 60% in physical trials), and this is even for objects from similar categories as the train set (eg mug, marker, etc). I worry that CROG has essentially overfit to the provided dataset, raising questions about the difficulty of the dataset as well as the transferability of the method to practical deployment.\n- Baselines are not tested on actual grasping performance, but just IoU with the dataset, which doesn't capture the full story of performance (indeed, this can be seen by IoU for CROG being very high, but physical success quite low). I feel the author's should have compared to a zero-shot baseline in physical trials, for example querying an off-the-shelf grasp planner like GraspNet-1B, DexNet, etc and masking the grasps by the segmentation mask from CRIS or SAM+CLIP. Spatial relations could be processed with a scene-graph type post-processing similar to the data collection method except operating on detected masks.\n\nComments \n- I feel the connection to CRIS should be made apparent earlier in the introduction, given how heavily the method is based on it.\n- It would be helpful for analysis to separate results by spatial relation and intrinsic properties of objects like color\n- It would be helpful to provide an exhaustive list of all the object categories in the appendix (the word cloud is nice, but doesn't enable inspecting the difference between train/test object categories)\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "ZRtUoBcTVBa", "reviewer": "Reviewer_ZgiR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a dataset and method for 4-DoF grasping with language instructions, termed referring grasp synthesis. The dataset is based on the OCID dataset and adds language annotation to the grasps using the CLEVR engine. A novel method CROG is proposed, which is jointly trained on vision-language-based segmentation and grasp quality prediction. The method compares favorably to baselines and is demonstrated on a real dual-armed grasping setup.", "summary_of_recommendation": "The authors introduce a novel approach as well as a dataset that is very useful for top-down language-based grasping. Extensive experiments and ablations have been carried out with different handling of segmentation, language, and grasp synthesis modules. Sufficient real robot experiments have been carried out.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Can you hypothesize how you would use depth data in the pipeline? Where would it help the most?\n- How can the method be extended to 6-DoF grasping problems?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- Grasping in the real world with language instructions is relatively unexplored, and this paper does a good job of benchmarking it.\n- The use of the CLEVr engine to provide annotations for the dataset is clever. \n- The proposed CROG method with joint training of segmentation and grasp quality masks is interesting and is shown to perform well\nExtensive comparison and ablation experiments have been carried out.\n\nWeaknesses:\n- The proposed method may not work with any arbitrary language queries outside the dataset relation set.\n- The authors could add a discussion about how language-based grasping could be applied to 6Dof grasping\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "HeQGFX148iq", "reviewer": "Reviewer_xdNw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work focuses on robotic grasp synthesis with natural language object referral. In particular, many existing works utilize a multi-stage pipeline that separates the parsing of semantics and the synthesis of grasps. This study introduces a new benchmark for grasping in cluttered indoor scenes based on natural language expressions. Additionally, it presents a new single-stage, end-to-end model based on CLIP to generate grasps, demonstrating promising performance in spatial reasoning. The manuscript also includes tabletop 4DoF picking experiments conducted in both simulated and real environments.", "summary_of_recommendation": "Weak accept. The manuscript introduces a novel multi-modal grasping dataset that includes vision, language, and grasp data. Experiment results suggest that the proposed single-stage grasp algorithm outperforms two-stage algorithms. Some limitations: the dataset has a limited diversity of object choices, and does not yet support open vocabulary grasp queries. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Line 114-116 “The dataset consists of 1763 indoor tabletop RGB-D scenes with high clutter, including 31 object categories from a total of 58 unique instances.” Is the dataset generated based on a fixed camera pose or is there randomization for the camera poses? \n2. For the physical experiment, has the authors experienced any difficulty when transferring the model that is trained purely with data generated in simulation to real? \n3. Will the OCID-VLG dataset be open-sourced for public use? \n4. Line 247: The proposed approach uses a fairly large input image dimension for CLIP (416x416 instead of the standard 224x224). What is the runtime of the pipeline (i.e. how long does it take to generate each grasp?)\n5. Line 290: The CLIP baseline struggles with grounding spatial concepts and the proposed approach seems to have alleviated this problem. Is there any experiment on a table top setup where there are a few repeated objects and the only possible way to distinguish them is via spatial relation? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\nThe new grasping dataset, OCID-VLG, based on OCID-Grasp, contains multi-modal information such as labeled grasps, vision data, and semantic parsing. If open-sourced, it could prove helpful to the research community for future open-vocabulary grasping work. The authors also clearly delineate the differences between this new dataset and existing ones.\n\nAreas for Improvement:\nThe proposed method does not generalize well to objects outside the training distribution, which consists of 31 objects. Notably, many grasp synthesis papers [1,2] can generalize to a large distribution of objects, despite lacking semantic labels. The use of class-agnostic segmentation in combination with a grasp synthesis network [3] has demonstrated high success rates in real-world bin picking scenarios. The success rate for the proposed open-loop planner appears relatively low comparing to the aforementioned works.\n\nThe proposed method is a direct adaptation from CRIS. Broadly speaking, it accomplishes something very similar to CLIPort, which could be considered a single-stage grasp synthesis model. The key difference between the proposed approach and CLIPort, as the reviewer understands it, lies in how the CLIP model is used. While CLIPort uses a frozen CLIP model, the proposed approach fine-tunes CLIP. Reference [4] suggests that fine-tuning CLIP may lead to a reduction in its performance on long-tail queries. Such a comparison is absent in the manuscript, and the reviewer believes that conducting this experiment would help readers better understand the effect of CLIP in the proposed approach.\n\n[1] Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., ... & Goldberg, K. (2017). Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv:1703.09312.\n[2] Zhu, X., Wang, D., Biza, O., Su, G., Walters, R., & Platt, R. (2022). Sample efficient grasp learning using equivariant models. arXiv preprint arXiv:2202.09468.\n[3] Danielczuk, M., Matl, M., Gupta, S., Li, A., Lee, A., Mahler, J., & Goldberg, K. (2018). Segmenting unknown 3D objects from real depth images using mask R-CNN trained on synthetic point clouds. arXiv preprint arXiv:1809.05825, 16.\n[4] Kerr, J., Kim, C. M., Goldberg, K., Kanazawa, A., & Tancik, M. (2023). Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.09553.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In overall, it is an interesting research paper. The task of language guidied grasping is new and useful for the community. The paper also introduces OCID-VLG, a novel and demanding dataset that explores complete vision-language-grasping scenarios within naturally cluttered environments. This dataset establishes a link with grasp annotations sourced from the OCID-Grasp dataset. 

**Strengths**:

- An end-to-end CLIP-based model for pixel-level segmentation and grasp synthesis.

- Benchmarking of CLIP-integrated grasping methods in the real world with language instructions 

- An interesting use of the CLEVr engine to provide annotations for the dataset

- The dataset is a useful platform for conducting vision-language grasping research

**Weaknesses**:

- The performance on grasping and grounding is not very high.

- Baselines are not tested on actual grasping performance, but just IoU with the dataset. The author's should have compared to a zero-shot baseline in physical trials.

- The dataset has a limited diversity of object choices, and does not yet support open vocabulary grasp queries. 

- Generalization ability is limited

- Only with 4DoF grasp proposals

Post-rebuttal: Please revise the paper in line with the rebuttal discussion for the camera-ready submission. It would be useful for the community if the created dataset is published with the paper publication.
Decision: Accept (Poster)