Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding
Authors: Eslam Mohamed BAKR
Yasmeen Youssef Alsaedy
Mohamed Elhoseiny
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D Visual Grounding
Multi-Modal
3D
3D Detection
Synthetic 2D generation.

Abstract: 3D visual grounding task has been explored with visual and language streams to comprehend referential language for identifying targeted objects in 3D scenes.
However, most existing methods devote the visual stream to capture the 3D visual clues using off-the-shelf point clouds encoders. The main question we address is “can we consolidate the 3D visual stream by 2D clues and efficiently utilize them in both training and testing phases?”. The main idea is to assist the 3D encoder by incorporating rich 2D object representations without requiring extra 2D inputs. 
To this end, we leverage 2D clues, synthetically generated from 3D point clouds, that empirically show their aptitude to boost the quality of the learned visual representations. We validate our approach through comprehensive experiments on Nr3D, Sr3D, and ScanRefer datasets. Our experiments show consistent performance gains against counterparts, where our proposed module, dubbed as LAR, significantly outperforms state-of-the-art 3D visual grounding techniques on three benchmarks.
Our code will be made publicly available.

Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding
Eslam Mohamed Bakr, Yasmeen Alsaedy, Mohamed Elhoseiny King Abdullah University of Science and Technology (KAUST)
(eslam. abdelrahman, yasmeen.. alsaedi, mohamed. elhoseiny) edu. sa
Abstract
The 3D visual grounding task has been explored with visual and language streams comprehending referential language to identify target objects in 3D scenes. How- ever, most existing methods devote the visual stream to capturing the 3D visual clues using off the-shelf point clouds encoders. The main question we address in this paper is "can we consolidate the 3D visual stream by 2D clues synthesized from point clouds and efficiently utilize them in training and lesting?". The main idea is to assist the 3D encoder by incorporating rich 2D object representations without requiring extra 21) inputs. To this end, we leverage 2D clues, synthetically generated from 3D point clouds, and empirically show their aptitude to boost the quality of the learned visual representations. We validate our approach through comprehensive experiments on Nr3D, Sr3D, and ScanRefer datasets and show consistent performance gains compared to existing methods. Our proposed module, dubbed as Look Around and Refer (LAR), significantly outperforms the state-ofi of- the-art 3D visual grounding techniques on three benchmarks, i.e., Nr3D, Sr3D, and ScanRefer. The code is available at https:/esslambbargguuuu Wtatuuubio/LARRgihhiiui
Introduction
Visiolinguistic modeling has attracted much attention over the past decade due impact on various areas,
including visual question answering [ 36), 3D scene
captioning 77, image-text matching [26] 44 visual
relation detection (25]. and scene graph generation
(42. Consequently, understanding natural language utterances and grounding them real-world scenar- ios is attractive due their importance in many real applications, such as robot navigation and interaction in indoor environment [2 [40].
dessel you. It whan
Visual Grounding (VG) has been explored in recent extend- ing object detection to a more complex task. How- ever, for real robots to intelligently ground a given utterance on natural scene in challenging environ- ment like a crowded indoor or outdoor scene, 3D rep- resentations are needed to understand the surrounding environment better. Therefore, 3D visual grounding task have been proposed by ReferIt3D [2] and Scan- Refer (5). The ReferIt3D (2) introduces two datasets containing natural and synthetic language utterances,
years in the context of 2D [6
55
Figure 1: Overview of our proposed approach, where we aim to localize referred object given an input utterance and 3D scene. Our novel approach generates
synthetic images for each object in the scene by placing virtual cameras surrounding the object. This novel approach enables to lever- age the rich 2D semantics representation without needing authentic extra 2D images not to limit po- tential application scenarios.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Table 1: Comparison of various 3D visual grounding models. Where GT indicates ground truth boxes used, Pred. indicates a pretrained detector is used to get the objects proposals. We show that Looking-outside [8]] and SAT [22 are using additional inputs such as the over all scene point clouds Spe, and extra 2D images 2D
respectively, Ope refers to object's Is point cloud.
RefitMD
Lang. Refer [3]
InstRel
TGNN
Looking
D3Net TransRef3D
3DYG
SAT
2
58
11
outside [8
[6]
[13
59
S2
Ours
Obj. Proposal
GT
GT
Pred.
Prod.
Pred.
Pred.
GT
Pred.
GT
GT
Lang. Encoder
RNN
DisilBert
GloVE
GLoVE
RoBERTa
GRU
DisülBert
GLoVE
BERT
BERT
Visual Input
Ore
Ope
Ope + SSC
OpC
Ope + Spc
Ore
OPC
OPc
Ope + +22mm
Ope
namely Nr3D and Sr3D, respectively, which aims determine the referent object from pre-defined objects set for each scene. In contrast, ScanRefer (5) aims to predict 3D bounding boxes for the referent object based on a given language description and 3D point clouds. Both ScanRefer [5] and ReferIt3D [ are built on the ScanNet dataset [8]. The critical difference, which made Referlt3D 2 more challenging, is that in each scene, several object instances belong the same fine-grained category as the referent, namely distractors.
Due to the sparsity and disorganization of the 3D point clouds, SAT [52 explores utilizing the 2D images provided by the ScanNet dataset [8] during training to assist point-cloud-language joint representation learning. SAT (52) proposes three variants; 1) The baseline, non-SAT, relies only on the 3D point clouds while tackling the grounding task. 2) Utilizing 2D semantics during the training phase only and masking them while inference, we termed it SAT. 3) Utilizing the 2D semanties in both phases, i.e., training and testing, we termed it SAT-GT. However, requiring extra 2D inputs in training inference limits potential application scenarios. To tackle this drawback, we hypothesized that incorporating synthetic 2D semantics will consolidate the 3D learned representation while not requiring any additional info. To this end, we propose a novel 2D Synthetic Images Generator (SIG) that projects the 3D point clouds into multi-view 2D images, as shown in Figure [] To the best of our knowledge, LAR the first method that augment learning of 3D tasks with synthetic 2D semantics. In addition, we propose novel mulli--ooda transformer-basee architecture. Our proposed architecture, LAR, on Nr3D outperforms SAT (52) by boosting the accuracy by 1.6% while exceeding all existing approaches that did not use any additional information than the 3D point clouds by wider margin, i.e., 5.0% to the closest one. Also, LAR architecture achieves state-of-the-arr results on Sr3D and ScanRefer benchmarks, 59.35% and 54.6%, respectively. Another variant is tailored from our architecture to be compared against SAT-GT, where geometry information and ground-truth class labels are concatenated to the 2D images; synthetic in case and original images in SAT case. Our LAR architecture outperforms SAT-GT by 11.7% ( 62% for our method and 50.3% for that). Our contributions are summarized as follows:
mulli-modee transformer- architecture, LAR, for the 3D visual grounding task that relies only on 3D point clouds, which exploits the synthetic 2D images generated by our SIG module.
We propose a novel End-to-Emt
We verify the effectiveness of our LAR architecture throughout extensive experiments with various benchmarks, i.e., Nr3D, Sr3D, and Sean which show that utilizing our 2D synthetic semantics significantly enhances the results.
Through detailed analysis, we examine our method's 's internal behavior and validity.
2
Related Work
Our novel architecture draws success from several areas, including 2D visual grounding, 3D visual grounding, and 2D semantics in 3D tasks.
-2D visual grounding. 2D visual grounding is locating an object based on a natural language description. Flickr30k (32) and ReferltGame (20] generated referring expressions for particular
object in an image used to identify which object is being referred to. Prior works [56] 27 29
have attempted to localize these referred objects based on the given expressions by focusing on the bounding- box level. In contrast, [53] 10 focus on the pixel- level predictions. Another 2D visual grounding taxonomy is the one-stage, also known as a single-stage framework, versus the two-stage framework. Single-stage methods (e.g., [88 49 50 fuse the linguistic query with each pixel/patch of the image, and generate set of possible bounding box candidates. In contrast, the two-stage framework [45] [4 [6] uses the visual image semanties generate object proposals. Afterward, the

---

Figure 2: Simplified overview of our 2D Synthetic Images Generator (SIG) module. First, we determine the prominent face of each object w.r. the scene center. Then, the camera is located at distance d' from that face and at a distance du from the room's floor. Finally, we randomly extend the region of interest by e.
grounding prediction is selected by comparing each proposal to the language query. Consequently, 2D visual grounding methods can guide learning 3D relationships. Driven by this, explore using 2D synthetic semantics to consolidate the 3D visual stream.
3D visual grounding. Table | summarizes the existing 3D visual grounding (3D VG) models whether ground truth object proposal used or a pre-trained detector is utilized to generate the proposals. Also, it summarizes whether extra info is used alongside the object's 's 3D point cloud or not. 3D visual grounding field has undergone extensive research and made considerable progress 19] A 3D visual grounding task uses language instruction along with 3D scene 8 as input order to estimate bounding box that allocates the most relevant object to the provided expression. However, the literature offers several extensions, such as using auxiliary input to improve localizing 3D objects [52]. Previous work 59 1 58 19). focuses on improving the language and 3D scene fusion by adopting two-stage framework. The first stage outputs 3D object proposals given a 3D point cloud representing scene. Secondly, visual and linguistic features fused to compute confidence score for each 3D object proposal. The first stage could be skipped by directly using the ground-truth proposals instead of predicting them [52]
over the last few years [2 5 12 17, [5, 37
13 58 19 52
6
21
do not assume the access to the ground-truth objects and use an object detector to extract the proposals. Intuitively, if the detector fails, the referring performance will significantly decrease. Therefore, objective related to the grounding task, which is learning 3I) visual-language joint representations by assuming access to the ground-trutt boxes. In addition, we explore the other setting that utilizes detector-generee generated proposals, which is mentioned in the Appendix.
13 2] In contrast, 66, 19 59
focus the main training
-2D semantics in 3D tasks. 2D semantics contains useful information that can be used detect 3D objects. The representations of visual scenes consist of point clouds, meshes, or volumes, while 2D images consist of pixel grids. Combining the best of two worlds can enhance 3D object localization. Researchers have attempted to solve the 3D detection problem by adopting different ways of representing RGB-D data 34
41 24 By either projecting the 2D image on 24 or fuse both 2D features with the 3D
48 23 15
the 3D scene [34] 48
(33. SAT [52) leveraged 2D semantics perform the 3D VG task. However, requiring extra 2D inputs limits
23 15
potential application scenarios. In contrast, in this work, we generate synthetic 2D images during the training phase, which is also used in the inference phase. The synthetic images enable us to consolidate the 3D visual stream by 2D semantics without requiring extra 2D inputs.
3 Look Around and Refer (LAR)
LAR aims to enhance the visual module, which in turn enhances the final objective.i.e. referential accuracy. By incorporating the synthetic 2D semantics alongside the 3D point clouds, LAR inherently captures the correlations between the 2D and the 3D streams. To design general framework to fit in any 3D application, when the input is 3D point clouds, the need for extra information, such as 2D semantics, should be bypassed. To this end, a 2D synthetic images generator, namely SIG, is introduced, which is discussed in Section 3.1. Then, we demonstrate how to learn robust representations by aligning the generated 2D semantics with the 3D ones by incorporating our SIG module into our proposed architecture, depicted in Section 3.2
3

---

Ch
Visual Module
Multi-Modal Module
Conahad
30 Head
Vixual Transformer
Faslon Transforent
Fusion
Coeaken
TR Magal
"This a set three windows high up in the
Figure 3: Detailed overview of LAR. The Visual Module incorporates rich 3D representation from the extracted 3D object points with the 2D synthetic image features using Visual Transformer. The 2D synthetic images first extracted by SIG and then processed by shared ConvNext backbone. Simultaneouslyy language descriptions are transformed into tokens and embedded into a feature vector. Then, the Multi-Modal Module fuses the output of the Visual Module separately by two Transformers. Finally, the output of the Multi-Modal Transformers is processed by Fusion Transformers.
3.1 Look Around: 2D Synthetic Images Generator
LAR exploits synthetic 2D semantics to reinforce the learned 3D representations. Consequently, this section will demonstrate our 2D Synthetic Images Generator (SIG) module that projects arbitrary 3D Point clouds representing an object to a 2D image. First, we introduce the essential operations of our module on single image, then show how we extend it to fulfill a multi-view setup.
A 3D scene, S IR Nx6, is represented by N points with spatial and color information, i.e., XYZ, and RGB, respectively. Following the previous works [2) [ 37 [52]. the object proposals x! for each scene are known, either generated by one of the off-the-she 3D object detectors manually annotated. The object proposal 2!"h R x6 is subset of the whole scene, where N' N and N represents the number of points that represent the object i.
Given a scene and an object proposal z our SIG module generates a 2D synthetic image :img represents the object The main idea is to place virtual camera in front of the fed object proposal, then project its points x. into the 2D plane to generate the corresponding 2D image x: inn The camera's 's location should be set carefully to generate a decent image representation for an object i. As shown in part (A) in Figure 2 after excluding the upper and lower faces from the fed object proposal, we get the center of the four remaining faces, then calculate the distance between each face center O², and the center of the scene Os, where ranging from 0 to 3. The nearest face I' from the scene center O₂ is the prominent one, colored with blue in Figure 2 which guarantees that we look at the object i from inside the room. The camera is located distance d from the room's 's floor; thus, the camera position is interpreted as Cam, = [d[ d', d'r), as shown in part (B) in Figure 2 The orientation of the camera frame is defined as follows. The z-axis is aligned with the principal axis of the camera, and two other orthogonal directions (x, y) align with the corresponding axes of the image plane. Failure cases are shown in Figure when the camera is located outside the room.
distance df from the prominent face !' and at
To project the points x: representing object i, from the scene coordinates to the camera coordinates, a transformation matrix M E 1R111 needs to be defined. Given the camera position Camp and the cen-
ter of the prominent face Or,, the transformation matrix M can be interpreted as,
y Vup Com,
Where the camera direction is defined by three vectors; V V and Vap, We define our virtual camera using the Unified Camera Model (UCM), which has five parameters I
[Tz, Yu Ca. Cys {]
7x çdt= [Tv cd+=]
Following the UCM, projection is defined follows: (xp"", ,))
+
Cyy
Afterward, the projected 3D point clouds are assigned to 2D grid map. To mitigate overfitting, we randomly extend the region of interest by as demonstrated in part (C) in Figure

---

Point clouds differ from 2D images, where every pixel contains spatial and texture details. Moreover, the object represented in a view-invariant way, unlike the images that are view-dependent. To mitigate this drawback in the image representation, multi-view setup is constructed as shown in Figure [ where v cameras are mounted in a circle around the prominent face I'. The pseudo-code for our SIG module is attached in the supplemental materials.
3.2 Refer: Visiolinguistic Transformer Architecture
We present our approach, depicted in Figure 3 for learning multi-modal representations of 3D scene associated with language instruction. First, the proposed visual module enriches the 3D representations by utilizing the 2D synthetic images generator, namely the SIG module, described in Section [.1 Thanks to it, we do not require access to any imagery sensor during training or testing. Then, the multi-modal module learns joint representations targeted to the final task, i.e., referring to an object in the scene based on the language instruction.
-Visual Module. Given a 3D scene S E Ng combined with agnostic multiple 3D object proposals rrf" E RN''x x6 we aim to classify each object proposal by predicting the semantic labels First, for each object in the scene S, we generate multi-view symthetic 2D images x. 2=!= E R Xx x3 from the 3D object's 's point cloud =!"". where v is the number of views for each object. H and W are the spatial dimensions for the generated images. Second, we employ PointNet++ [35] as a 3D encoder, producing 3D features E E Rxxdmmpee Hence, the Tiny-ConvNext (28 is adopted as
2D encoder, that produce V; iing E R x W' xdimimo. Where H' and W represent coarse spatial grid of spatial features, each represented by a diminn. Then, the spatial dimensions are squeezed img producing o; E R Buxdsmamg, Since our 2D encoder encodes multiple views e for the same object i, a fusing mechanism is needed to fuse the views' features. Various possibilities are explored to settle down the best fusion mechanism. We can interpret this as a fully connected layer to fully capture the views and cross-channel interactions. In contrast, to learn the views' interactions and neglect the cross-channel relations, ID-Conv. could be utilized to avoid involving tremendous parameters as in the fully connected case. Using the 1D-Conv. with kgize = v as a fusion layer produce Frimg E R. 1 Xdiimema, where kaize is the kernel size. The fused 2D features Fx:ng and the 3D features F.! are concatenated and fed to visual transformer, which is composed of several stacked transformer layers. The visual transformer is utilized to capture the cross-correlation between the 2D, and the 3D features outputting refined features, VPC and Ving, guided by three weighted auxiliary losses, i.c., alignment loss between //c and Vamg, and two visual classification losses.
-2D semanties. Inspired by SAT [52] we explore different 2D semantic information types combina- tions. Where we formulate our 2D semantics as follows:
E Lo(d LL((Wa=)"))
where zę is the predicted class labels, W, are learned projection matrices, LN is layer normalization, and é indicates concatenation. We encode each object individually using different semantic infor- mation types, i.e., visual semantics E and geometry semanties x:""0 R The geometry semantics, x7"-0 represents the bounding box coordinates of object i in the scene space and encode the virtual camera parameters, i.e., intrinsic and extrinsic parameters, used while generating the 2D image x; ,mg Thus, it can be interpreted as a positional encoding for the vision transformer, where the interaction among objects is captured. In contrast, SAT [52) utilizes a ground-truth 2D semantics, where the z/ denotes the manually annotated class labels, and the E:"g represents the embedding generated from real 2D images captured by imagery sensor g
-Multi-Modal Module. Using a pre-trained BERT model followed by a language classification head adopted from Referit3D [2]. we embed words query Q. After embedding each of the visual features using the aforementioned visual module, we fuse the input modalities Q, VPC. Vimg simultaneously by using two different transformers; 3D Multi-Modal Transformer and 2D Multi- Modal Transformer, as showen in Figure 3 In 3D Multi-Modal Transformer, we jointly encode the two input modalities 1/PC and Q understand better the referring expression that captures co- occurrence relationships between different objects in the scene. Meanwhile, the same words query Q are encoded with /img to enhance context-aware representation. A separate 2D and 3D referring heads are adopted from Referit3DD, which are auxiliary tasks. Finally, the output of the multi-modal transformers is processed by a Fusion Transformer that chooses the referred object from the fused features, FPC and pimo.
5

---

3.3 Losses
Our architecture, termed LAR, consists of two visual streams, i.e., 3D and 2D, in addition to one language stream. To refer to the object correctly guided by the input utterance, we must first classify the objects accurately. Thus, two auxiliary losses are employed to optimize the visual module; 3D classification loss Cās and 2D classification loss L In addition, We adopted Referit3D [2] auxiliary loss for language classification loss C to cooperate in the visual grounding task by recognizing the referred object class according to the description. Hence, two auxiliary grounding losses ( and Freff termed as early referring losses, are exploited before the fusion transformer. As a result of these early referring losses, the model learns the correlation between language description and the visual features separately, which enhances the performance. Additionally, we adopted object correspondence loss Leor from SAT to encourage the 3D model to distill the knowledge from the 2 stream. Finally, two grounding losses, i.e., C and Fref were added after the fusion transformer to predict the scores of each proposal. We optimize the whole model in an End-to- End manner, unlike SAT which learns the 2D features Ei separately, with the following loss function:
L= Nets (Cal 2 \mvef * (C C t-ree (Cre) E22) ) Acor * Lcor + Alang Clang (2)
where Acta is the object classification loss weight, AEre/ is the early referring loss weight, Aref is the final referring loss weight, Acor is the object correspondence loss weight, and Atang is the language classification loss weight. Our implementation details are detailed in the Appendix.
Experiments
4.1
Datasets
We evaluate our proposed 3D visual grounding approach, LAR, on three benchmarking datasets, Nr3D [2]. Sr3D 22, and ScanRefer [5). Nr3D, 3D Natural reference dataset (2), consists of 41.5K natural, rree-form utterances collected by humans using referring game between two humans. As opposed to Sr3D (2]. which consists of 83.5K synthetic utterances. Consequently, ScanRefer provides 51.5K utterances of 11K objects for 800 3D indoor scenes.
4.2 Evaluation Metries
we followed the same metrics followed by the previous work (2) [52], where three evaluation metrics are mainly used, i.e., the referring accuracy, visual classification accuracy, and language classification accuracy. In the case of Nr3d and Sr3d datasets, followed the previous work convention, assuming the proposals are generated with GT objects. Following this assumption, the 3D grounding problem classification problem, where the model objective is to classify the referred object
is reformulated among the other objects correctly, termed referring accuracy. The referring accuracy is calculated based whether the model picks the correct proposal from of X proposals. Alongside the main objective, which is the referring accuracy, two auxiliary metrics are defined to evaluate the performance of each stream individually, i.c., the visual classification accuracy and the language classification accuracy. Given agnostic GT-proposals, the visual classification accuracy is measured based on whether the model predicts the object's class correctly or not from predefined set of possible classes. Similarly, language classification accuracy is another auxiliary metric that assesses the performance of the linguistic branch. This branch aims to correctly predict the class of the referred object based on the input utterance.
4.3 Implementation Details
Input Configuration. For the 3D input, we randomly sample 1024 points per proposal from its point cloud segment. Furthermore, for the synthetic 2D input, we follow the algorithm, which is scrutinized in Section 3.I During the testing phase, the camera is set 2 meters away from the prominent face
I' i.e., d 2, and meter away from the floor, i.e., dup extension ratio depicted in Figure
1. In addition, the region of interest
(C), is set to 2%. The generated image is square with
size of 32 pixels. Five views are generated per object, where the angle 0 between each camera 30% demonstrated in Figure Thus the camera is set to the following angles ( (0, 3030 -00 260".- -000) )

---

Table Ablation Study of 2D synthetic image resolu-
tion and multi-view setup on Nr3D 2
Table 3: Comparison between different types of 2D
Image Multi- Encoder
Referit3 2
non-SAT
52
Size
Views
Ref. Acc. Cls. Acc.
Ref. Acc. Cls. Acc
semantics information as Eq. on Nr3D 22.
3D
35.6
57.4
37.7
59.1
+g000 | 1 2
SAT
[22
Ours
32
28.0
47.34
30.9
49.31
Ref. Acc.
Cls. Acc.
Ref. Acc. Cls. Acc
64
32.0
$6.93
33.7
16.14
(a)
37.7
59.1
38.1
58.2
128
34,4
62.15
35,5
63.32
/
32
29.7
50.11
31.1
51.40
(b)
42.3
61.0
42.2
32.9
63.20
64.02
(c)
/
/
46.2
61.0
46.3
63.3
128
3
35.1
64.85
36.4
65.13
(d)
/ /
43.2
61.2
44.5
62.1
32
5
31.1
53.18
33.7
56.72
(c)
/
/ /
47.3
60.6
48.9
65.42
128
5
36.0
65.19
38.2
65.40
-Ntwork and Training Configuration. A pre-trained model on the ImageNet dataset [9]. Tiny- ConvNext 28 is used extract the features of synthetic 2D images. Our 2D backhone is shared
across different views. We adopt the same 3D encoder in the previous works (1 [2] [3 37 [8]
[52]. PointNet++ (35), encode the sampled points per proposal. For the language encoder, we adopt the BERT encoder [T]] with three layers. We adopted the same transformer architecture used in [52] [5] for our visual, multi-modal, and fusion transformers. All models are trained for 100 epochs from scratch, using the weight initialization strategy described in [14) As mentioned in SAT [52] the initial learning rate set to 10-4 and decreases by 0.65 every ten epochs Adam optimizer 21
and mini-bate batch size of 32 per GPU are used for training all our models. We set the losses weights as follows, Adts = 5, Amref 0.5, Aref = 5, Apor = 10, and Atang 0.5.
-Software and Hardware Details. We use the python augmentation framework, Albumentations 4 for image augmentation. In order perform an online augmentation while training, we implemented our 2D Synthetic Images Generator (SIG), described in Section [.1] in pure Pythonic approach without using any 3D libraries, which also speedup the training. Our module implemented in Python using the PyTorch framework and four Nvidia V100 GPUs.
4.4 Ablation studies
We conducted five ablation studies to assess the proposed architecture and to validate the contribution of each component to achieve good performance. These components are 1) 2D multi-views and 2D resolution. 2) Types of 2D semantic information. 3) Camera vs. image augmentation 4) SIG Module vs. 2D real clues. 5) Noisy Cameras Extrinsic and Intrinsic. More ablations are stated the Appendix.
-2D multi-views and 2D resolution. To assess our 2D Synthetic Images Generator (SIG), we have deliberately removed the 3D stream, colored with yellow in Figure 3, 3D encoder, 3D heads, and rely only on our generated 2D images only while tackling the referring task. Thus, our architecture is simplified to include only the 2D encoder, 2D multi-modal transformer, and 2D heads. We trained this 2D architecture variant with different image resolutions and multi-view setup combinations. As shown in Table 2 The referring accuracy increased linearly while increasing the image size or increasing the number of views for the projected images. However, our multi-vien vieew setup has the dominant effect, whereas using only 64 image resolution with five views surpassing one shot with 128 resolution. Also, we found that using the combination of five different views, each has 128 resolution, surpassing Referit3D and non-SAT variant, where both use 3D semantics only, which shows the quality of the learned representation via our novel synthetic 2D images generator (SIG).
-Types of 2D semantic information. Inspired by SAT [52). we dissect the role of different 2D semantic, following Eq. I As shown in Table 3 the first row (a) represents 3D variant of our architecture where only the 3D encoder, 3D multi-modal transformer, and 3D heads are included. Therefore, this variant mimics the SAT [52) configurations. Starting from row to row e, our synthetic 2D semantics are incorporated alongside the 3D semantics. Also, we compare against SAT [52) to highlight the critical differences the meaning of each semantic information. ROI in our architecture is the synthetic 2D image generated from the 3d point clouds using our SIG module. In contrast, SAT [22) uses the original ScanNet images during the training phase. Unlike SAT, which uses the ground-trus truth classification labels during the training and masks them during testing,
7

---

Table Comparison between Camera augmentation and Image augmentation on Nr3D 2
Table 5: Comparison between SIG module and
real 2D clues utilized by SAT Nr3D 2
Cam Aug. Img. Aug,
Ref. Acc.
Cls. Acc.
Method.
Training
Inference
Ref. Acc.
45.9
60.33
SAT
Real 2D Images
47.3
X
48.2
62.69
SAT
Real 2D Images Real 2D Images
47.9
X
47.8
63.97
LAR
SIG
SIG
48.9
/
48.9
65.42
LAR
Real 2D Images Real 2D Images
46.8
Table 7: Benchmark-
Table 6: Showing LAR robustness against camera augmentation.
Table Benchmarking re- sults on Nr3D 22. while us- ing the ground truth classifi- cation.
ing results on ScanRe-
for 5
Training Testing Ref. Acc.
Cls. Ace.
Method Ref. Acc.
48.2
62.69
Refi33D
46.9
Method Ref. Acc.
Cls. Acc.
x
/ 46.7
61.32
non-SAT
48.2
48.9
65.42
SAT
53.8
SAT-GT
50.3
61
/
48.8
65.22
SAT
52.3
Ours
62
64
Ours 54.6
the predicted labels from the 2D classification head z'. Accordingly, we do not need to mask the 2D semantics, as we are using synthetic ones. As shown in row (e), the best accuracy is achieved when jointly incorporate the three semantic types. We also notice significant superiority in classification accuracy where our synthetic 2D images assist the 3D classifier in both phases; training and testing.
-Camera vs. Image Augmentation. We have two kinds of augmentation during training: 1) Image augmentation. 2) Camera augmentation.
For image augmentation, we followed the conventional techniques used by the major computer vision applications like horizontal and vertical flipping, blurring, color jitter, random scaling, random shifting. and random cropping. For camera augmentation, we range d from 1.5 meters, d"P from 0.5 to 2.5 meters, and from 2% to 15%. By changing the variables mentioned above, the extrinsic camera parameters are augmented. In addition, random noise added the intrinsic camera parameters. Table demonsttates the effect of each augmentation type alone and the effect of fusing both of them. The best accuracy is achieved by utilizing the two augmentation types.
-SIG Module vs. 2D real clues.
To validate SIG module, tailored variant of our architecture, where our SIG module is replaced by the actual 2D clues produced by Faster R-CNN [39]. As shown Table replacing our SIG module with the features produced by Faster R-CNN [39] degrades the performance by 2%. In addition, even when we compared unfairly to methods that use additional 2D information, even the variant of SAT, which utilize real images during training and testing (row #2), we still outperform it by 1%.
-Noisy Cameras Extrinsic and Intrinsic. Due to the significant improvement introduced by utilizing detailed analysis conducted to explore
camera augmentation, as shown in Table robustness against the camera's variations. Consequently, our 2D synthetic branch learns how to fuse information across different cameras in a data-driven way. Therefore, we can train the model to be robust noise camera models, shown in Table
Both cameras dying and noisy parameters experiments could be thought as irrelevant as LAR utilizes virtual cameras. Thus, no camera dropout nor noise can be introduced. However, our SIG module, depicted in Section 3.1 can be utilized in wide range of applications when these robustness characteristics against such noise are essential for these applications. For instance, in the autonomous driving world, our SIG module can employed to convert dense lidar point clouds into synthetic images, which will be used to train an arbitrary neural network that tackles specific tasks (e.g. object segmentation in cocoon setup 31 47 Then, the model will tested using real cameras instead of generating synthetic imagery input from the lidar.

---

Table 9: Benchmarking results on Nr3D and Sr3D datasets 2 All reported techniques are using the ground-truth object proposals. Additional input column emphasize the extra information used besides object's point cloud. where Spe and 2D Img indicate the over all scene point clouds and extra 2D images, respectively. The symbol indicates we retrain the model for a fair comparison.
Additiinnl Input
Method
Overall(
Easy
Hard
View-dop.
View-indep.
Overall(o)
Easy
Hard
Viow-dep.
View-indep.
Refert:3D A
35.6
43.6
27.9
32.5
37.1
40.8
44.7
31.5
39.2
40.8
Text-Gaided-GN::
InstanceRefer 58
Sye
38.8
46.0
31.8
34.5
41.9
48.0
51.1
40.5
45.4
48.
3DReTTaanSoouss
39.0
46.4
32.0
34.7
41.2
47.9
50.7
38,3
44,3
47.1
3DVG-Tannfommr
FFL-3DOG 12
41.7
48.2
35.0
37.1
44.7
TransReier3D 13
42.1
48.5
36.0
36,5
44,9
57.4
60.5
50.2
49.9
57.7
LanguagRReerr
37
43.9
51.0
36.6
41.7
45.0
56.0
58.9
49.3
49.2
56.3
Non-SAT
SAT 52
20mg
49.2
56.3
42.4
46.9
50.4
$7.9
61.2
50.0
49.2
58.3
SAT 52
20
47.3
55.8
41.4
46.1
49.9
56.6
60.6
49.7
48.7
57.4
4.5
Comparison to State-of-the-art
Moreover, we achieve state-of-ther ofee----ttherttttt results on the three well-known 3D visual grounding bench- marks, i... Nr3D (2). Sr3D (2). and ScanRefer [5]. By effectively utilizing our 2D synthetic images, LAR outperforms all the existing methods, which do not use any additional training dataset, by large margin of 5.0% and 1.9% Nr3D and Sr3D, respectively, depicted in Table 9 Where, LanguageRefer 37 achieves 43.9%, while LAR achieves 48.9% on Nr3D. For Sr3D, our model achieves 59.35%, which is 1.9% increase over TeansRefer3D [13] In addition, for the ScanRefer 2 benchmark, we also achieve state-of-the-an results, by boosting the accuracy by 2.3% from the closest counterpart architecture; SAT 12 mentioned in Table
retrained SAT [52] for a fair comparison, as we notice difficulties reproducing and verifying SAT Although we do not use an extra input during training as SAT, which uses the original ScanNet 2D images 8 we surpass SAT by 1.6%, 2.7%, and 2.3% on Nr3D, Sr3D, and ScanRefer, respectively.
results |
4.6 Discussion and Analysis
First, we will demonstrate our qualitative analysis. Then, we conducted two experiments to probe the robustness of the representations learned by our proposed approach LAR.
Qualitative Analysis. Figure depicts two corner cases in our projection module (SIG). While assigning the projected 3D) points into a 2D grid, more than one point may share the exact spatial location on the grid, which means only one will be stored in the grid, and the rest of the points will be discarded. As shown in part (A) in Figure 5] on the left column, the floor overlaps the other objects due to the conflict between points during the assignation step. To overcome this behavior, a simple priority rank is given for each point based on its height. Part (B) shows another failure case, where the prominent face falsely localized, which causes placing the camera outside the room or inside other objects. Therefore, effect. An example of an ambiguous description is shown Figure (ggure66par part A. The model incorrectly predicts "the chair the right rather than the chair the left." The model and the human annotations are occasionally confusing for referring to the objects in the view-dependent deseription. While successful cases can be in Figure part B. For more qualitative visualization, please refer the Appendix.
localize the prominent face relative to the room center to mitigate this
EEvaluation with Strundd-Truthe Class Labels.
Table [8] shows another variant, which is tailored from our architecture to be compared against SAT-GT. Where the predicted class labels are replaced by the ground-truth ones. Our LAR architecture outperforms SAT-GT by 12%.
Ret Acc.
-Zero-shot Cameras Dying Testing. We deliberately drop the cameras during testing to verify the robustness of our learned representations
Figure 4: Cameras dropout.
Referring to issues number and 2 from the official SAT implementation.

---

Prediction
Facing the row of chairs, it's furthest left
B
Figure 5: Qualitative results for projection corner cases. Part A shows failure in assignation step for the projected 3D points into the 2D grid. Part B, demonstrates failure the prominent face local- ization step.
The towel hanging closest to the mirror
Figure 6: Our qualitative results. Case A shows a failure due to view-dependent de- scription, While case B shows successful case.
against camera variations. As shown in Figure 4 starting from our best model, which is trained using five cameras, the referring accuracy drops to 44.6% then 40. 40%1,,, when we only use three cameras then one camera during testing, respectively. Driven by these results, camera dropout techniques can be invoked to overcome the performance drop, by deliberately drop random camera during the training; replace its input by duplicate view of another camera.
-imiitations Following the standard setup of Referit3D 2 we assume access to the object proposals of each 3D scene, similar SAT [52] LeneuageRRefer 37 TransRefer3D [13] and 3DVG-Trrnss
[59]. Therefore, we use the ground-truth proposals in our experiments. Following this assumption, the 3D grounding problem reformulated as a classification problem, where the model objective is
to classify the referred object among the other distractors correctly. However, LAR is compatible with the setting that uses detected-based proposals.
Conclusion
We propose an efficient 3D grounding architecture that leverages 2D knowledge to reinforce the leamned 3D representations. To avoid requiring 2D semantics neither during training testing, introduce a 2D synthetic image generator, termed SIG. Our SIG module generates high-quality 2D synthetic images. LAR, 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding, consists of two stages, i.c. Visual and Multi-Modk modules. LAR preserves the advantages of distilling the synthetic 2D knowledge the 3D stream without requiring any additional information. Our experiments show a significant superior performance, where LAR achieves state-of-the-ar results across three different 3D grounding benchmarks by outperforming the existing approaches, which do not rely on extra training data, by large margin, i.e., 5.0%, 1.9% and 2.3% on Nr3D, Sr3D and RefnRefer, respectively. Hence, despite the unfair comparison with SAT, which extra 2D images, LAR surpasses SAT by 1.6%, 2.7%, and 2.3% on Nr3D, Sr3D, and ScanRefer, respectively. Furthermore, verify the robustness of LAR and its generalization ability via zero-shot experiments to intrinsic and extrinsic camera parameters and dying cameras. More qualitative results are attached in the supplementary materials.
Acknowledgments and Disclosure of Funding
This work is funded by KAUST BAS/1/1685-01-0 and is also partially supported by the SDAIA- KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI).
10

---

References
[1] Ahmed AbdeIrcheem, Ujjwal Upadhyay, Ivan Skorokhodov, Rawan Al Yahya, Jun Chen, and Mohamed Elhoseiny. 3dreftransformer: Fine- Fine-grained object identification in real- world scenes using natural language. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages -9500 2022.
[2] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In European Conference on Computer Vision, pages 422-440. 440. Springer, 2020,
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. CoRR, abs/1 550.00066044,,,, 2015.
[4] Alexander Buslaev, Vladimir Iglovikov, Bugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A Kalinin. Albumentations: fast and flexible image augmentations. Information, 1 11(2):225, 2020.
[5] Dave Zhenyu Chen, Angel X Chang, and Matthias NieBner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European Conference on Computer Vision, pages 202-2211 Springer, 2020.
[6] Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and Angel X. Chang. Dãnet: A speaker-listener architecture for semi-supervised dense captioning and visual grounding in rgb-d scans, 2021.
[7] Zhenyu Chen, Ali Gholami, Matthias Niessner, and Angel X. Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 3193-3203, June 2021.
[8] Angela Dai, Angel X Chang, Manolis Sav Maciej Halber. Thomas Funkhouser, and Matthias Nießner. Scannet: Richly- annottad 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and paltern recognition, pages 248-255.
Ieee, 2009.
[10] Jiajun Deng. Zhengyuan Yang. Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to-end visual grounding with transformers. CoRR, abs/2 104.05414 2021.
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
[12] Mingtao Feng. Zhen Li, Qi Li, Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang. and Ajmal Mian. Free-form description guided 3d visual graph network for object grounding in point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 2021.
[13] Dailan He, Yusheng Zhao, Junyu Luo, Tianrui Hui. Shaofei Huang, Aixi Zhang, and Si Liu. Transrefer3d: andity-and-relation aware transformer for fine-grained 3d visual grounding. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2344-2352, 2021.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034 2015.
[15] Ji Hou, Angela Dai, and Matthias Nicßner. 3d-sis: 3d semantic instance segmentation of rgb-d scans, 2019. [16] Ronghang Hu, Huazhe Xu, Mareus Rohrbach, Jiashi Feng, Kate Sacnko, and Trevor Darrell. Natural lan- guage object retrieval. Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4555-4564, 2016.
[17] Pin-Hao Huang, Han-Huug Lee. Hwann- Hwann-Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1610-1618, 2021.
[18] Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Looking outside the box to ground language in 3d scenes. arXiv preprint X 12.08879, 2021.
[19] Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Looking outside the box to ground language in 3d scenes. CoRR, abs/21 12.08879, 2021.
[20] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg, ReferltGame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 787-798, Doha, Qatar, Oct. 2014, Association for Computational Linguistics.
[21] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.668 2014.
[22] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arx1v: 2017.
11

---

[23] Jason Ku, Melissa Mozifian, Jungwook Lee. Ali Harakeh, and Steven Waslander. Joint 3d proposal generation and object detection from view aggregation, 2018.
[24] Jean Lahoud and Bernard Ghanem. 2d-driven 3d object detection in rgb-d images. In Proceedings of the IEEE international conference on computer vision, pages 4622-4630, 2017.
[25] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12689-12697, 2019.
[26] Kunpeng Li, Yulum Zhang, K. Li, Yuanyuan Li, and Yun Raymond Fu, Visual semantic reasoning for image-lext matching. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages
4653-4661, 2019.
[27] Yongfei Liu, Bo Wan, Xiaodan Zhu, and Xuming He. Learning cross-modal context graph for visual grounding. CoRR, abs/1911 1.0042,, 2019.
[28] Zhuang Liu, Hanzi Mao. Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint arXiv:2201.03545 2022.
[29] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. CoRR, abs/1511.02221 1.2283,, 2015. [30] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1 11-20, 2016.
[31] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European Conference on Computer Vision, pages 194-210. Springer, 2020. [32] Bryan A. Plummer. Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier. and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phran correspondences for richer image-to- to-sentnce models. CoRR, abs/1505. 0487000 2015.
[33] Charles R. Qi, Xinlei Chen, Or Litany, and Lconidas J. Guibas. Imvotenct: Boosting 3d object detection in
point clouds with image votes, 2020.
[34] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 918-927, 2018.
[35] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[36] Mengye Ren, Ryan Kiros, and Richard S. Zemel. Image question answering: A visual semantic embedding model and new dataset. CoRR, Abs/1545.02074, 2015.
[37] Junha Roh, Karthik Desingh, Ali Farhadi, and Dieter Fox. Languagerefer: apatial-language model for 3d visual grounding. In Conference Robot Learning, pages 1046-1056. PMLR, 2022.
[38] Arka Sadhu, Kan Chen, and Ram Nevatia. scro-shot grounding of objects from natural language queries. CoRR, abs/1908.07122 2019.
[39] Amaia Salvador, Xavier Giró-i Nieto, Ferran Marqués, and Shin'ichi Satoh. Faster r-cnn features for instance search. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 9-16, 2016.
[40] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740- 11740, 2020.
[41] Shuran Song and Jianxiong Xiao. Deep sliding shapes for amodal 3d object detection rgb-d images, 2016.
[42] Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In Conference on Computer Vision and Pottern Recognition (CVPR), 2020.
[43] Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martin-Martin, Cewu Lu, Li Fei-Fei, and Silvio Savarese. Densefusion: 6d object pose estimation by iterative dense fusion, 2019.
[44] Liwei Wang, Yin Li. Jing Huang, and Svetlana Lazebnik. Learning two- wo-bbrnnhh neural networks for image- text matching tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):394 407, 2019.
[45] Liwei Wang, Yin and Svetlana Lazebnik. Learning two-branch neural networks for image-tex matching tasks. CoRR, abs// 704403470, 2017.
[46] Fanyi Xiao, Leonid Sigal, and Yong Jae Lee. Weakly-superviser visual grounding of phrases with linguistic structures. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pages 5945-5954, 2017.
[47] Enze Xie. Zhiding Yu, Daquan Zhou, Jonah Philion, Anima Anandkumar, Sanja Fidler, Ping Luo, and Jose M Alvarez. M 2bev: Multi-camera joint 3d detection and segmentation with unified birds-eye view representation. arXiv preprint arXiv:2204.088 05088, 2022.
[48] Danfei Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bounding box
estimation, 2018
12

---

[49] Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo. Improving one-stage visual grounding by recursive sub-query construction. CoRR, abs/2008. 01059, 2020.
[50] Zhengyuan Yang. Boqing Gong. Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate one-stage approach to visual grounding. CoRR, abs/1908.06354, 2019.
[51] Zhengyuan Yang. Yijuan Lu, Jianfeng Wang, Yin, Dinei Florencio, Lijuan Wang. Cha Zhang,
Zhang, and Jiebo Luo. Tap: Text-aware pre-training for text-vqa and text-caption. In Proceedings of the
IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 8751-8761, 2021.
[52] Zhengyuan Yang. Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: semanties assisted training for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
1856-1866, 2021.
[53] Linwei Ye. Mrigank Rochan, Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. CoRR,
2019.
[54] Licheng Yu, Zhe Lin. Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L. Berg. Mattnet: Modular attention network for referring expression comprehension. CoRR, abs/ abs//880.08118 2018
[55] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference Computer Vision, pages 69-85. Springer. 2016.
[56] Licheng Yu, Patrick Poirson, Shan Yang. Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. CoRR, abs/1608.002 00272, 2016.
[57] Licheng Yu, Hao Tan, Mohit Bansal, and Tamara Berg. A joint speaker-listener-reitte
model for referring expressions. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 7282-7290, 2017.
[58] Zhihao Yuan, Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. In- stancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1791-1800, 2021.
[59] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 2928 2937, 2021.
13


Reviews:
[{"review_id": "uQkGDV7IcED", "reviewer": "Reviewer_G6JP", "summary": "This paper presents LAR, a novel method for the task of 3D language-visual grounding. The key idea of LAR is to leverage synthetic 2D images as an additional modality of data to the downstream object reference task. LAR outperforms all baselines on the selected 3D grounding benchmarks that do not use additional training data.\n", "questions": "Same as 1, 2, 5 raised above in the weakness section\n", "limitations": "Same as 3 mentioned above in weaknesses. Additionally I would have liked to see if the setup is set closer with object detection: instead of assuming access to object proposal, predict objects directly given language description, since principally it seems language should also help bounding box proposal and localization. However, this is out of the scope of this paper and has not factored into my scoring.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths ===\n1. The paper is well written and the presentation is clear.\n2. Ablation studies on the design choices are mostly thorough.\n3. Strong empirical performance over previous methods.\n\n=== Weaknesses ===\n1. The proposed SIG module uses heavy heuristics and hand-engineering. I am wondering if this can be learned – given 3D scene point clouds and a single object proposal bounding box, predict (multiple) camera extrinsics. It seems this can be learned end-to-end as the projection function seems also differentiable.\n2. While LAR uses BERT embeddings, I am wondering if switching to a image-language embedding will improve performance given the task requires modeling language and visuals together in the first place.\n3. I am not sure if learning simple 1D conv kernel weights (seems more like learned weighted average…) is a good practice to fuse multi-view images, since it ignores the context.\n4. Typo in L181-182.\n5. The current model seems to ignore interaction among objects and predicts reference for them individually. I am wondering whether modeling objects together might make a difference in the performance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O3ghwFbVVY", "reviewer": "Reviewer_yWd9", "summary": "The paper proposes a method to enhance the visual part of a 3D visual grounding system by integrating 2D semantic information. The system receives a 3D point cloud, a text query, and point subsets for the candidate objects as input. The paper proposes to render each candidate point set into multiple virtual cameras and use a transformer to fuse 2D and 3D information. The visual information is merged further with the text to classify the objects for finding the desired candidate.\n", "questions": "I have a couple of questions regarding the paper:\n1. In the limitation section, the paper mentions using a detector instead of getting the ground truth point set proposals. How would this work with noisy detections? If points are cut out or added, this will drastically alter the information gained from the 2D encoder especially compared to a method like SAT, which uses ground truth 2D images. Also, if the goal is to use a robot, it would most likely be equipped with a camera and get a stream of 2D images. What was the rationale for not using this information and only using the synthetic 2D images instead? Line 110 mentions: \"SAT [51 ] leveraged 2D semantics to perform the 3D VG task. However, requiring extra 2D inputs limits potential application scenarios.\" What limitations does the paper refer to here? As the motivation seems to be mainly for VR and robotics applications, where cameras are used either on the VR device or on the robot itself, are images not widely available?\n\n2. The paper experiments show that it outperforms methods that have access to ground truth 2D images. How can this be explained? Are the single view images in the dataset taken from arbitrary angles? Is this paper just using a more advantageous camera angle? Or is this mainly due to the multi-view setting?\n\n3. The paper focuses on indoor scenes by how it selects the viewing directions of the virtual cameras. How would this work in more open-world settings where the camera does not always point outwards? Would this approach still work?", "limitations": "The paper contains a concise section on the technical limitations, namely using ground truth candidate point sets (see my mention regarding this under questions). Negative societal impacts are not discussed at all. \n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. In my opinion, the overall technical idea of the paper, to use 2D information together with 3D information for visual grounding, is novel and exciting. In contrast to 3D point cloud encoders, 2D feature encoders can be pretrained on large datasets and aggregate more information. Generating synthetic images through point cloud rendering also generates multiple views per object, increasing the chance of generating a dominant view of the object. \n2. The method achieves strong results in terms of classification accuracy, outperforming methods that use ground truth 2D images through rendering synthetic multi-view images. \n3. I appreciated the qualitative examples in the supplementary material of both success and failure cases. Maybe showing qualitative results of SAT and adding a figure to the main paper would increase the readers understanding of the significance of the results.\n4. The paper contains code and a supplementary video.\n\nWeaknesses:\n1. Section 3.1, lines 145 - 168. In my opinion, this section is overly wordy, explaining a well-known concept (pinhole projection), which takes space away from other parts of the paper and could use additional context. The paper explains how to project 3D points to a pinhole camera plane. As this is a very well-known topic in the literature, it suffices to mention which conventions are used for the extrinsic and intrinsic camera matrices and how the camera axes are defined. This should at most take a short paragraph and can even be shortened by using equations cleverly. \n\n3. In contrast, section 3.2 does not contain enough information, in my opinion. Equation 1 contains the definition of how image information is fused with bounding box information, but this is merely stated as fact, without giving intuition about why this might be beneficial. Similarly, in lines 213-223, the paper presents the loss terms without a proper introduction. Most losses are not defined or only are merely referring to previous papers. In my opinion, this makes the paper hard to understand and not self-contained. Shortening section 3.1, the paper should contain enough space to add the missing details. \n\n4. Section 3.3 Software Details: Here, the paper mentions for the first time that image augmentations were used, without specifying what kind of augmentations and why this is needed. The type of augmentations is mentioned later in Line 299, but there might be a better place.\n\n5. Section 4: The paper does not introduce the used evaluation metrics. These should be contained within the paper so that the reader can evaluate what the quantitative results represent.\n\n6. The paper does not contain a qualitative comparison of the detection performance with state-of-the-art. I am guessing that the quantitative score measures classification accuracy, but it is not apparent how large of a difference the improvement the method provides quantitatively.\n\nMinor issues, spelling, and grammar:\n* Figures 4, 5, and 6 are of different sizes and, therefore, are not aligned. Please align them such that the captions align.\n* Table 6, Overall and hard performance on Nr3D. The paper marks its approach as best, although SAT performs slightly better. Please change this. I do not think it is an issue that the presented approach does not outperform the reported numbers from SAT, especially regarding the reproducibility issues. One of the paper's strong points is that the SAT approach has been retrained in this case. \n\nI think the authors should address these weaknesses to polish the paper and make it more self-contained. If these issues are addressed, I am willing to increase my score.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Iqk7hy7sMC6", "reviewer": "Reviewer_8Se9", "summary": "This paper studies 3D Visual Grounding. Its goal is to identify from a 3D point cloud the 3D object referred in a sentence. The idea in this paper is to utilize RGB image as another input modality to help the grounding task. Different from the previous method SAT [51], the RGB info in the proposed method is obtained by rendering the RGB point cloud from multiple viewpoints, while SAT [51] obtains it from camera input from one single view. The paper further proposes a novel architecture to fuse the rendered RGB input and the point cloud, leading to better performance than SAT [51].", "questions": "Overall I feel that a lot of details are missing from the paper. I think adding those details are necessary to make the paper self-contained and easier to be understood. For example:\n1. Where do the 3D object proposals come from during training and inference?  How do you know what is the upper and lower faces of the bounding box as stated in line 136?\n2. Where does the predicted label z'_i in line 197 come from? From a trained image classifier? How is it trained? How many classes is it producing?\n3. What is the \"language classification head\" in line 203? What is its purpose?\n4. How exactly are the losses in line 213 to 220 defined?\n\n\nI am also curious to know what the performance would be if the proposed method uses the same RGB input as SAT, i.e. one single image from camera. That would make the proposed SIG more justified.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper designs a clever rendering pipeline (SIG) to render the input point cloud into multiple images, which are fed into the network together with the point cloud. This strategy provides more RGB signal than pure camera inputs, and together with the novel network that fuses multi-modal inputs, the proposed method outperform SAT (which requires an RGB camera input). This is an effective strategy, and as the number of provided view increases, the performance increases (Table2).\n\n\nWeaknesses:\n- I think the strategy designed to render the point cloud seems debatable, and it is possible that there is another object blocking the path from the camera center to the proposal object, resulting in occlusion. In this case the rendered image would be incorrect. \n- It seems that the proposed method requires a trained image classifier? If that is the case wouldn't that make the comparison with SAT [51] unfair because more information is given to train this framework? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EeE1WkA6Fk", "reviewer": "Reviewer_J6Di", "summary": "This paper proposes the SIG module to project RGB-D point clouds of object proposals to multiple views of RGB images, which acts as another stream of visual clues to help point cloud-based 3D visual grounding. In contrast to SAT, this method does not rely on actual 2D images to generate the 2D clues. The proposed method, also called LAR, fuses the aforementioned 2D visual tokens and 3D visual tokens by a visual transformer, and then the enhanced tokens in each stream are respectively fused with language tokens for the identification of referred objects, while these two streams of visual-linguistic tokens can be fused again by a fusion transformer for the final referring identification. The experiments on Nr3D, Sr3D, and Scanrefer datasets, show that the proposed SIG can consistently improve the performance in comparison to the SOTA. ", "questions": "It is encouraging if the authors can answer the questions listed in the paper's weaknesses.", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "--- Strengths ---\n\n1. The proposed SIG module can directly exploit the 2D visual clues from RGB-D point clouds, without acquiring real 2D images associated with point clouds, thus is more flexible than SAT. \n2. The experiments contain a list of comparisons, validating that the proposed SIG can consistently improve the performance than the way that SAT is used for exploiting 2D visual data.\n\n--- Weaknesses ---\n\n1. Other than the SIG module, the overall novelty of this paper is limited, in comparison to SAT. However, the proposed LAR model has to explicitly employ the SIG module both at the training and testing stages, it thus requires more computational budgets than the previous 2D-3D combined methods, such as SAT. The computational cost of the LAR model should be discussed.\n2. The SIG module may require accurate object proposals for the rendering of 2D images, which is okay for Nr3D and Sr3D, but for actual scenarios such as ScanRefer, how to maintain the reliability of SIG-generated 2D data is not guaranteed. How will the performance degrade if the proposals are predicted?\n3. Even with a bunch of sophisticated fusion techniques and extra computational cost introduced by the 2D visual stream, the proposed method just achieve marginal gains than SAT, which harms the significance of the claimed contributions.\n4. The presentation of this paper needs significant improvement. For example, \n    - Tab. 2: What do referit3D and non-SAT mean in this table?\n    - Tab. 3: is SAT$^\\dagger$ means the re-implemented SAT?\n    - How the camera and image augmentations are applied in the experiments other than Tab. 4. \n    - A number of typos and grammar errors throughout the whole paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept