3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding
Authors: Zehan Wang
Haifeng Huang
Yang Zhao
Linjun Li
Xize Cheng
Yichen Zhu
Aoxiong Yin
Zhou Zhao
Venue: EMNLP 2023 Main (2023)
Keywords: 3D visual grounding

Abstract: 3D visual grounding aims to localize the target object in a 3D point cloud by a free-form language description. Typically, the sentences describing the target object tend to provide information about its relative relation between other objects and its position within the whole scene. In this work, we propose a relation-aware one-stage framework, named 3D Relative Position-aware Network (3DRP-Net), which can effectively capture the relative spatial relationships between objects and enhance object attributes. Specifically, 1) we propose a 3D Relative Position Multi-head Attention (3DRP-MA) module to analyze relative relations from different directions in the context of object pairs, which helps the model to focus on the specific object relations mentioned in the sentence. 2) We designed a soft-labeling strategy to alleviate the spatial ambiguity caused by redundant points, which further stabilizes and enhances the learning process through a constant and discriminative distribution. Extensive experiments conducted on three benchmarks (i.e., ScanRefer and Nr3D/Sr3D) demonstrate that our method outperforms all the state-of-the-art methods in general.

3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding
Zehan Wang
Haifeng Huang" Yang Zhao? Linjun Li Yichen Zhu'
Xize Cheng'
Aoxiong Yin' Zhou Zhao 'Zhejiang University ByteDancc
(wangzehanOl, huanghaifeng @zju.edu.cn
Abstract
Ii's fatisuees IN. The thete ctlead acal dacwisdos. On the bic.
3D visual grounding aims to localize the target object in 3D point cloud by free-form lan- guage description. Typically, the sentences de- scribing the target object tend to provide infor- mation about its relative relation between other objects and its position within the whole scene. In this work, we propose relation-aware one- stage framework, named 3D Relative Position- aware Network (3DRP-Net), which can ef- fectively capture the relative spatial relation- ships between objects and enhance object at- tributes. Specifically, I) we propose 3D Relative Position Multi-head Attention (3DRP- MA) module to analyze relative relations from different directions in the context of object pairs, which helps the model to focus on the specific object relations mentioned in the sen- tence. 2) We designed soft-labeling strategy to alleviate the spatial ambiguity caused by re- dundant points, which further stabilizes and en- hances the learning process through constant and discriminative distribution. Extensive ex- periments conducted on three benchmarks (i.e., ScanRefer and Nr3D/Sr3D) demonstrate that our method outperforms all the state-of-the-art methods in general.
Rava cunse The camarr langlng
compate
This ki beows wiod desk
will wiaebs.
Figure I: 3D visual grounding is the task of grounding a description in a 3D scene. In the sentences, all the words indicating the relative positions of the target object are bolded. Notice that relative position relations between objects are crucial for distinguishing the target object, and the relative position-related descriptions in 3D space complex (e-g., "above", "on the left", "in front of". and "next to", etc.)
irregular and more complex terms of spatial po- sitional relationships, which makes it much more challenging to locate the target object.
In the field of 3D visual grounding, the previ- ous methods can be mainly categorized into two groups: the two-stage upproaches (Chen al., 2020; Achlioptas et al., 2020; Zhao al., 2021b; Yuan al., 2021; Huang al., 2022; Cai et al., 2022; Huang et al., 2021; Wang et al., 2023) and the one-stage approaches (Luo et al., 2022). The for- mer ones follow the detection-and-rant paradigm, and thanks to the ficxibility of this architecture, they mainly explore the benefits of different ob- ject relation modeling methods for discriminating the target object. The latter fuse visual-text fea- tures predict the bounding boxes of the target objects directly, and enhance the object attribute representation by removing the unreliable proposal generation phase.
1 Introduction
Visual grounding aims to localize the desired ob- jects based on the given natural language descrip- tion. With the rapid development and wide appli- cations of 3D vision (Xia et al., 2018; Savva et al., 2019; Zhu et al., 2020; Wang et al., 2019) in re- cent years, 3D visual grounding task has received more and more attention. Compared to the well- studied 2D visual grounding (Yang al., 2019; Kamath et al., 2021; I; Yang al., 2022; Li and Si- gal, 2021; Deng et al., 2021; Plummer al., 2015; Kazemzadeh et al., 2014), the input sparse point clouds in the 3D visual grounding task are more
However, these two methods still have limi- tations. For two-stage methods, the model per- formance is highly dependent on the quality of the object proposals. However, due to the spar- sity and irregularity of the input 3D point cloud, sparse proposals may leave out the target object, while dense proposals will bring redundant com- putational costs and make the matching stage too
"Equal contribution.
Corresponding author.

---

complicated to distinguish the target object. As for the one-stage methods, although the existing approach (Luo et al., 2022) achieves better perfor- mance, they can not capture the relative spatial rela- tionships between objects, which makes it often fail in samples that rely on relative relation reasoning. As shown in Fig. 1, the majority of sentences in 3D visual grounding contain relative spatial relation descriptions. Furthermore, due the spatial com- plexity of the 3D scene, there are various relative position-related descriptions from different orienta- tions. To further illustrate that relative position is a general and fundamental issue in 3D visual ground- ing tasks, we analyze the frequency of relative po- sition words in ScanRefer and Nr3D/Sr33,, and the results show that at least 90% of the sentences de- scribe the relative position of objects, and most of them contain multiple spatial relations. Detailed statisties can be found in supplementary materials.
clouds only capture surface of object, and the 3D object centers are likely be far away from any point. To accurately reflect the location of objects and learn comprehensive relation knowledge, we sample multiple key points of each object. How- ever, redundant key points may lead to ambiguity. To achieve disambiguation while promoting more stable and discriminative learning process, we pro- pose soft-labeling strategy that uses a constant and discriminative distribution as the target label instead of relying on unstable and polarized hard- label or IoU scores.
Our main contributions can be summarized as follows:
We propose novel single-stage 3D visual grounding model, called 3D Relative Position- aware Network (3DRP-Net), which for the first time captures relative position relation- ships in the context of object pairs for better spatial relation reasoning.
To alleviate above problems, we propose one- stage 3D visual grounding framework, named 3D Relative Position-award aware Network (3DRP-Net). Our 3DRP-Net combines and enhances the advan- tages of the two-stage approaches for relations mod- eling and the one-stage approaches for proposal- free detection while avoiding the shortcomings of both methods. For the relations modeling, we devise a novel 3D Relative Position Multi-head Attention (3DRP-MA) module, which can capture object relations along multiple directions and fully consider the interaction between the relative posi- tion and object pairs which is ignored in previous two-stage methods (Yuan al., 2021; Zhao et al., 2021b; Huang et al., 2021).
We design a 3D Relative Position Multi-head Attention (3DRP-MA) module for simultane- ously modeling spatial relations from different orientations of 3D space. Besides, we devise a soft- soft-labeling strategy to alleviate the ambigu- ity while further enhancing the discriminative ability of the optimal key point and stabilizing the learning process.
Extensive experiments demonstrate the ef- fectiveness of our method. Our 3DRP-Net achieves state-of-the- performance on three mainstream benchmark datasets SeanRefer, Nr3D, and Sr3D.
Specifically, we first extract features from the point cloud and description, and select key points. Then, the language and visual features interacted while considering the relative relations between objects. For the relation modeling, We introduce learnable relative position encoding in different heads of the multi-head attention to capture object pair relations from different orientations. Moreover, in sentences, the relative relations between objects are usually described as "Object I-Relation- -Relaatio-Obheet 2", such as "tv is on the tv cabinet" and "curtain is hanging on the window" in 1. The relation is meaningful only in the context of object pairs, thus our relative position encoding would interact with the object pairs' feature, better capture and focus on the mentioned relations.
Related Work
2.1 3D Visual Grounding
Recent works in 3D visual grounding can be sum- marized in two categories: two-stage and one-stage methods. We briefly review them in the following. Two-stage Methods. Two-stage approaches fol- low the detection-and-rank scheme. In the first stage, 3D object proposals are generated by pre- trained 3D object detector (Chen et al., 2020) or with the ground truth (Achlioptas et al., 2020). In the second stage, the best matching proposals would be selected by leveraging the language de- scription. Advanced two-stage methods achieve good performance by better modeling the relation- ships among objects. Referit3D (Achlioptas et al.,
Besides, as discussed in (Qi et al., 2019), point

---

Training phase
Inference phase
Aue 1MM JMN
Sofilabeling
Sarether
Fawd-forrood Network
Salert
ComeosAttarbin
Posiion
Fask-fieweed Setwork
30 Relarie Poeiion
3D Relative Position Multi-heal Attention
Vali-Ateerie
3D Relatve Roaitio
Dv
Linen
Lanpagee Eacode
Muli. Alsstiaes
Patrorsr doveive
Ale K
Samping
Actehv Aowniing
Relative Position Atlention
as that sccee IV. The ti cahinet nett window.
Linear
Linear
Linear
Zalate
Produc
K
V
Q
K
Deseciption.
Point clond
(Relative Position Ateenion)
Figure 2: 3DRP-Net is transformer-based one-stage 3D VG model which takes 3D point cloud and a description inputs and outputs the bounding box of the object most relevant to the input expression. In the stacked transformer layer, the 3DRP-MA captures the relative relations between points in the 3D perspective. Specifically, the two self-attentions based on 3DRP-MA capture the relative relations between objects, while the rross-attention between key points and seed points enhances the global position information.
2020) and TGNN (Huang et al., 2021) make use the graph neural network (Scarselli et al., 2008) to model the relationships between objects. 3DVG- Transformer (Zhao et al., 2021b) utilize attention mechanisms (Vaswani et al., 2017) to enable inter- actions between proposals, and the similarity ma- trix be adjusted based the relative Euclidean distances between each pair of proposals.
mation about the position should be injected for each token. In general, there are two mainstream encoding methods: absolute and relative position encoding. Absolute Position Encoding. The original trans- former (Vaswani et al., 2017) considers the absolute positions, and the encodings are generated based on the sinusoids varying frequency. Recent 3D object detection studies also use absolute position encodings. In Group-free (Liu et al., 2021b), the encodings are learned by the center and size of the predicted bounding box, while the Fourier function is used in 3DETR (Misra et al., 2021).
One-stage Methods. One-stage approaches avoid the unstable and time-consuming object propos- als generation stage under the detection-and-rant paradigm. The visual features extracted by the backbone are directly and densely fused with the language features, and the fused features are lever- aged to predict the bounding boxes and referring scores. 3D-SPS (Luo et al., 2022) first addresses the 3D visual grounding problem by one-stage strat- egy. It firstly filters out the key points of language- relevant objects and processes inter-model interac- tion to progressively down-sample the key points.
Relative Position Encoding, Recently, some ad- vanced works in natural language processing (He et al., 2020; Raffel et al., 2020; Shaw et al., 2018) and image understanding (Liu et al., 2021a; Hu et al., 2019, 2018) generate position encoding based on the relative distance between tokens. Rela- tive relation representations are important for tasks where the relative ordering distance matters. Our method extends relative position encoding 3D Euclidean space and enhances relative relation reasoning ability in 3D visual grounding.
Our work utilizes the advanced one-stage frame- work and introduces novel relative relation mod- ule to effectively capture the intricate relations be- tween objects, enabling our model to achieve supe- rior performance.
Method
2.2 Position Encoding in Attention
This section introduces the proposed 3D Relative Position-aware Network (3DRP-Net) for 3D visual grounding. In Sec.3.1, we present an overview our method. In Sec.3.2, we dive into the techni-
The attention mechanism the primary component transformer (Vaswani et al., 2017). Since the attention mechanism order-independent, infor-

---

cal details of the 3D Relative Position Multi-head Attention (3DRP-MA) module and how to compre- hensively and efficiently exploit the spatial position relations in the context of object pairs. In Sec.3.3 and Sec.3.4, we introduce our soft-labeling strategy and the training objective function of our method.
tial reasoning challenges, thereby compromising their performance. To address this limitation, we propose a novel 3D relative position multi-head attention to model object relations in the context of corresponding object pairs within an advanced one-stage framework.
3.1 Overview
3.2.1 Relative Position Attention
The 3D visual grounding task aims to find the ob- ject most relevant to given textual query. So there are two inputs in the 3D visual grounding task. One is the 3D point cloud which is represented by the 3D coordinates and auxiliary features (RGB val- ues and normal vectors our setting) of N points, Another input is a free-form natural language de- scription with words.
Before detailing our relative position attention, we briefly review the original attention mechanism in (Vaswani et al., 2017). Given an input sequence (II,,.., Jn. of elements where 24 E Rd, and the output sequence same length where 2 Rdd, Taking single-head attention, the output can be formulated as:
{i1, ..., 2n with the
The overall architecture of our 3DRP-Net is illustrated in Fig.2. Firstly, we adopt the pre- trained PointNet++ (Qi et al., 2017) to sample S seed points and K key points from the input 3d point cloud and extract the C-dimensional enriched points feature. For the language description input, by using a pre-trained language encoder (Radford et al., 2021), we encode the I.-length sentences to D-dimensional word features. Secondly, stack of transformer layers are applied for multimodal fu- sion. The features of key points are accordingly in- teracted with language and seed points to group the scene and language information for detection and localization. Our new 3D relative position multi- head attention in each layer enables the model to understand vital relative relations among objects in the context of each object pair. Eventually, we use two standard multi-layer perceptrons to regress the bounding box and predict the referring confidence score based the feature of each key point. As shown in Fig.2, 2, in the training phase, generate the target labels of referring scores based on the IoUs of the predicted boxes. During inference, we only select the key point with the highest referring score to regress the target bounding box.
qi = =x;W k2 = k, 8s = x.WV
(1)
giky T
cap(ais)
ai.j vd
21
1= cxp(aa,)) Uj
(2)
where Wo, WK, Wy E RR. represents the pro- jection matrices, 0j the attention weight from element to j.
Based on the original attention mechanism, we propose novel relative position attention that in- corporates relative position encoding between el- ements. Since the semantic meaning of relative relation "Object intelattio-Objee 2" also highly dependent on the object pairs involved, it is essen- tial for the position encoding to fully interact with object features in order to accurately capture the specific relative relations mentioned in the descrip- tion. To this end, the attention weight di, in our proposed relative position attention is calculated follows:
aij = bhil + gark
T qo pidbs)
+ r k2
(3)
v3d
where dij represents the relative distance from el- ement to element j, while dzi is the opposite. p(d) E [0, 2k) is an index function that maps con- tinuous distance to discrete value, as detailed in 1
3.2 3D Relative Position Multi-head Attention
When describing an object in 3D space, relations between objects are essential to distinguish objects in the same class. Given the spatial complexity of 3D space and the potentially mislcading similar relative positions between different object pairs, precise and thorough comprehension of the rela- tive position relationships is crucial for visual grounding. However, existing 3D visual grounding methods fail to effectively address complex spa-
Eq.4 p. "P()) p(-) E (2k) 1)xds is the learnable rela- tive position encoding. Considering typical object relation expression "Object I-Relation- 1-Relation-Objecl 2", our attention weight can be understood as sum of three attention scores on object pairs and rela- tion: Object I-to- -bjeett 2, Object I-to- -te-Relation, and Relation-to---A to Object 2.

---

3.2.2 Piecewise Index Function
Saft-label
The points in the 3D point cloud are unevenly distributed in Euclidean space, and the relative distances are continuous. To enhance the relative spatial information and reduce computation costs, we propose to map the continuous 3D relative dis- tances into discrete integers in a finite set. Inspired by (Wu et al., 2021), we use the following piece- wise index function:
30--
Diserimination
Obe-et
00-
044
Stability
00
ld. sign(() x min(k, la
Id E a (h- a))), Id > 1a(3/o)
p(d)
bnfld/o)
Figure 3: Comparison of various labeling strategies.
(4) where [ is a round operation, sign()) represents the sign of a number, i.e., returning for positive input, 11 for negative, and 0 for otherwise.
3.3 Soft-labeling Strategy
Due the object center are often not contained in the given point clouds, we select multiple key points for each object to better reflect its location. Therefore, as shown in Fig.3, there will be lots of accurately predicted boxes achicving high Intersce- tion over Union (IoU) of target object. Previous methods (Chen et al., 2020; Zhao et al., 202 Luo et al., 2022) use one-hot or multi-hot labels to su- pervise the referring score. The key points whose predicted box has the top N, highest IoU are set to and others are set to O, which can encourage the model select the most high-IoU proposals. However, the simple hard-labeling strategy results in two problems: Firstly, proposals with similar and high IoUs may be labeled differently as and 0, which can cause unstable training phase. Sec- ondly, it becomes difficult to distinguish between optimal and sub-optimal proposals, affecting the model's ability to accurately identify the most curate proposal.
Eq.4 performs a fine mapping in the range. The further over a, the coarser it is, and distances beyond 3 would be mapped to the same value. In the 3D understanding field, many studies (Zhao al., 2021a; Misra et al., 2021) have demonstrated that neighboring points are much more important than the further ones. Therefore, mapping from continuous space to discrete values by Eq.4 would not lead to much semantic information loss while significantly reducing computational costs.
3.2.3 Multi-head Attention for 3D Position
Till now, our relative position attention module can handle the interaction between object features and relative position information in continuous space. However, points in 3D space have much more com- plicated spatial relations than pixels in 2D images or words ID sentences. As shown in Table relying on a single relative distance metric lcads to insufficient and partial capture of inter-object relations. This makes it difficult to distinguish the target object when multiple spatial relations are described in the language expression. Therefore, we attempt to capture object relations from multi- ple directions. Specifically, we encode the relative distances under x, y. z coordinates, and the Eu- clidean metric, denoted as Dr,, Dy, Dz. and De. respectively. These four relative position metrics represent most of object relations in the language description (e.g., Dg for "left, right", Dy for "front, behind", D2 for "top, bottom", D. for "near, far"). Based on the architecture of multi-heat attention, each relative position encoding is injected into the relative position attention module of each head. Such 3DRP-MA allows the model to jointly at- tend to information from different relative relations in 3D space.
To tackle these issues, we introduce a soft- labeling strategy to smooth the label distribution and encourage the model to effectively distinguish the optimal proposal. To be specific, the soft- labeling function can be calculated as follow:
8; expp
20 1)
(5)
where E (o...., N.) represents the i-th highest IoU. We set a as [Ng/3] to control the smoothness of the distributions. The target label of the key- point whose predicted box's IoU is i-th highest and greater than 0.25 is set to 8,, and others set to O. Although this strategy is simple, its role is do more as one stroke, and the insight it provides is non-trivial.
For discriminative ability, the soft-labels en- hance the difference between the optimal and sub-

---

optimal proposals, which enforces the model to accurately identify the best key point for regressing detection box. In contrast, when hard-labels or loU scores are used as the target labels, there is little dif- ference between optimal and sub-optimal proposals from the perspective of learning objectives. For sta- bility, compared to hard-labels, our soft-labels can cover a broader range of accurate proposals with smoother label distribution, and excluding the pro- posals with low IoU further stabilizes the learning process. Additionally, compared to directly using IoU scores, the constant distribution in soft-labels provides a more stable loss across different sam- ples. For example, if we have two samples with vastly different target objects, such as large bed and small chair, the bed sample would have signif- icantly more key points selected, resulting in more proposals of the target object. Using loU scores as labels would ultimately lead to a much larger loss for the bed sample than the chair sample, which is clearly unreasonable.
4
Experiment
4.1 Datasets and Metries
ScanRefer. The ScanRefer dataset (Chen et al., 2020) annotates 800 scenes with 51,583 language descriptions based on ScanNet dataset (Dai et al.. 2017). Following the ScanRefer benchmark, we split the train/val/test set with 36,655, 9,508, and 5,410 samples, respectively.
Nr3D/Sr3D. The Nr3D and Sr3D are two sub- datasets in ReferIt3D (Achlioptas et al., 2020). They are also annotated on the indoor 3D scene dataset Scannet (Dai al., 2017). Nr3D contains 41, ,,503 human utterances collected by ReferltGame, and Sr3D contains 83,572 synthetic descriptions generated based on a "target-spatial relationship- anchor object" template.
Evaluation Metric. For ScanRefer (Chen et al., 2020), following previous work, we use Acc @ mloU as the evaluation metric, where m e {0.25, 0.5}. This metric represents the ratio of the predicted bounding boxes whose Intersection over Union (IoU) with the ground-truth (GT) bounding boxes larger than m. For Sr3D and Nr3D (Achlioptas et al., 2020), the ground truth bounding boxes are available, and the model only needs to identify the described object from all the bounding boxes. Therefore, the evaluation metric of these two datasets is accuracy, i.e., the percent- age of the correctly selected target object.
3.4 Training and Inference
We apply a multi-task loss function to train our 3DRP-Net in an end-to-end manner.
Referring Loss. The Referring loss Lref is cal- culated between the target labels discussed in S.c.3.3 and predicted referring scores S of K key- points with focal loss (Lin et al., 2017).
Keypoints Sampling Loss. Following the loss used in (Luo et al., 2022), we apply the key points sampling loss Las to make sure the selected key points are relevant to any object whose category is mentioned in the description.
4.2 Quantitative Comparison
We compare our 3DRP-Net with other state-of-the- art methods on these three 3D visual grounding benchmarks.
Detection Loss. To supervise the predicted bound- ing boxes, we use the detection loss Ldet as an aux- iliary loss. Following (Luo et al., 2022), the Laet consists of semantic classification loss, objectness binary classification loss, center offset regression loss and bounding box regression loss.
ScanRefer. Table 1 shows the performance on ScanRefer. 3DRP-Net outperforms the best two- stage method by +4.20 at A.c@0.25 and +1.40 at Acc@0.5 and exceeds the best one-stage method by +2.45 at Acc@0.25 and +2.47 A.c@0.5. Even when compared to 3DJCG, which utilizes an extra Scan2Cap (Chen et al., 2021) dataset to assist its training, our 3DRP-Net still shows superi- ority in all metrics. Specifically, for the "Multiple" subset, 3DRP-Net achieves +2.66 and +2.34 gains when compared with the advanced one-stage model in terms of Acc @0.25 and Acc@0.5, which vali- dates the proposed 3DRP-MA module is powerful for modeling complex relative position relations in 3D space and significantly contributes to distin- guishing the described target object from multiple interfering objects.
Language Classification Loss. Similar to (Chen et al., 2020), We introduce the language classifica- tion loss Lreat to enhance language encoder. Finally, the overall loss function in the training process can be summarized as
L = =n + Lz + a.Lddet + ac La (6)
where the balancing factors a1, 0g ag, 04 are set default as 0.05, 0.8, 5, 0.1, respectively, and the Lref and Lies are applied on all decoder stages following the setting in (Qi et al., 2019).

---

Table I: Comparisons with state-of-the-eee methods on ScanRefer. We highlight the best performance in bold.
Unique
Multiple
Overall Acc
Methods
Acc600.25
Acc000.5
Acc6:0225
Acc@0.55
A.c
ScanRefer
67.64
46.19
32.06
21.26
38. 97
26. 10
TGNN
68.61
56.80
29.84
23.18
37.37
29,70
InstanceRefer
77.45
66.83
31,27
24.77
40.23
32.93
Two-stage:
SAT
2D nssist
73.21
50.83
37.64
25.16
44.54
30.14
Tran-orrnfformer
77.16
58.47
38.38
28.70
45,90
34.47
MVT
77.67
66.45
31,92
25.26
40.80
33,26
3DJCG
Scan2Cap
78.75
61.30
40.13
30.08
47.62
36.14
ViI.DDee
81.58
68.62
40.30
30.71
47.94
37.73
3D-SPS
81.63
64.77
39.48
29.61
47.65
36.43
One-stage:
3DRP-Net (Ours)
83.13
67.74
42.14
31.95
50.10
38,90
Table Comparisons with state-of-the-art methods on Nr3D and Sr3D. We highlight the best performance bold.
Nr3D
Sr3D
Method
View Dep
View Indep
View Dep
View Indep
Easy
Hard
Overall
Easy
Hard
Overall
Neft
43.6
27.9
32.5
37.1
35.6
44.7
31.5
39.2
40.8
40.8
InstanceRefer
46.0
31.8
34.5
41.9
38.8
51.1
40.5
45.4
48.1
48.0
3DVG- Tranrrrrnform
48.5
34.8
34.8
43.7
40.8
54.2
44.9
44.6
51.7
51.4
LanguageRefer
51.0
36.6
41.7
45.0
43.9
58.9
49.3
49.2
56.3
56.0
SAT
56.3
42.4
46.9
50.4
49.2
61.2
50.0
49.2
58.3
57.9
3D-SPS
58.1
45.1
48.0
53.2
51.5
65.4
56.2
49.2
63.2
62.6
MVT
61.3
49.1
54.3
55.4
55.1
66.9
58.8
58.4
64.7
64.5
ViL3DRel
70.2
57.4
62.0
64.5
64.4
74.9
67.9
63.8
73.2
72.8
3DRP-Net(Ours)
71.4
59.7
64.2
65.2
65.9
75.6
65.5
74.9
74.1
Nr3D/Sr3D. Note that the task of Nr3D/Sr3D dif- ferent from ScanRefer, which aims to identify the described target object from all the given ground- truth bounding boxes. Therefore, the soft-labeling strategy and the keypoint sampling module are re- moved. We only verify the effectiveness of 3DRP. MA on these two datasets. Besides, the data aug- mentation methods ViL3DRel (Chen al., 2022) are also used our training phase for fair compar- ison. The accuracy of our method, together with other state-of-the-art methods, is reported in Ta- ble 2. 3DRP-Net achieves the overall accuracy of 65.9% and 74.1% on Nr3D and Sr3D, respectively, which outperforms all existing methods by large margin. In the more challenging "Hard" subset, 3DRP-Net significantly improves the accuracy by 2.3% in Nr3D and + +1.6% Sr3D, again demon- strating our method is beneficial for distinguishing objects by capturing the relative spatial relations.
Relation Modeling Module. We compared our proposed 3DRP-MA with the relation modules other 3D visual grounding methods. For fair com- parisons, we also introduce distances in x, y, z coordinates and Euclidean space to other relation modules. The results are provided in Table 3, com- paring rows 1, and our 3DRP-MA is far su- perior the relation modules in EDVG-Tram and 3DJCG, and the performance improvement mainly comes from the subsets that rely on relative re- lationship reasoning for localization, namely the "ee"-Rel" and "Multi-Rel subsets.
Relative Position Encoding. In Sec.3.2.3, we dis- cuss the complexity of relative relations in 3D space and propose four relative position encod- ings based on relative distance in x.y.z coordinates (Dæyz), and the Euclidean metric (De), respec- tively. From Table 3, both Dayz and D), can bring significant improvement for subsets that require relative relation reasoning. Row 6 demonstrates that considering relative relations from multiple directions further helps capture comprehensive and sufficient object relations and distinguish the target object from multiple distractors.
4.3 Ablation Study
We conduct ablation studies to investigate the con- tribution each component. All the ablation study results are reported on the ScanRefer validation set.

---

Table 3: Ablation studies on relation position encoding and different relation modeling modules. Vone-Ree/One Rel/Multi-Ren represent subsets that contain zero/one/multiple relation descriptions in the original Multiple set of ScanRefer, and the relative percentage improvements compared to the different settings are marked in green.
Row
De
Days
Rel Module
Overall
Multiple
None-Rel
One-Rel
Multi-Rel
3DVG-Trane Transformer
36.85
30.16
(4.899))))5%)
32.51(+5.519)
28.03(+6.60%)
2
v
v
3DJCG
36.43
29.62
35.51(+1.159)
3187(+7.629%)
27.35(+9.25%)
3
3DRP-MA
32.74
26.39
34.18(45.09%)
2-.39(420:825)
23.94
4
V
x
3DRP-MA
36.43
30.26
35.47
32.54 1%)
28. (0(+3,33))
X
3DRP-MA
37.13
30.56
35.30(+1.769))
32.87(+4.35%)
28. (8.46(+4)99%)
6
V
3DRP-MA
38.90
31.91
35.92
34.30
29.88
Table 4: Ablation studies 3DRP-MA MA in each trans- former layer and pair-aware relation attention.
Table 5: Ablation studies the labeling strategies.
Strategy
N.
Acc@0.25
Acc@0.5
Row OI-R R-O2 SA, CA SAa Acc@0.25
A.c@0.5
Original
48.20
38.06
loUs
/
48.83
38,46
Linear
48.82
37.50
2
v
x
d
v
48,30
37.56
47.36
37.25
3
v
V
46.70
36.10
Hard
47.29
37.68
/
x
48.72
37.59
8
47.30
37.26
5
v
v
V
50.10
38.90
49.13
38.46
Soft
24
50.10
38.90
49.64
38.55
Pair-aware relation attention. The typical de- scription of a spatial relation can be expressed as "Object 1-Relation- Objecct 2". Our pair-aware rela- tion attention can be considered the sum of two scores: Object -to-Relation (O1-R) and Relation- to-Object 2 (R-02). To further verify the superi- ority of capturing the relation in the context of an object pair, we ablate the two scores, and the results are illustrated in Table 4. From rows 1, 2 and 5, both O1-R and R-O2 terms benefit the 3D visual grounding task by capturing the relative relations, and the joint use of Ol-R and R-O2 provides more comprehensive understanding of spatial rela- tion description and leads to the best performance.
ber of soft labels. To further demonstrate that our proposed strategy improves stability and discrimi- nation, we also use IoUs score as label. The "orig- inal" setting directly uses loUs as label, while the "linear" setting stretches IoUs linearly to the range of O to 1 to enhance discrimination. Compared to hard- lerd-llabling and IoUs methods, our soft-labeling strategy improves discrimination and stability. Us- ing the "original" IoUs method lacks discrimination power and stability due to the unbalanced loss on different samples. Even using linear scaling to en- hance discrimination power; this instability cannot be eliminated. Our method alleviates these prob- lems with a discriminative constant distribution and shows comprehensive superiority in Table 5.
3DRP-MA in each layer. We study the effect of each 3DRP-MA module in the transformer layer. SA,, CA and S/42 respectively denote whether to replace the self-attention before interacting with seed points, the cross- -ross-attention for key points and seed points, and the self-attention before interact- ing with language. Row 3 to 5 in Table 4 add each 3DRP-MA in turns and the performance is gradu- ally improved to 50.10% and 38.90%.
Conclusion
In this paper, we propose a relation-aware one- stage model for 3D visual grounding, referred to as 3D Relative Position-award Network (3DRP-Net). 3DRP-Net contains novel 3DRP-MA modules to exploit complex 3D relative relations within point clouds. Besides, we devise soft-labeling strat- egy achieve disambiguation while promoting a stable and discriminative learning process. Com- prehensive experiments reveal that our 3DRP-Net outperforms other methods.
Soft-labeling Strategy. Table 5 presents the per- formance of different labeling strategies. In hard- labeling, N₂ represents the number of key points whose IoU is in the top N, and greater than 0.25, which are labeled as 1. In soft-labeling, N, is a hyperparameter Eq.5, which controls the num-

---

6 Limitations
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0360 03654.
The datasets of 3D visual grounding task are all stem from the original ScanNet dataset which brings generalization to other scene types into ques- tion. More diverse benchmarks are important for the further development of the field of 3D visual grounding.
Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. 2018. Relation networks for object de- tection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3588- 3597.
Acknowledgments
Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. 2019. Local relation networks for image recogni- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3464-3473.
This work was supported in part by National Nat- ural Science Foundation of China under Grant No.62222211, Grant No.61836002 and Grant
Pin-Hao Huang, Han-Hung Lee, Hwann-Tzomn Tzong Chen, and Tyng-Luh Liu. 2021. Text-guided graph neural networks referring 3d instance segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1610-1618
No.62072397.
References
Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. 2022. Multi-view transformer for 3d visual ground- ing. In Proceedings of the IEET/CVF Conference on Computer Vision and Pattern Recognition, pages 15524 15533.
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia,
Mohamed Elhoseiny, and Leonidas Guibas. 2020.
Referit3d: Neural listeners for fine-grained 3d ob- ject identification in real-world scenes. In Euro- pean Conference on Computer Vision, pages 422- 440. Springer.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. 2021. Miltr-modulaatdd detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790.
Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 2022. 3djeg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16464-16473.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014, Referitgame: Referring to objects in photographs of natural scenes. In Proceed- ings of the 2014 conference on empirical methods in natural language processing (EMNLPP), pages 787- 798.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. 2020. Scanrefer: 3d object localization in rgb-d scans using natural language. In Euro- pean Conference on Computer Vision, pages 202- 221. Springer.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv: arXiv:1412.6000 1412.6980.
Shizhe Chen, Pierre-Louis Gubur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. 2022. Language conditioned spatial relation reasoning for 3d object grounding, arXiv preprint arXiv:221..00 100646.
Muchen Li and Leonid Sigal. 2021. Referring trans- former: A one-step approach multi-task visual grounding. Advances in Neural Information Process- ing Systems, 34: 19552-19644.
Zhenyu Chen, Ali Gholami, Matthias Nießner, and An- gel X Chang. 2021. Scan2cap: Context-aware dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 3193-3203.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias NieBner. 2017. Scannet: Richly-annotated 3d reconstructions of indoor scenes. Proceedings of the IEEE con- ference on computer vision and pattern recognition. pages 5828-5838
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021a. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022.
Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wen- gang Zhou, and Houqiang Li. 2021. Transvg: End- to-end visual grounding with transformers. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 1769-1779.
Ze I Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. 202 Group-free 3d object detection via transform- ers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2949-2958.

---

Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haib- ing Ren, Hao Shen, Huaxia Xia, and Si Liu. 2022. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Put- lern Recognition, pages 16454-16463.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Xin Wang, Qiuyuan Huang. Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang. William Yang Wang. and Lei Zhang. 2019. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6629 6638.
Ishan Misra, Rohit Girdhar, and Armand Joulin. 2021. An end-to-end transformer model for 3d object detec- tion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2906-2917.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrant correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649
Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng, Yichen Zhu, Aoxiong Yin, and Zhou Zhao. 2023. Distilling coarse-to-fine semantic match- ing knowledge for weakly supervised 3d visual grounding. arXiv preprini arXiv:2307.09267.
Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. 2019. Deep hough voting for 3d object detec- tion in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277-9286.
Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. 2021. Rethinking and improv- ing relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 10033-10041.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30.
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. 2018, Gibson env: world perception for embodied agents. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9068-9079.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR.
Li Yang, Yan Xu, Chunfeng Yuan, Wei Liu, Bing Li, and Weiming Hu. 2022. Improving visual grounding with visual- verification and iterative rea- soning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9499 9508.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li. Peter Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21 21(140):1-67.
Zhengyuan Yang, Boqing Gong, Liwei Wang. Wenbing Huang, Dong Yu, and Jiebo Luo. 2019. fast and accurate one-stage approach to visual grounding. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 4683-4693.
Junha Roh, Karthik Desingh, Ali Farhadi, and Dieter Fox. 2022. Languagerefer: lpatual-language model for 3d visual grounding. In Conference Robot Learning, pages 1046-1056. PMIR.
Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. 2021. Sat 2d semantics assisted train- ing for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1856-1866.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. 2019. Habitat: A platform for embodied research. Proceedings of the IEFF//VV International Conference on Computer Vision, pages 9339-99377.
Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang. Zhen Li, and Shuguang Cui. 2021. In- stancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring, In Proceedings of the IEEE/CV International Conference on Com- puter Vision, pages 1791 1791-
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008 The graph neural network model. IEEE transactions on neural networks, 20(1):61-80 80.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. 2021a. Point transformer. In the
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprini arXiv: TrXiv:8003.0255.
Proceedings of the IEEE/CVF International Confer- ence on Computer Vision pages 16259-16268.

---

Lichen Zhao. Daigang Cai, Lu Sheng, and Dong Xu. 2021b. 3dvg-transton transformer: Relation modeling for vi- sual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2928-2937.
A Qualitative Analysis
In this section, provide some visualization sults in ScanRefer (Chen al., 2020) for qualitative analysis.
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. 2020. Vision-language navigation with self- supervised auxiliary reasoning tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 10012-10022.
A.1 Analysis on Success Cases
To better understand our 3DRP-Net, we visualize some success cases and comparisons with the other one-stage method (Luo et al., 2022) in Figure 4. From (a,b,c), both 3D-SPS (Luo et al., 2022) and our 3DRP-Net accurately locate the target object when the description does not involve too many relative position relations and there not many interfering objects in the scene. However, as shown in (d.e,f), when the relative position relation be- tween objects is necessary for distinguishing the target object from multiple objects of the same cat- egory, the previous one-stage method 3D-SPS is often confused by distractors. By modeling the relative position in 3D space, our 3DRP-Net is able to fully leverage the relative position descriptions in the sentence for reasoning, which bring more precise localization.
A.2 Analysis on Failure Cases
To conduct a comprchensive qualitative evaluation, we further elaborate on the failure cases and discuss them in detail. These reasons for our 3DRP-Net prediction errors can be roughly summarized into three categories:
Ambiguous annotations. Due to the com- plexity and irregularity of 3D scenes, ambigu- ous descriptions are difficult be completely avoided in 3D visual grounding datasets. There may be multiple objects a scene that match the description, but only one of them is considered correct by the annotation. As shown in the cases (1,2,3) of Figure 5, both ground-truth objects and our predicted objects semantically match the natural language de- scriptions, but according the ground truth box annotations, our predictions are com- pletely wrong-
Challenging target object. In 3D point clouds, some objects are inherently diffieult identify because of obscured missing sur- faces. In case of Figure 5, the described target object is cabinet, but the point cloud the ground truth box is seriously missing,

---

Groend-Truth
3D SPS
SDRI Net(Ours)
A) This is toflet is made of purochin.
b). rectangsar winden office soble surouuded with black office
<) This balhroem ranigg bnown. It smonth
grown and black chuir pullid out from desk Thore is loryboard of
<). chair faxing up and left. bnow..
Da The objuct is cubinet is dinecily to your thoough door.
Description
Figure The visualization results of some success cases. untth/correc//ncorreet boxes.
The bluc/green/red colors indicate the ground
case 6, the auxiliary object is "chair of the cubicles", which is challenging for the model to recognize.
It This te long refee Thoe and mevitor en of
It The competer dask bus asdal logs in the orner of mom.
This is weokon dork with stpe The Tekk cloe wall.
B Statistics of Relative Position Words
To further illustrate that relative position relation a general and fundamental issue in 3D visual grounding task, we count some common words representing relative spatial relations in three 3D visual grounding datasets (i.e., ScanRefer (Chen et al., 2020), Nr3D (Achlioptas et al., 2020) and Sr3D (Achlioptas et al., 2020)) in Figure 6 and 7. From Figure 6, in ScanRefer, least 97% de- scriptions contain relative position relations, and more than 63% sentences use multiple relative posi- tion relations to indicate the target object. Besides, about 90% sentences utilize the relative position words in Nr3D, and almost all the samples in Sr3D require relative position relations between objects for localization. As shown in Figure 7, in ScanRe- fer and Nr3D, which collected human utterances descriptions, most of the commonly used relative position words appear in the sentences. This further demonstrates the importance of modeling relative position relations from different perspectives.
cabinet
chuir.
bed.
chaiove
Figure 5: The visualization results of some failure cases. The ground-truth boxes are labeled in blue and the in- correctly predicted boxes are marked in red.
which makes it very difficult to identify the cabinet in the scene.
Challenging auxiliary objects. 3D visual grounding task often requires the relations be- tween the target object and auxiliary objects to assist the localization. The challenging aux- iliary objects may result in an incorrect pre- diction. As shown in case 5 of Figure 5, the target table is on "the left of the bed", but the left and right side of bed difficult to distinguish, which requires identifying the direction of the bed according the position of pillows. This reasoning process is too com- plex for our model, and our prediction actually found the table on the right side of a bed. In
C Implementation Details.
We adopt the pre-trained PointNet++ (Qi al., 2017) and the language encoder in CLIP (Rad- ford et al., 2021) extract the features from point clouds and language descriptions, respec- tively, while the rest of the network is trained from

---

ScanReler
Nr0
Sr3D
25
Ratho%
Katto
SEST
540
2440 AUI
312
0314
Namber
Number
Namber
Figure 6: Ratio sentences containing the specific number of relative position words in three 3D visual grounding datasets.
scratch. We set the dimension d in all transformer layers to 384. The layer number of the transformer is set to 4. Our model is trained in an end-to-end manner with the A (Kingma and Ba, 2014) optimizer and batch size of 15 for 36 epochs. The initial learning rates of all transformer layers and the rest of the model are set and we use the cosine learning rate decay strategy to schedule the learning rates. The seed point num- ber M and keypoint number Mo are set to 1024 and 256. For the soft-labeling strategy, the label number N₂ is assigned as 24. In the piecewise in- set the
and 1e
7 1 and the 3 is assigned 20. When calculating the relative position index, the coordinates of all points are linearly scaled [0, 100].
dex function,
In the ablation study, we further divided the "Multiple" set of ScanRefer into "Non-Rel/One- Rel/Multi-Rell subsets according to the number of relational descriptions in the sentences. Specifi- cally, follow the statistical method in Sec. B to count some common words representing relative spatial relations.
Prior Methods for Comparison
In order to validate the effectiveness of the pro- posed 3DRP-Net, Sec. 4.2 comprehensively com- pare it to many previous state-of-the-art methods: !) Referlt3DNet (Achlioptas al., 2020) 2) Scan- Refer (Chen al., 2020); 3) TGNN (Huang et al., 2021); 4) InstanceRefer (Yuan al., 2021); 5) LanguageRefer (Roh et al., 2022); 6) SAT (Yang et al., 2021); 7) 3DVG-Trans (Zhao et al., 2021b); 8) MVT (Huang et al., 2022); 9) 3D-SPS (Luo et al., 2022): 10) 3DJCG (Cai et al., 2022); 11) Vil (Chen et al., 2022)

---

(a) SeanRefer
30
25
Frequeacy
right
left
side
next to
contrr
located
hetweeen
from
under
Eacing
against
above
behind
nex
far
by
below
up
over
attach
sooth
west
cast
head of
down
straight
benenth
eni
Word
(b) Nr3D
30
25
Frequcee55%
20
10
in
close
right
loft
from
nect to
sile
facing
corner
under
above
Gar
by
behind
between
agninst
end of
ont
over
lecated
below
dowa
straight
hend of
beneath
attach
east
north
south
Word
(c) Sr3D
40
30
Frequecc
20
15
10
from
füt
elose
near
on in
next side
between right
left
over
under
beneath
below
behind
up
down
by
straight
north
south
corner
head of
end ot
located
against
attach
Word
Figure 7: Frequency of some commonly used relative position words in three 3D visual grounding datasets.


Reviews:
[{"review_id": "Aa7zOgXWmr", "reviewer": "Reviewer_fZVS", "summary": "", "questions": "The dataset is not big. Are there other datasets you can also try out your method?\n\nCan you scale up your method to several orders of more data?\n\n== post rebuttal\n\nThe authors provided their explanations on the difficulties of addressing these questions. However, there is no new results or insights. I maintain my original rating.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1： Developing a foundation model like PaLI-X, RT-2 and Kosmos-2**\n\nFrom the task-setting perspective, these methods (PaLI-X, PT-2, Kosmos-2) are primarily based on 2D vision. However, the real world is 3-dimensional, and human perception of the world also encompasses depth information. Compared to 2D vision, the language-3D vision we are focusing on has unique and crucial advantages, including more spatial information, comprehensive object shape information, and precise localization within the real 3D world. Moreover, 3D vision also presents its own challenges, such as the sparse and irregular nature of 3D point clouds and the more complex spatial relationships understanding and reasoning.\n\nFrom the model development perspective, these 2D vision-language methods (PaLI-X, PT-2, Kosmos-2) benefit from well-studied image model architectures and strong pre-trained 2D-vision models of different downstream tasks (such as detection, question answering, and grounding). However, in the realm of 3D vision, larger-scale datasets and pre-trained models are still being explored. Considering the unique advantages and challenges of 3D vision, as well as its wild applications and vast potential, we believe the exploration of plain 3D visual grounding architectures with stronger relative relation modeling capability is meaningful and promising.\n\n**Q2: “The dataset is not big. Can you scale up your method to several orders of more data?”**\n\nCurrently, the larger-scale datasets for 3D-language tasks are very scarce. ScanRefer, Nr3D, and Sr3D are the biggest and most used datasets for 3D visual grounding. Other advanced methods for 3D visual grounding (all compared methods in our paper) also rely on these datasets.\n\nWe firmly believe that larger datasets and more powerful pre-trained models are crucial for the development of 3D-language filed. This will be an important direction for future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper has several notable contributions, which constitute compelling reasons for its acceptance:\n\nIt introduces an innovative single-stage 3D visual grounding model, termed 3D Relative Position-aware Network (3DRP-Net). This model incorporates relative position relationships in the context of object pairs, thus enhancing the capability for spatial relation reasoning.\n\nIt has also developed a unique 3D Relative Position Multi-head Attention (3DRP-MA) module. This mechanism allows simultaneous modeling of spatial relations from multiple orientations in 3D space.\n\nThe paper also proposes a soft-labeling strategy to mitigate the ambiguity typically associated with identifying optimal key points in 3D objects. \n\nFinally, the paper validates the effectiveness of the proposed model through a series of comprehensive experiments. The 3DRP-Net model attains state-of-the-art performance on three widely-used benchmark datasets: ScanRefer, Nr3D, and Sr3D.", "reasons_to_reject": "While the paper did a good job taking a traditional approach -- developing a specialized model with ground truth data. A better approach is to develop a foundation model without relying on a large amount of labeled data. For example, the recent work on PaLI-X, RT-2, Kosmos-2 are all along this direction. \n\nPaLI-X: On Scaling up a Multilingual Vision and Language Model\nhttps://arxiv.org/abs/2305.18565\n\nRT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control\nhttps://arxiv.org/abs/2307.15818\n\nKosmos-2: Grounding Multimodal Large Language Models to the World\nhttps://arxiv.org/abs/2306.14824", "questions_to_authors": "The dataset is not big. Are there other datasets you can also try out your method?\n\nCan you scale up your method to several orders of more data?\n\n== post rebuttal\n\nThe authors provided their explanations on the difficulties of addressing these questions. However, there is no new results or insights. I maintain my original rating.", "paper_topic_and_main_contributions": "3D visual grounding is the problem to localize the target object in a 3D point cloud given a free-form language description. It has important applications in robotics. This work is premised on the observation that typical linguistic descriptions of target objects often incorporate details about their spatial relationships with other objects and their location within the entire scene. The paper develops a 3D Relative Position Multi-head Attention (3DRP-MA) module that can effectively understand and map the spatial interrelationships between objects, paying due attention to the attributes and orientations of object pairs in a three-dimensional context. This facilitates model attention on specific object relations as cited in the linguistic descriptions, thereby enhancing object attribute representation.\n\nFurthermore, in an effort to accurately pinpoint the center location of 3D objects, the authors propose a methodology for densely sampling key points of objects. They also introduce a soft-labeling strategy to mitigate the ambiguity arising from redundant points, thereby fortifying and refining the learning process via a steady and discerning distribution.\n\nThe method is evaluated through extensive experimentation on three benchmark datasets - ScanRefer and Nr3D/Sr3D, where the proposed 3DRP-Net outperforms existing state-of-the-art models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j6tohRvsfJ", "reviewer": "Reviewer_Qbww", "summary": "", "questions": "See reasons to reject.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.1: “The 3DRP-Net is built upon the one-stage model 3D-SPS. Comparing results in the ablations with 3D-SPS shows that the two components do not work or have limited improvement”**\n\nOur approach does not simply improve the modules and training strategies under the structure of 3D-SPS, and the main architecture of our model is also different from 3D-SPS.\n\nThe difference in architecture between Ours and 3D-SPS: 3D-SPS introduces a complex structure (The TPM module in Figure 4 of 3D-SPS paper) which encodes 3D-vision and language modalities via two parallel branches and devises sophisticated modality interaction, along with a progressive key point selection strategy to filter out target objects. In contrast, our approach only sequentially stacks two vanilla transformer decoders to integrate scene and language information into key-point features. Therefore, the ablation results of our approach are not comparable to the results of 3D-SPS.\n\nWhy design such an architecture: Our experiments found that the progressive key-point selection strategy in 3D-SPS limits the comprehensive modeling of relative positions between objects. The text information in 3D-SPS interacting with the vision feature will disturb the consistency of the textual condition of the object relation modeling in different layers. The sophisticated architecture of 3D-SPS is not suitable for exploring more effective relation modeling modules. Our further experiments show that simply replacing the attention in 3D-SPS with 3DRP-MA, the Acc@0.25, and Acc@0.5 under ScanRefer would drop from 47.65, 36.43 to 39.89, 30.01.\n\nIn summary, our contribution is not only a new relative relationship modeling module and soft label strategy for the one-stage model but also a plainer and more adaptable one-stage architecture for 3D visual grounding.\n\n**Q1.2: Comparisons between Table 5 and Row 3 in “Table 2” with 3D-SPS**\n\nConsidering that Table 2 in our paper is the performance comparison on Nr3D and Sr3D, we suppose there is a typo. You may mean Row 3 in Table 3.\n\nAs we discussed in Q1.1, in addition to the relative relationship module and soft label strategy, the network architecture of our method is different from 3D-SPS. Besides, the progressive key-point strategy in 3D-SPS is not used in our method, in order to capture the relations between objects more comprehensively.\n\n**Q2: “Related work sections do not discuss existing works sufficiently.” “ViL3DRel should be discussed and compared in depth”**\n\nThanks for your suggestion, we will add more discussion in the related work of the future version.\n\nFor ViL3DRel, its relationship modeling (the “Spatial Attention Matrix” in their paper) does not consider object features. Therefore, its spatial matrix would highlight all object pairs that have a similar relative relation according to the language inputs. However, our method realizes the interaction among object-relation-language and can pay more attention to the exact object relation mentioned in the sentence.\n\nMoreover, ViL3DRel adopts a continuous relative position encoding, which results in a computational complexity of $O(n^2 d)$, where $n$ is the number of tokens and $d$ is the dimension of the token. In the two-stage approach, each object is used as a token, so the $n$ is relatively small. In the one-stage method, many points are sampled as tokens, the $n$ will become very large. Therefore, adopting the relative position module of ViL3DRel in a one-stage method will also introduce a lot of additional computational complexity.\n\n**Q3.1: triplet relations in the sentences such as “object 1 is between object 2 and object 3”**\n\nThis work mainly explores a plain model structure with strong relative relation modeling ability and a novel effective training strategy. Modeling the relationship between object pairs multiple times via the stacked L-layer transformer layer can implicitly model the relationship among multiple objects. More explicit relation modeling among more objects will be an interesting direction for future work. Considering we are the first to explore relative position modeling under the plain one-stage framework, we believe the current method is inspiring and would be a good starting point for future works.\n\n**Q3.2.1: “Why the authors use learnable embedding to encode the relative distances”**\n\nThe discretized relative position embedding is a more computationally efficient method. Considering $n$ tokens with $d$ dimension, using discretized position representations can reduce the complexity of calculating the Relation Attention Matrix from $O(n^2 d)$ to $O(nkd)$. Since the one-stage method uses each point as a token, the $n$ is much larger than $k$. Our experiments show using discretized position embedding can effectively reduce computational complexity with no performance loss. Using continuous position encoding takes 50 hours in training on a single 3090 and the Acc@0.25 and Acc@0.5 on ScanRefer is 50.07, 38.91, while using the discretized relative position embedding, the training time reduces to 30 hours with 50.10, 38.90 accuracies.\n\n**Q3.2.2: “The round operation would make the two distance embeddings the same.”**\n\nIn our implementation, we linearly scaled the coordinates of all points to [0:100]. The difference between 2.00 and 2.49 in the 3D space is also indistinguishable for humans, and the round operation is fine-grained enough for understanding relative relations in 3D scenes.\n\nIn addition, the coordinates of points can be linearly scaled to a larger scale, which brings finer-grained relative distance embeddings, enabling more detailed relationship modeling in larger scenes. At the same time, by adjusting the hyperparameters in Equation 4, we can control the balance between computational complexity and granularity of position embeddings.\n\n**Q3.3 “$D_{xyz}$ might not correspond to relative directions such as left, right, etc.”**\n\nThanks for your suggestion, the current statement is indeed vague and ambiguous. $D_{xy}$ represents the relative distance on the horizontal plane, and $D_z$ represents the vertical relationship. Since $D_x$ and $D_y$ embedding would interact with object features (which include object orientation information), the model can learn the correspondence between x-y coordinates and the left, right direction of the object. We will clarify this in the future version.\n\n**Typos:**\n\nThanks for your suggestion, we will add more descriptions about the notation of relative improvements in the title of Table 3. The grammar mistake in L322 in Sec 3.3 will also be revised.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow.\n\n- The 3d relative position multi-head attention provides a simple but effective way to inject location information into transformers.\n\n- The proposed model outperforms existing methods, and the ablations demonstrate the effectiveness of the new components.", "reasons_to_reject": "A. The 3DRP-Net is built upon the one-stage model 3D-SPS and improves it with the proposed relation-aware attention and soft-labeling training. However, comparing results in the ablations with 3D-SPS shows that the two components do not work or have limited improvement.\n\nTable 2: The main difference between the model in row 3 and 3D-SPS should be the training strategy. However, the result in row 3 is much worse than 3D-SPS, indicating the proposed training strategy does not work.\n\nTable 5: Since 3D-SPS used a hard-labeling training strategy, the main difference between the models in Table 5 Hard block and the 3D-SPS is the proposed relation-aware attention. However, the results in Table 5 are slightly worse than 3D-SPS on Acc@0.25, indicating that the proposed relation-aware attention is not that useful. \t \n\nB. The introduction and related work sections do not discuss existing works sufficiently though some of them are compared in the experimental tables, such as SAT and ViL3DRel. \n\nIn particular, ViL3DRel is one of the most similar works that also uses 3d relative spatial relations in transformer attention layers, which should be discussed and compared in depth.\nIt is unclear whether the approach in ViL3DRel could work in the one-stage framework and how much gains the proposed method can outperform ViL3DRel.\n\n\nC. The method contains some major limitations that are not discussed or better designs are required.\n\n1) The proposed method only captures pairwise spatial relations between objects, however, there are triplet relations in the sentences such as “object 1 is between object 2 and object 3”. \n\n2) It is unclear why the authors use learnable embedding to encode the relative distances, which can lose fine-grained information. For example, it may fail if the goal is “find the closest chair to the door” and the two chairs are in 2.0 and 2.49 distances to the door, since the round operation would make the two distance embeddings the same.\n\n3) The D_{xyz} might not correspond to relative directions such as left, right etc. since the object orientations are unknown. The relative directions should be relevant to the object orientations.", "questions_to_authors": "See reasons to reject.", "paper_topic_and_main_contributions": "The paper proposes a 3D relative position-aware network (3DRP-Net) for 3d object grounding given language sentences. 3DRP-Net is a one-stage model with two new components compared to existing one-stage methods: i) a 3D relative position multi-head attention to enhance the understanding of 3d pairwise object relation; ii) a soft-labeling strategy to improve training with weighted losses for predicted object proposals. 3DRP-Net achieved state-of-the-art performance on three benchmarks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The presentation of table 3 can be improved. It took me quite a while to understand the gains marked in green in row 1-5, which denote the relative improvements of row 6 over each row. It would be easier to understand if the relative performance decreases are presented for each row.\n\n- Grammar mistake: L322 in Sec 3.3 \"Due to\" -> \"Because\"."}}, {"review_id": "XPUGTPqhkz", "reviewer": "Reviewer_bfbG", "summary": "", "questions": "A. Is there a better way to capture object spatial relations than the proposed pairwise mechanism? Perhaps accounting for multi-to-multi relationships.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: “All evaluations are run on indoor scenes of everyday environments in a western setting”**\n\nExisting visual grounding datasets are all constructed based on indoor scenes. Our work focuses on proposing a 3D visual grounding architecture with stronger relative relation modeling capabilities and compares it with previous visual grounding methods. Extending visual grounding to more scenarios will be a very important development direction in the 3D-language field. Once more datasets of other scenarios are released, we will evaluate the applicability of previous methods and ours in more scenarios.\n\n**Q2: “Is there a better way to capture object spatial relations than the proposed pairwise mechanism?”**\n\nModeling the relationship between objects pair multiple times via the stacked L-layer transformer layers can implicitly model the relationship among multiple objects. More explicit multi-to-multi relation modeling will be an interesting direction for future work. Considering that we are the first to explore relative position modeling under the plain one-stage framework, we believe our method is inspiring and would be a good starting point for future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is generally well-written, the motivation is clear and straightforward.\n\n2. The method proves to be substantially better than existing baselines though thorough evaluations on ScanRefer and Nr3D/Sr3D datasets.\n\n3. Model offers structural simplicity compared to existing ones (though this is largely inherited from the previous 3D-SPS model with minor modifications.", "reasons_to_reject": "1. All evaluation are run on indoor scenes of everyday environments in a western setting. It would be interesting to see whether the model can adapt to different scenarios/out-of-distribution cases.", "questions_to_authors": "A. Is there a better way to capture object spatial relations than the proposed pairwise mechanism? Perhaps accounting for multi-to-multi relationships.", "paper_topic_and_main_contributions": "This work introduces the new model 3DRP-Net, aimed at improving 3D visual object grounding given point cloud of the scene and natural language input. \n\nThe proposed model uses a relatively novel one-stage structure to avoid unstable and time-consuming object proposals generation stage while employing relative position encoding instead of absolute position encoding. \n\nEvaluation is run on the benchmark datasets ScanRefer and parts of ReferIt3D. Automatic evaluations show model significantly outperforming SOTA methods.\n\nThe method enhances spatial reasoning abilities of the 3D-SPS model, making it better suited to reference resolution and 3d spatial relation reasoning.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Fig.3 can be more clear if supplemented with an x-axis and related captions."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new model for referring expression grounding in 3D point clouds. Their model uses attention to model spatial relationships between objects, and relate them to the description.

Most reviewers agree that there are multiple noteworthy and exciting model contributions that result in compelling performance.

Most reviewers agree that the experiments used to evaluate the model are comprehensive and sufficient, and the work is overall sound.
Decision: Accept-Main