Language Conditioned Spatial Relation Reasoning for 3D Object Grounding
Authors: Shizhe Chen
Pierre-Louis Guhur
Makarand Tapaswi
Cordelia Schmid
Ivan Laptev
Venue: NeurIPS 2022 Accept (2022)

Abstract: Localizing objects in 3D scenes based on natural language requires understanding and reasoning about spatial relations. In particular, it is often crucial to distinguish similar objects referred by the text, such as "the left most chair" and "a chair next to the window". In this work we propose a language-conditioned transformer model for grounding 3D objects and their spatial relations. To this end, we design a spatial self-attention layer that accounts for relative distances and orientations between objects in input 3D point clouds. Training such a layer with visual and language inputs enables to disambiguate spatial relations and to localize objects referred by the text. To facilitate the cross-modal learning of relations, we further propose a teacher-student approach where the teacher model is first trained using ground-truth object labels, and then helps to train a student model using point cloud inputs. We perform ablation studies showing advantages of our approach. We also demonstrate our model to significantly outperform the state of the art on the challenging Nr3D, Sr3D and ScanRefer 3D object grounding datasets.

Language Conditioned Spatial Relation Reasoning for 3D Object Grounding
Shizhe Chen', Pierre-Louis Guhur', Makarand Tapaswi', Cordelia Schmid', Ivan Laptev' Inria, École normale supérieure, CNRS, PSL Research University, "IIIT Hyderabad
https://cshizhe http://cssiizee.ihbbbbbb.. http://.oooooocsvvvlttttttt html
Abstract
Localizing objects 3D scenes based on natural language requires understanding and reasoning about spatial relations. In particular, it is often crucial to distinguish similar objects referred by the text, such as "the left most chair" and "a chair next to the window". In this work we propose language-conditiones transformer model for grounding 3D objects and their spatial relations. To this end, we design a spatial self-attention layer that accounts for relative distances and orientations between objects in input 3D point clouds. Training such a layer with visual and language inputs enables to disambiguate spatial relations and to localize objects referred by the text. To facilitate the cross-modal learning of relations, we further propose a teacher-student approach where the teacher model is first trained using ground-truth object labels, and then helps to train a student model using point cloud inputs. We perform ablation studies showing advantages of our approach. We also demonstrate our model to significantly outperform the state of the art on the challenging Nr3D, Sr3D and SeanRefer 3D object grounding datasets.
Introduction
To carry out human instructions in the real world, robots should understand natural language and be able to ground mentioned objects in 3D environments. Following this objective, recent research is
shifting from object grounding in 2D images I| 2 3 5
o to the 3D object grounding task 7 [] where objects referred by a sentence should be localized in a 3D point cloud.
Language expressions often refer to objects by their relative spatial locations in 3D scenes. Figure illustrates example scenes and corresponding sentences where object grounding requires disambigua- tion between objects of the same class. For instance, "the backpack closest to the piano" requires to compare relative distances among objects, while "choose the door on the left when facing them" " requires to infer the correct viewpoint and understand relative directions. Such complexity and diversity of the spatial language makes 3D object grounding highly challenging.
Given the critical role of the spatial language, many existing methods attempt to model spatial relations for 3D object grounding. Early work TC 2
explicitly build 3D visual graphs based on distances between objects and apply graph neural networks to learn relationships. However, since only nearest neighbors are considered the graph, it remains difficult to infer relationships between distant objects such as "farthest", More recently, transformer architectures [9] [3] [14, [1] [[]] have been used, as they have the potential learn relations between pairs of objects with multi-hean self- attention mechanism [17]. Enabling transformers to better understand 3D spatial relations expressed by natural language, however, remains an open research problem.
In addition, 3D object grounding suffers more from the scarcity of training data compared to its 2D counterpart. The success of 2D object grounding models [1S [9] can be largely attributed large-scale datasets with image-text pairs
20 whereas the limited amount of 3D scene-
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(b) Of the two brown wooden doors, choose the door on the left when facing them.
(a) The backpack closest to the piano.
Figure 1: Example sentences that refer to objects in 3D scenes. The green box denotes the ground- truth object, the blue box is the prediction from our model, and the purple one is from a baseline model without explicit spatial reasoning and knowledge distillation.
language pairs increases the difficulty of 3D object reasoning. To address this challenge, Yang er al. [44 propose to use 2D-3D alignments assist the training of 3D models, but require additional high-quality 2D images and camera parameters which are not always available.
In this work. we propose Vision-and and-Langaage 3D Relation reasoning model (ViL3DRel) to tackle the above issues in 3D object grounding. We design a new spatial self-attention layer for the transformer architecture to enhance 3D spatial understanding. This layer encodes relative distances and orientations for all pairs of objects, and explicitly learns spatial attention to capture spatial relations referred by the the language expressions. We propose sigmoid softmax function to effectively fuse the spatial attention with standard self-attention. Rotation augmentation is used to further improve view invariant reasoning. To alleviate the negative effect of noisy object features, further propose a teacher-student training approach. The teacher and student share the same model architecture, but different inputs. The teacher uses ground-truth object labels, which enables to better learn the relation reasoning. The learned relation knowledge is distilled to the student taking point cloud features as input. Our code and models are available on the project webpage [22]
To summarize, our contributions are three-fold:
We propose ViL3DRel model for the 3D object grounding task. It uses new spatial self-attention to explicitly encode pairwise 3D spatial relations in the transformer layer for better language conditioned spatial understanding.
We employ a reacher-studens training strategy, which facilitates the cross-modal learning of relations. The student with point cloud inputs benefits from the relation reasoning model of the teacher trained with ground-truth object label input.
We evaluate our ViL3DRel approach on the challenging Nr3D, Sr3D and ScanRefer benchmarks. Our model significantly outperforms state-of-the-ard approaches, with 9.3, 8.3 absolute gains on Nr3D and Sr3D given ground-truth object proposals, and 4.47 points on ScanRefer using the same detected object proposals compared to the previous work [6
2 Related Work
2D and 3D Object Grounding. The object grounding task aims to localize objects in 2D images or 3D point clouds given sentence, Existing approaches can be categorized into two groups, namely one- and two-stage frameworks. The one-stage methods densely fuse the text features with patch- or point-level visual representations to directly regress the bounding box [8 23). These approaches flexible to detect various objects given the input sentence. The two-stage methods adopt
detection-then- then-matching pipeline II 7 8 24 where the detection stage generates object proposals, and then the matching stage selects the best proposal according to the sentence. The decoupled object perception and cross-modal matching make the two-stage methods easier to analyze [5].
The benchmark datasets for 3D object grounding are ReferIt3D (Nr3D and Sr3D) and Scan- Refer 77. which are all built upon scenes and object annotations in the ScanNet dataset [25] Graph-based approaches 26 e are widely adopted in early works infer spatial relations. The
object graph is constructed by connecting each object with its top nearest neighbors 3 10 I based

---

on Euclidean distance. Inspired by the success of transformers 17. recent works 9
14
15
16
23
have adopted transformers for 3D object grounding. BEAUTY-DETR 28 and 3D-SPS
2 are
one-stage methods. However, most works follow the two-stage framework with pre-detected object proposals. LanguageRefer [T5] converts the cross-modal task into a language modeling problem with predicted object labels. SAT [4] adopts multimodal transformer and transfers 2D semantics to as- sist the training of the 3D model. Multi-view transformer T6 aggregates object representations from multiple views to improve view robustness. Perhaps most similar to our work, 3DVG-Transformer 13 proposes a coordinate-guided attention module to encode spatial distances among objects. In contrast, we introduce spatial self- module, which explicitly encodes both relative distances and relative orientations among objects. It also conditions spatial relations on the language and presents more effective strategy for attention fusion.
Transformers in -att-anndd--nnuae.e.e Transformer-bant based architectures have led to significant improvements in various vision-and-large -ann-aggge tasks such as text-video retrieval ing I8
20 visual ground-
2 and vision-and- languug naviga- Most of the methods project textual and visual inputs into a sequence of tokens, and use multimodal transformers to learn cross-modal semantic relationships. While relative positional encoding (RPE) B55 has been explored mostly separately in vision and language transformers, we here integrate RPE with cross cross-modal modal transformer and show its benefits for resolving spatial object relations referred by text.
30I, image captioning 3T, vision
tion 3 R
typically used to compress a large network (teacher) into compact model (student). The common objective is to let the student model mimic the soft logits of the teacher model. Beyer et al. [37] show that a consistent and patient teacher is essential in knowledge distillation. Jiao et al. [38] further demonstrate that the intermediate representations learned by the teacher are beneficial. In contrast to distilling knowledge from a heavy model to a light one, our teacher network first learns cross mooal object relations using ground object labels. We then transfer this knowledge to the target student network that uses noisy inputs.
Knowledge Distillation. Knowledge distillation 36
3 Method
Given a sentence S, the goal of 3D object grounding is to detect an object referred in S by locating its 3D bounding box By E IR in a 3D point clouds Pecene. We assume Pscene R. Kx6 contains K points each represented by XYZ coordinates and RGB values. We follow the detection-then- matching framework 7 R
16 for 3D object grounding and assume to be given a list of ON) ) obtained via automatic 3D instance segmentation or ground-truth annotations (depending on the evaluation setup). Each object O is represented by a subset of K; points P; C Pacene, P. E R xx Our work is focused on interpreting spatial relations and selecting the target object T,,T E [1, N] among N object proposals. We first present an overview of our ViL3DRel model in Section 3.1 We then introduce our cnnuuin-condittoned spatial self- lelf-attention module and the teacher-studen training approach in Sections 3.2 and 3.3 respectively.
13
object proposals (O,
3.1 Architecture Overview
Figure 2 shows an overview of our ViL3DRel model, consisting of four modules: text encoding, object encoding, multimodal fusion and grounding head as described next.
Text encoding. Given the sentence S with M word tokens, we use pre-trained BERT model 39 to encode S into a sequence of word features (Sels. ( MM),, 8: E Rdd, where Sats is a special classification token and d is the dimensionality of the feature.
Object encoding. For each object O,, we first normalize the coordinates of its point cloud P. into
unit ball, and then use PointNet++ [40] to compute the object feature 0f E IRd We also obtain the object center c; Ic2, Cy. cz] E R and the object size 2 = [Zz, Zy. 2g] E R from object points P. as the mean and the spatial extent of P. respectively. We use a linear projection layer to obtain the absolute 3D location feature as 4 Wifc; 24] Rd
Multimodal fusion. A stack of transformer layers 177 are applied to fuse the text and object modality features. Each transformer layer is composed of a spatial self-attention layer, a cross-attention layer and ferd-forward neural network (FFN). Our new spatial self-attention layer aims to improve the understanding of spatial relations among objects referred by the sentence. Assume of is the input embedding for object proposal O before the i-th layer, we first add it with its absolute 3D location feature I to enhance the spatial information. The spatial self att-ettentinn then generates contextualized object representations o'. We will describe the proposed spatial self-attention in details in Section 3.2 The cross-attention layer takes of as queries and text features (Scles "1, S.M) ) as keys and values
3

---

UP
Grounding Head
Feed-forwane Network
Self
SigSof-
orfs
Spatial Attention Matrix
Cross-Attention
Attention
:
%
's
Spatial Self-Attention
By
Scaled Dot-Product
gi
FIR
"The correct backpack farthest from the plano the comeer"
Point
Linear Wx
Linear Wg
Linear Ws
0f
0
+ Sets
G1 ... CV
Sentencee S
Object Proposals (O,...,s))
object
token
positions
Figure 2: Left: overview of the ViL3DRel model; Right: language spatial self-attention
to learn cross-modal relations. The final FPN uses two fully connected layers to encode each output tokens after the attention layer. For detailed explanation of Transformer attention and FFN layers please refer to [17
Object grounding. We use a two-layer feed-forward neural network as the object grounding head to predict the target object. Given the output embedding of from the last multimodal fusion layer, the grounding head generates a scalar score for each object proposal O and applies softmax function to obtain the probability P;- The object proposal with the maximum probability is predicted as target.
3.2 Spatial Self-Attention
The standard self-attention relies on token embeddings to learn relations among tokens. However, the input embeddings of object proposals are mixed with semantic features and absolute 3D locations, making it difficult to accurately infer spatial relations among objects such as relative distances or orientations in Figure [ As spatial relations among objects referred by language important to distinguish object instances, we propose to inject language conditioned spatial attention that explicitly captures pairwise spatial relations complement the standard self-attention. We first describe the standard self-attention and then introduce our proposed spatial self- attention as illustrated the right part of Figure
Given the feature matrix X Nxd for N object proposals, the self-attention mechanism first computes the query, key and value embeddings from X as Q XWQ, K = XWK, V XWy respectively, where Wo. WK, Wv E Rdddd, are learnable parameters and d, is the dimensionality of the output embedding. It then calculates attention weights given the query and key embeddings and aggregates the value embeddings as follows:
QKT Vah
Ω softmax
SelfAttm Q,K,V) 2VV,
where Ω2 is N x N attention matrix, whose elements wi, is the attention weight between the i-th and j-th object proposal. In order to capture more diverse relations, multi-head self-attention used where each head computes an independent SelfAttn( Q, K, V) and the outputs from all heads are concatenated.
To model spatial relations among objects, we propose use explicit pairwise spatial features f0 E R j.j E []. For each pair of objects (O,, Oý), we compute their Euclidean distance dy = =|c - c2ll2 well as horizontal and vertical angles On, O,, of the line connecting object centers 6 and Cj. The computation details of the angles are provided in Section of supplementary material. We then define the pairwise spatial feature fij as:
fij Id.. sin (9,), cos cos(( sin().), cos((.)).
We automatically generate language conditioned weight of to select relevant spatial relations for each object proposal O as we mainly care about spatial relations described in the text for an object:
gi W (sale - of),
(3)

---

Feed freddforwwrr Network
Hidden State Distillation .*
Feed-forward Network
Cross-Attention
Cross-Attention
Attention Distillation
Spatial Self-Artention
Spatial Self-Attention
pos
Text BERT
Text BERT
obj color
obj color
Point Net++
Point Net++ obj pcd
sentence
obj label
obj label
sentence
obj pcd
Teacher Model
Student Model
Figure 3:
Teacher student learning with hidden state and attention distillation.
where Ws E Rdxb is learnable parameter and we omit the bias term for simplicity. We then define the spatial relevance for (Os, O₃) as:
wij g fij-
(4)
We modulate the self-attention in Eq ① with the above language conditioned spatial relevancy using the sigmoid softmax (sigsoftmax) fusion function:
a(wft) exp(wi,)
Wij Σi= (w))) exp(wx))
where (-) is the sigmoid function. In this way, we compute the new self-attention matrix Ω [Wig]Nx N which explicitly considers 3D relative spatial locations, absolute spatial locations and object appearances in relation reasoning. We also use the multi-head mechanism support different types of spatial relations and compute multiple spatial attentions to fuse with the standard self-attention. We concatenate the outputs from all heads in the spatial self-attention layer.
3.3 Teacher-Student Training
Incorrect estimation object classes from P. can deteriorate the cross modall alignment and spatial relation reasoning. To improve object grounding in point clouds previous work [14] relies on additional supervision in 2D images. Here, instead, propose new teacher-student training approach using no additional training data. The teacher and student models share the same transformer architectures except for the object encoding module. The teacher uses ground- truth object semantic features, while the student uses 3D point clouds. Since there is less cross-modal gap between the teacher's inputs and the sentence, the teacher model can excel at learning language relevant spatial relations among objects and aligning objects with the sentence. Such knowledge can then be distilled to the student model improve its training. Figure illustratts the teacher-student training pipeline. We describe the inputs of the teacher and the knowledge distillation objectives in the following.
Teacher's inputs. We use ground- class labels and dominant colors of the object as the object representations for the teacher model, since the color is the most widely used attributes in the sentence, Specifically, we encode the ground-trute -tunt object class label using pre- pre-trained Glove word vectors 41 To obtain the dominant colors of the object, we Gaussian Mixture model on the RGB values of all points in the object, where the mixture component is set to 3. We linearly project the mean value each component into color embedding and use mixture weights of the components to average all the color embeddings. The final object representation is the sum of the class label embedding and the averaged color embedding.
Knowledge distillation objectives. We transfer the attention weights and hidden states in the multimodal fusion module of the teacher model to the student model. The self-attentions learned by the teacher capture spatial relations among objects, and the -ross-attentions measure cross-modal matching, which are both essential for 3D object grounding but hard to learn given noisy object features. We facilitate student learning by forcing it to mimic all the attention matrices of the teacher using the following objective function:
Lattn
M 12
(6)
5

---

where S2 and S2. are attention weights in student and teacher model respectively, MSE(-) is the mean square error function, L is the number of multimodal fusion layers and H is the number of heads in self-attention. In addition to distilling the relation knowledge in attention weights, we also distill the output embeddings of the transformer layers:
N
Chidden
MSE 18 )
(7)
LN
Training, We follow the previous works 8 7 to use multiple auxiliary losses in training the transformer model, including a 3D object grounding loss Log. sentence classification loss Csent and two object classification losses Lo and d...- Liseut relies on Sets to predict the target object class from the sentence. Coby and Libs use unimodal object representation of and multimodal fused object representation of respectively to predict the classes for input object proposals. Details are in Section A of supplementary material. Therefore, the overall training objective is as follows:
L == Log + Lsent + Labj + Cob + AeCattn + Xb, Chsdden
(8)
where Aa, An are two parameerr to balance the losses.
4
Experiments
4.1
Datasets
Nr3D dataset [8 contains 37,842 human-writter sentences that refer to annotated objects in the 3D indoor scene dataset ScanNet (5 The dataset includes 641 scenes with 511 (resp. 130) scenes for training (resp. validation). It covers 76 target object classes. The annotated sentences are designed to refer to objects with multiple same-class distractors in the scene. The sentences are split into "casy" and "hard" subsets in evaluation, where the target object in "easy" subset only contains one same-class distractor in the scene while it contains multiple ones in the "hard" subset. According to whether the sentence requires a specific viewpoint to ground the referred object, the dataset can also be partitioned into "view depedent" and "view independent" subsets.
Sr3D dataset X constructed using templates to automatically generate sentences. The sentences only utilize spatial relations to distinguish objects of the same class. It has 1,018 training scenes and 255 validation scenes from ScanNet and 83,570 sentences in total. The dataset can be split in the same way Nr3D during evaluation.
ScanRefer dataset [7] has 51,583 human-written sentences for 800 scenes in ScanNet. We follow the official split and use 36,665 and 9, 9,508 samples for training and validation respectively. According to whether the target object is a unique object class in the scene, the dataset can be divided into "unique" and a "'mlttipe" subset.
4.2 Experimental Setting
Evaluation Metrics. We evaluate models under two evaluation settings. One uses ground-truth object proposals, which is the default setting in the Nr3D and Sr31) datasets. The metric is the accuracy of selecting the target bounding box among the proposals. The other setting does not provide ground-truth object proposals and requires the model to regress 3D bounding box, which is the default setting for the ScanRefer dataset. The evaluation metrics are acc @0.5 and acc @cc @0.,, which is the percentage of correctly predicted bounding boxes whose IoU is larger than 0.25 or 0.5 with the ground-truth.
Implementation details. For the model architecture, we set the dimension d = 768 and use 12 heads for all the transformer layers. The text encoding module is three-layer transformer initialized from BERT 39 and the multimodal fusion module contains four layers. The object encoding module PointNet++ 40% samples 1024 points for all the objects. These architecture parameters are the same as in previous work 14. H to ensure a fair comparison. We first train the PointNet++ for object classification on ScanNet, which achieves 61.9% accuracy on the validation set. Its weights remain fixed during the following training steps. Rotation augmentation is used to increase the viewpoint invariance. The hyper- papemmparaeeters in the loss function Eq () are set to A and ' 0.02. We train the model with batch size of 128 and a learning rate of 0.0005 with warm-up of 5000 iterations and cosine decay scheduling. The AdamW algorithm Ap used in the optimization. We train for 50 epochs for the teacher model and 100 epochs for the student model on Nr3D and ScanRefer datasets. The training epochs are reduced to half on the Sr3D dataset, as it larger and easier to converge. All models trained on single NVIDIA RTX A6000 GPU.

---

Table 1: Grounding accuracy (%) on the Nr3D dataset with ground -oound-truth object labels. Dist stands for Distance; Ort for Orientation; MHA for multi-head spatial attention; RotAug for Rotation Augmentation; sigs for sigsoftmax in Eq ; and *-' means not applicable.
Spatial Relation Reasoning
Rot Aug
Color
Overall
ViewDep
ViewIndep
Dist
Ort MHA
Fusion
RI
X
X
53,5
51.4
54.6
R2
X
V
55.1
53.8
55.8
R3
V
/
62.4
58.3
64.5
R4
V
sigs
v
66.0
53.8
72.0
R5
X
/
sigs
/
v
71.3
68.5
72.6
R6
V
/
sigs
/
/
67.7
65.2
69.0
R7
V
bias
/
-
55,4
46.8
59.6
R8
V
/
ctx
56.4
50.8
59.1
R9
V
v
sigs
v
74.4
71.3
75.9
4.3
Ablation Studies
We carry out extensive experiments the Nr3D dataset to demonstrate the effectiveness of our proposed spatial self-attention and teacher-student training. The ablations on the ScanRefer dataset are provided in Section B of supplementary material.
4.3.1 Spatial Relation Reasoning
We first evaluate the proposed spatial self-attention using ground-truth object labels, which decouples the object perception from spatial relation reasoning. Table M results on the Nr3D dataset. Baselines. Our baseline RI the teacher model the left part of Figure 3 that excludes object color, rotation augmentation and the proposed spatial self-attention. This baseline achieves similar performance to the state-of-th fe-th-eart LanguageRefer [I5] (overall accuracy: 54.3%) which also uses ground- ground-truth object labels. Row R2 shows that adding color information helps object grounding as sentences often contain color attributes. The rotation augmentation brings additional gains and im- proves accuracy from 55.1% (R2) to 62.4% (R3). It more effective for view independent sentences (+8.7%) than for view dependent sentences (+4.5%) because rotation augmentation mainly improves view invariance. We consider R3 strong baseline and use it to demonstrate improvements of our proposed model.
Pairwise spatial features: distance vs. orientation. Rows R4-R5 in Table compare different pairwise spatial features defined in 2 R4 only uses pairwise distances, while R5 only uses pairwise orientations. We can see that pairwise distances are beneficial for view-independent sentences which contain distance-related spatial relations such as "nexi to" and "farthext from ", but do not improve performance for view-dependent sentences. contrast, pairwise orientations significantly boost the performance for view-dependent sentences (+10.2% This shows that explicitly encoding the relative orientations facilitates the learning of view dependent spatial relations such as "in front of"" and "to the left of The relative distance and orientation features are also complementary. Their combination achieves the best performance as shown in our full model (R9).
Table provides more systematic analysis of R3-5 and R9 in Table to show the contribution of spatial relation modeling to different types of sentences. We categorize sentences into four groups according to whether the sentence describes spatial relations terms of distances or orientations. We can see that the explicit pairwise distance modeling contributes most the distance-only sentences but has little influence on samples with orientation-relatte sentences. On the other hand, the pairwise orientation modeling can significantly improve prientation-relatee sentences by 10.5%. Combining both pairwise distance and orientation modeling achieves the best performance on all categories.
Table 2: Comparison of pair- wise spatial feature computa- tion methods.
Overall
object center
74.4
bottom center
74.4
boxes MIP
57.4
Pairwise spatial feature computation. We compute pairwise distances and angles using the coordinates of object centers and design the spatial features shown in Eq 2 In Table 2 we further
7

---

by spatial relation types: Dist( Ort) only which only contains distance(orientattion) descriptions; Dist & Ort which contains both distance and orientation descriptions; and the Others which not contain spatial relation descriptions.
Table 3: Performance breakdown models in Table
Dist
Ort
Overall
Dist only
Ort only
Dist & Ort
Others
62.4
63.5
61.2
57.7
63.9
X
66.0
72.6
58.9
55.1
68.8
71.3
73.8
71.7
67.7
69.1
74.4
77.8
74.0
69.1
72.6
#Rthest
14
montorr
desk
desk
desk
desk
desk
door
whitssbard
clock
file cabinet
monitor
office char
16
"Facing whiteboard, &t on the right side of the room farthest from the whiteboard."
Figure 4: An example of the learned cross-attention (upper) and self-attention (bottom) at different transformer layers the teacher model. The query the target object proposal (desk 17). The darker color denotes higher attention weights from the query to the keys.
compare the proposed method with two variants show its effectiveness. The first variant uses the bottom center of objects to compute pairwise vertical angles which ignores the height of objects and could be more accurate boxes of two objects and uses a multi-layer perceptron (MLP) to learn pairwise spatial relations. We can see that using the bottom center achieves similar performance as using the object center, which suggests that the object center is sufficient to capture the vertical spatial relations of objects in the textual descriptions. The learnable MLP, however, achieves much worse performance than our approach. This indicates that it is challenging to implicitly learn pairwise spatial relations and the proposed spatial features designed by domain knowledge are beneficial.
measure vertical relations. The second variant concatenates bounding
Multi-head spatial attention. R6 in Table T uses single-head attention for the spatial relevance wi in (5) but keeps multi-head for the standard attention weight w Compared to the full model with the multi-head spatial attention, such single-head spatial attention achieves significantly worse performance. This suggests that the multi-head is beneficial for learning spatial relations.
Attention fusion. We use sigmoid softmax function 5
aggregate the spatial attention and the standard self- attention weights. We compare the proposed fusion mech- anism with standard relative positional encoding methods B35 in vision transformers: bias mode and contextual mode. The bias mode adds the spatial attention weight bias term in the standard self-attention, while the con- textual mode further considers the interaction with queries by injecting the pairwise spatial features into the key em- beddings. From the results in R7 and R8, we can see that the standard relative positional encoding methods fail to model 3D spatial relations among objects; they even per-
Table 4:
Performance comparison of dif- ferent textual encoders.
Encoder
#Layers
Overall
3
74.4
6
73.7
BERT
9
73.2
12
52.3
Glove+GRU
3
45.7
form worse than the strong baseline (R3), which doesn't
rely on relative positional encoding. The proposed sigmoid softmax function is by far more effective to modulate the standard self-attention with spatial relations.
Text encoding. We follow recent works [14] [6]] to use the first three layers of BERT textual encoder, while [[] uses weaker textual encoder (Glove+GRU), To compare these different textual encoders, Table provides results of our proposed models under the setting in Table | We can see that more BERT layers not lead better performance. The reason might that representations from higher BERT layers are less generalizable to different domains. Moreover, more layers more prone

---

Table 6: Grounding accuracy (%) on Nr3D and Sr3D datasets with ground -ruth object proposals.
Nr3D
Sr3D
Method
View Dep
View Indep
View
View Indep
Overall
Easy
Hard
Overall
Easy
Hard
Dep
ReferIt3D
35.6
43.6
27.9
32.5
37.1
40.8
44.7
31.5
39.2
40.8
ScanRefer 7
34.2
41.0
23.5
29.9
35.4
TGNN 10
37.3
44.2
30.6
35.8
38.0
InstanceRefer 181
38.8
46.0
31.8
34.5
41.9
48.0
51.1
40.5
45.4
48.1
FFL-3DOG 12
41.7
48.2
35.0
37.1
44.7
3DVG-Trans
40.8
48.5
34.8
34.8
43.7
51.4
54.2
44.9
44.6
51.7
TransRefer3D
42.1
48.5
36.0
36.5
44.9
57.4
60.5
50.2
49.9
57.7
LanguageRefer 15
43.9
51.0
36.6
41.7
45.0
56.0
58.9
49.3
49.2
56.3
SAT [4]
49.2
56.3
42.4
46.9
50.4
57.9
61.2
50.0
49.2
58.3
3D-SPS 2X
51.5
58.1
45.1
48.0
53.2
62.6
56.2
65.4
49.2
63.2
Multi-view 16
55.1
61.3
49.1
54.3
55.4
64.5
66.9
58.8
58.4
64.7
Vil.3DRel (Ours)
64.4
70.2
57.4
62.0
64.5
72.8
74.9
67.9
63.8
73.2
to overfit and harder to optimize. The pretrained BERT model achieves much better performance compared to the GRU model trained from scratch.
4.3.2 Teacher-Student Training
Table Grounding accuracy (%) on the Nr3D dataset with ground-truth object proposals.
The teacher model with ground-truth object labels learns well how to perform spatial relation reasoning given the input sentence. Figure A provides a qualita- tive example of the learned cross- and self-attention weights at all multimodal fusion layers in the teacher model. For the pross-attention weights, that the teacher first aligns the object proposal with the correct words in the text, and then gradually shifts its attentions to relation words to perform spa- tial relation reasoning. For the self-attention weights, in the first layer, the teacher mostly attends to nearby objects be aware of the context; then in the second layer, it focuses more on the reference object mentioned in the sentence (e.g., the whiteboard); finally, it concentrates on the same-class distractors distinguish them. Such reasoning steps are promising to ease the training of the student model. In the following, we evaluate the proposed teacher-student training to distill relational knowledge from the teacher student with point cloud inputs extracted from the ground-truth object proposals.
init.
battn
Lhidden
Overall
Teacher
74.4
can
X
X
58.1
X
X
62.6
Student
X
V
X
63.6
X
X
V
62.1
X
64.4
Table presents any knowledge from the teacher model. In the second row, we use the weights in the teacher model to initialize the student model since the teacher and student share the same architecture except for the input object representations. Such simple weight initialization already facilitates the training of the student, and achieves 4.5% gains. The attention distillation in (6) and hidden state distillation 7 are both beneficial to improve the student model compared to first row without knowledge distillation. It is more effective to distill the relation knowledge in the attention weight matrices. The combination of Lattn and Lhidden is helpful and achieves the best performance. We empirically find that combining weight initialization and the two distillation losses performs similar to using the distillation losses alone.
presents the performance of different student models. The first row in the student block does
4.4 Comparison with State-of-the-Art Methods
Table O compares our ViL3DRel model with ttate-of-the-ee art methods on Nr3D and Sr3D datasets. the compared works use ground-truth object proposals, but no ground-trufe labels. Our model achieves significant improvements over the previous best method [16], with 9.3% and 8.3% absolute gains on Nr3D and Sr3D respectively. We outperform
[ by even larger margins on the two datasets. This work also spatial information, but doesn't 't use spatial attention module.

---

Table Grounding accuracy (%) on ScanRefer with detected object proposals. VN and PG denote the VoteNet and PointGroup models pretrained on 18 classes of the ScanNet dataset, while Optimized denotes end-to-end training of the object detector on the ScanRefer dataset.
Unique
Multiple
Overall
Method
Det
acc 0.25
acc @0.5
acc acc@0.25
acc@0.5
acc @0.25
acc 0.5
ScanRefer 7
65.00
43.31
30.63
19.75
37.30
24.32
TGNN [T]]
Opti-
68.61
56.80
29.84
23.18
37.37
29.70
3DVG-Trans
13
mized
77.16
58.47
38.38
28.70
45.90
34.47
3D-SPS
81.63
64.77
39.48
29.61
47.65
36.42
Non-SAT
14
VN
45
68.48
47.38
31.81
21.34
38.92
26.40
SAT 14
VN
45
73.21
50.83
37.64
25.16
44.54
30.14
InstanceRefer
|||
PG
15
77.45
66.83
31.27
24.77
40.23
32.93
Multi-view [16
PG
45
77.67
66.45
31.92
25.26
40.80
33.26
ViL3DRel (Ours)
PG
81.58
68.62
40.30
30.71
47.94
37.73
UpperBound
PG
43
88.63
74.47
78.82
60.37
80.64
62.98
Table and Table g present results on ScanRefer dataset with ground-truth object proposals and detected object proposals, respectively. For fair comparison, model to generate object proposals, which trained on 18 classes ScanNet. To be noted, the upper block in Table optimizes the detection stage utilizes single-stage pipeline. These methods use more data to train object proposals, so the comparison to these methods is not entirely fair. There is large gap between the ground-truth and detected proposals (over 12% for our method), because the detected proposals might not include the target or reference objects in the sentence. However, our ViL.DRel model still outperforms the state of the in both settings. Though our proposed spatial relation module can also be plugged into transformer-basee 3D object detectors 44 improve the 3D object proposal generation, we leave it to future work. Figure I compares ViL3DRel model and a baseline without spatial sell- attention and seacher-student training. We present more qualitative comparisons in Section of supplementary material.
pre-trained PointGroup [
Table 7:
Grounding accuracy (%) ScanRefer with ground- truth object proposals.
Overall
ReferIt3D 8
46.9
Non-SAT
48.2
SAT 14
53.8
ViL.DRRll (Ours)
59.8
5 Conclusion
In this work, propose ViL3DRel model for 3D object grounding. It contains newly designed spatial self-attention module to improve language conditioned spatial relation reasoning in the transformer layer, which explicitly considers relative distances and orientations among objects. A teacher-student training approach is further proposed to transfer relation knowledge from teacher with ground-truth object labels outperforms the state of the art on Nr3D, Sr3D and ScanRefer datasets.
student with point cloud inputs. The proposed model significantly
Beyond the application of object grounding, the general approach of incorporating priors into self-attention layer for transformer combine multimodal input can be of wide interest. Since our model belongs to the two-stage framework, is limited by imperfect object proposals the first detection stage. We also did not explore explicitly extract object orientations more accurately estimate pairwise spatial relations as the automatic object poses prediction remains challenging problem. In addition, the evaluation a rich diversity of environments though we carried out extensive ablations to mitigate the problem. This work has minimal ethical, privacy and safety concerns.
suffer from the fact that existing datasets do represent
Acknowledgments and Disclosure of Funding
This work was granted access to the HPC resources of IDRIS under the allocation 101002 made by GENCI. It was funded in part by the French government under management of Agence Nationale de la Recherche as part of the "Investissements d' l'avenir" program, reference ANR 19-P31A-0001 (PRAIRIE 31A Institute), the ANR project VideoPredict (ANR-21- (ANR-21-FAI1-0000-0) and by Louis Vuitton ENS Chair Artificial Intelligence.
10

---

References
[1] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, pages 85- Springer, 2016.
[2] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, pages 11-20, 2016.
[3] Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L Yuille. Clevr-ref+: Diagnosing visual reasoning with referring expressions. In CVPR, pages 118844114, 2019.
[4] Zhenfang Chen, Peng Wang, Lin Ma, Kwan-Ye Yee K Wong, and Qi Wu. Cops-ref: A new dataset and task compositional referring expression comprehension. In CVPR, pages 10086-10095, 2020.
[5] Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, 2019.
pages
[6] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In CVPR, pages 9982-9991, 2020.
[7] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In ECCV, pages 202-221. Springer, 2020.
[8] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In ECCV, pages 422-440. Springer, 2020.
[9] Dailan He, Yusheng Zhao, Junyu Luo, Tianrui Hui, Shaofei Huang, Aixi Zhang, and Si Liu. Transrefer3d: Entity- andily-and-rllation aware transformer for fine-grained 3d visual grounding, In ACM MM, pages 2344-2352, 2021.
[10] Pin-Hao Huang, Han-Hung Lee, Hwann- -zonn Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation. In AAAI, volume 35, pages 1610-1618, 2021.
[11] Zhihao Yuan, Xu Yan, Yinghong Liao. Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In ICCV, pages 1791-1800, 2021.
[12] Mingtao Feng, Zhen Li, Qi Li, Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang, and Ajmal Mian. Free-form description guided 3d visual graph network for object grounding in point cloud. In ICCV, pages 2722-3 2021.
[13] Lichen Zhao, Daigang Cai, Lu Sheng. and Dong Xu. 3dvg trang-rranfoomer: Relation modeling for visual grounding on point clouds. In ICCV, pages 2928 2937, 2021.
[14) Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: 2d semantics assisted training for visual grounding. In ICCV, pages 1856-1866, 2021.
[15] Junha Roh, Karthik Desingh, Ali Farhadi, and Dieter Fox. Languagerefer: Spatial-language model for 3d visual grounding. In CoRL, pages 1046-1056. PMLR, 2021.
[16] Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual grounding. In CVPR, 2022.
[17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 30, 2017.
[18] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Ander-modulatee detection for end-to-end multi-modal understanding. In ICCV, pages 1780-1790. 2021.
11

---

[19] Sanjay Subramanian, Will Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. In ACL, 2022.
[20] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 123(1):32-73, 2017.
[21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, PMLR, 2021.
[22]
Project webpage. https://cshizhh: httpc/shhhh...iiihb....eeeeocc.e (ttthub.o//rrojecss//ilIl
.html
[23]
Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In CVPR, 2022.
[24] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1307-1315, 2018.
[25] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-ane kichly-annotated 3d reconstructions of indoor scenes. Proceedings of the IEEE conference computer vision and pattern recognition, pages 5828-5839, 2017. 2017.
[26] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. ICLR, 2018
[27] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. TOG, 38(5):1-12, 2019.
[28] Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Looking outside the box to ground language 3d scenes. arXiv preprint arXiv:2112.08879) 2021.
[29] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, pages 7464-7473, 2019.
[30] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Tubedetr: Spatio-temporal video grounding with transformers. CVPR, 2022.
[31] Marcella Comia,, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed- -emory transformer for image captioning. In CVPR, pages 10578-10587. 2020.
[32] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. AAAI, volume 34, pages 13041-13049, 2020.
[33] Shizhe Chen, Loerre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. NeurlPS, 34, 2021.
[34] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for iision-and-languaa navigation. CVPR, 2022.
[35] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In ICCV, pages 10033-10041, 2021.
[36] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge a neural network. arXiv preprint 150300350..,,31,, 2015.
[37] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher patient and consistent, arXiv preprint 2rX1v:2106.05237. 2021.
12

---

[38] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In EMNLP Findings, pages 4163-4174, 2020.
[39] Jacob Devlin, Ming- deep bidirectional transformers for language understanding. NAACL, 2019.
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of
[40] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. NeurlPS, 30, 2017.
[41] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods natural language processing (EMNLP), pages 1532-1543, 2014.
[42] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
[43] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In CVPR, pages 4867-4876, 2020.
[44] 7e Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2949-2958, 2021.
[45] Charles R Qi. Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, pages 9277-9286, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the conclusion section.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the conclusion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [No] We will release the code, data and instructions upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the experiment section.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See the supplementary material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See the experimental section.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "pSOYWwp_N4", "reviewer": "Reviewer_KM3K", "summary": "This paper proposes a new method for 3D object grounding with a transformer that combines input of a BERT text encoder and a Point++ object encoder to match the detected object proposals with the input sentence description.\n\nThe key insight here is that explicitly encoding spatial relations between detected objects significantly help with spatial reasoning. Based on this observation, the method incorporates a spatial self-attention layer that computes spatial relevance from language inputs and the explicitly defined pairwise object relations (of relative distance and angle). This spatial relevance is then used to weight the self attention between objects. \n\nCombining this novel spatial reasoning layer with teacher-student knowledge that distills knowledge from a teacher network with access to ground truth labels and color of objects, the proposed method is able to achieve significantly better performance than prior works on multiple benchmarks, where reasonable control of variables are attempted. Thorough ablation are also performed to justify the importance of each of the novel design choices.\n", "questions": "### Design Choices\n- What is the motivation of computing Euclidean distance from the centroids, instead of between the bounding boxes? It seem to me that distance between bounding boxes might describe proximity between objects more unambiguously as opposed to centroid-to-centroid distances.\n- Similarly, it seems that the vertical angles between objects might be very sensitive to height of the objects. Would it be better to decouple the impact of dimension e.g. by computing this angle based on the center of the bottom face?\n- I wonder what's the benefit of hardcoding spatial relations as opposed to, say, using a MLP to automatically compute spatial relations from the bounding boxes of the objects, probably in their local coordinate frame?\n- I imagine many text descriptions are probably related with the room architecture itself. Would there be benefit in also trying to encode object-room relations explicitly?\n- I know its mentioned in the supplementary that determining the orientations from noisy inputs is challenging. Still, it appears to be that most natural language descriptions of directions (left, right, etc) is highly depending on the orientation of the objects, so the model will need to reason about orientation in some way. Have you tried to more explicitly handle orientations when defining and encoding the spatial relations?  If, does inclusion of them degrades the performance because of the ambiguity? If not, what are some possible ways to potential handle such cases?\n\n### Evaluation\n- As mentioned, some discussion of alternative ways to design the spatial relation layer might be helpful, but definitely not necessary.\n", "limitations": "Only one limitation that is common to all method of this class is mentioned towards the end. More discussion of limitation specific to this method would help.\nQualitative failure cases are provided in the supplementary, with analysis. One example (Supp Figure 2, right) shows a weakness specific to this method regarding object orientation.\n\nI agree that there is no apparent negative social impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First and foremost: I have not worked on the problem studied in this paper and was not familiar with the related works until reviewing this paper. So this is more of a common-sense review judging from the technical soundness and quality of evaluation, as well as quick skim through of some of the relevant works discussed. I'll defer the judgment about the magnitude of contribution to reviews who are more familiar with this problem.\n\nThat being said, I like this work in general: it has a very clear motivation (improving spatial reasoning for 3D object grounding) and proposes an effective solution that clearly outperforms relevant prior works. My concern about this paper is largely around the magnitude of the contribution and if that have significant impact on future works. Again, I don't think I am really qualified to judge about this, but based on other points below, I lean towards accepting this paper.\n\n### Strengths:\n+ Simple yet important key observation about the importance of explicitly encoding pairwise spatial relations, solid design of a novel module based on this observation.\n+ A generalizable teacher-student training method that can be extended to other methods for the problem studied.\n+ Exceptional results: significantly outperforms relevant prior works on multiple benchmarks and even outperforms works with more assumptions on input.\n+ Extensive ablations validating each of the design choices.\n+ Good writing quality, easy to follow.\n\n### Weaknesses:\n- Magnitude of contribution might be slightly limited since it follows a rather standard pipeline. The novel part of this paper can, to some extent, be seen training tricks / inductive biases to help the model better picking up certain information (spatial relation) that is important to the task.\n- Following the previous point: it might be argued that the techniques introduced in this network might be limiting eventually if solutions emerge to better capture the spatial relations from the object attributes alone.\n- Ablations only justify that each of the component help with the final result, without discussion of alternative method that might achieve the same goal. I think this understandable though - should not expect a work that explores an idea to exhaustively study the design space.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "__TrIvMY7XR", "reviewer": "Reviewer_9xAT", "summary": "This paper focuses on improving spatial relationship reasoning in the task of 3D object referring expression comprehension. The key ideas are a dedicated spatial attention mechanism integrated into the basic transformer architecture, and a teacher-student training mechanism. The paper significantly outperforms existing baselines on ReferIt and ScanRefer datasets.", "questions": "L159: Can you be more specific about the horizontal and vertical angles?\n\nL165: Why do you use sigsoftmax, in contrast to, for example, softmax？\n\nL235: Is there any reference on using only the first 3 layers of BERT to encode the sentence? Why don't use all layers? Also, it seems that some baselines (InstanceRefer) use (perhaps) weaker textual encoders (GloVe+GRU), can you provide an ablation study on the textual encoder module?\n\nL239/L205: If the pointnet is pretrained and fixed, why are you adding $L_{obj}^u$?\n\nL240: The author should explain the details of the rotation augmentation.\n", "limitations": "What are the limitations of this paper?", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly motivated and well written. The model is based on a straightforward transformer architecture, and integrates spatial attention models. Although I would argue that the color feature and spatial relationship feature used in the paper is a little ad-hoc but it is simple and effective. I have only very few questions and suggestions about the weakness of the paper.\n\nPerhaps the biggest weakness of the paper is that, based on my understanding, the base architecture (e.g., the R1 model in Table 1) is not an existing baseline model. Thus, the authors are both changing the basic architecture (c.f., InstanceRefer and Multi-View) and add-ons (teacher-student, spatial attention). To better understand the effectiveness of the proposed modules, the authors should consider integrating the presented techniques with other baselines too.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7J_h_iFInSw", "reviewer": "Reviewer_yLyV", "summary": "The paper proposes a method for language grounding in 3D scenes (i.e. detecting a specific 3D object instance based on an input natural text describing the object in the context of the scene).  The method is based on a transformer architecture, with the key novelty lying in a spatial self-attention mechanism and a teacher-student distillation training setup.\n\nThe method is evaluated on the ScanRefer and ReferIt3D datasets, with comparisons against ablations to demonstrate the value of method components, and also against baselines taken from prior work to demonstrate the relative improvement in 3D object grounding.  The experiments report grounding accuracy and show measurable improvements over prior work.", "questions": "I would like the authors to address the identified weakness above, at least in terms of the broad strokes of how the approach performs for input text exhibiting different degrees of spatial relation disambiguation.", "limitations": "The conclusion mentions the two-stage nature of the approach as a limitation (i.e. that object detections are assumed to be given, and may be imperfect).  The conclusion also briefly states that the presented work only has minimal potential negative societal impact.  I would agree with this statement given the domain (3D interiors), but I would still point out that inherent dataset biases may misrepresent the rich diversity of environments in which people around the world may live.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The paper is clearly motivated in the need to better model 3D spatial relations in the context of the language grounding problem. The described spatial self-attention mechanism is to my knowledge a novel contribution in this area\n+ The exposition is overall quite clear, and the experiments present a thorough spectrum of ablations that do a good job of analyzing the impact of different method components (in particular, the attention to contrasts between view-dependent vs view-independent text inputs is appreciated)\n\nWeaknesses\n- Given the big focus on spatial relations, I would have liked to see a more systematic analysis of the results dependent on the need for attending to spatial relations to disambiguate the specific object instance that the input text describes (as illustrated well in Fig 1).  For example, a comparison of overall performance on sentences like those seen in Fig 1 vs sentences that do not use spatial relations to disambiguate would offer more detailed evidence that the proposed architecture is indeed better able to ground objects in such cases.  In addition, a breakdown by spatial relation types (involving distance, involving orientation, and involving both) would be quite informative.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "02Vdl4VPUS", "reviewer": "Reviewer_cFMa", "summary": "This paper proposes a language-conditioned transformer model for localizing objects in 3D scenes and reasoning about relative spatial distances based on natural language description. The model is based on the transformer architecture and includes spatial attention layers and multi-head design to learn and reason about relative distances and orientations among multiple objects. The spatial attention output is fused with standard self-attention by the sigmoid softmax function proposed by the authors. The model is trained using a teacher-student approach accompanied by rotation augmentation to facilitate learning under scarce data and to produce reasonable inference with point cloud inputs. The final model is tested on object grounding tasks Nr3D, Sr3D, and ScanRefer3D, and outperformed state-of-the-art models.", "questions": "As mentioned in the weaknesses above, clarifying the language conditioned weight, BERT text feature input, and auxiliary tasks would be helpful. In addition, it would be helpful to include more details about the cross-attention computation in the model.", "limitations": "The object proposal in the two-stage pipeline is critical because it constrains the hypothesis space for cross-modal matching. Thus the prospect of improvement in the object proposal stage would be beneficial.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. This paper addresses a challenging and interesting task of 3D object reasoning based on natural language description, with a focus on distinguishing similar objects referred by the text using different spatial relationships. \n2. Using multi-head attention, the proposed language-conditioned transformer model effectively learns to capture and reason about different spatial relationships among objects in the self-attention layers. \n3. The model is systematically evaluated on multiple datasets and achieves strong performance, and the learned attention is clearly demonstrated in Figure 4. \n4. The ablation study is carefully designed and shows the value of different components the authors introduce in the paper.\n\nWeaknesses: \n1. In equation 3, it is not very intuitive what g (language conditioned weight) means. What is the role of the classification token s_cls being an input in the conditioning? \n2. Related to the previous point, in Figure 2 (left), the Text BERT serves as input to both the spatial self-attention and cross-attention. It is not clear if the self-attention takes the text features (s_1, …, s_M) as input or only takes the s_cls token.\n3. In line 189, the authors state that the teacher model’s input object representation is the sum of object class label embedding and the averaged color embedding. Does the sum introduce any difficulty during training, especially when it requires the model to use object appearance (such as color) rather than spatial relationships to locate objects?\n4. It is nice to see the visualization of the teacher model in Figure 4. It will be compelling if the authors show that the student model attention layers behave similarly.\n5. The loss function in equation 8 is very complicated. There is little detail about the auxiliary losses and the rationale for including them in the objective.\n6. The final model only marginally outperforms 3D-SPS, according to Table 5. Although the authors state that the comparison is not fair due to detection optimization or single-stage pipeline, it would be helpful to reflect on how to improve the current detected object proposal.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept