Exploiting Contextual Objects and Relations for 3D Visual Grounding
Authors: Li Yang
Chunfeng Yuan
Ziqi Zhang
Zhongang Qi
Yan Xu
Wei Liu
Ying Shan
Bing Li
Weiping Yang
Peng Li
Yan Wang
Weiming Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D Visual Grounding
Contextual Object
Contextual Relation

Abstract: 3D visual grounding, the task of identifying visual objects in 3D scenes based on natural language inputs, plays a critical role in enabling machines to understand and engage with the real-world environment. However, this task is challenging due to the necessity to capture 3D contextual information to distinguish target objects from complex 3D scenes. The absence of annotations for contextual objects and relations further exacerbates the difficulties. In this paper, we propose a novel model, CORE-3DVG, to address these challenges by explicitly learning about contextual objects and relations. Our method accomplishes 3D visual grounding via three sequential modular networks, including a text-guided object detection network, a relation matching network, and a target identification network. During training, we introduce a pseudo-label self-generation strategy and a weakly-supervised method to facilitate the learning of contextual objects and relations, respectively. The proposed techniques allow the networks to focus more effectively on referred objects within 3D scenes by understanding their context better. We validate our model on the challenging Nr3D, Sr3D, and ScanRefer datasets and demonstrate state-of-the-art performance. Our code will be public at https://github.com/yangli18/CORE-3DVG.

Exploiting Contextual Objects and Relations for 3D Visual Grounding
Li Yang',, Chunfeng Yuan I+, Ziqi Zhang's Zhongang Qi? Yan Xu Wei Liu', Ying Shan? Bing Li Weiping Yang', Peng Lis.6, Yan Wang*., Weiming Hu
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA
2 ARC Lab, Tencent PCG. "The Chinese University of Hong Kong
'Education Management Information Center, Ministry of Education Allibaba Group
&Zejjiang Linkheer Science And Technology Co., Ltd.
School of Artificial Intelligence, University of Chinese Academy of Sciences Schooo of Information Science and Technology, ShanghaiTech University
Abstract
3D visual grounding, the task of identifying visual objects in 3D scenes based on natural language inputs, plays a critical role in enabling machines to understand and engage with the real-world environment. However, this task is challenging due the necessity to capture 3D contextual information to distinguish target objects from complex 3D scenes. The absence of annotations for contextual objects and relations further exacerbates the difficulties. In this paper, propose novel model, CORE-3DVG, to address these challenges by explicitly learning about contextual objects and relations. Our method accomplishes 3D visual grounding via three sequential modular networks, including a text-guided object detection network, a relation matching network, and a target identification network. During training, we introduce pscudo-label self- generation strategy and a weakly-supervised method facilitate the learning of contextual objects and relations, respectively. The proposed techniques allow the networks to focus more effectively on referred objects within 3D scenes by understanding their context better. We validate model on the challenging Nr3D, Sr3D, and ScanRefer datasets and demonstrate state-of-the-art performance. Our code will be public at ullhtps: /ttp::////ttbbtb// com/ yangli18/CCRR--30
1 Introduction
Teaching machines to interpret and interact with the real world through vision and language has been long-standing pursuit in artificial intelligence research. An important and fundamental task towards this goal is visual grounding, which involves detecting the referred visual object based on natural language input. While previous works have largely focused on visual grounding within 2D images [ 2 4 5). there has been a growing interest in extending this task to point clouds of 3D scenes, giving rise to 3D visual grounding 7
In 3D scenes, objects exhibit diverse spatial distributions, and it is common for objects of the same category to coexist. To clearly describe the target object, language descriptions often provide the contextual information, i.e. the contextual objects as well as their relations to the target object within the environment. For instance, in Figure [a), the text describes the target chair by specifying its relative position to the door and table. The interpretation of such contextual information from textual descriptions to visual representations is erucial for 3D visual grounding.
"Equal contribution, Cooresppondigg author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Relation matching network
The black chair table, closest to the door.
Text gext-guided object
Target dentification network
detection network
Target object
Contextual objects and relations
Labeled
Unlabeled
(8) The contextual information for the referred object.
(b) The proposed framework 3D visual grounding.
(a) Language descriptions often provide the necessary contextual information accurately depict the target objects. (b) Our proposed framework for 3D) visual grounding.
Figure 1:
Existing methods for 3D visual grounding typically involve matching a large set of object proposals with the textual description. These methods either use graph networks [6 g g or transformer architectures [10 [2 13 [E] to implicitly model contextual information between object proposals while performing cross- modsl fusion with the text. Despite the effectiveness, their optimization objectives mainly focus on the final matching of object proposals with the textual description, without paying special attention to the alignment with the contextual information in the text. Consequently, important contextual cues may be overlooked during target inference, resulting in less reliable results.
In contrast, humans naturally possess remarkable ability to infer the target object based on language description. We can easily capture the relevant contextual information in the 3D scene to reason about the target indicated by the description in a two-step manner: Firstly, we locate all the objects mentioned in the text. Then we figure out the relations between these objects and compare them with the textual description, which enables to ultimately identify the target referred to in the description. Motivated by this, we improve 3D visual grounding by mimicking this human reasoning process. Specifically, (1) we explicitly detect all mentioned contextual objects in addition to the target object, focusing the network on these objects to capture contextual information for target inference. As most current datasets do not have annotations for contextual objects, we propose pseudo-label self-generation strategy, which leverages object semantics learned at the phrase level to generate learning labels for contextual objects. (2) We explicitly model the contextual relations between the candidate targets and the contextual objects as features of an adjacency matrix, which initially encode various spatial relations of objects. To facilitate the learning of contextual relations, we introduce weakly-supervised learning loss that explicitly distinguishes related and unrelated relations for the target object.
The above ideas are embodied as a novel transformer-base framework that explicitly exploits Contextual Objects and RElations for 3D Visual Grounding (CORE-3DVG). Our framework. illus- trated in Figure [bb), comprises three modular networks that divide the 3D visual grounding process into three inference steps. Firstly, text-guided object detection network performs joint detection of both the target and contextual objects. Next, a relation matching network models the representations of their contextual relations. Finally, the target identification network pinpoints the referred object based on the modeled contextual relations.
To summarize, our contributions are threefold:
We propose to explicitly model and learn the objects and relations mentioned in the context to focus the network on contextual information for 3D visual grounding
We introduce novel framework for 3D visual grounding, dubbed CORE-3DVG, which tackles the task of 3D visual grounding through three sequential modules. Our framework first detects all mentioned objects in the scene, then models their contextual relations, and finally performs context-based target inference.
We evaluate our method on the challenging benchmarks including Nr3D, Sr3D, and Scan- Refer. Our method outperforms the previous state-of-the-art counterparts by significant margins. Extensive experiments validate the efficacy of our proposed techniques.

---

2 Related Work
2D/3D Visual Grounding. Visual grounding aims to locate target objects in 2D images or 3D point clouds based on the language imput. Existing methods are generally categorized as one-stage methods [15 16
17
4 [8] and two-stage methods [ 3 20
21 22 23
24
25 26 6 71. One-
stage approaches fuse textual features with visual representations to generate target predictions in shot. Two-stage methods follow two-step detection and matching process: they first leverage the off-the-shelf detectors [27 28 29 30 to generate object proposals, which are then matched with language input select the best match the estimated target.
The prevalent methods in 3D visual grounding adopt a two-stage scheme. Early works 6 use graph-based methods to model spatial relations among object proposals. With the rise of transformers [31], recent works [T 12
g
13
14
18 have adopted transformers for feature fusion incorporates 2D visual grounding to improve the
and cross cross-modal fusion. Among them, SAT
12
performance of 3D visual grounding. Multi-view transformer
[4] models different views in visual
feature modeling.
Context Modeling for Visual Grounding. Modeling contextual information is important for visual grounding, especially for 3D visual grounding. While early works use graph neural networks or transformers to model contextual information, they fall short in encoding various spatial relations between objects. Recent 3DVG-rransormerr [1]] and ViL3DRel (32) explicitly encode the objects' distances or orientations into their self-attention computation to better model 3D spatial relations.
In contrast to prior works, our method focuses on the explicit learning of contextual objects and relations and exploiting this information for target inference.
Method
3.1 Framework
Our method focuses on the learning of contextual objects and relations, and exploits such information to achieve reliable 3D visual grounding. Figure illustrates the overall framework. Given a 3D scene's point cloud and language description, initially encode their features into F, E MxC and Fi E _ respectively, where and L represent the number of features and denotes the feature dimension. Based on these encoded features, 1) A text-guided object detection network first detects semantically related objects in the 3D scene, including target and contextual objects. 2) Then, construct the spatial relation features in a pairwise manner for these detected objects and input them into the relation matching network for modeling contextual relations. 3) Finally, utilizing the modeled contextual features, the target identification network effectively match the detected objects with the textual information to find the referred object.
3.1.1 Text-Guided Object Detection Network
The object detection network, as illustrated in Figure aa), consists of Na transformer decoder layers. The network initiated with set of K object queries [01, .2. oo IR KxC, which are generated from the point cloud features F. based on their correlation with the textual features Fi (see the supplementary material for more details). In each decoder layer, the initial object queries are first input to a self-attention layer to enable feature interaction between candidate objects. The queries are then passed through two cascaded cross- aats-nattntion layers to gather textual and visual features successively. Next, a feed-forward network (FFN) updates the object queries. For the resulting K object queries, two FFNs are applied to predict their object bounding boxes [6,, b2 6k]]T k and confidence scores p2, _p]'"' k x1 respectively.
To avoid duplicate object detections, we employ non-maximum suppression (NMS) and retain the top scoring detections. We represent the bounding boxes of the retained detections as [b,, b.. b E RTx4 and their corresponding object queries as Jō 52 ... or I TxC_ These detections correspond to the most relevant objects detected for the language description, in- cluding both target and contextual objects. For the training of this detection network, we propose pseudo-label seff-generttion strategy to facilitate learning of unlabeled contextual objects, which detailed in Section 3.2
3

---

(g Tect Tect-guized Object Detection Network
By) Raidion Matching Network
Fi g200
Updend rinion festures
prendan
Juaus ancade
PFN
Octecion knnee Epee
corree Irullsss
Terkel festuee gdue
Weake superioa rosting
Cest Feet
Psesedo fober eefggrrrttonn strrtegy
Target
Thalggu
Tha Nolkan
co 10.00
ukialide surppirass Hur
Desition
rext to the rocke
Soat gaidee
00
Cane Care
aeeeddrau phoue
clac
Figure 2: The overview of our framework. Our framework comprises three modular networks that divide the 3D visual grounding process into three inference steps.
3.1.2 Relation Matching Network
The relation matching network is developed to model the target object's contextual relations that match the textual description. Specifically, for each pair of detected objects 6;), we devise multiple spatial relation features that consider different aspects, including distance orientation, volume & dimension, and perspective (see the supplementary material for details). This results in x T spatial relation features [rf,] (islssisT for the T detected objects, where T. E R represents the relation feature with b, as the candidate target and b, as the contextual object. We perform linear projection on each feature rf and its corresponding contextual object query o,, and then concatenate them as the initial contextual relation feature:
rl. = Concat rig. W22 0j),
where Wi RC/2XC, and W2 E C/2xCC are the linear projection weights.
As depicted Figure 2 b), the relation matching network takes the contextual relation features PigllstssT as input and applies N, transformer decoder layers to model contextual relations that match the textual description. To achieve this, each decoder layer contains a self-attention layer for intra-modal information interaction among relation features, a cross- -ooss-tttention layer on textual features for mrosl-modal fusion, and an FFN transform the relation features. The relation matching network outputs the updated relation features (isiiri and we apply two-layer FFN to generate their respective matching scores w.r.t. the textual information, denoted (m.,)]i ST.ST. weakly- supervised learning approach is proposed to supervise these matching scores and thereby learn the contextual relations associated with the referred object (detailed in Section 3.3)
3.1.3 Target Identification Network
We propose the target identification network to find the referred object with fine-grained alignment to the textual description by leveraging the information of the detected objects and their contextual relations. As shown in Figure 2(c), with the detected objects as candidates, we input their object queries [ō1, 02. 6T|T" R RCC into the target identification network. This network aligns the object queries with the textual features via N₂ transformer decoder layers. Each decoder layer includes a self-attention layer and a cross-attention layer to enable intra-modal and cross-modal feature fusion, followed by FFN layer refine the object query representations, We take the output object queries from the last decoder layer and apply a two-layer FFN to estimate their confidence scores the target object, denoted [pi,
P TXX1
To exploit contextual relations for target discrimination, we propose context-aware self-attention mechanism incorporating relation features to better model contextual information among candidate objects. Specifically, the self-attention layer contains multiple attention heads to process the input object queries. In each attention head, the object queries are first lincarly projected as the query, key,

---

and value embeddings, represented by Q. K, V E RTxx, respectively (dy is the dimension of the embeddings). The conventional self-attention method computes the attention of the i-th object the j-th object by correlating their query and key embeddings, i.e. softmax, (Q(i) ( /Vde). Our self-attention mechanism extends this attention computation by considering the modeled contextual relations between objects. To accomplish this, we calculate an additional query embedding of the object queries as Qr E gd'xd, and perform linear projection on the contextual relation features (rijl to obtain their key and value embeddings as Rg, R. E Rxxxxdd respectively. The attention of the i-th object to the j-th object is then computed as follows:
(Q(()
attma.j = softmax,( (oftmax,(((()))) Q()) +
Ra.
(2)
22
The resulting self-attention matrix allows each object to focus more on the objects that have important contextual relations between them. Moreover, we aggregate the value embeddings of both the object queries and their associated relation features for attention-base feature fusion:
out; = -, attm,, : (V()) + (i,),)
(3)
In this manner, each attention head of the self-attention layer achieves context-based feature aggre- gation, and the outputs from all attention heads are concatenated to update the input object queries. Consequently, the updated object queries can explicitly encode information about related contextual objects and relations. We incorporate the context-aware self-attention layer in each decoder layer of the target identification network, enabling a more effective comparison of modeled object queries with textual information to discern the referred object.
3.2 Pseudo-Label Self-Generation for Contextual Object Learning
Our method starts with text-guided detection network to detect all the objects mentioned in the textual description. However, since the contextual objects associated with the referred targets are typically unlabeled, the object detection network lacks the requisite supervision for effective training. To address this issue, we propose a pseudo-label self-generation strategy that facilitates the learning of contextual objects within the network.
As shown in Figure a). we first extract the noun phrases for physical objects from the input text and divide them into the labeled nown phrase (of the target object) and the unlabeled noun phrases (of the other contextual objects). We input these extracted noun phrases into the object detection network along with the same initial object queries and point cloud features. For the labeled noun phrase, the network produces K detected bounding boxes [b1, b2, b IT E IR x6 and confidence scores f2.. ... PK]T E IR xx1, Given the ground- -rund bounding box of the target object as byt E R we find the best matching detection result (δ } (with index 1)) as the positive sample. Then we define the following losses to supervise the detection results of the labeled noun phrase:
Luloc = Lbox (bge, bn); Σ An -log(1 - -px),
Lacls = -log(p,)
Reptn
where the loss function Loox consists of the GloU loss and the L1 loss, i.e. (:oxe::-)) AgiouLgiou(:: (-.-) + >11 ()) (Agiou and ΛL1 are hyper-parametee parameters to balance the two losses).
During training, based on the supervised learning for labeled noun phrases, we develop the network's s ability to detect objects for various noun phrases. We thereby apply the detection nerwork to the unlabeled noun phrases and utilize the output detection results to create pseudo labels for contextual objects. In particular, for each of the U unlabeled noun phrases, we process the detection results with NMS and select the detections with the highest scores. We use bī.5 E R and př R to denote the bounding box and unnormalized score of the j-th detection for the i-th unlabeled noun phrase. This gives U x H object detections [ (i.,),sssss,ss: for contextual objects. As shown in Figure aa),
to serve as the pseudo labels a), we combine these pseudo labels with the target's ground-trutt label byt to supervise the network's 's detection results ((br,pt) for the entire sentence (see Section I. Similar to DETR (34], we use the Hungarian algorithm to find the best matching detections for the combined labels, where and Øs.j denote the matching indexes for the target label
5

---

and each pseudo label, respectively. Then the supervision losses on the detection results are computed
as:
H
Lior Lbox (bgx: (7)
Σ
Σ
Loox (D7,g) Ber.s):
wis
i=1
1=1
U
(5)
Leks -log((+)
Σ
Σ
wij log(fo:.)
Σ KET R (1:5)
An loog(1 pk),
exp(p5.)
where Wi
Σn exp(ri,) sigmoid(p?,)) denotes the learning weight for each pseudo label. We
use Wiy to encourage learning pseudo labels higher scores under each unlabeled noun phrase while also considering the individual confidence of each pseudo label to mitigate the impact of learning unreliable pscudo labels. With Equation (5), the network learns to understand the object phrases within the input text for associated object detection. During inference, the network detects target and contextual objects based on the input text without the need to manually extract their noun phrases.
3.3 Weakly-SSperrised Contextual Relation Learning
Our relation matching network outputs score matrix [maj]isijss for all pairs of ' detected objects, representing the matching scores between their contextual relations and the textual description. However, since there are no explicit labels for contextual relation learning, we propose weakly- supervised method find the contextual relations associated with the referred object. The overall optimization objectives for contextual relation learning are:
1. For the target object, there exists at least one contextual relation with a high matching score.
2. For the related contextual object, its contextual relations should have lower matching scores when paired with objects that are not the target.
To accomplish this, we first match the T detected objects with the ground truth target bet and find the best matching detection ba (indexed by a), which the model shall later infer as the detected target. Then we take the a-th row from the score matrix SA..g.. obtaining [masliza as the contextual relation matching scores between the detected target 6, and other objects. From these scores, we find the maximum score Ma,8 with column index 8 = arg max, max,((ma..))l (Ima.,]izaa) We thereby consider the object bg as the most related contextual object for the target 6,. Following the optimization objectives I and 2, devise the following losses to supervise the matching scores of the a-th row and f3-th column:
Lrell Σ
max(0, A Ma.j ma,9),
ita
(6)
Lre12 Σ
max(0, A mi,3 mo,s),
where the hyper-parameter A defines the score margin. The losses Lrell and Lre12 ensure that the contextual relation Ta, (between objects ba and b3) has a matching score surpassing the scores of other contextual relations (with target b, contextual object by) by margin of A. This encourages the network to explicitly differentiate related and unrelated contextual relations for the target object. We utilize the learned contextual relation features for target inference, detailed in Section 3.1.3
3.4 Training
In addition the previous losses for learning contextual objects and relations, we also supervise the target identification network on its estimated grounding scores [pi.rt, Pr for the T candidate object detections. We take the a-th detected object that best matches the ground-truth label by. the positive sample, and treat other detections negative samples. We compute the binary

---

cross-entropy loss on their grounding scores as follows:
T
Lvg - logg(p))
+
An log pk)
We take the sum of losses for all the modular networks to train the entire model. Following the previous works [6 7 we apply auxiliary losses to the predictions from every decoder layer of the modular networks, The overall training loss is:
Na (Chloe
N₂
C
Ciile + Cloc Labs) +
Σ (Crenl + Cta22)
1 Ng
vg
(8)
where the superscript indexes the losses of predictions from multiple decoder layers.
4 Experiments
4.1 Datasets
Nr3D [6] is built 3D indoor scene dataset ScanNet (35). It contains 41,503 coman-annooatedd text descriptions, covering 76 object categories and 707 indoor scenes. The dataset is divided into "Easy" and "Hard" subsets depending
whether there objects that share the same category the target in the scene. Based on whether a specific viewpoint required to infer the target, the dataset is divided into "View-dep." and "View-indep." subsets.
Sr3D (6) contains 83,572 descriptions that are automatically generated using specific templates. Similar to the Nr3D dataset, the Sr3D dataset is divided into multiple subsets for evaluation.
SeanRefer 7 provides 51,583 text descriptions of 11,046 objects in 800 3D scenes from the SeanNet. On average, there are 13.81 objects and 64.48 text descriptions per scene. The official division takes 36.665 samples as the training set and 9,508 as the test set. According to whether the target object category is unique in the scene, the dataset is divided into "Unique" and "Multiple" subsets.
4.2 Implementation Details
We utilize the AdamW optimizer 36 train model with batch size of 24. For visual feature encoding, we utilize the PointNet++ network (37] with an initial learning rate of 10-3, The rest of the model has an initial learning rate of 10-4, and the weight decay value is
10-4, We employ the first three layers of the RoBERTa 3S to extract text features. We train our model for 120 epochs on the Nr3D dataset. 60 epochs on the Sr3D dataset, and 100 epochs on the ScanRefer dataset. The ppper-parameter A in Equation i set to 0.4, and we set the weight hyper-parameterr Agiou and AL1 for the Lbox loss. The numbers of decoder layers for the text-guided object detection, relation matching, and target identification networks are set to N₂ = 3, N, 2. and N₂ 3, respectively. Following the previous work [39], we apply rotation data augmentation for the 3D point cloud scenes and augment supervision with detection prompts during training. We measure accuracy using A.e50.25 for both Nr3D and Sr3D datasets, where a predicted bounding box is considered correct if its IoU with the ground truth target exceeds 0.25, For the ScanRefer dataset [7]. we employ two metrics: Aco@0.25 and A.c@0.5.
4.3 Comparison with State-of-the-Art Methods
Table presents the comparative results of our method with the current state-of-the-art methods on the Nr3D, Sr3D, and ScanRefer datasets. Our method consistently outperforms the previous methods. For the Nr3D and Sr3D datasets, it is standard practice to use ground-truit object boxes as object proposals infer the referred object. However, considering that ground-truie boxes are usually unavailable in practical scenarios, our model is designed to locate objects of interest using a dedicated detection network. Thus, for a fair comparison, we quote the evaluation results from BUTD-DETR 39. where previous methods are evaluated with detected object proposals by a pre- trained 3D object detector [40] The quoted results are marked with i and we denote this evaluation setup "det" in Table] I On the Nr3D and Sr3D datasets, shown in Table I our method exhibits
7

---

Table I: Comparison with the state-of-the-art methods on Nr3D, Sr3D, and SeanRefer.
Nr3D Acc@0.25 (det)
Sr3D
ScanRefer
Method
Acc@0.25 (det)
Acc@0.25
Acc@0.5
Referlt3DNet 6
24.0
27.7'
26.4
16.9
ScanRefer t
35.5
22.4
TGNN S
37.4
29.7
InstanceRefer 9
29.91
31.51
40.2
32.9
FFL-3DOG [4]
41.3
34.0
LanguageRefer [1]
28.6
39.5t
3DVG-Transformer
45.9
34.5
SAT 2D [2]
31,71
35.4
44.5
30.1
BUTD-DETR (3
43.3
52.1
52.2
39.8
CORE-3DVG
49.57
54.30
56.77
43.84
Table Comparison with the state-of-the-arr methods on different subsets ScanRefer.
Unique
Multiple
Overall
Method
Acc@0.25
Acc@0.5
Acc@0.25
Ace@0.5
Acc@0.25
Acc@0.5
Neferlt3DNet
53.8
37.5
21.0
12.8
26.4
16.9
ScanRefer
63.0
40.0
28.9
18.2
35.5
22.4
TGNN[8]
68.6
56.8
29.8
23.2
37.4
29.7
InstanceRefer()
77.5
66.8
31.3
24.8
40.2
32.9
FFL-3DOGG4T
78.8
67.9
35.2
25.7
41.3
34.0
3DVG-Transformer
77.2
58.5
38.4
28.7
45.9
34.5
SAT 2D[12)
44.5
30.1
BUTD-DETEY 39
84.2
66.3
46.6
35.1
52.2
39,8
CORE-3DVG
84.99
67.09
51.82
39.76
56.77
43.84
significant performance advantages the previous methods. Compared with the recent BUTD- DETR, which employs a larger transformer encoder and decoder network, our method achieves absolute improvements of 6.27% and 2.20% on the Nr3D and Sr3D datasets, respectively.
Jable/]rrovides detailed performance comparison between our method and previous approaches on the ScanRefer dataset. Our method shows superior performance across various test settings and evaluation metrics. Notably, on the challenging "Multiple" subset, where multiple objects share the same category as the target objects, method achieves 51.82% % in Acc@0.25 surpassing the leading BUTD-DETR (39 by an appreciable margin 5.22%. Our explicit modeling of contextual information enables better handling of such challenging scenarios. For the simpler "Unique" subset, where the target object is unique, we also achieve higher or comparable results compared to previous methods. Our overall accuracy on the ScanRefer dataset reaches 56.77% (Acc@0.25) and 43.84% (Acc@0.5), respectively, outperforming all the previous methods.
4.4 Ablation Studies
In this section, we conduct the ablation studies on the Nr3D 6 dataset to validate our method.
Learning Contextual Objects and Relations. Table presents the ablation study the proposed explicit learning of contextual objects and relations. The first row of Table represents the baseline model, which achieves an overall accuracy of 34.66% (Ace@0.25). Based on this baseline, first introduce the learning of contextual objects, resulting in a significant improvement of 9.76% in overall accuracy. The improvements are consistently observed across all test subsets, as shown the second row of Table 3 While the contextual relations are not modeled, the contextual objects alone provide crucial contextual information for target inference. the third row of the table, we separately introduce the learning of contextual relations and observe an improvement of 2.67%. The "View-dep" subset exhibits the largest improvement of 14%, highlighting the effectiveness of relation learning for context modeling. Finally, by incorporating the learning of both contextual objects and relations, we achieve an overall accuracy of 99.57%. This surpasses the baseline model by 14.91 and achieves the best performance among these ablation variants, shown in the last of Table

---

Table 3: The ablation study of learning contextual objects and relations.
Leamiing context. objects
Learning context. relations
View- dep.
View- indep.
Easy
Hard
Overall
40.50
29.04
32.13
35.59
34.66
49.85
39.20
40.75
45.78
44.42
/
43.34
31.55
38.27
36.98
37.33
53.91
45.39
48.19
50.07
49.57
Table The ablation study of pseudo-label gereretierttnn strategy.
View- dep.
View- indep.
Method
Easy
Hard
Overall
40.50
29.04
32.13
35,59
34.66
Supervised learning on labeled noun phrases
45.05
31.87
33.17
40.23
38.33
+ Learning pscudo labels for unlabeled noun phrases
49.85
39.20
40.75
45.78
44.42
Table 5: The ablation study of initial spatial relation features.
Distance & Orientation
Volume & Dimension
Perspective- related relations
View- dep.
View- indep.
Easy
Hard
Overall
49.85
39.20
40.75
45.78
44.42
52.38
40.99
43.53
47.70
46.57
4
51.62
40.70
42.94
47.20
46.05
/
52.49
42.58
46.46
47.81
47.44
52.55
43.42
46.85
48.28
47.90
Pseudo-Label Self-Generation Strategy for Contextual Object Learning. In Table in- vestigate the pseudo-label generation strategy used for contextual object learning. First, we only employ the detection loss defined in Equation 4 to learn the objects corresponding labeled noun phrases. The second row of Table 4 shows an overall accuracy improvement of 3.67% compared to the baseline model in the first row. This suggests that augmenting the textual diversity for object detection training also contributes to improved visual grounding. After that, we utilize the pseudo labels generated for unlabeled noun phrases to guide the learning of contextual objects. The third row of Table demonstrates that this technique boosts the model's accuracy from 38.33% to 44.42% (+ 6.09%) and exhibits significant performance enhancements across test subsets. These results further emphasize the importance of learning contextual objects, it allows the model to effectively capture contextual information for target inference.
Initial Spatial Relation Features. We develop a diverse set of spatial relations between objects as initial features for modeling contextual relations. Table 5 presents an ablation study of these relation features. The first row of Table 5 shows an overall accuracy of 44.42% when the initial spatial relation features are not used. Then, we individually incorporate the relation features related to "Distance & Orientation". "Volume & Dimension", and "Perspective-related relations", improving the overall accuracy by 2.15%, 1.63%, and 3.35% respectively. Notably, the perspective-relann relation feature achieves the greatest improvement, especially in the "View-dep." test subset (from 40.75% 46.46%), highlighting effectiveness in reasoning textual deseriptions associated with specific viewpoints. Finally, combining all these relation features in the last row of Table yields the highest accuracy of 47.90%.
Weakly-Supervised Contextual Relation Learning. In Table 6 we conduct an ablation study on the learning of contextual relations. The first two rows of Table 6 show an overall accuracy improvement of 3.48% when directly using the initial spatial relation features. However, as shown in the third row of Table the relation matching network alone fails improve performance without the inclusion of the weakly-supervised losses in Equation 6 By incorporating weakly- learning of contextual relations, we achieve an overall accuracy improvement of 1.77% and observe appreciable performance advancements across all test subsets, as depicted in the last row of Table 6 These results clearly demonstrate the necessity of the weakly-superviss learning in effectively modeling contextual relations.

---

Table 6: The ablation study of contextual relation learning.
Initial spatial relation features
Relation matching network
Weakly-supervised learning losses
View-
View- indep.
Easy
Hard
Overall
dep.
49.85
39.20
40.75
45.78
44.42
52.55
43.42
46.85
48.28
47.90
52.55
43.24
46.90
48.13
47.80
53.91
45.39
48.19
50.07
49.57
(a) The lamp next the red curtains.
(b) Choose the monitor on the desk that close the coats hanging
[c) Blue empty kid chair by green table right underneath the animals curtains
[d] The table white with monitors
Figure 3: 3D visual grounding results with associated contextual objects and relations.
4.5 Visualization
In Figure we visualize the visual grounding results of our method for various descriptions and 3D scenes. We green bounding boxes to represent the detected target objects and red bounding boxes to indicate the contextual objects that exhibit relation matching scores greater than 0.4. The results illustrate the efficacy of our method in comprehending contextual information within the text infer the referred target object. For example, in Figure B(a), when given the description "The lamp next to the red curtains", our model successfully locates the correct lamp and the adjacent red curtains. thereby avoiding misclassification of the lamp near the bed. This explicit utilization of contextual information enables more reliable inference of the target object.
Conclusion
In this paper, we presented 3D visual grounding framework that focuses on learning contextual objects and relations. The proposed framework consists of three sequential modules: text- gext-guided object detection, relation matching, and target identification networks. The innovative techniques of pseudo-label self- seff-gnneration and weakly-supervised learning facilitate the learning of contextual objects and relations. We verify the effectiveness of contextual objects and relations learning through extensive experiments and achieve leading performances multiple benchmarks, including Nr3D, Sr3D, and ScanRefer.
Despite the remarkable performance achieved, the generalization ability of our method is still limited since our model has been only trained on datasets with limited object semantics and corpus. The performance of our algorithms could be influenced by the types of devices used to capture the point clouds and the types 3D scenes. In the future, we plan to further dig into the generalization ability of 3D visual grounding frameworks and extend our method more practical and realistic scenarios.
Acknowledgements. This work is supported by the National Key R&D Program of China (No. 2022ZD01185011 the Beijing Natural Science Foundation (JQ21017, L223003, M22005, 4224091) 1), Natural Science Foundation of China (Grant No. 61972397, 62222206, 6203601 1, 62192782, 61721004, U2033210, 62372451, 62192785, 62372082, 62202469). the Major Projects of Guang- dong Education Department for Foundation Research and Applied Research (201 (0DD000))) Guangdong Provincial University Innovation Team Project (2020KCXTD045).
10

---

References
[1] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 69-85. Springer, 2016.
[2] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan I. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11 11-20, 2016.
[3] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang. Xin Lu, Mohit Bansal, and Tamara Berg, Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1307-1315 1315, 2018
[4] Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jicbo Luo. A fast and accurate one-stage approach to visual grounding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4683-4693 2019.
[5] Li Yang, Yan Xu, Chunfeng Yuan, Wei Liu, Bing Li, and Weiming Hu. Improving visual grounding with visual- -ssaa-linguistic verification and iterative reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9499-9508, 2022.
[6] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real woaldworld scenes. In Computer Vision-Eii ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 422 440. Springer, 2020.
[7] Dave Zhenyu Chen, Angel Chang, and Matthias NieGner. Scanrefer: 3d object localization in rgb-d scans using natural language. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX, pages 202-221. Springer, 2020.
[8] Pin-Hao Huang, Han- Hunn Lee, Hwann- Hwann-Tzong networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1610-1618, 2021.
Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural
[9] Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. In- stancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1791-1800, 2021.
[10] Dailan He, Yusheng Zhao, Junyu Luo, Tianrui Hui, Shaofei Huang, Aixi Zhang, and Si Liu. Transrefer3d: and-reltinno aware transformer for fine-graines 3d visual grounding. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2344 2352, 2021.
[11] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg- bovg-transformer: Relation modeling for visual grounding on point clouds. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928 2937, 2021.
[12] Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jicbo Luo. Sat 2d semanties assisted training for 3d visual grounding. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1856-1866, 2021.
[13] Junha Roh, Karthik Desingh, Ali Farhadi, and Dieter Fox. Languagerefer: visual grounding. In Conference on Robot Learning, pages 1046-1054 PMLR, 2022.
model for 3d
[14] Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15524- 15533, 2022.
[15] Xinpeng Chen, Lin Ma, Jingyuan Chen, Zequn Jic, Wei Liu, and Jicbo Luo. Real-time referring expression comprehension by single-stage grounding network. arXiv preprint crXiv:1812.00444 2018.
[16] Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, and Bo L.. real-time cross-modality correlation filtering method for referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10880-10889, 2020.
[17] Zhengyuan Yang. Tianlang Chen, Liwei Wang, and Jiebo Luo. Improving one-stage visual grounding by recursive sub-query construction. In European Conference Computer Vision, pages 387-404 Springer, 2020.
11

---

[18] Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16454-16463, 2022.
[19] Ronghang Hu, Marcus Rohrbach, Jacoh Andreas, Trevor Darrell, and Kate Saenko. Modeling relationships in referential expressions with compositional modular networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11 1115-1124, 2017.
[20] Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image- text matching tasks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):394 407,
2018.
[21] Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, and Anton Van Den Hengel. Parallel attention: unified framework for visual object discovery through dialogs and queries. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4252-4261, 2018.
[22] Hanwang Zhang, Yulei Niu, and Shih-Fu Chang. Grounding referring expressions in images by variational context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4158-4166, 2018.
[23] Peng Wang, Qi Wu, Jiewei Cao, Chunhua Shen, Lianli Gao, and Anton van den Hengel. Neighhourhood watch: Referring expression comprehension via gogguage-guide graph attention networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1960-1968, 2019.
[24] Richang Hong, Daqing Liu, Xiaoyu Mo, Xiangnan He, and Hanwang Zhang. Learning to compose and reason with language tree structures for visual grounding. IEEE transactions on pattern analysis and machine intelligence, 2019.
[25] Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha. Learning to assemble neural modulc tree networks for visual grounding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4673-4682, 2019.
[26] Sibei Yang, Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression comprehension. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4644-4655 2019.
[27] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-can: Towards real-time object detection with region proposal networks. IEEP Transactions on Pattern Analysis and Machine Intelligence, 39(6): 137-1199, 2016.
[28] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv: 120400 2018.
[29] Kaiming He. Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969 2017.
[30] Li Yang, Yan Xu, Shaoru Wang, Chunfeng Yuan, Ziqi Zhang, Bing Li, and Weiming Hu. Pdnet: Toward better one-stage object detection with prediction decoupling. IEFE Transactions Image Processing, 31:5121-5133, 2022.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 6008. 2017.
[32] Shizhe Chen, Poeire-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. In Advances in Neural Information Processing Systems, volume 35, pages 20522-20535, 2022.
[33] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, lan Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 658-666, 2019.
[34] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagonuyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 213-229. Springer, 2020.
[35] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annon annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9928-5839, 2017.
12

---

[36]
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2017.
[37] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++:: Deep hierarchical feature learning point sets in metric space. Advances in neural information processing systems, 30, 2017.
[38] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeit and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[39] Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transtormers for language grounding in images and point clouds. Computer Vision-ECCV 2022: 17th Eurupean Conference. Aviv, Israel, October 23-27. 2022, Proceedings, Part XXXVI, pages 417-433. Springer, 2022.
[40] Liu, Zheng Zhang, Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2949 2949-2958, 2021.
[41] Mingtao Feng. Zhen Li, Qi Li, Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang. and Ajmal Mian. Free-form description guided 3d visual graph network for object grounding in point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3722 3731, 2021.
13


Reviews:
[{"review_id": "oLQR8y6UQz", "reviewer": "Reviewer_Dhe4", "summary": "This paper proposes to explicitly model object relations mentioned in the language description for 3D visual grounding task. The proposed method designs a relation matching network and weakly-supervised contextual relation learning strategy to perform contextual learning. Finally, they design a target inference network to output the referred object and give performance of model on three benchmarks.\n\n", "questions": "See weakness.", "limitations": "none", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThis paper proposes to explicitly model the relation mentioned in the context for 3D visual grounding.\n2.\tThe paper introduces pseudo-label self-generation method and weakly-supervised contextual relation learning to address the unlabeled objects.\n3.\tExtensive ablation experiments demonstrate the effectiveness of the proposed modules.\n", "weaknesses": "1.\tThe comparison of experimental results misses the ViL3DRel method and 3D-SPS[17] method which use ground truth of all proposals in a scene for training. Maybe you can use ground truth of all proposals in a scene for a fair comparison.\n2.\tIt seems that the paper uses the b_gt to supervise the detection results of labeled noun phrase. Why not supervise the detection results of the language description?\n3.\tThe overall optimization objectives of contextual relation learning is very unreasonable in line 210. For a target object, is there only one relationship possible?\n4.\tWhy does the detection network have the ability to detect the unlabeled objects without any other annotations in section 3.2? Maybe you can explain it more clearly.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable suggestions and comments. Below are our responses to the raised questions. \n\n**Q1: Comparison with methods using ground truth object proposals.**\n\n**A1:** In our approach, we focus on reasoning about the target objects from point clouds without relying on pre-provided object proposals. Consequently, we did not compare with methods that utilize ground truth proposals in our original manuscript. \nBut your suggestion can help us better understand the effectiveness of our proposed modules. Following your suggestion, we compare our method with ViL3DRel[27], 3D-SPS[17], and the Multi-view [13] method. For a fair comparison, as in previous works, we use PointNet++ to encode point cloud features of ground truth object proposals and take them as inputs to train our model. As shown in the following table, our approach consistently outperforms previous leading methods across various settings. Even compared with the ViL3DRel (w/ KD) that has applied knowledge distillation based on a powerful teaching model, our method still achieves better performance. We will add more detailed results in the revised paper. \n\n| Method            |  Overall  |    Easy   |    Hard   | View-dep. | View-indep. |\n|-------------------|:---------:|:---------:|:---------:|:---------:|:-----------:|\n| 3D-SPS            |    51.5   |    58.1   |    45.1   |    48.0   |     53.2    |\n| Multi-view        |    55.1   |    61.3   |    49.1   |    54.3   |     55.4    |\n| ViL3DRel (w/o KD) |    58.1   |     -     |     -     |     -     |      -      |\n| ViL3DRel (w/ KD)  |    64.4   |    70.2   |    57.4   |    62.0   |     64.5    |\n| **CORE-3DVG (ours)**  | **66.60** | **72.64** | **60.80** | **65.79** |  **66.90**  |\n\n\n**Q2: Why not use the $b_{\\rm gt}$ to supervise the detection results of the language description?**\n\n**A2:** Sorry for the ambiguity. The target object label $b_{\\rm gt}$ is indeed employed to supervise the detection results of the entire language description (as shown in Eq. 5). We develop our model to first detect all semantically mentioned objects (including target and contextual objects) and subsequently identify the target object by analyzing inter-object relations. The target object label $b_{\\rm gt}$ plays a pivotal role in model training in the first detection stage. We will clarify this during revision. \n\n**Q3: The question about the overall optimization objectives of contextual relation learning in line 210. Does it mean that there is only one relationship for a target object?**\n\n**A3:** No, it does not mean that there's only one relationship for a target object but **at least** one relationship. We will modify the statement in line 210 as \"For the target object, there exists at least one contextual relation with a high matching score.\" \nSpecifically, given the absence of annotations for contextual objects and their relations, we employ weak supervision to exploit important contextual information. The objective 1 aims to learn the contextual object that is most related to the target object. For the most related contextual object, the objective 2 ensures that its relation to the target object is distinct from its relations to others, helping to identify the target object. We will improve the clarity of this in the revised manuscript. \n\n\n**Q4: Explain more clearly in Section 3.2 why the detection network has the ability to detect unlabeled objects without any other annotations?**\n\n**A4:** Sorry for the lack of clarity here. We propose to generate pseudo labels for unlabeled objects, enabling the network to learn to detect them. Specifically, for the entire language descriptions, we only have labels for the referred target objects, but lack labels for other objects that are also mentioned in the descriptions. This makes it difficult for the network to learn to detect all the mentioned objects, which limits the follow-up grounding performance. \n\nTo circumvent this, we extract the object phrases from the language descriptions, thus obtaining a diverse set of phrases corresponding to labeled target objects and unlabeled contextual objects. Using the phrases and labels of the target objects, we train the network to detect objects based on input noun phrases. As a result, the network is also able to detect the unlabeled contextual objects based on their noun phrases. These detection results are utilized as pseudo labels, which facilitate the learning process to detect all mentioned objects (both target and unlabeled objects) based on the complete language description."}, {"review_id": "Ae6HCGf8VR", "reviewer": "Reviewer_DS3o", "summary": "This work presents a novel approach for learning 3D visual grounding: the task of localizing and identifying objects given natural language inputs. The language prompts in this problem often involve using other contextual objects and their relations to the target object. The proposed approach breaks down this problem into an intuitive set of transformer-based architecture components: \n- a module to localize the objects corresponding to the nouns in the prompt and produce object features\n- a module to model the spatial relations between the detected objects conditioned on the text prompt\n- a module to combine the object features and learned spatial relations with the text prompt to predict the target object\n\nPrior work has generally been evaluated on ground truth 3D bounding boxes and on 3D detector outputs. This work takes the practically feasible approach of evaluating prior work on 3D detector outputs, and in this case the proposed work outperforms prior methods on three datasets by a significant margin. Extensive and detailed ablation studies justify the design of each of the components of the architecture.", "questions": "- What are more detailed reasons for why exactly the proposed approach which breaks down the problem into smaller sub-problems works better than prior work? This is valuable insight that would strengthen the paper\n- Is it possible to show qualitative examples comparing the proposed approach with prior methods?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n- Most prior work does not treat 3D visual grounding as an end-to-end learning problem, relying on pre-trained detectors for 3D bounding box proposals, whereas the proposed approach does. \n- The most successful prior work that tries to use transformers for this task, BUTD-DETR, uses a transformer encoder-decoder setup like DETR that implicitly combines the 3D bounding box, visual feature and text feature information with cross-attention. The proposed work breaks the visual grounding problem using multiple transformer-based modules, into text-guided object detection, modeling spatial relations between objects, and target object inference based on the output of the other two modules. This is empirically shown to be more superior to BUTD-DETR.\n- Explicitly detecting objects based on a text prompt is challenging because the labels for visual grounding only contain annotations for the target object and not the context objects. This work proposes an effective pseudo-labelling technique to address this.\n\nQuality\n- The proposed work has significant performance gains over prior methods, verifying the effectiveness of its design.\n- The qualitative results demonstrate high performance, including cases with complex text prompts describing context/target object relations and small objects.\n\nClarity - The paper is well written but quite dense due to having to contain descriptions for all the separate modules and their design.", "weaknesses": "Clarity\n- The draft appears to have been compiled with an altered value for the general line spacing value for the text paragraphs. The text appears vertically more tight than the standard NeurIPS. This is against the submission guidelines and should be fixed.\n- Section 3.2 about pseudo-label generation was a bit hard to follow, it would be beneficial to revise it for clarity or include a visual explanation using a diagram in the supplement.\n\nQuality\n- While in aggregate the performance is better than prior work, the current draft could benefit from more detailed quantitative comparisons with prior work beyond the results in Table 2 for unique vs multiple objects. How can we further characterize the examples where the proposed approach performs better than prior work e.g. small objects or cluttered scenes? Why exactly does this representation outperform prior work?\n- This similarly follows for qualitative examples as well, it would be beneficial to visually compare and understand the advantages of the proposed approach with prior work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and insightful comments. Below, we address each of the concerns raised.\n\n**Clarity** \n\n**Response:**\nThanks for the kind reminder. We will revise the paper and move some details to the supplementary material to make it more concise and clear. The text in Section 3.2 will be improved, and we will include a visual diagram in the supplementary material to provide a clearer understanding of the pseudo-label generation process. \n\n**Quality** \n\n**Q1: More detailed quantitative comparisons with prior work beyond the results in Table 2.**\n\n**A1:** Thanks for the suggestions. \nTo provide a more comprehensive comparison between our method and the prior work BUTD-DETR, we evaluate them on the ScanRefer dataset from various aspects, which includes the number of mentioned objects, the object size, and the total number of objects in each scene). \n\n- **The number of mentioned objects**: We first extract the noun phrases of mentioned objects from the language descriptions, and divide the dataset into subsets based on the number of noun phrases. We evaluate the performance of our method and BUTD-DETR on each subset, as shown in the table below. It can be seen that our method performs similarly to BUTD-DETR when only one object is mentioned, but significantly outperforms it when two or more objects are mentioned. This suggests that our method excels at utilizing contextual object information for target grounding, as descriptions with multiple nouns typically describe the target along with its associated contextual objects.\n| |  #Nouns=1 |  #Nouns=2 |  #Nouns=3 | #Nouns>=4 |\n|----------------------|:---------:|:---------:|:---------:|:---------:|\n| BUTD-DETR            | **50.42** |   53.16   |   51.15   |   51.09   |\n| **CORE-3DVG (ours)** |   50.14   | **56.78** | **58.68** | **56.51** |\n\n- **The object size**: We also divide the dataset into subsets based on the sizes of target objects and compare our method with BUTD-DETR separately on these subsets. The table below shows that our method brings larger improvements over BUTD-DETR in localizing smaller targets than larger ones. Identifying small targets solely based on their appearances is more challenging, but our explicit modeling of contextual objects and relations improves the identification of small targets. \n| Object size          |  ≤$0.4^3$ | $[0.4^3\\sim 0.8^3]$ |  >$0.8^3$ |\n|----------------------|:---------:|:-------------------:|:---------:|\n| BUTD-DETR            |   45.79   |        47.53        |   65.34   |\n| **CORE-3DVG (ours)** | **51.09** |      **53.24**      | **68.70** |\n\n- **The total number of objects in each scene**: We use the total number of objects in each scene to roughly indicate the level of clutter and compare our method with BUTD-DETR on subsets with different ranges of object numbers. From the results listed in the table below, our model achieves higher improvements on subsets containing 30\\~40 and 40\\~50 objects, particularly a significant 7.84 point improvement on the subset of 40\\~50 objects. This demonstrates our advantages in identifying the target when lots of objects are present in the scene. Our method is developed to first detect semantically mentioned objects and focus on these objects to capture contextual information, which can reduce distractions from unrelated objects and better handle cluttered scenes. \n|#Objects|    ≤20    |   20\\~30   |   30\\~40   |   40\\~50   |    >50    |\n|----------------------|:---------:|:---------:|:---------:|:---------:|:---------:|\n| BUTD-DETR            |   57.30   |   57.56   |   50.39   |   49.08   |   45.60   |\n| **CORE-3DVG (ours)** | **61.26** | **62.05** | **55.67** | **56.93** | **49.28** |\n\n**Q2: Visually compare the proposed approach with prior work.**\n\n**A2:** Thanks for your advice. We have submitted a PDF in the global author rebuttal, which presents the visual grounding results of our method compared to the ground truth and the results of BUTD-DETR on both the Nr3D and ScanRefer datasets.  Overall, our method demonstrates a better understanding of the contextual description about the target, and is able to pinpoint the target with the help of the modeled contextual object and relations. \n\n**Questions** \n\n**Q1: Reasons for breaking down the problem.**\n\n**A1:** \nOur idea of decomposing the 3D grounding problem into sub-problems is motivated by the human cognitive process of localizing visual targets based on language descriptions. When we humans interpret a textual description to identify a target object in a scene, we first need to identify the objects mentioned in the text, and then determine the correct target in relation to its contextual objects based on the described object relations. \n\nBy breaking down the 3D grounding into sub-problems, we can also explicitly model the contextual objects and relations associated with the target object. For free-form language descriptions, there are no explicit labels for objects and their relations outside the referred target. This necessitates specialized modeling methods for learning both the detection of contextual objects and the understanding of their relations with the target. Hence, our approach models these two sub-problems separately.  Compared to prior works, our explicit modeling of contextual objects and relations allows us to distinguish targets more intuitively and accurately, resulting in better performance and interpretability. We will revise the paper to elucidate the advantages of our approach over previous work more clearly. \n\n\n**Q2:Qualitative examples comparing the proposed approach with prior methods.**\n\n**A2:**\nWe have included qualitative examples in the attached PDF in the global author rebuttal. These examples showcase the performance of our method in comparison to the ground truth and the prior method BUTD-DETR, highlighting the advantages of our method in exploiting contextual objects and relations for target reasoning. "}, {"review_id": "vmsamCJy2s", "reviewer": "Reviewer_2pN3", "summary": "This paper introduces the CORE-3DVG method, which consists of three consecutive modules: Text-guided object detection network for detecting all candidates, Relation matching network for matching the relationships among candidate objects, and Target inference network for selecting the referred object. The proposed method incorporates a self-generation strategy to avoid the need for additional supervision. As a result, the CORE-3DVG method achieves state-of-the-art (SOTA) performance across multiple datasets. ", "questions": "Pls refer to weaknesses. ", "limitations": "This paper discusses the limitations of generalization ability.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The underlying idea of CORE-3DVG involves the explicit extraction of candidate objects and the reliable modeling of their relationships. The proposed module for extracting nouns/weakly supervised relational information appears to be a plausible and effective solution to avoid the need for additional data. By incorporating this approach, the CORE-3DVG method achieves a reliable and guaranteed modeling of candidate object relationships.\n\n2. The improvement achieved by CORE-3DVG on multiple general datasets is significant, as supported by additional ablation experiments that demonstrate the effectiveness of each module.\n\n3. The overall coherence and comprehensibility of the paper make it easy to follow.", "weaknesses": "1. The paper lacks essential analysis regarding the pseudo labels. \n    * For methods like CORE-3DVG, which rely on pseudo labels to avoid additional supervision, the quality of the pseudo labels is undeniably crucial. While the appendix provides detailed information about the generation process and the main text explains how the pseudo labels are generated, and the experiments indirectly demonstrate the usefulness of the pseudo labels, it is necessary to conduct a thorough analysis of the pseudo labels. \n    * For example, it would be valuable to assess the accuracy of localizing objects other than the target noun under the given annotation with only target nouns. \n    * Additionally, it is important to evaluate whether the relationships among candidate objects are correctly modeled under weak supervision. \n\n2. Lack of detailed comparisons and discussions in the paper. Most of the methods listed in Table 1 claim to model the relationships between the target and surrounding objects. For instance, even Instancerefer includes an explicit relationship modeling module and scores. I am curious whether the significant performance improvement is indeed attributed to the explicit pseudo labels (which makes the analysis in point 1 particularly important). If so, the authors should enhance the discussion in this regard to further establish the positioning of CORE-3DVG among a range of 3DVG papers. A thorough discussion of the similarities and differences with previous methods would not diminish the contributions of this paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive suggestions and comments. The following are our responses to the comments. \n\n**Q1: The lack of essential analysis regarding pseudo labels.**\n\n**(a) The accuracy of localizing objects other than the target.**\n\n**A1 (a):** Thanks for your suggestions. For a language description, correctly localizing mentioned contextual objects is indeed important for pinpointing the target object. However, the datasets (Nr3D, ScanRefer) with natural (human-annotated) descriptions lack the labels for mentioned contextual objects, making it infeasible to directly assess the accuracy of localizing such unlabeled objects other than the target. In light of this, we use the Sr3D dataset as a benchmark for this evaluation. Sr3D is built upon the ScanNet dataset and synthesizes template-based descriptions that define a relationship between a target and a single surrounding object (from ScanNet objects). Sr3D has the labels of the target object and its surrounding object.  \n\nTo evaluate the ability to detect objects other than the target, we take the model trained on the Nr3D dataset (based on pseudo labels) and evaluate its accuracy in localizing surrounding objects on the Sr3D test set. Given that our detector produces detection results for all mentioned objects for input descriptions, we take these detection results to compute the recall and mean IoU for localizing the surrounding objects. The evaluation results are listed in the table below. Although our model is not directly trained on the Sr3D dataset, it is able to detect surrounding objects with high recall and mean IoU. This demonstrates the effectiveness of pseudo labels for learning objects other than the target. \n|                                | Recall (IoU>0.25) | Mean IoU |\n|--------------------------------|:------:|:--------:|\n| Object Detections (all)        |  85.84 |   51.77  |\n| Object Detections (score>0.25) |  74.75 |   45.89  |\n| Object Detections (score>0.5)  |  71.29 |   43.92  |\n\nMoreover, for the dataset subsets where surrounding objects are successfully detected or not (denoted by \"Context Detected\" and \"No Context Detected\", respectively), we compute the accuracy of 3D visual grounding separately. As shown in the table below, the performance is significantly higher when surrounding objects are successfully detected. This illustrates the importance of detecting contextual objects other than the target object for 3D visual grounding.  \n|  Dataset Subsets    | Acc\\@0.25               |\n|---------------------|:-----------------------:|\n| Context Detected    |          50.31          |\n| No Context Detected |          35.46          |\n\n**(b) Evaluate whether the relationships among candidate objects are correctly modeled under weak supervision.**\n\n**A1 (b):** \nSimilarly, we use the model trained on the Nr3D dataset and evaluate its accuracy of relation modeling on the Sr3D dataset.  In our method, we perform weak supervision on the matching scores of the modeled object relations, with the aim of helping the network distinguish the target from other objects. To evaluate these modeled relations, we analyze their matching scores for identifying the target. \nSpecifically, we calculate the relation matching scores between all detected objects and the detected surrounding object. Then, we compute the top-K accuracy of target identification (which indicates whether the target object is among the top-K matched objects of the surrounding object). From the table below, we can see the modeled relations alone are effective in recognizing targets from other objects. \n|  Top-1 Acc |  Top-2 Acc |  Top-4 Acc|  Top-6 Acc|  Top-8 Acc |\n|:-----:|:-----:|:-----:|:-----:|:-----:|\n| 40.0% | 53.9% | 69.7% | 78.6% | 84.9% |\n\nWe further evaluate the final grounding accuracy on dataset subsets where the target object is matched by the top-K scoring relations of the detected surrounding objects (denoted by \"Top-K Subset\"). As illustrated in the table below, when the target is more accurately associated with  the modeled relations, the model also exhibits higher accuracy in 3D visual grounding. These results further demonstrate the importance of accurately modeling contextual relations for final target identification. \n|              | Acc\\@0.25 |\n|--------------|:--------:|\n| Top-1 Subset |   81.2   |\n| Top-2 Subset |   76.9   |\n| Top-4 Subset |   72.2   |\n| Top-6 Subset |   68.9   |\n| Top-8 Subset |   66.6   |\n\n**Q2: Lack of detailed methodological comparisons and discussions in the paper.**\n\n**A2:** Thanks for your advice. We will add a more detailed comparison and discussion in the revision. \nMost previous works typically match a large set of pre-prepared object proposals with the language description for target identification, without localizing the mentioned contextual objects related to the target, which is important as we analyzed above. These previous works largely rely on graph networks and transformer architectures to model the relationships between objects. For example, InstanceRefer considers k nearest neighbors for each candidate object to construct the object graph and perform feature fusion from local neighborhoods, which could be less flexible to infer relations such as \"farthest from\" in a language description. \n\nIn contrast, our method proposes to explicitly model contextual objects and relations, enabling a more reliable target reasoning. (1) The explicit pseudo labels help our network find the semantically related objects for a given description, allowing us to focus on these objects to capture contextual information. \n(2) Based on (1), we devise various spatial relation features for detected objects and perform weak supervision to model important contextual relations, which helps distinguish the target object. \nOur analysis above and the ablation studies in the paper illustrate the importance and efficacy of the proposed methods. We will add the above results and analyses to the revision. "}, {"review_id": "JCkk0ZbTsf", "reviewer": "Reviewer_oYfR", "summary": "This paper presents a transformer-based method for learning underlying spatial relationships in scenes for 3D visual grounding. \nSpecifically, three neural modules are proposed to tackle this challenge: 1) a language-guided object detector; 2) a relation matching network; 3) an attention-based target inference module.\nTo facilitate learning the contextual relationships, this work introduces a pseudo-label generation strategy and a weakly supervised training objective during training.\nExtensive experiments show the effectiveness of the proposed method, surpassing previous SOTA by a significant margin.", "questions": "Please see the Weaknesses section.", "limitations": "The limitations are properly discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper is well-written in English. The presentation is clear and very easy to follow. All necessary technical details for reproducing the method are provided in the paper and supplementary material.\n+ The method is valid and sound. Learning the spatial relationships is essential for 3D visual grounding, and it was handled well by the proposed network.\n+ The proposed method achieves SOTA results on established benchmarks, showing great improvements over previous methods.", "weaknesses": "I didn't find any significant technical flaws in the paper. But I do have some concerns that prevent me from giving a higher recommendation.\n\n1) The inputs to the relation matching module seem highly engineered, as detailed in the supplementary material. Although the auxiliary features are shown to be effective for the final visual grounding accuracy, I wonder if the authors can justify the impact brought by those features vs the proposed network itself. That is, if the same set of features was fed to a similar architecture, e.g. BUTD-DETR, would it achieve better performance than before?\n2) The object detector receives language inputs. How does it perform against other transformer-based 3D detector, e.g. 3DETR? If the language inputs were removed from the detector but fed into the later modules, how would the performance change?\n3) Regarding the experiments on ScanRefer, can the authors provide results on the online benchmark? The performance on the test set is also critical for validating the claim of achieving the SOTA performance.\n4) Can the authors provide more qualitative results on ScanRefer?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and the positive feedback on our paper. We appreciate your insights and concerns, and below are our responses to the points you raised.\n\n**Q1:  The impact of designed spatial relation features (inputs to the relation matching module).**\n\n**A1:** \nThe spatial relation features are devised to directly represent the commonly described relationships between objects. It is worth noting that these relation features are closely tied to our approach. We calculate these features for the objects detected by the prior text-guided detection network. If the contextual objects are not explicitly detected, it will be difficult to compute these relation features and demonstrate their full effectiveness. \n\nTo evaluate the isolated impact brought by these spatial features, we incorporate them into the BUTD-DETR. The table below illustrates that incorporating these relation features improves performance in BUTD-DETR. However, this improvement is less significant compared to what our method achieves. Our method learns to detect the contextual objects along with the target, allowing for better utilization of their spatial relation features for target grounding. \n| Method          |  Acc@0.25 | \n|----------------------|:---------:|\n| BUTD-DETR            |   43.3   |\n| BUTD-DETR (+relation feats.) | 45.59 |\n\n**Q2: Performance of the text-guided object detector.**\n\n**A2:** We compare the performance of our model's detection network with 3DETR on the ScanNet dataset. As shown in the table below, our detection network does not perform as well as 3DETR.  This could be because we employ a compact detection network with only 3 transformer decoder layers, which may not be advantageous for general object detection. To enhance the final grounding performance, it may be beneficial to focus on improving the performance of the detection network in the model.\n\n| Method          |  mAP@0.25 | \n|----------------------|:---------:|\n| 3DETR            |   61.1 |\n| The detection network in our model| 57.3|\n\nFor our method, we remove the language inputs from the detection network and evaluate its grounding performance on the Nr3D dataset. Removing the language inputs leads to significant performance degradation. Without the description information in the first detection stage, the model may not be able to detect related objects, thus limiting the follow-up grounding performance. This illustrates the importance of accurate object detection in the first stage. \n| Method          |  Acc@0.25 | \n|----------------------|:---------:|\n| CORE-3DVG            |   49.57 |\n| CORE-3DVG (w/o language for detection)| 20.68|\n\n**Q3: Evaluation on online ScanRefer benchmark.**\n\n**A3:** \nThank you for your advice. We have submitted our method to the online ScanRefer benchmark. **Our method, CORE-3DVG, achieves **the best performance** in overall Acc\\@0.25 (60.11%), surpassing the second-ranked method by a significant margin of 5.78 points.** Our performance in overall Acc\\@0.5 is nearly on par with the current leader, achieving 45.27% compared to their 45.45%. It's worth noting that our method employs a compact detection network with only 3 decoder layers, which may limit its advantages in evaluations requiring more precise localization. Nevertheless, we still achieve the best performance in both Acc\\@0.25 and Acc\\@0.5 on the more challenging \"Multiple\" subset. Our method also outperforms all published methods across all settings. These results further validate our claim of SOTA performance.\n\nThe table below compares our method with leading published methods. For more detailed comparisons, please visit the official ScanRefer benchmark website. We will update the paper with these results. \n| Method           |   Unique  |   Unique  |  Multiple |  Multiple |  Overall  |  Overall  |\n|------------------|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|                  |  Acc@0.25 |  Acc@0.5  |  Acc@0.25 |  Acc@0.5  |  Acc@0.25 |  Acc@0.5  |\n| ScanRefer        |    68.6   |    43.5   |    34.9   |    21.0   |    42.4   |    26.0   |\n| TGNN             |    68.3   |    58.9   |    33.1   |    25.3   |    41.0   |    32.8   |\n| InstanceRefer    |    77.8   |    66.7   |    34.6   |    26.9   |    44.3   |    35.8   |\n| D3Net            |    79.2   |    68.4   |    39.1   |    30.7   |    48.1   |    39.2   |\n| HAM              |    78.0   |    63.7   |    41.5   |    33.2   |    49.7   |    40.0   |\n| 3DVG-Transformer |    77.3   |    57.9   |    43.7   |    31.0   |    51.2   |    37.0   |\n| 3DJCG            |    76.8   |    60.6   |    43.9   |    31.2   |    51.3   |    37.8   |\n|**CORE-3DVG (ours)**| **85.57** | **68.67** | **52.75** | **38.50** | **60.11** | **45.27** |\n\n \n**Q4: More qualitative results on ScanRefer.**\n\n**A4:**\nWe have included qualitative examples on ScanRefer in the attached PDF of the global author rebuttal. These examples help to provide a better understanding of how our method performs in various scenarios. Additionally, we compare our method with BUTD-DETR to highlight the advantages of leveraging contextual objects and relations for target localization. "}, {"review_id": "W6Yq1hJuoN", "reviewer": "Reviewer_VN72", "summary": "The paper proposes to supervise the detection of contextual objects and contextual relations in a given language sentence to help improve language grounding performance. Specifically, the authors devise three modules: a) Language guided detection module is responsible for detecting all mentioned objects. Since the supervision for contextual objects is not readily available, this paper proposes a pseudo-label generation strategy to supervise them. b) Relation Matching Network which is responsible for enhancing relations between the target and contextual objects, with supervision derived from another pseudo labelling strategy proposed in the paper. c) Target Inference Network which is responsible for utilizing the detection queries from module (a) and relationship matrix from (b) to produce the final grounding output. Experiments show superior performance over SOTA baselines with significant margins. Ablations validate the claims made in the paper.", "questions": "My main questions are regarding the details of the proposed architecture as mentioned in the limitations section. I think addressing how the authors could improve the clarity of the paper would be the most important for improving my ratings of the paper. The proposed model performs very well and I think the results would be valuable to the community, hence I want to recommend acceptance. However, the paper  lacks clarity is very hard to read which can be significantly improved.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- SOTA performance by a significant margin\n- Nice and thorough ablations, clearly showing what matters and what doesn’t", "weaknesses": "I read this paper multiple times and yet I do not understand it clearly. I think the paper writing can be improved to aid understanding. Below are some of my confusions:\n- About Text-guided detection module:\n    - My understanding is that the authors want to supervise the model with detection boxes for both target and contextual objects. Grounding datasets do not label contextual objects and hence the authors propose to use pseudo-label generation for the contextual objects. For that, they train the model to detect “noun phrases”. For eg: “The chair next to the table, closest to the door”, the model will be given a text utterance of “the chair” and will be asked to predict boxes of “the chair”. \n        - Q1: do you use chair boxes of all chairs in the scene or the ground truth “chair” mentioned by the utterance “the chair next to ….. door”. If latter, wouldn’t it induce ambiguity in training because the model will be penalized for predicting extra chairs?\n        - Q2: Instead of using pseudo labels, why not use the ground truth of the noun classes for the contextual objects as well. For example, in the above utterance, supervise the model with “all tables”, and “all doors” for contextual objects. This supervision is readily available in all referential datasets including Referit3D/ScanRefer via Scannet annotations.  Why pseudo-labels are better since anyway the model is supervised with predictions from “the table” supplied to detection model which will, in best case, produce all tables already available in detection setup. Is there any quantitative comparison for this?\n        - Q3: How is the training exactly performed? Do you train first for some epochs to make the model train on noun phrases and then start using pseudo labels? And to get the pseudo labels, for each forward pass, you would need to run several passes through the model to get pseudo labels for each noun phrase?\n        - Also, I think the idea of training on detection phrases/noun chunks is proposed by GLIP [1] and BUTD-DETR [34]. The idea of improving performance by training on intermediate/contextual objects is proposed in MDETR and BUTD-DETR [34]. It might be good to explicitly state that.  I think the novelty here is how to get the labels for intermediate objects for datasets like NR3D/ScanRefer where such a supervision does not exist.\n\t[1]: Li, Liunian Harold, et al. \"Grounded language-image pre-training. 2022 IEEE.\" CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021.\n[2]: Kamath, Aishwarya, et al. \"Mdetr-modulated detection for end-to-end multi-modal understanding.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n<start> Minor Comments\n- About Relation matching Network:\n    - I think Section-3.3 could use a figure to make it more clear to understand. \n- Target Inference Network:\n    - The name “inference” is slightly misleading as it gives an impression that this is only used at test time and is not involved in training\n    - I think Figure-2 does not indicate  any connection going from section-b to section-c i.e. how (and if) relation matching network’s output is used in target inference network.\n- Overall, a pseudo code on how everything connects could aid the clarity tremendously I think. \n\n<end> Minor Comments\n\n- The dataset on which ablations are conducted is missing (which can be inferred to be NR3D from the numbers, but would be nicer to mention). Additionally, in my initial readings, I was confused why the final row in Ablation Table-4 gets 44.42% instead of the reported NR3D number of 49.57 (which later I realized is because Table-4 is concerned with ablation the jump from 34.66 -> 44.42 in Table-3).\n\n- The related works section is not thorough and well-written. Some suggestions should be to discuss works related to supervising contextual objects and enhancing focus on contextual relations from both 2D and 3D community; identifying gaps in the literature and properly motivating how the proposed work is different.\n\n- Minor Writing/Formatting errors:\n    - Implementation details section is duplicated in main paper as well as supplementary\n    - “References” heading duplicated", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your comprehensive feedback on our paper. Your questions and suggestions have been carefully addressed to improve the clarity and quality of the paper. Please find our responses below. \n\n**About Text-guided detection module:**\n \n**A1:** \nWe supervise the model with the ground truth box of the \"chair\" mentioned by the utterance \"the chair next to… door\". We understand the concern regarding the model being penalized for predicting extra chairs. In our experiments, we empirically found that even with this supervision, the model is still capable of detecting multiple chairs in the scene, without overfitting to predict only one chair. This could be due to the difficulty of training the model to exactly match the object phrase \"the chair\" with just one labeled chair.  Additionally, we employ a smaller loss weighting hyper-parameter to alleviate this penalty during training.  \n\n**A2:**\nGiven the free-form language descriptions, the extracted noun phrases are not strictly the same as the standard noun classes. The extracted noun phrases might use different words to describe the object category, and they can encompass descriptive terms, including attributes and other modifiers.  As a result, it is not straightforward to associate labeled objects of specific categories with the noun phrase extracted from language descriptions. In our method, the extracted noun phrases of target objects naturally form a diverse set of object phrases with corresponding labeled objects. We train the network using this data to detect objects for various object phrases, which can be directly used to generate pseudo-labels for other objects mentioned in language descriptions.\n\n**A3:**\nSorry for the ambiguities here. We do not devise a complex training process, and we start to use  pseudo labels from the beginning of model training. Considering the pseudo labels may not be initially accurate, we have devised a learning weight based on the score of each pseudo label (as shown in Eq. 5), which mitigates the impact of learning unreliable pseudo labels. To obtain the pseudo labels, we form a batch of extracted noun phrases and input them into the text-guided detection network for an extra forward pass. We will revise the related part to improve clarity. \n\n**A4:**\nWe acknowledge the contributions of GLIP [1] and BUTD-DETR [34] in training on detection phrases/noun chunks, as well as the idea of training on intermediate/contextual objects as presented in MDETR and BUTD-DETR [34]. In the revised paper, we will explicitly mention these works. The novelty of our approach, as you rightly pointed out, involves the methodology we employ to obtain pseudo labels for intermediate objects. This is particularly important in datasets like NR3D/ScanRefer where such explicit supervision is absent. We will make this distinction more explicit in the revised manuscript to give due credit to prior works and clearly highlight our unique contributions. \n\n**Minor and Other Comments**\n\n- **Relation Matching Network:** We will incorporate a new figure in Section 3.3 to visually represent the processes involved, making it more intuitive.\n- **Target Inference Network:** \nThanks for your advice and we will rename the module to avoid confusion. We will update Figure-2 to show how the output of the relation matching network is fed into the target inference network, to offer a clearer understanding of the flow.\n- We will add a pseudo code in Section 3.4 to provide a clearer overview of how the components are connected and their interactions.\n- In the experiment section, we will explicitly mention the dataset used for conducting the ablation study, and provide clearer illustrations of the experimental settings for the tables. \n- Thanks for your suggestions. We will improve the related work section as suggested. This should provide readers with a clearer picture of where our work fits into the existing literature.\n- Thank you for pointing these out. We will fix these errors.  Additionally, we will thoroughly review the paper for any other formatting or typographical errors.\n\n**Questions**\n\nThank you for your constructive feedback and for recognizing the potential value of our work to the community. We acknowledge the concerns raised about the clarity of the paper, especially regarding the architecture details. We will revise the paper to provide a clearer description of the architecture and improve overall readability. We appreciate your recommendation and will ensure that the necessary revisions are made to enhance the paper's clarity."}]

Author rebuttal:
We sincerely thank all reviewers for their constructive feedback and valuable insights. We have carefully addressed each concern raised and will incorporate the suggested changes to improve the clarity and quality of our paper.

In response to the concerns raised, we have made more comprehensive comparisons to demonstrate the superiority of our method. Additionally, we have provided detailed explanations of our methodology. We have also enhanced our analysis regarding pseudo labels, and conducted a thorough evaluation of our method's performance under different scenarios. Furthermore, we submitted our method to the online ScanRefer benchmark, where it achieved state-of-the-art performance, further validating our claims. 

We believe that these revisions will significantly enhance the paper's clarity and its potential value to the community. We appreciate the opportunity to clarify our contributions and thank the reviewers for their time and effort in reviewing our work.

Meta-review: The paper proposes a new model for 3D visual grounding which explicitly learns to detect context objects and relations. Reviewers appreciated the novelty of the idea, good experimental results over state of the arts, and thorough ablations provided in the rebuttal. Three out of five reviewers gave the weak accept scores after the rebuttal. The remaining two reviewers did not reply to the authors, but the AC considered the authors have adequately addressed their concerns in the rebuttal including the in-depth analysis on the quantitative results and pseudo labels. Therefore, the AC recommends accepting the paper. The authors should revise the paper according to reviewers’ comments in their final version.
Decision: Accept (poster)