RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation
Authors: Changli Wu
Qi Chen
Jiayi Ji
Haowei Wang
Yiwei Ma
You Huang
Gen Luo
Hao Fei
Xiaoshuai Sun
Rongrong Ji
Venue: NeurIPS 2024 oral (2024)
Keywords: 3D Referring Expression Segmentation
Spatial Awareness Modeling
Rule-guided Supervision
Primary area: human-AI_interaction

Abstract: 3D Referring Expression Segmentation (3D-RES) aims to segment 3D objects by correlating referring expressions with point clouds. However, traditional approaches frequently encounter issues like over-segmentation or mis-segmentation, due to insufficient emphasis on spatial information of instances. In this paper, we introduce a Rule-Guided Spatial Awareness Network (RG-SAN) by utilizing solely the spatial information of the target instance for supervision. This approach enables the network to accurately depict the spatial relationships among all entities described in the text, thus enhancing the reasoning capabilities. The RG-SAN consists of the Text-driven Localization Module (TLM) and the Rule-guided Weak Supervision (RWS) strategy. The TLM initially locates all mentioned instances and iteratively refines their positional information. The RWS strategy, acknowledging that only target objects have supervised positional information, employs dependency tree rules to precisely guide the core instance’s positioning. Extensive testing on the ScanRefer benchmark has shown that RG-SAN not only establishes new performance benchmarks, with an mIoU increase of 5.1 points, but also exhibits significant improvements in robustness when processing descriptions with spatial ambiguity. All codes are available at https://github.com/sosppxo/RG-SAN.

RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation
Changli Wul.2, Qi Chen 1", Jiayi Ji Haowei Wang", Yiwei Ma You Huang', Gen Luo', Hao Fei', Xiaoshuai Sun' Rongrong Ji Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University. 361005, P.R. China Shanghai Innovation Institute, Shanghai, P.R. China
3 Youtu Lab, Tencent, Shanghai, P.R. China National University of Singapore
Abstract
3D Referring Expression Segmentation (3D-RES) aims to segment 3D objects by correlating referring expressions with point clouds. However, traditional ap- proaches frequently encounter issues like over-segmentation or mis-segmentation, due to insufficient emphasis on spatial information of instances. In this paper, we introduce Rule-Guided Spatial Awareness Network (RG-SAN) by utilizing solely the spatial information of the target instance for supervision. This approach enables the network to accurately depict the spatial relationships among all entities described in the text, thus enhancing the reasoning capabilities. The RG-SAN consists of the Text-driven Localization Module (TLM) and the Rule-guided Weak Supervision (RWS) strategy. The TLM initially locates all mentioned instances and iteratively refines their positional information. The RWS strategy, acknowledging that only target objects have supervised positional information, employs depen- dency tree rules to precisely guide the core instance's 's positioning. Extensive testing the ScanRefer benchmark has shown that RG-SAN not only establishes new performance benchmarks, with an mloU increase of 5.1 points, but also exhibits significant improvements in robustness when processing descriptions with spatial ambiguity. All codes are available at https: / gighub...... com/ sosppxo/RG---pp SAN
1
Introduction
3D Referring Expression Segmentation (3D-RES) is an emerging field that segments 3D objects in
point cloud scenes based given referring expressions [24]. Gaining significant attention for its
applications in autonomous robotics, human--mchhin interaction, and self-driving systems, 3D-RES demands deeper understanding than 3D Referring Expression Comprehension (3D-REC) (5) 73 I
75 70), which focuses only on locating the referring objects via bounding boxes. 3D-RES, on the other hand, requires identifying instances and providing precise 3D masks.
Early 3D-RES approaches [24] 73 adopted a two-stage paradigm, starting with an independent text- agnostic segmentation model for generating instance proposals, followed by linking these proposals with textual descriptions. This paradigm, separating segmentation and matching, proved suboptimal in performance and efficiency. Recent explorations have shifted towards end-to-end paradigm. For instance, 3D-STMN (5) achieved efficient segmentation by directly matching superpoints with text, while 3DRefTR 4 integrated 3D-RES and 3D-REC into unified framework using multi-task
'Equal Contribution.
t CCoreesponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

approach, boosting inference in both tasks. Despite these advancements, limitations persist, primarily due to over-reliance on textual reasoning and insufficient modeling of spatial relationships between instances. For example, as shown in Fig. [ without spatial modeling, it's challenging to understand and correctly segment the intended chair scenarios involving complex spatial terms like "far away".
To tackle this issue, the core is to assist textual reasoning by modeling the spatial relationships of core instances. By effectively identifying these spatial relationships within expressions, a substantial improvement can be achieved in comprehending spatial arrangements. Neverthe- less, this endeavor is not without its challenges. While accurate positional information crucial for ensuring precise modeling of spatial relation- ships, accurately regressing instance positions from textual information is far from simple task. Furthermore, our available positional in- formation is limited to the target instance, leav- ing us without supervisory signals for other in- stances referenced in the expression.
(81.
& (75.8016)
(98.20.10) 18)
& (75.9018
(80.60.13)
There is chair near the desk. It away from the sofa.
To overcome these challenges, we propose the novel Rule-Guided Spatial Awareness Network (RG-SAN), utilizing the spatial information of the target instance for supervision. This enables the network to accurately depict spatial relation- ships among all text-described entities, thereby significantly enhancing the model's inference and pointing capabilities. RG-SAN consists of two main components: the Text-driven Localiza- tion Module (TLM) and the Rule-guided Weak Supervision (RWS) strategy. TLM initially lo- cates all mentioned instances and iteratively refines their positions, ensuring continuous improvement in location accuracy. RWS, leveraging dependency tree rules, precisely guides the positioning of core instances. This focused supervision significantly improves the handling of spatial ambiguities in referring expressions. Extensive testing on the ScanRefer benchmark shows that RG-SAN not only sets new performance standards, with a mloU increase of 5.1 points, but also greatly enhances robustness in processing spatially ambiguous descriptions.
Figure I: Illustration with a target object and mul- tiple auxiliary objects, associated with referring expression. The target marked in green represents the main referred instance, while targets in other colors indicate other mentioned entities. This vi- sual highlights the challenge of effectively com- pleting semantic reasoning in the absence of spatial inference.
To sum up, main contributions follows:
We introduce RG-SAN, a novel approach for modeling spatial relationships among all entities in expressions, which enhances the model's referring ability in 3D-RES.
We propose the TLM for precise localization of all instances mentioned in expressions, and RWS, utilizing only the target instance's location for supervising the spatial positioning of all instances.
Extensive experiments on the ScanRefer benchmark demonstrate the effectiveness of the proposed RG-SAN, showing significant improvements in performance and robustness in 3D-RES tasks.
2 Related Work
2.1 3D Referring Expression Comprehension and Referring Expression Segmentation
Referring Expression Comprehension (REC) is proposed to locate the referred target from short
description visual space by bounding boxes 774
59
29], which is part of vision-language tasks [12
10
18
69
68
15
40
41 Recent works in 3D-REC can be divided into two parts, two-stage and
single-stage. As for two-stage methods 5 1 75 73 72 13, 3D object proposals are generated
directly from ground-truth [ or extracted by pre-trained 3D object detector [52] in the first stage,
2

---

and then assigned to language in the second stage. In the other way, some methods adopt one-stage paradigm [47 26 70 66]. enabling end-to-end training.
Referring Expression Segmentation (RES) need fine-grained vision- laniuag alignment [36 37 16 351, proposed to locate the referred target by masks [27 61 25
TGNN [24] introduce 3D-RES by extending the bounding box annotations of ScanRefer [S] to masks by incorporating the instance masks from ScanNet and proposed two-stage pipeline. Further, 3D-STMN [65] proposed an end-to-end method that matches the text and superpoints to get the 3D segmentation of the target object directly.
2.2 3D Human-Al Interaction
ScanQA 3 has notably advanced visual question answering in 3D scenes, enhancing the human-Al in-
teraction experience. Meanwhile, 3D-LLM [2T] 3D-VisTA 777. NaviL.MM [76., and BridgeQA (51)
have further propelled this task. Li et al. 38 (3). Lu et al. 46 have explored how Al understands
human instructions like gestures and language to locate targets. 3D-VisTA [77] introduced new paradigm for large-scale 3D vision- lasion-language pre-training, greatly enhancing Al's understanding of 3D vision-language and advancing various downstream tasks. Works like 3D-LLM (21), Chat3D [64] 22). NaviLLM [76] and Scene-LLM [4] have extended the capabilities of multimodal large language models to the 3D realm, endowing embodied intelligence with the rich knowledge and capabilities of LLMs, thus ushering in the era of large models in Human-Al Interaction.
2.3 Weakly Supervision in Vision-and-I Langnaagd-Language Language
In the field of Vision Language, weakly supervised S3 34 4 have gained significant attention and great progress. These approaches aim to tackle the challenge of limited or incomplete annotations by leveraging alternative supervised data or weakly labeled data. For weakly supervised visual question answering (VQA), Kervadec et al. (28) employ weak supervision in the form of object-word alignment as a pre-training task. Trott et al. [62] use object counts in images as weak supervision to guide VQA for counting based questions. Gokhale et [I7 employ logical connective rules to augment training datasets for yes-no questions. Weakly supervision from captions has also been recently. Especially, for RES, some methods [3 localize the target object only using readily available image-text pairs.
employed for visual grounding tasks
[9] 50 2
3 Method
In this section, we provide a comprehensive overview of the RG-SAN. The framework is illustrated First, the features of visual and linguistic modalities are extracted parallel (.Tc.3.T). Next, we demonstrate the process of TLMM (Sec. 3.2.T.. Finally, we outline the RWS and the training objectives (Sec. 3.3
in Fig. [
3.1 Feature Extraction
3.1.1
Visual Encoding
Given a point cloud scene Poloud E (3++7) with N₂ points. Each point comes with 3D coordinates along with F-dimensional auxiliary feature that includes RGB, normal vectors, among others. We first employ a Sparse 3D U-Net 19] to extract point-wise features, represented as P Peloud E RA,xC,, Then, we follow Sun et al. [60] and Wu et al. (65] to obtain N, superpoints (IC, M
32 from the original point cloud. Finally, we directly feed point-wise features P. Peloud into
Nr to obtain the superpoint-level features Sp E NN,,C, superpoint pooling layer based on (K,
3.1.2 Linguistic Encoding
Given a free-form plain text description of the target object, consisting of Nr words (c:).N we utilize a pre-trained MPNet model (58) to extract dimeinssion word-level embeddings, represented as Eg E N.C.
3

---

Pruttion
Superisiion
Extraction
Pelatier Posttion Eaceding
Absolute Fostlon Encoding
Addition
Murtppcation
Text-driven Localization Module
Refinemen
Prnx
Self Self-Attention nttinn
Cross-Attenn Attintiion
Test-driven Inttializaion
Rule-guided Weak Supervision
Pe
G1 Pestinn
There is large, gray, bookshelf
Li Lons
corner of the up It small couch vitting
ME:
Toraei
DICE
Depeatencyy Farding
Hulegalded Target
Figure 2: An overview the proposed RG-SAN, This model analyzes a point cloud and a textual description with Nr tokens, extracting superpoints and word-level features. The TLM assigns spatial positions to tokens, facilitating multimodal fusion. The RWS strategy enables model to learn positions of all mentioned entities using only the supervision of the target position.
3.2 Context-driven Spatial Awareness
In this section, we address key limitation in prior works that interact point clouds with text without considering spatial positioning [65] 47 70). Unlike these methods, which often lose spatial informa- tion due to unordered point cloud features, leading to ambiguous spatial relationship understanding, our approach is distinct. In 3D-RES, spatial information inherently sparse and dynamic, depending on the specific target object described in the text, rather than the dense, static sampling of an entire point cloud scene [31]
To address this issue, we propose to facilitate interactions between textual entities and point clouds within 3D space, rather than merely at the semantic level. Specifically, objective to fully leverage semantic and spatial contextual information to accurately predict the spatial positions of all mentioned nouns within the point cloud.
Therefore, we introduce the Text-driven Localization Module (TLM) to initialize the positions of entity nouns in the text and continuously update and refine these positions through iterative multimodal interactions.
3.2.1 Text-driven Localization Module
Given the superpoint features Sp and word embeddings Eg, we first project the features into the same dimension, and enhance the word-level embeddings by Dependency-Drin -rivenn Interaction (DDI), following Wu et al. [65]
Ep DD W lang); $= W vies
(1)
where Wang C and W oss E CDD denote learnable parameters, and the subscript of E
and É represents the round number.
Text-driven Initialization. The key is to map the text into 3D geometric space in meaningful way. Specifically, we enhance entity position prediction within point clouds through an interactive text-point cloud process. We do this by calculating feature similarity across modalities to accurately

---

estimate the spatial probability distribution for each mentioned entity:
E Eo WE, E,
SWSWs s.
Ag Sim Sz)
21=1 Sim Sim Sj)
where Eg denotes the initial word embeddings. S denotes the superpoint features, WE, Ws E DDD are learnable parameters, Ary E IR denotes the probability of the i-th word token being located at the j-th superpoint, and Sim(-,- represents the similarity function, which in this case is defined as Sim (E, S) eepppxs /VD).
Following this, we utilize the spatial probability distribution A to predict the approximate positions of the mentioned entities, well as their corresponding representations:
P
0
Aig Pj.
j=1
N1
S. = SW
Eo
A.yS..
j=1
where Pj is the position of the j-th superpoint, P is the initial spatial position of i-th word token which will be refined iteratively as formulated in Sec. 3.2.2 W. E R denotes learnable parameters, and Eo,; denotes the updated representation of the i-th word token. The sharing of distribution A during centroid computation allows the entity representations benefit from the guidance provided by spatial information, leading to more accurate understanding of the 3D spatial relationships. Subsequently, the text and point clouds undergo multiple rounds of multimodal interactions, continually updating the embeddings and positions of the entities.
Iterative Position Refinement. After l-round multimodal interactions, the word tokens E,, referred to as textual segment kernels, become increasingly precise, theoretically resulting in more accurate position predictions. A straightforward approach would involve replicating the initial interaction method by regressing position information in each round. However, following the methodologies of Redmon et al. (56) and Lai et al. (31) rather than directly optimizing the final position, we adopt more manageable strategy of iteratively learning offsets. To this end, we refine the positions of textual tokens based on the evolving textual segment kernels. As depicted in Fig. 2 we employ a Multilayer Perceptron (MLP) to predict a position offset AP; MLP(ÉE+2) e Ntx3 from the
updated textual segment kernels Ê This offset is then added to the previous textual positions Pf:
Pf [+1
P APP.
This method allows for gradual refinement of position predictionn, making the optimization process more effective and leading to progressively more accurate positioning with each iteration.
3.2.2 Spatial Awareness Aggregation
Once the positions of noun entities are obtained, techniques like positional encoding (63 can be used to further refine the positions.
67
3
30
Absolute Positional Encoding (APE). To initiate, we follow the approach of the original trans- former 63 to encoded the positions of both superpoints and text tokens to obtain positional encodings B E R M,xP xD and B V D using absolute positional encoding (APE):
B APE(P?). B APE(P().
(6)
These positional encodings facilitate spatial-award self-attention in the textural segment kernels É,;
E = Attention E, B É, + B Ex),
where Attention(-) uses the technique of Vaswani et al. 63 and B denotes the absolute positional encoding of Eg.
5

---

Next, we enhance textual and superpoint features with absolute positional encoding, and use them as Queries and Keys for subsequent multimodal aggregation:
Concat Bi) querys
(8)
K Concar (S,B;) B;) keys
where B E R.XD B E RR D denote the absolute positional encoding of segmentation kernels and superpoints, respectively, and W Wquery. Wkey E R2Dx2D denote learnable parameters.
Relative positional encoding (RPE). For the further interaction with superpoint features, adopt
well-established relative positional encoding techniques [67 31 30 6), such as Table-based RPE (67
31 and 5D Euclidean RPE [6. which are formalized as follows:
[] RPE Q + K()),
where Bi 5l E R denotes the relative positional bias of the i-th Q relative to the j-th K, RPE(-) denotes the operation of relative positional bias and [-] denotes the indexing operation.
Thus, we can perform multimodal aggregation enhanced with relative positional encoding:
Q 'QK"" VD
K.K" + +B²) (SW val):
E = softmax
(10)
where W vot E Dx denote learnable parameters, B; E NR. denotes the relative positional bias, and E denotes the updated segmentation kernels. This methodology significantly enriches the interaction between linguistic and 3D visual data, enabling more nuanced spatial understanding in our model.
3.3 Rule-guided Weak Supervision
3.3.1 Rule-guided Target Selection
In the preceding sections, initially predicted the locations of all entities mentioned in the text. Ideally, supervised training would require position labels for each entity. However, we only have access to the location information of the target instance. This constraint leads us to adopt a weak supervision approach, focusing solely on the position of the referring instance for training. This approach introduces significant challenge: accurately identifying the referring instance among the mentioned nouns. To address this, we utilize a pre-processed dependency tree, as outlined in Manning et [88] to accurately pinpoint the core noun, typically the subject of the sentence. We have developed set of manual rules, based on this more general dependency tree, to enhance the identification process. These rules are specifically designed to guide the accurate positioning of core instances. The implementation of these rules is outlined in Algorithm
Algorithm Rule- guilegguied Target Selection
The dependency tree G (V,E) of the textual description, where V = [token} denotes the set of nodes, & = { (relation, head, tail)} denotes the set of relations between nodes. Output: The index of Target Instance node ygt
Input:
I: Initialization to root: =
2: find E, with Vi as its head
3: if (E, (nsubj. compound)) (V, & {which, that)) then
4: i E,'s tail index
5: end if
6: V; (theree, this, it, object) then
7: find &, with V; as its head
8: i E,'s tail index
9: end if
V; {sset sets, color, shape) then
11: find the first E,'s relation E (compound, nmod, dep)
it- E,'s head index
13: end if

---

3.3.2 Training Objectives
Given the index of the target instance, we can directly obtain the corresponding segment kernel E E D and position P|",, which are then supervised by the target ground truth.
Then we perform matrix multiplication between E and S to get the predicted instance response maps, which can be formulated as
M = a (B44 -s")
(11)
Mask = M > 0.5,
(12)
where M4+1 E RRV,,, Mask E {0,1)% are the predicted response map and the instance mask corresponding to the target.
Given ground-truth binary mask of the referring expression Y E {0, 1, we get the corresponding superpoint mask Y (0, 1I by superpoint pooling follewed by 0.5-threshold binarization, and then we apply the binary cross-entropy (BCE) loss on the final response map M4+1 following Sun al. [60]. The operation can be written as:
Y L(o(AvvPool (C;))),
(13)
Chee BCE(M Y*),
(14)
where AvgPool(.) denotes the superpoint average pooling operation, and Y denotes the binarized mask value of the i-th superpoint KG. I(.) indicates whether the mask value is higher than 50%.
To tackle foreground-background sample imbalance, we can use Dice loss 49
d DICE(M Y*).
(15)
To supervise the position Plot 141 the center of the superpoints of the target instance Pat
Epos
LI(P 1+1 Pot).
(16)
In addition, we add simple auxiliary score loss Liccore for mask quality prediction following Sun al. [60].
Overall, the final training loss function L can be formulated as:
AbreLree Adiceoaiion + X Xpos
(17)
Ascore Lscores
where Abees Adioos Apet and Ascore. are hyperparameters used to balance these four losses.
4 Expriment
4.1 Experiment Settings
In our experiment, we utilize the pre-trained Sparse 3D U-Net method extract point-wise features from point clouds [60]. We also employ the pre-trained MPNet model (58) as our text encoder. For the rest of the network, training is conducted from scratch. We set an initial learning rate of 0.0001 and apply learning rate decay epochs 26, 34, and 46, each with decay rate of 0.5. Our experiments use default of multiple rounds L. batch size of 32, and a maximum sentence length of 80. We set Abce Adice 1. A.po Ascore 0.5. All experiments on a single NVIDIA Tesla A.00 GPU, ensuring consistency in our computational process.
conducted using Py PyTrch
Dataset and Evaluation Metrics
We evaluate our method using the ScanRefer dataset, recent 3D referring dataset [ 24), comprising 51,583 English natural language expressions referring to 11,046 objects across 800 ScanNet scenes [ Following Chen et al. 5 our evaluation metrics include mean Intersection over Union (mloU) and Acc@kloU) "Unique" refers to cases where the target instance is the only one of class, and "Multiple" indicates situations where there is at least one more object of the target's 's class.
7

---

Table 1: The 3D-RES results ScanRefer. The mloU and accuracy reevaluated machine. "We reproduce results by extracting points within the boxes as segmentation mask predictions using their official codes.
Unique (~-19%)
Multiple (~-81%)
Overall
Inference Time
Method
0.25
0.5
mfoU
0.25
0.5
mloU
0.25
0.5
miot
Stage-
Stage-2
All
Multi-task
EDA-box2mask
70
84.7
56.9
50.0
37.0
55.2
40.0
35.0
3DRefTR-SP
87.9
69.8
51.6
41.9
57.0
46.1
40.8
388ms 405ms
3DRefTR-HR
43
89.6
77.0
52.3
43,7
57.9
48.7
41.2
Un
29.6
SegPoint
20
41.7
Reason3D
23
88.4
84.2
74.6
34.1
$7.9
41.9
42.0
Single-task task
TGNN
X3
37.5
31.4
27.8
TGNNF
69.3
57.8
50.7
31.2
26.6
23.6
38.6
32.7
28.8
26862ms
235ms
27097mss
InstanceRefert
08
81.6
72.2
60.4
29.4
23.5
21.5
40.2
33.5
30.6
509ms
672ms
118
X-RefSeg3D
40.3
33.8
29.9
JDVG-Transforung
75
79.5
58.0
49.9
42.0
30.8
27.0
49.3
36.1
31.4
3D-SPS'
84.8
65.6
54.7
41.7
26.7
50.1
37.6
32.1
3DRESTR
43
79.0
54.2
40.2
22.1
46.0
26.9
28.7
STMN
89.3
84.0
74.5
46.2
29.2
31.1
54.6
39.8
39.5
2833ms
RG-SAN (Ours)
89.2
84.3
74.5
55.0
35.4
37.4
61.7
44.9
44.6
295ms
4.3 Quantitative Comparison
In our experiments on the ScanRefer dataset, our proposed RG-SAN demonstrates significant improvements in nearly all metrics the single-task leaderboard, as shown in Tab. T Notably, RG-SAN shows substantial gains compared to the state-of-the of-tth-rrttt single-task model 3D-STMN, with increases of 5.1 points in mloU and 7.1 points in Acc @oc@0r 0.25. This highlights capability. A more detailed examination reveals that the majority of these improvements occur in scenarios with multiple disruptive instances, where RG-SAN achieves a remarkable 6.3-point increase mloU. This setting, where the target instance is among other instances of the same type, demands discriminative reasoning from the model. The significant performance validates the enhanced referring capabilities empowered by spatial reasoning. Our proposed RG-SAN also outperforms multi-task models 770 43 including LLM-based models (0 23], in most 3D-RES metrics, despite those models benefiting from more annotated data.
model's inferencing
Moreover, RG-SAN has competitive inference costs, being only 12ms slower than the efficient 3D-STMN and faster than all other compared models, demonstrating its high performance with minimal computational increase.
4.4 Ablation Study
4.4.1
Text-driven Localization Module
We conduct an ablation study on the Text-driven Localization Module (TLM), as illustrated in Tab. 2 Simultaneously, we perform fine-grained analysis of various initialization schemes for embeddings and positions. The term "w/o TLM" denotes the approach of not modeling positional information and instead directly using text embeddings for interaction. "MAFT" refers the direct adaptation of the method proposed in (31] The "Project" method involves initializing embeddings based on text-driven embeddings and then projecting each textual token directly into 3D position, while the "Random" method randomly assigns a position to each textual token. Finally, we utilize the initialization technique called Text-driven Initialization (TI), which simultancously initializes both embeddings and positions in text-driven manner. Tab. clearly shows that, under identical conditions, TI outperforms the others in all metrics. This indicates that TI more effectively leverages positional information from the visual scene, leading to more precise initial positions for the textual tokens. Consequently, this reduces the complexity of the subsequent iterative refinement process, thereby enhancing the overall accuracy of our model in spatially aligning text with point cloud data. Additionally, Tab. demonstrates that proper initialization leads to superior performance of TLM compared to the methods without TLM.

---

Table 2: Ablation study of Text-driven Localiza- tion Module (TLM), where "w/o TLM" means not using TLM.
Table 3: Ablation study of positional encoding. where "w/o Pos. Supervision" means not super- vising the positions, and "w/o PE" means not using any positional encoding.
Method
Init. of Embeddings
Init. of Positions
Multiple miot'
Overall mloU
positional encoding
Multiple
w/o TLM
Text-driven
32.5
40.3
0.25
0.5
mloU
0.25
0.5
mloU
MAPT 31
Zero
Random
29.7
37.9
wio Pos. Supervision
45.4
27.3
30.4
54.4
38.2
38,9
Project Random
Text-driven Text-driven
Project
30.3
38.8
w PE
46.1
31.2
32.8
54.6
41.1
Random
30.1
38.8
46.0
40.7
Text-driven
Text-drivca
34.1
42.3
5D Eaclidean RPE
46.7
32.5
33.3
54.6
43.9
41.7
Table-hased RPF
47.2
33.7
34.1
55.6
43.9
42.3
4.4.2 Positional Encoding
We compare various positional encoding methods previously employed in (57 6 These methods include Fourier Absolute positional encoding (APE), 5D Fuclidean Relative positional encoding (5D Euclidean RPE) 6 and Table-based Relative positional encoding Ffable-base RPE) [31]. Tab. reveals that Table-based RPE surpasses the other methods, suggesting that combining semantic information with relative relationships is advantageous. Additionally, we observe that employing only absolute positional encoding can result in lower performance than not using any positional encoding at all. This may be attributed to the inherent limitations of absolute positional encoding in capturing relative positional information. By complicating the semantic features, it introduces challenges in the model's s training process, underscoring the importance of choosing the right positional encoding technique for effective performance.
4.4.3 Rule-guided Weak Supervision
We conducted experiments employing various weakly supervised text kernel selection strategies to evaluate their efficacy in leveraging target annotations. The strategy labeled as "w/o RWS" involves selecting the token based on attention weight within the cross-attention module [65], while "Root" entails selecting the root token of the dependency tree. Table 4 illustrates that utilizing the root node as supervision slightly outperforms the "w/o RWS" baseline. This is likely due to the root node providing consistent supervision, whereas Topl tends select different nodes variably, which complicates the training process. In contrast, our Rule-guided Target Selection (RTS) strategy, based on dependency tree rules locate subjects, aligns more effectively with the structural nature of the text. It precisely identifies the target entity's 'st' position, significantly enhancing annotation utilization and effectively directing model training. This leads to notable improvement in model performance.
Furthermore, we conduct an ablation study on the impact of the position loss weight Lpos. detailed in Tab. 5 We observe that increasing the weight generally improves performance, peaking weight of 0.3, beyond which performance begins to taper off. This finding highlights the importance of balancing the weight of the position loss optimize the model's effectiveness.
4.4.4 Comparison with MAFT
MAFT 31 has played a pivotal role in 3D instance segmentation by incorporating spatial position modeling, offering valuable insights into how spatial information can improve model performance. Inspired by this approach, we extend spatial information into the text space to better align visual and textual semantics, specifically targeting spatial relationship reasoning 3D-RES. Our approach introduces two key innovations that distinguish it from MAFT:
Unlike MAFT 31,, which initializes queries with zeros and uses random initialization for posi- tional information, we employ text-driven queries and positional information model the spatial relationships of entities in the expressions. This allows our model to capture the spatial context better, resulting in 4.4 4.4-point improvement in mIoU, as shown in Tab.
In contrast to (31], which supervises the positions of all target instances, 3D-RES supervises only the core target word. Our novel RWS method constructs spatial relationships for all noun instances using only the target word's positional information, resulting in a 2.3-point improvement in mloU, as demonstrated Tab,

---

Table 4: Weak Supervision Strategy in RWS, where "w/o RWS" means using attention-bast based Top1 approach in 665] instead of our RWS, and "RTS" refers to our Kule-guided Target Selec- tion strategy.
Table 5: Ablation study of the weight of Loss.
Multiple 0.5
Overall mlol
Weight of Lipee
0.25
0.1
54.7
34.9
36.9
61.3
44.3
44.0
Strategy
Multiple 0.5
Overall
0.2
0.25
mloU
0.25
0.5
mloU
0.5
55.0
35.4
37.4
61.7
44.9
44,6
61.8
w/o RWS
47.2
33.7
34.1
55.6
43.9
42.3
2.0
54.3
33.8
36.6
61.0
43.7
43,9
Root
53.5
30.4
34.7
60.7
40.9
42,5
RTS
55.0
35.4
37.4
61.7
Description
Original Scene
Ground
3D-STMN
Predicted Instance Locations
This gular It is above
2,97, 6.77)
small thin table.
20.44.822-007
(b) There is large, gray bookshelr the corner of the by the door, has a small couch sitting up
1:255, 0339
(.7)
(-LA7, 8.14, 41.3
(e) There is square brown chair. It is the with
JAAK -LO8, 4.54 H.79, .0.4, 8.46)
Figure 3: Visualization of all the nouns in the textual description. Our RG-SAN can segment instances corresponding to different nouns, while 3D-STMN indiscriminately assigns all nouns the target instance. Zoom in for best view.
4.5 Qualitative Comparison
We conduct a qualitative analysis the ScanRefer validation set as shown in Fig. comparing our proposed RG-SAN with 3D-STMN (65) to highlight our model's exceptional referring capability. Fig. 3 demonstrates our model's ability accurately segment not only the target objects but also other nouns mentioned in the text. Unlike 3D-STMN, which misattributes all nouns to a single target, RG-SAN distinctly recognizes and locates each noun. For example, in Fig. (c), our model successfully identifies the target chair through relative positioning, even with similar objects in the scene, and accurately recognizes a coat as a supporting element the description. This ability extends to Fig. (a) and (b), where RG-SAN correctly segments multiple auxiliary nouns into their corresponding instances, demonstrating its robust generalization for complex texts and precise localization for multiple entities. Such capabilities enhance the model's understanding of complex semantic scenes, significantly improving its ability to refer to specific entities accurately.
5 Conclusion
In this paper, we present RG-SAN overcome the limitations of traditional 3D-RES methods, particularly their lack of spatial awareness. Specifically, the TLM is introduced to model and refine positional information, while the RWS designed to employ dependency tree rules to accurately guide the position of the target object. Combining TLM with RWS strategy, RG-SAN significantly improves segmentation accuracy and robustly handles spatial ambiguities. Extensive experiments conducted on the ScanRefer benchmark demonstrate the superior performance of RG-SAN. This underscores the importance of incorporating spatial awareness into segmentation models, paving the way for future advancements in the domain.
10

---

Acknowledge
This work was supported by National Science and Technology Major Project (No. 2022ZD01 18201), the National Science Fund for Distinguished Young Scholars (No.62025003), the National Natural Science Foundation of China (No. U22B2051, No. U21B2037, No. 62072389, No. 62302411, 1, No. 623B2088), the Natural Science Foundation of Fujian Province of China (2200000)003) and China Postdoctoral Science Foundation (No. 2023M732948).

---

References
[1] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for line-grained 3d object identification in real-world scenes. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 422 440. Springer, 2020.
[2] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812, 2017.
[3] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Seanqa: 3d question answering for spatial seene understanding. proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129-19139) 2022.
[4] Pratyay Banerjee, Tejas Gokhale, Yezhou Yang. and Chitta Baral. Weaqa: Weak supervision via captions for visual question answering. arXiv preprint arXiv:2012.02:55 2020.
[5] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202-221. Springer, 2020.
[6] Shizhe Chen, Pierre-Louis Gubur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in Neural Information Processing Systems, 15:20522-205355 2022.
[7] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated reconstructions of indoor scenes. Proceddings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
[8] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- 2018
tional transformers for language understanding. arXiv preprini
[9] Zhiyuan Fang, Shu Kong. Zhe Wang, Charless Fowlkes, and Yezhou Yang. Weak supervision and referring attention for temporal-textual association learning, arXiv preprint arXiv:2006. // 1747, 2020.
[10] Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception cognition. In Forty:firt International Conference on Machine Learning, 2024.
[11] Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing, 2024.
[12] Hao Fei, Shengqiong Wu, Mcishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. Enhancing vidco-language representations with structural spatio-temporal alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[13] Mingtao Feng Zhen Li. Qi Li, Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang. and Ajmal Mian. Free-form description guided 3d visual graph network for object grounding in point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 3722-3731, 2021.
[14] Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403. /1000.,.1401,,111 14011, 2024.
[15] Xuri Ge, Fuhai Chen, Joemon M Jose, Zhilong Ji. Zhonggin Wu, and Xiao Liu. Structured multi- modal feature embedding and alignment for image-sentence retrieval. In Proceedings of the 29th ACM international conference multimedia, pages 5185-5193, 2021.
[16] Xuri Ge, Songpei Xu, Fuhai Chen, Jie Wang, Guoxin Wang, Shan An, and Joemon M Jose. 3shnet: Boosting image sentence retrieval via visual semantic spatial hiff-highhighting.. Information Processing Management, (4): (4):103716) 103716, 2024.
[17] Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Vqa-lol: Visual question answering under the lens of logic. In European conference on computer vision, pages 379-396. Springer, 2020.
[18] Yunpeng Gong. Liging Huang, and Lifei Chen. Person re- re-identification method based on color attack and joint defence. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4313 4322, 2022.
12

---

[19] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 9224-9232, 2018.
[20] Shuting He, Henghui Ding, Xudong Jiang, and Bihan Wen. Segpoint: Segment any point cloud via large language model. In European Conference on Computer Vision, pages 349-367. Springer, 2025.
[21] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the world into large language models. Advances in Neural Information Processing Systems, 36:20422-20994, 2023.
[22] Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint arxiv:231.008166 2023.
[23] Kuan-Chih Huang, Xiangtai Li, Lu Qi. Shuicheng Yan, and Ming- Hsuan Yang. Reason3d: Searching and reasoning 3d segmentation via large language model. arXiv preprint arXiv:2405.17427. 2024.
[24] Pin-Haa Huang, Han-Hung Lee. Hwann-Tzone Chen, and Tyng-Lui Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 1610-1616 2021.
[25] Ziling Huang and Shin' ichi Satoh. Referring image segmentation via joint mask contextual embedding learning and progressive alignment network. In Houda Bouamor, Juan Pino, and Kalika Bali, editors. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7753- 7762, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/v 18653/v1/2023. amnlp-main.44 URL httth:ts//calaaaeeee
www./rrg.........eo 2023. ewnlpemaamma.e.ee wwm.mainn.4.
[26] Ayush Jain, Nikolaos Gkanatsios. Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In European Conference on Computer Vision, pages 417 433. Springer, 2022.
[27] Kanishk Jain and Vineet Gandhi. Comprehensive multi-modal interactions for referring image segmentation. In Findings of the Association for Computational Linguistics; ACL 2022, pages 3427-3435, 2022.
[28] Corentin Kervadee, Grigory Antipov, Moez Baccouche, and Christian Wolf. Weak supervision helps emergence of word-object alignment and improves vision-language tasks. arXiv preprint 1rXiv:1912.03063, 2019.
[29] Yongmin Kim, Chenhui Chu, and Sadao Kurohashi. Flexible visual grounding. In Samuel Louvan, Andrea Madotto, and Brielen Madureira, editors, Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics: Student Research Workshop. pages 285-299, Dublin, Ire- land, May 2022. Association for Computational Linguistics. doi: 18.18653/v acllll URL https: betpp://aclanttologe www.eey..rg/0000 org/2022. acl- SIW.22
[30] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8500-8509, 2022.
[31] Xin Lai, Yuhui Yuan, Ruihang Chu, Yukang Chen, Han Hu, and Jiaya Jia. Mask-attention-free transformer for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3693-3703, 2023.
[32] Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4558-4567, 2018
[33] Jungbeom Lec, Sungjin Lee, Jinscok Nam, Scunghak Yu, Jacyoung Do and Tara Taghavi. Weakly supervised referring image segmentation with intra-chunk and inter-chunk consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21870-21881, 2023.
[34] Hui Li, Mingjie Sun. Jimin Xiao, Eng Gee Lim, and Yao Zhao. Fully and weakly supervised referring expression segmentation with end-to-end learning. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[35] Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang. Qi Tian. and Siliang Tang. Fine Fininee semantically aligned vision-languag pre-training. Advances in neural information processing systems, 35:7290-7344 -303, 2022.
13

---

[36] Jumnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imann pre-training for unified
understanding and generation. In International conference on machine learning, pages 12888-12900 12900. PMLR, 2022.
[37] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pages 19730-1972 19742. PMLR, 2023.
[38] Pengfei Li, Beiwen Tian, Yongliang Shi, Xiaoxue Chen, Hao Zhao, Guyue Zhou, and Ya- Fa-Qin Zhang. Toist: Task oriented instance segmentation transformer with noun-pronoun distillation. Advances in Neural Information Processing Systems, 15:17597-17611, 2022.
[39] Yang Li, Xiaoxue Chen, Hao Zhao, Jiangtao Gong, Guyue Zhou, Federico Rossano, and Yixin Zhu. Understanding embodied reference with touch-linc transformer. In ICLR, 2023.
[40] Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Transfrmer-empooeree invariant grounding for video question answering. IEEE Transactions Pottern Analysis and Machine Intelligence, 2023.
[41] Yicong Li, Na Zhao, Junbin Xiao, Chun Feng, Xiang Wang, and Tat-seng Chua. Laso: Languago-guin guiddd affordance segmentation on 3d object. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14251-14260, 2024.
[42] Zhihao Liang. Zhihao Li. Songcen Xu, Mingkui Tan, and Kui Jia. Instance segmentation in 3d scenes using semantic superpoint tree networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2783-2792 2021.
[43] Haojia Lin, Yongdong Luo, Xiawu Zheng, Lijiang Li, Fei Chao, Taisong Jin, Donghao Luo, Chengjie Wang. Yan Wang. and Liujuan Cao. A unified framework for 3d point cloud visual grounding. arXiv preprint arXiv:2308 1887, 2023.
[44] Fang Liu, Yuhao Liu, Yuqiu Kong, Ke Xu, Lihe Zhang, Baocai Yin, Gerhard Hancke, and Rynson Lau. Referring image segmentation using text supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22124-22134, 2023.
[45] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke 7ettlemoyer, and Veselin Stoyanov. Roberta: A rohustly optimized bert pretraining approach. arXiv preprint arviv / 1907. / 1692, 2019.
[46] Ziyang Lu, Yunqiang Pei, Guoqing Wang, Peiwei Li, Yang Yang, Yinjie Lei, and Heng Tao Shen. Scaneru: Interactive 3d visual grounding based embodied reference understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1936-39444 2024.
[47] Junyu Luo, Jiahui Fu, Xianghao Kong. Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16454-16463, 2022.
[48] Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for compidational linguistics: system demonstrations, pages 55 55-60, 2014.
[49] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565-571. Ieee, 2016.
[50] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Goy-Chowdhury. Weakly supervised video moment retrieval from text queries. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11592-11601, 2019.
[51] Wentao Mo and Yang Liu. Bridging the gap between 2d and 3d visual question answering: A fusion approach for 3d vqa. arXiv preprint arXiv:2402.15933 2024.
[52] Charles Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277-9286, 2019.
[53] Charles Ruizhongtai Qi. Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning point sets metric space. Advances in neural information processing systems, 30, 2017.
14

---

[54] Zhipeng Qian, Yiwei Ma, Jiayi Ji, and Xiaoshuai Sun. X-refseg3d: Enhancing referring 3d instance segmentation via structured cross-modal graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4551 4551-4559, 2024.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748- 8863, PMLR, 2021.
[56] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.
[57] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang. and Bastian Leibe. Mask3d for 3d semantic instance segmentation. arXiv preprint arXiv:22 2022.
[58] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tic-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding, Advances in Neural Information Processing Systems, 33:16857- 16867, 2020.
[59] Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner. Sameer Singh, and Anna Rohrbach. ReCLIP: A strong zero-shot baseline for referring expression comprehension. In Smaranda Muresan, Preslav Nakow, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Papers), pages 51 5198-5215, Dublin, Ireland, May 2022. Association for Computational I. Linguisties. doi: 10. 10.186633/1120022..ccl-.m.......
URL
hawpolloggyyoogge/eeee org/2022 dell-long.33 .37
[60] Jiahao Sun, Chunmei Qing. Junpeng Tan, and Xiangmin Xu. Superpoint transformer for 3d scene instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2393-24001 2023.
[61] Yucheng Suo, Linchao Zhu, and Yi Yang. Text augmented spatial aware zero-shot referring image segmentation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguisticx: EMNLP 2023. pages 1032-1043, Singapore, December 2023. Association for Computational Linguistics. doi: 1188556112222333 1202555/in122.....nnnnneeeee findings-emnlp.77 73. /ttL/ttts///acaattiiin org/ f findings-emmlp73
[62] Alexander Trott, Caiming Xiong, and Richard Socher. Interpretable counting for visual question answering. arXiv preprint 1rXiv:1712.08697, 2017.
[63] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[64] Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Pata-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08766 2023.
[65] Changli Wu, Yiwei Ma. Qi Chen, Haowei Wang, Gen Luo, Jiayi Ji, and Xiaoshuai Sun. 3d-stmn: Dependency-driver superpoint-text matching network for end-to-end 3d referring expression segmentation. arXiv preprint ar3iv:2308. 16632, 2023.
[66] Changli Wu, Yihang Liu, Jiayi Ji, Yiwei Ma. Haowei Wang, Gen Luo, Henghui Ding, Xiaoshuai Sun, and Rongrong gregres: Generalized referring expression segmentation. 2024. URL https 206644
//arxiv. , orgg abs/
[67] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 10033 10041, 2021.
[68] Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards semantic equivalence of tokenization multimodal I1m. arXiv preprint wwXiv:2400600127.. 2024.
[69] Shengqiong Wu. Hao Fei, Leigang Qu. Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal IIm. In Proceedings of the International Conference on Machine Learning, 2024.
[70] Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. Eda: Explicit text dext-decuupinng and dense alignment for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19231-19242. 2023.
15

---

[71] Wei Xu. Chunsheng Shi, Sifan Tu, Xin Zhou, Dingkang Liang, and Xiang Bai. A unified framework for 3d scene understanding. arXiv preprint arXiv:2407.0326. 2024.
[72] Zhengyuan Yang, Songyang Zhang. Liwei Wang. and Jicho Luo. Sat: 2d semantics assisted training for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1856-1866, 2021.
[73] Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. In- stancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1191-1800, 2021.
[74] Zhi Zhang, Helen Yannakoudakis, Xiantong Zhen, and Ekaterina Shutova. CK-transformer: Commonsense knowledge enhanced transformers for referring expression comprchension. In Andrcas Vlachos and Isabelle Augenstein, editors, Findings the Association for Computational Linguistics: EACL 2023, pages 2586 2596, Dubrovnik, Croatia, May 2023, Association for Computational Linguistics. doi: 1118855311122233 /tt.222....nnnneeee/l....
findings- eacl. 196
[75] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2928-2937, 2021.
[76] Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang. Towards learning a generalist model for emhodied navigation. arXiv preprint arXiv:2312.020100 2023.
[77] Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng. Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 291 1-2921, 2023.
16

---

Appendix
A
The Critical Role of Spatial Information in 3D-RES Tasks
Our analysis underscores the pivotal role spatial relations play in 3D-RES tasks. We assessed the ScanRefer dataset's referring expressions, classifying examples into two categories: those with spatial relation terms (e.g., "left", "right", "next", "bottom" and "side") as spatially related, and those without as spatially unrelated. Our findings revealed that spatially related samples form about 92% of the dataset, highlighting the prevalence of spatial descriptors. Additionally, the Sr3D dataset consists entirely of spatially related descriptions, and in the Nr3D dataset, a significant 90.5% of entries utilize spatial prepositions [.. This evidence demonstrates the necessity of spatial descriptions in accurately identifying objects within scene through natural language, emphasizing the essential need for effective spatial relation modeling in 3D-RES tasks.
B 3D-RES on ReferIt3D Dataset
We extended the 3D-RES task on the ReferIt3D dataset [ (which is also in English) by integrating instance masks from ScanNet and conducting relevant experiments, as shown in Tab. [ In contrast to the original setup of ReferIt3D, we refrained from using ground truth bounding boxes masks as input during our experiments, which significantly increased the level of difficulty. Nonethe- less, our model achieved remarkable Acc 50 gains of 5.3 points for Sr3D and 2.9 points for Nr3D, accompanied by mloU gains of 5.2 points for Sr3D and 1.0 points for Nr3D.
SeanRefer
spatial
wlo spatial
8%
92%
It is worth highlighting that our results demonstrate excep- tional performance in terms of Acc 50 and mloU. This can be attributed to the incorporation of spatial informa- tion, which enhances the accuracy of segmentation results and addresses the challenges of over-segmentation and under- segmentaennnnnn encountered in previous approaches.
Figure 4: Statistics of samples in the ScanRefer dataset based on the presence of spatial relation descriptions, where "spatial" represents samples with spa- tially related descriptions, while "w/o spatial" denotes spatially unrelated sam- ples.
C More Ablation Studies
C.1 Number of Multiple Rounds
We investigated the impact of varying the number of TLM rounds in our model. Analyzing rows two to five in Tab. 7 reveals consistent pattern: performance improves with more rounds, reaches its peak at six, and then slightly declines. Fewer layers result in insufficient capacity, while excessive number of layers increases the risk of overfitting. Therefore, selecting six layers strikes a balance that yields the best model performance.
In addition, we conducted ablation experiments to remove the iterative position refinement process at each layer. The results, shown in the first row of Tab. 7 clearly demonstrate the effectiveness of iterative refinement, leading significant improvement.
C.2 The Textual Backbone
In Tab. g, we compare the effects of commonly used natural language encoders. It can be observed that our method demonstrates robustness with respect to the selection of the NLP backbone. And we achieve the best performance using MPNet (58). The underperformance of CLIP (55 is under- standable, considering its optimization over large dataset of text-image pairs. While CLIP excels at extracting representations the sentence level, it encounters difficulties in comprehending intricate
17

---

Table 6: Results of 3D-RES tasks on ReferIt3D.
Method
hand
View
View Indep
Overall 0.5
Inference
0.25
mioU
0.25
mloU
0.25
0.5
mloU
0.25
0.5
mloU
0.25
miot
Time
Nr3D
TGNN
29.2
22.3
21.0
22.5
19.6
17.4
22.2
18.1
17.0
27.6
22.4
20.3
25.7
20.9
19.1
26599ms
3D.STMNN
32.6
35,4
28.4
29.4
25.8
27.66
RG-SAN
15.8
34,5
33,5
34.5
23.3
24.0
37.3
26.3
26.3
41.5
36.1
29.8
40.1
28.7
28.6
289ms
Sr3D
TONN
22.9
STMM
49.4
38.2
36.3
41.9
31.0
30.1
45.5
33.5
31.9
47.2
36.2
34.6
47.2
36.1
34.4
281
RG-SAN
55.8
43.6
41.5
46.7
36.3
35.2
42.6
34.7
32.5
53.5
41.7
39.9
53.1
41.4
39.6
293ms
Table 7: Number of multiple rounds.
Number of Rounds
Multiple 0.5
Overall 0.5
Iter. Refine
0.25
mloU
0.25
mloU
6
55.2
33.0
36.3
61.8
42.9
43.7
56.3
30.3
35.6
62.8
40.6
43.1
55.2
34.5
37.1
61.7
44.1
44.3
55.0
35.4
37.4
61.7
44.9
44.6
9
53.1
35.8
37.0
60.0
45.2
44.2
spatial relationships within sentences. This limitation hampers its performance situations involving multiple objects, resulting in relatively poorer results.
C.3
The Visual Backbone
pretrained by the classic work 3D-V (77 and another superpoint-based backbone, SSTNet 42 as detailed in Tab. 9 Our findings indicate that the performance with PointNet++, SSTNet and our employed SPFormer are quite comparable, demonstrating the adaptability and effectiveness of our proposed modules across different backbone architectures.
We explored alternative visual backbones, including the PointNet++ 53
D
More Qualitative Analysis
More qualitative comparison results illustrated in Fig. Fig.5laa and Fig. demonstrating the remark- able discriminative ability of our RG-SAN compared to 3D-STMN. Fig. 5 showcases RG-SAN's
superior performance in accurately localizing target objects, especially in challenging scenarios that require understanding complex positional relationships described in the text. For instance, Fig. 5 (b) illustrates scenario with numerous distractors and complex textual description, where 3D-STMN fails, causing over-segmentation. In contrast, RG-SAN accurately discerns and localizes the target object amidst distractions, achieving higher-quality segmentation. It is important highlight that when faced with descriptive text that involves spatial relationship reasoning among multiple instances mentioned, seen in all cases in Fig. 5 our RG-SAN demonstrates the capability precisely locate and identify the target object. In contrast, 3D-STMN lacks comparable complex reasoning abilities in such scenarios.
Table 8: Ablation study comparing text encoders.
Multiple
Overall
Text Encoder
0.25
0.5
mloU
0.25
0.5
mIoU
BERT
53.2
34.8
36.4
60.0
44.1
43.7
RoBERTa 45
53.1
34.8
36.7
60.0
44.0
44.0
CLIP
55
52.3
33.4
35.0
58.2
42.1
42.5
MPNet
58
55.0
35.4
37.4
61.7
44.9
44.6
18

---

Table 9: Ablation study comparing visual backbones.
Multiple
Overall
Text Encoder
0.25
0.5
mloU
0.25
0.5
mloU
SSTNet 8
53.7
34.3
34.9
59,4
42.5
43.2
PointNet++ 45
54.1
34.6
36.1
60.3
44.2
44.0
SPFormer
55
55.0
35.4
37.4
61.7
44.9
44.6
In Fig. we visualized the predicted masks of the mentioned instances of our RG-SAN. As can be seen in Fig. 6 (b), even if the "coat" category is not present in the training labels, our RG-SAN is still able to accurately identify the mentioned "coat" in the point cloud scene. This is because we align the word features from the textual modality and the point cloud features from the visual modality in a fine-grained manner through weak supervision. This aligmment brings them into the same feature space, enabling the model to have strong generalization capabilities for unknown semantic categories. This paves the way for future research weak supervision and open vocabulary.
Furthermore, as seen in Fig. (f), our RG-SAN is even able to accurately recognize the plural form of the entity noun "couches" mentioned in the descriptive text, while successfully identifying the target object. This capability enables the model to have a more precise and efficient understanding of spatial relationships associated with multiple auxiliary objects, such as "between" and "among", showcasing the powerful spatial relationship modeling ability of our RG-SAN.
E
Analysis of Target Word Positioning Capability in LLMs
We attempt to utilize LLMs, specifically LLAMA 2 70B example, for target word positioning. To achieve this, we construct a command template for LLMs, which includes the input description token list and demonstration examples. Such general template is designed as follows:
a Given word list, find the target word the list:
['the', 'trash', 'can', 'is', 'directly', 'right',
'the', 'brown', 'tables', 'turned", 'sideways", 5.7] => 'can', 2
(18)
('there', 'is', 'a', 'dark', 'brown', 'wooden', 'and', 'leather', 'chair',''' 'placed', 'in', 'the', 'table', "th"''the,, 'kitchen,"" " | => 'chair', [LIST]: =>
where [LIST] is replaced by the input description token list, and " =>> 'can', 2 denotes the target token the first example is "can" whose index in the token list 2.
For the ScanRefer dataset, LLAMA 70B produces approximately 80% of target word positions that align with the results obtained from our RWS module. In the remaining portion, our RWS module demonstrates higher accuracy. This partially indicates that there is still room for improvement in LLAMA 2 70B's s ability identify target word positions. Conversely, our rule-based approach benefits from efficient utilization of explicit dependency relationships and exhibits certain advantages. Additionally, LLAMA 2 70B poses significant computational burden. Taking this into consideration, our adopted RWS approach outperforms LLAMA 2 70B in terms of both accuracy and efficiency.
F Limitations and Broader Impact
Despite the strong performance of RG-SAN, we identify several limitations that call for further improvement. A primary limitation is its difficulty in accurately localizing plural nouns. This issue arises from the method of using a single point for localization, which proves challenging for plural entities in certain contexts. In future work, we will explore using multiple points to delineate boundaries for more precise localization of plural nouns.
The second limitation of our study concerns the model's inadequate robustness towards damaged point cloud data. The occurrence of damaged or incomplete data within point cloud datasets presents
19

---

Map
Description
Original Scene
Ground Truth
RG-SAN
3D-STMN
(a) necycling bin
the corner. is th- right of's copier.
(b) The chair is one chairs round tab-
chair closest
the open dourway sitting sideways relation the the wood and glass wall cabinet.
1.00
front of bookshelf, is the
(e)
0.75
(d) This the brown chair the back
0.50
the room where the
two chalk hoards
brown easy chair.
(e) The kitchen cabinet
between the dishwa- shor and stove, The
kitchen cabinet a beo-
we reclangular
(0 The chair is
desk that between
the two blackboards.
The chair is black with
cuerved back.
(g) chair
front of the desk, in between beds.
Figure 5: Qualitative comparison between the proposed RG-SAN and 3D-STMN. Zoom in for best view.
significant challenge, one that our current model is not sufficiently equipped to address. This lack of robustness can impair the model's 's ability to process such data accurately, leading to unreliable results in scenarios involving incomplete corrupted point clouds. Future work will aim to enhance the model's resilience and capability handling and compensating for data imperfections.
RG-SAN is expected to stimulate further development and application of multimodal 3D perception, especially practical scenarios such as embodied intelligence and autonomous driving. However, when it comes to practical applications, particularly those involving safety and privacy, rigorous testing is required ensure compliance with relevant laws and regulations.
G Ethics Statement and Licenses
In our work, there are no human subjects and informed consent is not applicable. Additionally, use publicly available text data from the ScanRefer Dataset (https://daveredrz wwwpsdrruumiiiiiii io/ ScanRefer), which is licensed under Creative Commons Atrrbbutorr-VnnCCCmneerra----ii 3.0 Unported License which allows us to use the dataset for non-commercial purposes. For point cloud data, used the publicly available ScanNet Dataset which is licensed under the ScanNet Terms of Use and the code is released under the MIT license. Both the licenses of ScanNet allow us use the dataset and code for non-commercial purposes. In the appendix, we use the ReferIt3D Dataset (https: : com/ref t3d/ rtreereferr))) for the dataset for
com wtl.cooo/oaaaae Caraaneee/Saaneeeeeee
extra experiments, which licensed under the MIT license which allows non-commercial purposes.
20

---

Predicted Masks of Mentioned Instances
Description
Original Scene
Ground Truth
(a) This is a rectang- ular tv, is above a small thin table.
(b) There a square brown chair. It is the one with a coat on it.
(c) This is armchair with four legs. This is placed in front of the laptop.
(d) This is a brown armchair. It is across from a desk.
(e) The mini fridge is small and white. It to the left of the cabinet.
(f) The coffee table is between two orange couches. The coffee table is gray with a square shape.
(g) This is a white dre- sser. It is under tv.
(h) There is a tall rect- angular black cabinet. It is next to the bookshelf.
Figure 6: The visualization of the predicted masks of mentioned instances of our RG-SAN. Zoom in for best view.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: The abstract and introduction accurately summarize the paper's key contribu- tions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

The paper primarily focuses on the experimental exploration of model struc- tures rather than theoretical derivations, hence it does not provide full of assumptions proofs for theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper fully discloses the necessary information for reproducing the main experimental results, including an anonymized open-source link in the abstract and the use of public datasets. Detailed experimental settings are provided Sec.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
23

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The paper provides open access the code through open-source link included the abstract, which, along with the description the supplemental material, offers sufficient instructions to reproduce the main experimental results.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
httpeeeuuubmssiioooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ctipss.c//ubbic//uiie gwwies/gguee//eeeeeoeooi.ocsssee CodeeSummeee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the necessary training and test details, including data splits, hyperparameters, their selection process, type of optimizer, etc., both in the experimental section and the appendix, ensuring a clear understanding of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper presents the results from single run for each experiment, which is consistent with previous works on the same task.
Guidelines:
The answer NA means that the paper does not include experiments.

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides detailed information on the computer resources used for each experiment within Sec, and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics ptttps://neuriss httpss/htttp//oopp.....p..b.b//p
Answer: [Yes]
Justification: The research conducted in the paper adheres to the NeurIPS Code of Ethics, ensuring that all aspects of the work, including the methodology, data handling, and reporting, conform to the ethical guidelines provided.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
25

---

Justification:
The paper includes discussion on the potential societal impacts of the work in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The research exclusively utilizes open-source, public datasets and does not involve high-risk models or scraped data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper credits the creators and original owners of all used assets and explicitly states the licenses and terms use in Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
26

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce any new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or human subjects research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing human subjects research.
Guidelines:
27

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "7bz2rdQmNq", "reviewer": "Reviewer_6dEW", "summary": "The paper proposes a new framework for 3D referring expression segmentation. The main contributions include analyzing the spatial information among objects and rule-guided target selection. Extensive experiments validate the effectiveness of the proposed method.", "questions": "1. The text-driven localization module is similar to the architecture in [22]. Have you tried to report their performance in ScanRefer in table 1?\n2. In equation (2), how do you initialize W_E and W_S to obtain the initial representations?\n3. In terms of equation (5), what is the intuition of adding positional encoding to position features? Does the order of the visual feature affects the final attention output?\n4. In equation (6) the positional encoding is added while in equation (9) and (10) it is concatenated. Why are these different?\n5. For section 3.3.1, do you have separate evaluation on how your algorithm performs in terms of finding the target?\n6. The explanation for table 4 points out the Top1 tends to select different nodes variably (line 281), but RTS is also choosing different nodes?", "limitations": "The authors addresses the limitations and societal impact.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors develop the method to achieve the state of the art performance on ScanRefer benchmark for 3D Referring Expression Segmentation. The authors conduct detailed experiments and comparison to validate the design.", "weaknesses": "1. Both the idea of incorporating the spatial information is not new. For spatial relation, section 3.2.2 and section 3.2.3 are very similar to [22]. Equation (7) to equation (10) are almost identical to equation (5) to equation (7) in [22]. Equation (12) is similar to equation (8) in [22]. What's more, there have been a lot of work on spatial information in 3D visual grounding, such as [6].\n2. The writing needs improvement. Some notations are not well explained. For example, K_i in line 120 and c_i in line 123 are not introduced. P^t in equation (3) with two subscripts is inconsistent with P^t in equation (5). Table t in line 166 and q in equation (7) are not introduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Using spatial information is not new. Sec. 3.2.2 and 3.2.3 and Eq. (7) to (12) resemble those in [22]. Spatial information in 3D visual grounding has been explored, like [6].\n> \n\nA1: Thank you for your valuable feedback. Indeed we acknowledge that our use of positional encoding is based on previous methods[51][22]. However, our primary contribution lies in modeling the spatial positions and relationships of noun entities within sentences for the 3D-RES task, which has also been recognized by reviewers p7dx, 8gkL, and MnE2. This approach has two core differences compared to [22]:\n\n(1) Unlike [22], which uses zero initialization for queries and random initialization for positional information, our queries and positions are text-driven from the start. This improves mIoU by 4.4 points, as shown in Tab. 2 and the newly added Tab. A.\n\n(2) Unlike [22], which supervises all target instances' positions, in 3D-RES, only the core target word is supervised. Our novel RWS method constructs spatial relationships for all noun instances using only the target word's positional information, improving mIoU by 2.3 points, as shown in Tab. 4.\n\n| Method | Initialization method of Queries | Initialization method of Position | Multiple mIoU | Overall mIoU |\n| --- | --- | --- | --- | --- |\n| MAFT [22] | Zero | Random | 29.7 | 37.9 |\n|  | Text-driven | Random | 30.1 | 38.8 |\n| RG-SAN w/o RWS | Text-driven | Text-driven | 34.7 | 42.3 |\n| RG-SAN (Ours) | Text-driven | Text-driven | 37.4 | 44.6 |\n\nTable A: Comparison of MAFT [22] with our RG-SAN in ScanRefer Dataset.\n\n**In summary, our core innovation lies in constructing spatial positional information, rather than just using positional encoding, as done in [6].** Using positional encoding is a regular operation after generating spatial information. We also explored other positional encodings, such as 5D Euclidean RPE, achieving similar results in Tab. 2.\n\nFollowing your suggestion, we will further compare and analyze our work with the highly relevant and interesting study [22] in the new version to clarify our contributions.\n\n---\n\n> **Q2: The writing needs improvement: K_i (line 120) and c_i (line 123) are not defined, P^t in Eq. (3) with two subscripts is inconsistent with P^t in Eq. (5), and Table t (line 166) and q (Eq. 7) are not introduced.**\n> \n\nA2: Thank you for your detailed feedback. We adopted the representation from [28] to formulate our approach as concisely as possible. Following your suggestion, we will make improvements in new version.\n\n---\n\n> **Q3: The TLM module resembles [22]'s architecture. Have you compared their performance on ScanRefer in Tab. 1?**\n> \n\nA3: Thank you for your constructive suggestions. We conducted a detailed comparison with [22] in Q1A1 and reported the suggested performance. Our proposed RG-SAN improves by 6.7 points, demonstrating its effectiveness. We will include this discussion in new version. Your suggestions will enhance the robustness of our contributions.\n\n---\n\n> **Q4: In Eq. (2), how do you initialize W_E and W_S to obtain initial representations?**\n> \n\nA4: Thank you for your attention to the details of our paper. In Eq. (2), W_E and W_S are initialized randomly. We will include this information in new version to enhance clarity.\n\n---\n\n> **Q5: For Eq. (5), what is the intuition of adding positional encoding to position features? Does the order of visual features impact the attention output?**\n> \n\nA5:  Thank you for your constructive question. Adding absolute position encoding is common in computer vision [51]. Changing the input order of visual features does not affect the final attention output because (1) the attention mechanism is order-invariant, and (2) the position encoding is tied to the visual tokens' 3D positions (xyz), so altering input order does not impact these positions.\n\n---\n\n> **Q6: Why is positional encoding added in Eq. (6) but concatenated in Eqs. (9) and (10)?**\n> \n\nA6: Thank you for your insightful inquiry. Our experiments show that addition and concatenation for positional encoding in Eq. (6), (9), and (10) yield similar results, as shown in Tab. B. The differences are negligible, so either approach can be used without significantly impacting the outcome.\n\n| Eq. (6) | Eqs. (9), (10) | Unique mIoU | Multiple mIoU | Overall mIoU |\n| --- | --- | --- | --- | --- |\n| Cat |  Cat | 74.6 | 37.4 | 44.6 |\n| Cat | Add | 74.7 | 37.4 | 44.7 |\n| Add | Cat | 74.5 | 37.4 | 44.6 |\n| Add | Add | 75.1 | 37.5 | 44.8 |\n\nTable B: Ablation of positional encoding usage, where \"Cat\" denotes concatenation, while \"Add\" denotes direct addition. \n\n---\n\n> **Q7: Evaluation of RTS for finding the target.**\n> \n\nA7: Thank you for your question. We previously evaluated RTS's ability to find the target using LLAMA2 70B in Sec. F of the supplementary materials, achieving an 80% match rate. However, LLAMA2 is not entirely accurate, making it an unreliable benchmark.\n\nTo better validate RTS, we annotated the text of 9,508 Val set samples to mark the target word positions. RTS achieved an accuracy of 93.4%, compared to 63.7% for the Top1 method, confirming our algorithm's effectiveness.\n\nWe will include this evaluation in new version and open-source the annotations to ensure reproducibility.\n\n---\n\n> **Q8: The explanation for Tab. 4 points out Top1 tends to select different nodes variably (line 281), but RTS is also choosing different nodes?**\n> \n\nA8: We apologize for the confusion. In line 281, \"different\" refers to predicted nodes that are not the target noun word. Top1 often selects nodes other than the target word, such as adjectives or verbs, leading to semantic confusion and an accuracy of only 63.7%.\n\nIn contrast, RTS accurately identifies the target word based on syntax, regardless of its position, achieving an accuracy of 93.4% as Q7A7 points out. This precise selection enhances semantic accuracy and significantly improves performance, as shown in Tab. 4.\n\nWe will revise the description in new version to improve clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tMU8VofWcW", "reviewer": "Reviewer_p7dx", "summary": "This paper presents the Rule-Guided Spatial Awareness Network (RG-SAN) for 3D referring expression segmentation (3D-RES), offering a novel approach to understanding spatial relationships in the visual-language perception domain. It aligns 3D and linguistic features not only at the semantic level but also within geometric space. The proposed network incorporates modules for textual feature extraction, text-driven localization, and rule-guided weak supervision. In the experimental setup, the model builds upon the efficient Superpoint Transformer for 3D feature extraction, as developed by Sun et al. (2023). The experimental results are promising, particularly in the overall 0.25 threshold setting on the ScanRefer dataset. Extensive ablation studies demonstrate the effectiveness of the proposed modules, while vivid visualizations showcase the method's impressive generalization capabilities.", "questions": "1. Does \"object center\" in this paper refer to the geometric centroid (center of the bounding box) or the center of mass (mean coordinate of all points)?", "limitations": "The authors have discussed the limitations, and I think it is somewhat okay for this work. It would be even better if an analysis of bad cases could be included.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. RG-SAN approaches 3D-language semantic alignment and spatial perception from a novel perspective. It not only aligns language features with 3D point cloud features at the semantic level but also explicitly assigns spatial positions to textual entity words within the geometric space. This explicit alignment helps address the spatial ambiguity inherent in directional natural language, allowing RG-SAN to achieve a more precise understanding of spatial relationships described in text.\n2. The authors conducted comprehensive experiments on the 3D-RES task, with particularly notable performance improvements on the ScanRefer dataset. It is impressive that the method significantly enhances performance while maintaining rapid inference speed, which is beneficial for real-time applications of this task.\n3. The authors performed detailed ablation studies on the proposed TLM and RWS modules, thoroughly examining the settings and hyperparameter choices. Additionally, they conducted ablation studies on the visual backbone and text backbone in the supplementary materials. These comprehensive ablations help readers understand the efficacy and rationale behind the proposed modules.\n4. In the supplementary materials, the authors provided a statistical analysis of the importance of spatial information for the 3D-RES task. This quantitative analysis supports the motivation of the paper and gives readers a clearer understanding of the role of spatial information in this task.\n5. The authors' visualizations are illustrative. In particular, Figure 3 demonstrates RG-SAN's text-guided spatial understanding and localization capabilities, showcasing excellent generalization.\n6. The authors have committed to open-sourcing their method, providing a link that includes the source code and an engaging video demo. This openness will promote development and knowledge sharing within the community.", "weaknesses": "1. RG-SAN adopts superpoints as fundamental visual units for feature extraction and segmentation. While previous works have employed similar approaches, I am interested in understanding the segmentation quality of superpoints themselves. For instance, how many superpoints exist solely within individual objects? This is crucial because if a superpoint spans across two objects, it inevitably affects the segmentation results.\n2. RG-SAN trains spatial awareness networks using the centroid coordinates of target objects. Here, does \"object center\" refer to the geometric centroid or the center of mass (where the former denotes the center of the bounding box and the latter denotes the mean coordinate of all points belonging to the object)? Given the inherent sparsity of point clouds, these two centers may exhibit significant differences.\n3. The statistical analysis of the importance of spatial information for 3D-RES should be included in the main text. This will help readers understand the motivation of the paper from both qualitative and quantitative perspectives.\n4. Although this paper discusses its limitations, it does not provide failure cases or corresponding analyses. Specifically, for the segmentation of plural nouns, it remains unclear whether only one object is recognized or if segmentation fails altogether. It would be beneficial to include either qualitative or quantitative analysis in this regard. Including this part would make the paper more comprehensive and facilitate follow-up research and improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and acknowledgment of our paper's strengths. We're pleased you appreciate our approach of attributing 3D spatial properties to text for 3D multimodal spatial perception modeling, recognize the promising performance of our model on ScanRefer, and underscore the effectiveness of our proposed methods. Now, we'll address the specific concerns you've raised to provide further clarification:\n\n---\n\n> **Q1: RG-SAN adopts superpoints as fundamental visual units for feature extraction and segmentation. While previous works have employed similar approaches, I am interested in understanding the segmentation quality of superpoints themselves. For instance, how many superpoints exist solely within individual objects? This is crucial because if a superpoint spans across two objects, it inevitably affects the segmentation results.**\n> \n\nA1: Thank you for your insightful feedback. In practice, due to the fine granularity of superpoints and their tendency to aggregate semantically similar points, most superpoints cover only a single object. To verify this, we conducted a statistical analysis of the superpoints in the ScanRefer [5] dataset. If a superpoint contains points from more than one object, it is classified as containing multiple objects; otherwise, it is categorized as containing a single object. Our analysis reveals that 99.55% of the points are within superpoints that cover a single object, with a missing probability of less than 0.5%. This indicates that the issue of multiple objects within a single superpoint has a negligible impact on the final results and does not warrant special attention.\n\n---\n\n> **Q2: RG-SAN trains spatial awareness networks using the centroid coordinates of target objects. Here, does \"object center\" refer to the geometric centroid or the center of mass (where the former denotes the center of the bounding box and the latter denotes the mean coordinate of all points belonging to the object)? Given the inherent sparsity of point clouds, these two centers may exhibit significant differences.**\n> \n\nA2: Thank you for your valuable comments. We use the centroid of all superpoints belonging to an object, representing the average coordinates of these superpoints. We conducted comparative experiments between this centroid setting and another centroid setting. As shown in Table A, the results indicate no significant differences between the two approaches.\n\nWe will include this discussion in the revised version to enhance the clarity of the paper.\n\n| Setting | Unique mIoU | Multiple mIoU | Overall mIoU |\n| --- | --- | --- | --- |\n| Center of Box | 74.8 | 37.4 | 44.7 |\n| Center of Mass | 74.5 | 37.4 | 44.6 |\n\nTable A: Comparison of the Center of Box and Center of Mass.\n\n---\n\n> **Q3: The statistical analysis of the importance of spatial information for 3D-RES should be included in the main text. This will help readers understand the motivation of the paper from both qualitative and quantitative perspectives.**\n> \n\nA3: Thank you for your suggestion. We will include the statistical analysis on the importance of spatial information for 3D-RES in the paper to enhance the rigor of the paper.\n\n---\n\n> **Q4: Although this paper discusses its limitations, it does not provide failure cases or corresponding analyses. Specifically, for the segmentation of plural nouns, it remains unclear whether only one object is recognized or if segmentation fails altogether. It would be beneficial to include either qualitative or quantitative analysis in this regard. Including this part would make the paper more comprehensive and facilitate follow-up research and improvements.**\n> \n\nA4: Thanks for your insightful suggestion. We will include visualizations of failure cases and provide a qualitative analysis in the new version.\n\n---\n\n> **Q5: Does \"object center\" in this paper refer to the geometric centroid (center of the bounding box) or the center of mass (mean coordinate of all points)?**\n> \n\nA5:  Thank you for your detailed question. We are referring to the centroid, where the center coordinate is defined as the mean of the coordinates of all superpoints belonging to the object.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KGb71gZVlQ", "reviewer": "Reviewer_8gkL", "summary": "This paper presents a novel and high-performing 3D referring segmentation network. Specifically, it approaches the problem from both 3D spatial relationships and natural language spatial descriptions, innovatively using explicit spatial position modeling and multimodal interaction. This allows the query corresponding to textual entities to understand both semantics and spatial locations. Additionally, the use of weak supervision techniques enables the model to achieve strong generalization capabilities even under incomplete annotations. Comprehensive experiments further validate the superior performance of the proposed method.", "questions": "Will the complete code for the paper be open-sourced for additional exploration?", "limitations": "The authors discuss the limitations in the appendix.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I commend the authors for their insightful paper, particularly the proposed text-guided spatial perception modeling approach. This method aligns with human cognitive habits and has the potential to significantly advance the field of multimodal 3D perception. Several notable advantages are highlighted:\n1. The paper deeply explores text-conditioned 3D spatial perception from both 3D spatial relationships and natural language structure perspectives, advancing the community's exploration of multimodal spatial perception modeling.\n2. The proposed TLM module effectively addresses the challenge of explicit spatial reasoning in previous end-to-end segmentation paradigms, significantly improving segmentation performance while maintaining high inference speed.\n3. The RWS module demonstrates data efficiency, generalizing capabilities to all entities without requiring mask labels for all textual entities.\n4. The experiments are comprehensive, evaluating the model's performance on both the ScanRefer and ReferIt3D datasets, thoroughly validating its robust performance.\n5. The ablation studies are detailed, thoroughly analyzing the proposed TLM and RWS modules, as well as the backbone selection and hyperparameter settings.\n6. The video demo in the open-source link is engaging, and the visualizations in Figure 3 of the paper are intuitive, effectively illustrating the core ideas and powerful performance of the proposed method.", "weaknesses": "1. More details can be added regarding the superpoint feature extraction and text feature processing sections.\n2. The paper mentions that the Sparse 3D U-Net used as the visual backbone is pre-trained. On which datasets was it pre-trained? Would using different pre-trained backbones result in performance variations?\n3. Figure 2 has too many colors, making it somewhat cluttered and potentially confusing for readers. It is recommended to simplify and optimize the color scheme.\n4. It is suggested to include some bad cases to enhance the completeness of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and recognition of our work. We appreciate your acknowledgment of our exploration of text-conditioned 3D spatial perception and the effectiveness of the TLM and RWS modules. We're also glad you found our video demo and Figure 3 visualizations clear and insightful. Now, let's address your specific concerns and provide further clarification:\n\n---\n\n> **Q1: More details can be added regarding the superpoint feature extraction and text feature processing sections.**\n> \n\nA1: We appreciate your valuable suggestions. We will provide a detailed description of superpoint feature extraction and text feature processing in the new version.\n\n---\n\n> **Q2: The paper mentions that the Sparse 3D U-Net used as the visual backbone is pre-trained. On which datasets was it pre-trained? Would using different pre-trained backbones result in performance variations?**\n> \n\nA2: Thank you for your insightful question. The 3D U-Net we used has been pre-trained on 3D instance segmentation tasks [45]. Additionally, following your suggestion, we explored alternative backbones, including PointNet++ [39], used by the classic work 3D-VisTA [a], and another superpoint-based backbone, SSTNet [28], as detailed in Table A. Our findings indicate that the performance with PointNet++ [45] and our employed SPFormer [45] are comparable, demonstrating the adaptability and effectiveness of our proposed modules across different backbone architectures. We will include this discussion in the final version.\n\n| Visual Backbone | Unique mIoU | Multiple mIoU | Overall mIoU |\n| --- | --- | --- | --- |\n| SSTNet [28] | 73.9 | 33.9 | 42.0 |\n| PointNet++ [39] | 75.5 | 36.1 | 44.0 |\n| SPformer [45] |  74.5 | 37.4 | 44.6 |\n\nTable A: Ablation study of the Visual Backbones.\n\n[a] 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment. ICCV 2023.\n\n---\n\n> **Q3: Figure 2 has too many colors, making it somewhat cluttered and potentially confusing for readers. It is recommended to simplify and optimize the color scheme.**\n> \n\nA3: Thank you for your detailed feedback. We will optimize Figure 2 in the next version to enhance its clarity and readability.\n\n---\n\n> **Q4: It is suggested to include some bad cases to enhance the completeness of the work.**\n> \n\nA4: Thank you for your valuable suggestions. We will include bad cases and corresponding analyses in the new version.\n\n---\n\n> **Q5: Will the complete code for the paper be open-sourced for additional exploration?**\n> \n\nA5: Thank you for your interest. In the paper, we have already provided the code via an anonymous link and also uploaded a copy in the supplementary materials. We also commit to releasing the complete code once the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hQxWWHXYDq", "reviewer": "Reviewer_MnE2", "summary": "This paper presents RG-SAN, a new method for 3D Referring Expression Segmentation. It combines spatial reasoning with textual cues to segment 3D objects accurately. RG-SAN uses a Text-driven Localization Module and a Rule-guided Weak Supervision strategy. It outperforms existing methods on the ScanRefer benchmark. It handles spatial ambiguities well and sets a new precision standard for 3D scene understanding. Extensive ablation studies and visualizations validate the effectiveness of the proposed modules.", "questions": "1 Will the code for RG-SAN be open-sourced in the future?\n\n2 Could the authors provide a quantitative comparison of RG-SAN with a broader range of 3D visual grounding methods?\n\nOthers please look at weaknesses.", "limitations": "Limitations are discussed [line 576].", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1) The motivation of this paper is clear. Figure 1 shows how spatial relationship info in natural language matches with 3D scenes, making the motivation obvious. Figure 4 reinforces this motivation from a quantitative perspective through statistical analysis. \n\n2) I agree that spatial relationship reasoning is crucial for understanding 3D scenes. The complex spatial relationships in natural language give important spatial clues. The proposed Text-driven Localization Module (TLM) performs spatial reasoning explicitly, aligning with how humans understand 3D scenes, which makes sense to me.\n\n3) The Rule-guided Weak Supervision (RWS) module enables localization and segmentation of auxiliary object nouns in natural language without any supervision. This aspect is interesting and shows the model's generalization capability.\n\n4) The appendix includes an analysis of how well large language models (LLMs) can localize target words, comparing this with the RWS module's results. This comparison further validates RWS and offers new insights into using LLMs for the 3D-RES task.\n\n5) The paper reports extensive experiments on common 3D-RES datasets like ScanRefer and ReferIt3D, achieving state-of-the-art performance. Ablation studies also show the effectiveness of the TLM and RWS modules", "weaknesses": "1) The paper does a good job comparing 3D-RES methods. But traditional 3D Visual Grounding methods using bounding boxes are more mature. I'd like to see how these older methods perform on this task. This would give a more complete quantitative comparison with the proposed method.\n\n2) The appendix talks about LLMs for target word localization. But it doesn't compare them directly with 3D Visual Grounding methods based on LLMs. Could LLM-based approaches be better? I'd like to see a comparison between specialized lightweight models and general LLMs.\n\n3) The paper should provide details on the superpoint feature extraction mentioned in line 119. I'm curious if superpoint features cover all targets. If not, what's the missing rate?\n\n4) The layout of Tables 2 and 3 is off, and the font in Figure 2 is too small. This makes it hard to read the details.\n\n5) The text processing procedure isn't detailed enough. For example, the interaction process of the DDI module. I'd recommend including this description in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and recognition of our contributions. We greatly appreciate your commendation of our clear articulation of motivation and quantitative analysis. We're pleased you acknowledged our method's effectiveness in spatial relation reasoning for 3D scenes, expressing interest in our rule-guided weak supervision strategy. Now, let's address the specific concerns you've raised and offer further clarification:\n\n---\n\n> **Q1: The paper does a good job comparing 3D-RES methods. But traditional 3D Visual Grounding methods using bounding boxes are more mature. I'd like to see how these older methods perform on this task. This would give a more complete quantitative comparison with the proposed method.**\n> \n\nA1: Thank you for your insightful suggestion. Based on your advice, we adapted the high-performing methods 3DVG-Transformer and 3D-SPS from 3D-REC for 3D-RES and tested their performance, as shown in Table A. Our method still demonstrates a significant advantage, outperforming by over 10 points.\n\n| Method | Unique mIoU | Multiple mIoU | Overall mIoU |\n| --- | --- | --- | --- |\n| 3DVG-Transformer [a]* | 49.9 | 27.0 | 31.4 |\n| 3D-SPS [b]* | 54.7 | 26.7 | 32.1 |\n| RG-SAN (Ours) |  74.5 | 37.4 | 44.6 |\n\nTable A: Comparison with 3D Visual Grouding methods. * we reproduce results by extracting points within the boxes as segmentation mask predictions using their official codes.\n\n[a] 3DVG-Transformer: Relation modeling for visual grounding on point clouds. ICCV 2021\n\n[b] 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. CVPR 2022\n\n---\n\n> **Q2: The appendix talks about LLMs for target word localization. But it doesn't compare them directly with 3D Visual Grounding methods based on LLMs. Could LLM-based approaches be better? I'd like to see a comparison between specialized lightweight models and general LLMs.**\n> \n\nA2: Thank you for your valuable suggestion. We compared our model with LLM-based 3D RES models SegPoint [c] and Reason3D [d], as shown in Table B, and our model still demonstrates a significant advantage, leading by more than 2.6 points.\n\n| Method | Unique mIoU | Multiple mIoU | Overall mIoU |\n| --- | --- | --- | --- |\n| SegPoint [c] | - | - | 41.7 |\n| Reason3D [d] | 74.6 | 34.1 | 42.0 |\n| RG-SAN (ours) |  74.5 | 37.4 | 44.6 |\n\nTable B: Comparison with LLM-based methods.\n\n[c] SegPoint: Segment Any Point Cloud via Large Language Model. ECCV 2024\n\n[d] Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model. Arxiv 2024\n\n---\n\n> **Q3: The paper should provide details on the superpoint feature extraction mentioned in line 119. I'm curious if superpoint features cover all targets. If not, what's the missing rate?**\n> \n\nA3: We appreciate your insightful feedback. The superpoint generation mechanism in RG-SAN ensures that no instances are missed, as the superpoints comprehensively cover the entire scene. Superpoints are essentially fine-grained fragments that group semantically similar points together. They do not overlap with each other and collectively constitute the whole scene, guaranteeing that all objects are included within superpoints. This setup ensures that every object is included within the superpoints, as detailed in the upper left corner of Figure 2 in our paper. Therefore, the issue of missing objects does not arise in our framework.\n\n---\n\n> **Q4: The layout of Tables 2 and 3 is off, and the font in Figure 2 is too small. This makes it hard to read the details.**\n> \n\nA4: Thank you for your suggestions. To improve readability and facilitate understanding, we will adjust the layout of Tables 2 and 3 and enhance Figure 2, including resizing the font, in the new version.\n\n---\n\n> **Q5: The text processing procedure isn't detailed enough. For example, the interaction process of the DDI module. I'd recommend including this description in the main text.**\n> \n\nA5: Thank you for your suggestion. We will include the details of DDI interactions in the new version to enhance understanding for readers.\n\n---\n\n> **Q6: Will the code for RG-SAN be open-sourced in the future?**\n> \n\nA6: Thanks for your interest. In the paper, we have already provided the code via an anonymous link and also uploaded a copy in the supplementary materials. We also commit to releasing the code once the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude to the reviewers for their valuable feedback and positive comments on our paper. Their insightful reviews have greatly contributed to improving the clarity and overall quality of our work.

We appreciate Reviewer **6dEW** $\color{red}{(Rating:\mathbf{3},\ Confidence:\mathbf{3})}$ for acknowledging the strengths of our paper. Specifically, they mention our new framework for 3D RES and highlight the analysis of spatial information among objects. They also recognized the excellent performance of our method and the thorough validation of the designed modules.

Reviewer **p7dx** $\color{red}{(Rating:\mathbf{7},\ Confidence:\mathbf{5})}$ appreciated our novel perspective of explicitly assigning spatial positions to text for 3D-language modeling and acknowledges our extensive comparative experiments with state-of-the-art methods and detailed ablation studies. Furthermore, they highlighted the clarity of our motivation and the thoroughness of our statistical analysis. Additionally, we are grateful for their recognition of the comparison of qualitative results with previous models. Such acknowledgment reinforces the validity of our research findings.

Reviewer **8gkL** $\color{red}{(Rating:\mathbf{7},\ Confidence:\mathbf{5})}$ acknowledged our in-depth exploration of text-conditioned 3D spatial perception, addressing both 3D spatial relationships and natural language structure. They appreciate the TLM module's role in enhancing performance and high inference speed in explicit spatial reasoning for 3D scenes. Furthermore, they acknowledge the data efficiency and generalization capabilities of our RWS module. Additionally, they acknowledge that our video demo and the visualizations in Figure 3 intuitively demonstrate the capabilities of our model.

Reviewer **MnE2** $\color{red}{(Rating:\mathbf{6},\  Confidence:\mathbf{5})}$ commended the clear articulation of our motivation and the quantitative analysis presented. They agree that spatial relation reasoning is crucial for understanding 3D scenes and recognize that our proposed method effectively extracts spatial relationships from complex language descriptions, enabling text-centric spatial reasoning. Additionally, they express interest in our rule-guided weak supervision strategy, which demonstrates the ability to perform localization and segmentation using natural language object nouns without any explicit supervision. The reviewer also supports our extensive experiments, which thoroughly validate the effectiveness of the proposed modules.

We sincerely thank the reviewers for recognizing these strengths, and we appreciate their positive feedback on the clarity, novelty, and effectiveness of our proposed methods. Their comments have further motivated us to address the concerns and improve the weaknesses pointed out in their reviews. We are committed to addressing their concerns and providing a detailed response in our rebuttal.

Meta-review: The authors present a novel framework for 3D referring expression segmentation. In the words of one of the reviewers, "The authors' exploration of how spatial relationships in natural language correspond with 3D scenes tackles a crucial and challenging area, especially compared to purely visual 3D segmentation. Spatial and relational reasoning is one of the major hurdles in cross-modal 3D vision today, and it’s great to see the authors making strides in this direction. I’m confident this work will inspire further progress in embodied intelligence". The authors have addressed all of the reviewers' concerns, and there is agreement that this is a high-quality and well-executed work with novel technical contributions.
Decision: Accept (oral)