LESS: Label-Efficient and Single-Stage Referring 3D Segmentation
Authors: Xuexun Liu
Xiaoxu Xu
Jinlong Li
Qiudan Zhang
Xu Wang
Nicu Sebe
Lin Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: Referring 3d segmentation
label-efficient
single-stage
cross-modal
Primary area: machine_vision
TL;DR: We design the first single stage referring 3D segmentation network, which trains on simpler labels and perform better.

Abstract: Referring 3D Segmentation is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. Previous works perform a two-stage paradigm, first conducting language-agnostic instance segmentation then matching with given text query. However, the semantic concepts from text query and visual cues are separately interacted during the training, and both instance and semantic labels for each object are required, which is time consuming and human-labor intensive. To mitigate these issues, we propose a novel Referring 3D Segmentation pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. Specifically, we design a Point-Word Cross-Modal Alignment module for aligning the fine-grained features of points and textual embedding. Query Mask Predictor module and Query-Sentence Alignment module are introduced for coarse-grained alignment between masks and query. Furthermore, we propose an area regularization loss, which coarsely reduces irrelevant background predictions on a large scale. Besides, a point-to-point contrastive loss is proposed concentrating on distinguishing points with subtly similar features. Through extensive experiments, we achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels. Code is available at https://github.com/mellody11/LESS.

LESS: Label-Efficient and Single-Stage Referring 3D Segmentation
Xuexun Liu Xiaoxu Xu Ku'", Jinlong Li Qiudan Zhang's Xu Wang" Nicu Sebe', Lin Ma *College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, 518060, China. University of Trento, Italy.
3 Meituan Inc., China.
Abstract
Referring 3D Segmentation is visual-language task that segments all points of the specified object from 3D point cloud described by a sentence of query. Previous works perform two-stage paradigm, first conducting languago-agnostee instance segmentation then matching with given text query. However, the semantic concepts from text query and visual and both instance and semantic labels for each object are required, which is time consuming and human-labor intensive. To mitigate these issues, propose novel Referring 3D Segmentation pipeline, Label-E Efffccifffc and Single-Stage, dubbed LESS, which only under the supervision of efficient binary mask. Specifically, we design a Point-Word Cross-Modal Alignment module for aligning the fine-grained features of points and textual embedding. Query Mask Predictor module and Query-Sentence Alignment module are introduced for coarse geeeeegrrnndddeeeee alignment between masks and query. Furthermore, we propose an area regularization loss, which coarsely reduces irrelevant background predictions on a large scale. Besides, point-to-pon contrastive loss is proposed concentrating on distinguishing points with subtly similar features. Through extensive experiments, we achieve state- of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mloU using only binary labels. Code is available at wihttt:/// :/gg h/////gttt//// thub com/mellody11//
separately interacted during the training,
Introduction
Referring 3D Segmentation task aims to segment the specific object from a 3D point cloud scene with free-form natural language expression. It allows users to interact and analyze 3D data through verbal instructions or queries. This approach is particularly beneficial in applications that necessitate direct interaction with 3D environments, such augmented reality (AR) systems, embodied-AL, robotics, virtual reality (VR) environments, and in fields like architecture and medical imaging, where precise identification and segmentation of objects based on descriptive queries can significantly enhance user experience and operational efficiency.
29 mainly leverage two-stage workaround, as shown in Fig (a). They typically adopt a 3D instance semantic segmentation network to get the instance proposals at the first stage. The predicted instance proposals will be utilized to match with the queries and finally get the final prediction mask according to the matching score. Although those method have achieved remarkable performance, there still exist some problems. First, owing to the large-scale and
Previous Referring 3D Segmentation
B
"qqual contributions. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Pre-seg ree-seegmnnlaiion Slage
Inslance Malching Stage
Single Stage
Instanoe label
Semantic label
Predicted Mask
Binary Label
Predicted Score
Ground Truth
Loss
Mask Predictor
Instance Predictor
3D
U-Net
Text Encoder
U-Net
Text Encoder
Thech is lo the right of the round table Query
The chairis to the right Khe lablle
Instance proposals
Query
(a) Previous two-stage method
(b) Our single stage method
Figure I: Comparison between the two-stage method and our single-stage method. (a) The two-stage method initially performs instance segmentation with instance labels then semantic labels to get the instance proposals and bases on the provided query match the most relevant instance proposal. (b) Our single-stage method only utilizes the binary mask of the described object for training and integrates language and vision features during feature extraction.
irregular 3D point clouds, some instance proposals may leave out the target in the pre-segmennattinn stage. Besides, lacking of linguistic guidance in the segmentation stage fails to focus on the objects that are more essential the referring task. Moreover, existing Referring 3D Segmentation utilizes both instance labels and semantic labels to segment target proposal rather than binary mask used in referring image segmentation, which is more time consuming and labor intensive.
To address the aforementioned problems, we propose a 3D Segmentation method, namely LESS, which is under the supervision of binary mask, as shown in (b).. We first process the query with text encoder to get the word-level feature and sentence-level feature. Then we extract the multi-modal feature with the guidance text feature through 3D sparse U-Net. Finally, the mask predictor aligns multi-modal features with language features, and directly predicts the mask of the described object. Here only object mask serves as the label supervise the whole training procedure.
and Single-Stage Stage Referring
However, 3D point cloud inherently provide a higher level of complexity and large scale. There exists numerous different objects in a single 3D scene compared to the referring image segmentation task. Besides, binary mask has less semantic meanings compared to instance labels and semantic labels. These challenges make it difficult to supervise our model to localize and segment target objects with only binary mask. Therefore, we propose to alleviate these problems by some ways, as shown in Fig 2 Firstly, to facilitate fine- fineigrained aligmment between points and words, we propose Point-Word Cross-Modal Alignment module. The PWCA module utilizes cross-modal attention in the multi-modal context align textual features extracted by the text encoder with point cloud features, followed by further extraction of useful multi-modal information using robust 3D sparse convolutional layers. Thus we can extract more semantic meaning fused feature. Meanwhile, we employ Query Mask Predictor, which utilizes the extracted multi-man modlll features to decode Icarnable query embeddings, generating candidate masks. By introducing the Query-Sentence Alignment modules, we can compute similarity between the decoded query embeddings and sentence features, using the similarity as weights to perform weighted summation of the candidate masks to produce the final mask. To address the significant interference caused by multiple objects and backgrounds, propose area regularization loss and point-to, to-poin contrastive loss. Area regularization loss reduces irrelevant background predictions by constraining the probabilities of the predicted mask, while point-to-poine contrastive loss constrains the distances between positive and negative points in the latent space to achieve better segmentation.
To summarize, contributions are as follows:

---

We propose new Referring 3D Segmentation method LESS, which directly performs Referring 3D Segmentation single stage to bridge the gap between detection and matching under the supervision of binary mask. To the best of our knowledge, LESS is the first work investigating label-efficient and single-stage in Referring 3D Segmentation task.
Our LESS utilize a Point-Word Cross-Modal Alignment module to align fined-grained point and word features. Besides, we employed Query Mask Predictor module and Query- Sentence Alignment modules for coarse- gairreeg alignment between masks and sentences. Moreover, the area regularization loss and the point-to- to-point contrastive loss are introduced to better support to eliminate interference caused by multiple objects and backgrounds.
Extensive experiments confirm the effectiveness of our method. Our method outperforms the existing sate-of-the-ar method on ScanRefer dataset with only the supervision of binary mask. Our LESS and its results provide valuable insights to improve further research of label-efficient and single-stage Referring 3D) Segmentation.
2 Related Works
2.1 Referring 3D Segmentation
Referring 3D Segmentation has previously received limited exploration, however, with the advance- ments in multi-modal learning and embodied AI, is set to attract increasing interest in the future. TGNN [13 is the first to introduce Referring 3D Segmentation task. They initially trained an instance segmentation network, followed by a Graph Neural Network (GNN) to learn features instances and
their relationships guided by linguistic information. Building on TGNN, X-RefSeg [29] developed a cross- modal graph. They employed an GNN to model the texture and spatial relationships of instances, and refining the results through inference and matching processes.
2.2 3D Visual Grounding
3D visual grounding aims to locate the object within point clouds mentioned by free-form natural language descriptions. Most methods follow a two-stage detection cereioonn-eee--eeee theettheneaee matehing pipeline. Initially, they utilize pre-trained 3D detector [28 23 or segmenter [15 33 to extract object representations. Subsequently, these methods align text features with object features
identify the best-matched object. Researchers primarily concentrate on the second stage which involves modeling object relationships and exploring feature fusion between language and objects. Methods employed include multi-modal feature concatenation 3 I attention-basel
multi-modal feature alignment [44; [1].
graph neural network-bard based reasoning [13 [1], and the alignment of visual and language features aided by 2D images [40] 42 38 Other researchers have investigated single-stage approaches for 3D visual grounding. 3D-SPS 25 views the task as key point selection, progressively identifying keypoints with the guidance of language and directly locates the target. BUTD-DETR [4] employs a transformer decoder [2] to identify described objects using language cues and proposal boxes.
Building on this, EDA (36] enhances dense alignment between objects and point clouds by explicitly decoupling textual attributes from sentences.
The distinction between 3D visual grounding and Referring 3D Segmentation lies in the latter's s enhanced localization precision, offering significant value in applications like robotic grasping. While traditional single-stage 3D visual grounding methods regress 3D bounding box, our single- stage approach decodes a binary mask for the entire point cloud scene, presenting a more complex challenge.
2.3 Referring Image Segmentation
Referring image segmentation is visual task that involves pixel-level segmentation of an image's s target object based on referring expression. Early approaches [12] 20:
[8] 4 17] utilize CNNs and RNNs to extract features and then perform simple concatenation for multi-modaa feature fusion. adopt transformer models for more effective feature extraction focus on identifying optimal positions for language-vision alignment. Additionally, some methods 34 5 have enhanced alignment between language and pixel features by designing specialized loss functions. Other approaches [21 45) treat referring image
Subsequent research [16 19:
and fusion. Recent studies
39 10 43
3

---

Seg Lass
rraa Loss
Contraslive Loss
PWCA
W
M
Sparse Feature Extractor
Qeery
The to right?
Vi
the lablle
PWCA
V
V
Sigmoid Function
Predictor xm
QSA
T
W
PW CA
Element wise addition
Querres
Miltixx multiplication
Text Encoder
W
S
Point-WWor and Cross-Modal Alggmment
uery Mask Predictor
Query- Sentence A Alggmeent
Mask Predictor
Querias Embedding
Multi Modal Features
Atemtion Mask
Queries Emholding
t D -- 4
Proposi
Similanty Score
anh
Sentence Features
f
MLP
Query D Decoder
Shared MLP
O
0.2
Cross A ltention
V.
W
0 D-O k Querries
Point Features
W Festures
Pasadd Pentunes
Figure 2: Overview of our LESS framework. Given a point cloud scene P, we use sparse 3D feature extractor to extract multi-scale feature V4. The query sent to a text encoder and we obtain the word features W and sentence features S. Meanwhile, we introduce a PWCA module aligns the word features W with the multi-scale point cloud features V,. After that, an mn-ayyer QMP module adopted decode K learnable queries Qo base the fused feature F, and output query embeddings Qm and proposal masks Mm. Finally, QSA module aligns the query embeddings Qm with sentence features S. i.c., computes the similarity scores R that filter the proposal masks Mm the final mask prediction M.
segmentation an auto-regressive vector generation problem, creating masks from generated closed vectors.
Methods for referring image segmentation have been extensively explored, yet they cannot be directly applied Referring 3D Segmentation due to the inherent challenges of point cloud scenes. Additionally, unlike referring image segmentation methods that typically employ pre-trained visual encoders, Referring 3D Segmentation lacks such resources, compelling reliance on limited supervisory signals for model training.
3 Method
The overall framework of our proposed LESS is shown in Fig which leverages Point- Word Cross- Modal Alignment and Query-Sentent Alignment facilitate multi-modal interaction. In this section, we start by introducing our visual and text feature extractor in S.c Then Query Mask Predictor and Query- Alignment detailed in See 3.2 and S.c Finally in Sec 3.4 our area regularization loss and point-to-point contrastive loss.
introduce
3.1 Visual and Text Feature Extractor
Sparse 3D Feature Extractor. The point cloud scene P E Nx6 contains N points in the scene, and each point represented with six dimensions RGBXYZ attributes. We first voxelize points into regular voxels and adopt sparse 3D U-Net (31 15 to extract point-wise fused feature F E R XC Here the encoder part of the U-Net has 5 stages and the feature from the i-th encoder stage denoted as V,.

---

Text Encoder. Given the query sentence T with L words, a text encoder is used to embed the query into C-dimensional feature vectors. In this paper we choose GRU [6), BETR [8] and RoBERTa [22] as our text encoder respectively and fine-tune the BERT or RoBERTa during training. Finally, we can get both word features W E LxC and sentence features S E TRC after the text encoder.
Point-Wore Cross-Modks Alignment. Due to the lack of such rich annotations as 13 29 it is crucial for our model learn the relationship between fine-grained word-level features and point- level features in such a point-level segmentation task. Meanwhile, we notice that leveraging the rich convolutional layers in the encoder to excavate multi-modal context is effective way extract language-awan visual feature. Therefore, we design Point- Cross-Modal Aligmment (PWCA) module, as shown in the lower left part in Fig2) which contains a standard cross-attention module 32 and an M-P-Tanh gate. Cross-attention module aligns point-wise and word-wise feature to get the language-awes aware visual feature. nonlinear tanh gate is adopted to prevent the fused signal from overwhelming the original signal. Given the multi-scale point cloud features (V; E RNN. 15 simply project the word feature W E IR LxC to W; E RLXC. PWCA can be formulated as follow:
fi= we
V = Tanh (CrosssstVss,, W.)) + V,, i (I,....,, 5),
(1)
where: indicates the i-th stage of the encoder part of sparse 3D feature extractor. Here we use V; as the query and W; as the key and value for cross attention.
3.2 Query Mask Predictor
Inspired by [3 30], Query Mask Predictor (QMP) module, as shown in 2 takes fused feature F E R R-NxC' xC: and learnable queries Qu E KxC as input and progressively distinguishes the referring target by multi-layer cross-modal transformers. Finally, we extract the proposal masks Mn. E TR& K based on queries embeddings Qm E KxC and fused feature F.
Here query decoder is comprised m-layer masked cross-attention modules (32), where the fused features F are served as keys and values and A;-1 E {0, 1|KNN is utilized as the attention mask. Therefore, in the j-th layer, the learnable queries Qj- - capture multi-modal contextual information from fused feature F via
Query Decoder. As illustrated in the lower middle section of Fig 2
query decoder to obtain the query embeddings Qj E IR K
Mask Predictor. First the fused feature F E NNC is processed by a Shared MLP, which indicates each Query Mask Predictor layer shares the same MLP. We perform the matrix multiplication on the new F and Q2 E KR to generate proposal mask predictions Mj- E R After applying sigmoid function whose threshold of 0.5, we can get the new binary attention mask Aj E {0, 1) K
Finally, the Query Mask Predictor is formally described as follows:
Qj Query Decoder,
F),
(2)
M, 2, @ Shared- (M
(3)
1 a (M,) 0.5
A, = 0 otherwise
where T denotes the transpose operation, @ represents matrix multiplication, and refers to the sigmoid function. Here we need to note that the Qo randomly initialized. Therefore we can leverage the function 3 to initialize the attention mask Ag-
3.3 Query-Sentence Alignment
The Referring 3D Segmentation task involves the segmentation of a solitary target object according to the query. Previous modules focus on extracting language-aware visual feature from aligning point-wise feature and word-wise feature, which lacks a comprehensive perception of the entire query sentence. Therefore, we adopt Query- Query-Sentence Alignment (QSA) to better align the query feature with sentence-level feature. We perform the matrix multiplication on Qm R and S E R to get the their similarity score R E [0, 1]K, Formally, Query-Sentence Alignment can be represented as:
R = Softmax (Qm @ S).
(5)
5

---

The final mask prediction M E IN is produced by weighted sum of similarity score R and proposed mask prediction Mn E K Finally, we use sigmoid function and threshold of 0.5 to produce the final predicted binary mask Y (0. 13:
M R M
Y 8-10
{
(M) 2 0.5
(6)
otherwise
3.4 Loss Function
Segmentation Loss. Different from previous work [13 29 we take the Referring 3D Segmentation task as segmentation task with only binary mask Y E (0.1) 1) Here we utilize the Binary Cross- Entropy (BCE) loss function to compute the segmentation loss, which can be formulated as:
Seg (M)
Area Regularization Loss. For Referring 3D Segmentation task, each query always corresponds to one target object in the point cloud scene. The target objects occupy a smaller area in the large scale of 3D point cloud. As a result, the predicted mask often includes backgrounds other objects. To address this, we propose region regularization loss, which promotes the network to predict smallest mask by minimize the output probability of each point, formulated as:
1 N (M))
Carea
(8)
By combining with the segmentation loss, we intend to segment only the most probable regions while reducing segmentation of large-scale irrelevant background areas.
Point-to-Point Contrastive Loss. Area regularization loss uniformly penalizes the predicted probabilities of all points, which can reduce the majority of the background points. However, the network struggles to differentiate between objects that possess characteristics similar to those described target object in the latent space. Therefore, we propose point-to-pon contrastive loss 26 that pull the points from the described object together and push away the rest points:
1 P
exp(P, Pavg/T)
Cpp =
exp P: Parg/T)+ + E/A exp(Px.NN/77
where P is the positive point set from the described object, A is the negative point from the background, P, denotes the 1.2-normalized feature vector of i -th positive points from F, while N, denotes the 1.2-normalized feature vector of j-th negative points from F, P avg is the average feature vector of positive point P avg = Σ.
= Ps and 7 is the hyper-parameter The contrastive loss promotes the network to distinguish the described object from the adjacent background points in a fined-grained manner.
The overall loss function is the weighted sum of the aforementioned three loss functions:
L Aueg + Xarea Caren Xp2p Sp2ps
(10)
where {Asegy Anreny }pap} is {1, 0.055 in practice balance the contrastive loss because of the large amount of points.
4 Experiments
4.1 Dataset and Experiment Settings
ScanRefer [3] is a dataset for 3D referring expression comprehension tasks such as 3D visual grounding and 3D referring instance segmentation. It contains 51, queries of 11, 046 objects from 800 ScanNet scenes. Each scene contains 13.81 objects and 64.48 queries on average.
ScanRefer.

---

Table 1: Quantitative results of different methods ScanRefer B
validation set. "Supervision"
indicates the type of supervision. Ins. denotes instance labels and Sem. indicates semantic labels. Mask represents binary labels. Bold indicates the best.
Method
Backhone
Label Effort
Supervision
mloU
A.c
A.5
TGNN TGNN
GRU
Ins. Sem.
26.10
35.00
29.00
Two
BERT
Ins. Sem.
27.80
37.50
31.40
20 min
Stage
X- X-RefSeg
GRU
Ins. Sem.
29.77
39.85
33.52
X- X-RefSeg
BERT
Ins. Sem.
29.94
40.33
33.77
LESS (ours)
GRU
Mask
32.19
51.00
26.41
Single Stage
LESS (ours)
BERT
2 min
Mask
32.44
51.41
29.02
LESS (ours)
RoBERTa
Mask
33.74
53.23
29.88
The evaluate of label effort is base on single sample.
Table Module ablation on ScanRefer dataset.
Table Loss ablation on ScanRefer dataset.
PWCA
QSA
mloU
A A@25
A@50
Larea
Sppp
mioU
A@25
A@51
(a)
32,66
51.71
27.20
25.86
40.85
16.81
33.44
52.73
28.92
31.04
49.61
24.72
33.74
53.23
29.88
(c)
33.74
53.23
29.88
we use the mean intersection-over-unioo (mloU), and Ace@kIOU the evaluation metries. The mloU is the average of the IoU over all test samples and the Acc @ KIOU measures the accuracy of test samples with an IoU higher than the threshold where {0.25, 0.5). We use A@25 and A@ 50 for brevity in some of the following tables.
Evaluation Metric. Following previous work 13
29
Implementation Details. We adopt the 3D spares U-Net B1
3D feature extractor.
explore multiple text encoders, i.e., GRU (6, BERT g and RoBERTa (22) in our experiments for comparative analysis. For BERT and RoBERTa, them during training. We set the number of queries K to 20 and use a single layer for QMP module. We set initial learning rate of 2e-5 for the text encoder and le-4 for the others. We reduce the learning rate by multiplicative factor of 0.95 each epoch and adopt Adam (24] as our optimizer. The weights of our loss function {Aueg, Aarea Ap2p] is to {1, 0.05). We train for 64 epochs with batchsize of 14, and all experiments are implemented on PyTorch 27
use the official pre-trained weights and fine-tune
4.2 Comparison with State-of-the-Art Methods
As shown in Tab we evaluate our LESS against the previous two-stage methods. LESS outperforms the previous SOTA method using GRU 6 and BERT (8] by an impressive progress of 2.42%, and 2.50% on mloU, and 11.15% and 11.08% on 0.25 respectively. Moreover, we conduct an extra experiment using RoBERTa [22] outperforming the best method of 3.8% and 12.9% on mloU and Acc@0.25 respectively. Such results demonstrate the potential of our method. We also reported the comparison between method and the two-stage approach in terms of label effort. We find that our label-efficient method saves more than 90% label effort compared to existing methods.
However we find that the performance of our LESS has gap on A.c@0.5 compared to previous methods. Previous methods employed a segmentation matiiiinnnmathingggn strategy. Once matching successfully, the IoU between predicted mask and ground truth is mostly higher than 0.5, which is beneficial for the A.5 In contrast, our single-stage method without instance labels and semantic labels can not extract the more accurate instance candidates as prior knowledge to assist referring 3D segmentation task. Therefore, is acceptable for our method to perform lower than previous methods on Acc@0.5 with fewer supervisory signals.
7

---

Query
Input
+ Sos
Srea
+ 22
GT
(a) this tan cabinet. it is Jeft of a chair.
(b) the lamp white shade. the lamp is behind a black chair.
(c) this is brown chair facing to the right. it sturdy and made of
brown material
(d) this is tan table. is in between trash cans.
(e) this lamp is the top left. is white.
Figure 3: Final predictions using different combinations of loss functions. The queries and input scenes are shown in column 1 and 2. Columns 3 to 5 indicate the gradual addition of loss functions.
4.3 Ablation Studies.
We conducted several ablation studies to evaluate the effectiveness of the key components in our proposed network. In these studies, all other components and consistent with the aforementioned experiments, except for the component being ablated.
settings are kept
Module Ablation. In this ablation study, we evaluate the effectiveness of the QSA and PWCA modules. As indicated in Table the ablation model (a) only retain the sparse 3D feature extractor and query mask predictor. Here we perform element-wise addition of the word features W to the multi- scale features {V4) instead of PWCA and project the proposed mask prediction Mm. E IR K to the final mask prediction M R via an MLP instead of QSA. We set model (a) as the baseline of our experiment. Compared to model (a), model (b) adopts PWCA module and we find that the model performance increase greatly from 32.66% 33.44%. This observation proves that PWCA facilitate fine-graine mne-grained cross-moda alignment between points and words, which is more effectively to leverage the rich convolutional layers the encoder to excavate multi-modal context. When we introduce QSA to model (b), shown in model (c), we can find that the performance of mloU is improved from 33.44% to 33.74%. Such results indicates that the fine-grained point-word alignment of PWCA and the coarse- -rarise--g the capability in capturing multi-modal context.
alignment of QSA effectively coupled to enhance
Loss Ablation. As demonstrated in 3.4 we refine predicted masks from coarse to fine by introducing two loss functions, i.e., area regularization loss Larea and point-to-point contrastive loss Lypp- We successively add the loss functions and the results are shown in Tab 3 The model (a) is only supervised by segmentation loss L When we introduce the Carea into model (a), as shown in model (b), the performance greatly increase from 25,86% to 31.04% mloU, which indicates our

---

Table 4: The impact of linguistic features different granularities. Word represent word-level features, and Sentence represent sentence-level features.
Word
Sentence
mIoU
Acc@0.25
Ace@0.
31.56
49.50
25.84
33.02
52.33
27.75
/
33.74
53.23
29.88
Table 5: The impact of the number of queries.
Table 6: The impact of the number of layers.
Num of queries
mloU
A@ A@25
A@
Num of layers
mloU
A@25
A A@50 50
20
33.74
53.23
29.88
33.74
53.23
29.88
60
32.62
51.96
27.70
3
32.45
51.09
27.70
100
33.16
52.98
28.82
32.66
52.18
28.33
area loss can exclude significant number of irrelevant background points. Moreover, we successively add the contrastive loss p-ppp as shown in the model (c). The performance is improved from 31.00 1.04% to 33.74%, which proves that the contrastive loss can make the model more focused on the target area rather than others.
Qualitative results are shown in Fig indicates that: When only the segmentation loss Seg is applied (column 3), the predicted masks include many points from other regions. ii) After adding the Larea, most of the irrelevant points disappear (row row d). iii) After incorporating the Lypp. objects that were previously difficult to distinguish due to their similarity are successfully separated, and the predictions are close to the ground-truth.
Both quantitative and qualitative experiment demonstrate that our proposed loss function effectively reduces large background misclassifications and distinguish objects or points with the similar characteristics.
4.4 Extension Experiments
Linguistic Features at Different Levels of Granularity. In this section, we will investigate the impact of linguistic features at different levels of granularity, as shown in Table The first row represent that leverage word-level features both PWCA module and QSA module as text features. The second row indicates we utilize sentence-level features in both modules. We can find that sentence-only method even outperforms the word-only one. Referring 3D Segmentation task involves the segmentation of a solitary target object. This mandates more profound and thorough comprehension of the semantic information conveyed by the sentence, extending beyond a mere focus on its individual words. The last row indicates the word-level features are utilized in PWCA module and sentence-level features are leveraged in QSA module. We can find that the performance of it outperforms two introduced above, which proves that fine-grained word-level feature is also helpful in extracting the 3D language-aware visual feature.
Layer and Query Number of QMP. layer numbers in the QMP module. As shown in Tab Tabl we find that too many queries and layers do not bring performance gains for Referring 3D Segmentation task. As result, we set 20 queries and QMP layer as the default configuration after balancing performance and efficiency,
We also investigate the performance on different query and
4.5 Limitations
The limitations LESS due to the inherent complexity of 3D point clouds and the ambiguous queries, although we have made significant improvements on previous methods. The scarcity of detailed semantic annotations are still challenging our model from distinguishing multiple similar objects. These limitations could guide future work.

---

Conclusions
In this paper. we propose LESS, label-efficient and single-stage approach for Referring 3D Segmen- tation. Specifically, our LESS enhances feature extraction by integrating multi-mode features and employs progressive constraints on predicted masks, achieving fine-gres fine-grained alignment between points and words and distinguishing between points or objects with similar characteristics. Comprehensive experiments demonstrate that our single-stage method outperforms existing two-stage approaches on ScanRefer dataset, using only the binary labels as supervision. Though our framework still has some limitations, we believe that solving the Referring 3D Segmentation task only using the binary labels new and promising research, and we hope that LESS can serve simple but powerful bascline to inspire future research on Referring 3D Segmentation.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants 62371310, part by the Guangdong Basic and Applied Basic Research Foundation under Grant 152311515 in part by the Stable Support Project of Shenzhen (Project 122222201221222220000
We also thank the support by the MUR PNRR project FAIR (PE00000013) funded by the NextGener- ationEU, the PRIN project CREATIVE (Prot. 2020ZSL9F9), and the EU Horizon project ELIAS (No. 101 120237). We also acknowledge the CINECA award under the ISCRA initiative, for the availability of partial HPC resources support.
References
[1] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine- tine-grained 3d object identification in real-world scenes. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 422-440. Springer, 2020.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[3] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202-221. Springer, 2020.
[4] Ding-Jie Chen, Songhao Jia, Yi-Chen Lo, Hwann- Tzong Chen, and Tyng-Luh Liu. See-through- text grouping for referring image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7454-7463, 2019.
[5] Yong Xien Chng, Henry Zheng, Yizeng Han, Xuchong Qiu, and Gao Huang. Mask grounding for referring image segmentation. arXiv preprint arXiv:2312.12110 2023.
[6] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.3555,
2014.
[7] Angela Dai, Angel X Chang, Manolis Savva, Macicj Halber, Thomas Funkhouser, and Matthias NieBner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
[8] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.044 04805. 2018.
[9] Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 16321-16330, 2021.
10

---

[10] Guang Feng, Zhiwei Hu, Lihe Zhang, and Huchuan Lu. Encoder fusion network with co- attention embedding for referring image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15506-15515, 2021.
[11] Dailan He, Yusheng Zhao, Junyu Luo, Tianui Hui, Shaofei Huang, Aixi Zhang, and Si Liu. Transrefer3d: Entity-and-relationn aware transformer for fine-grained 3d visual grounding. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2344-2352, 2021.
[12] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October / 11-4, 2016, Proceedings, Part I 14, pages 108-124. Springer, 2016.
[13] Pin-Hao Huang, Han-Hung Lee, Hwann-Tzaee Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1610-1618, 2021.
[14] Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In European Conference on Computer Vision, pages 417-433. Springer, 2022.
[15] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and Pattern recognition, pages 4867-4876, 2020.
[16] Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Restr: Convolution- free referring image segmentation using transformers. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 18145-18154, 2022.
[17] Jinlong Li, Zequn Jie, Xu Wang. Xiaolin Wei, and Lin Ma. Expansion and shrinkage of localization for weakly-supervised semantic segmentation. In Advances in Neural Information Processing Systems, volume 35, pages 16037-16051, 2022.
[18] Jinlong Li, Zequn Jie, Xu Wang, Yu Zhou, Xiaolin Wei, and Lin Ma. Weakly supervised semantic segmentation via progressive patch learning. IEEE Transactions on multimedia, 2022.
[19] Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. Advances in neural information processing systems, 34: 19652-16664, 2021.
[20] Ruiyu Li, Kaican Li, Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Referring image segmentation via recurrent refinement networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5745-5753, 2018.
[21] Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18653-18663, 2023.
[22] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint urXiv:I900) 1692, 2019.
[23] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong, Group-free 3d object detection via transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 2949-2958, 2021.
[24] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 17Xiv:1711.05100 2017.
[25] Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16454-16463, 2022.
11

---

[26] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.0377 07744878 2018.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[28] Charles Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference Computer Vision, pages 9277-9286, 2019.
[29] Zhipeng Qian, Yiwei Ma, Jiayi Ji, and Xiaoshuai Sun. X-refseg3d: Enhancing referring 3d instance segmentation via structured cross-modal graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 38, pages 4551-45599 2024.
[30] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 8216 22266.823.222 IEEE, 2023.
[31] Jiahao Sun, Chunmei Qing, Junpeng Tan, and Xiangmin Xu. Superpoint transformer for 3d scene instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2393-2401, 2023.
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[33] Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang D Yoo. Softgroup for 3d instance segmentation point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 7708-2717, 2022.
[34] Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11686-11699 2022.
[35] Changli Wu, Yiwei Ma, Qi Chen, Haowei Wang, Gen Luo, Jiayi Ji, and Xiaoshuai Sun. 3d-stmn: sppendency-drived superpoint text matching network for end-to-end 3d referring expression segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5940-5948, 2024.
[36] Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. Eda: Explicit text-decoupling and dense alignment for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19231-19242, 2023.
[37] Xiaoxu Xu, Yitian Yuan, Jinlong Li, Qiudan Zhang. Zequn Jie, Lin Ma, Hao Tang, Nicu Sehe, and Xu Wang. 3d weakly supervised semantic segmentation with 2d vision-language guidance. arXiv preprint arXiv:2407.09826, 2024.
[38] Xiaoxu Xu, Yitian Yuan, Qiudan Zhang, Wenhui Wu, Zequn Jie, Lin Ma. and Xu Wang. Weakly-supervised 3d visual grounding based on visual linguistic alignment. arXiv preprint arXiv:2312.09625. 2023.
[39] Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18155 18 18165, 2022.
[40] Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: 2d semantics assisted training for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1856-1866, 2021.
[41] Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 1791-1800, 2021.
12

---

[42] Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description multiple 3d objects. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 15225-15236 2023.
[43] Zicheng Zhang, Yi Zhu, Jianzhuang Liu, Xiaodan Liang, and Wei Ke. Coupalign: Coupling word-pixel with sentence-man mask alignments for referring image segmentation. Advances in Neural Information Processing Systems, 35:14729-147777 2022.
[44] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg- transformerr:: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 2928-2937, 2021.
[45] Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. In European Conference Computer Vision, pages 598 615, Springer, 2022.
13

---

Appendix
A
Semantic and Instance Labels vs. Binary Labels
The differences between semantic labels, instance labels, and binary labels are shown Fig
Semantic labels assign a category ID to each point in scene, with points belonging to the same category (e.g., chairs) sharing the same ID. Instance labels assign object ID each point within the same object (c.g., a specific chair), distinguishing different objects by assigning different instance IDs. Binary labels assign a Boolean value to each point in the scene; points that are part of the described object are assigned value of 1, and points that are not are assigned a value of O.
Therefore, compared to binary labels, the annotations of semantic labels and instance labels are more time consuming and labor intensive.
Semantic Label
Instance Label
Binary Label
Nx1
Z Nx1
{0, 10,1)xx Nx1
Figure 4: There different types of labels.
More Qualitative Results
We present our success cases and failure cases Fig 5 Our method accurately segments objects with clear queries. However, ambiguous descriptions can still confuse our model and leads to segment both the referred object and other similar objects.
Success cases Query
Ours
GT
Query
Ours
GT
(b) chair black and in the comee. chair left the others.
a cushion sofa in brown color. near the wall.
the coffee table between orange couches, the coffee table gray square.
this is a small round can. it the right of a totlet.
Failure cases
the booksheelf is left my sofa chair. the bookshelf has cream color with four sections.
are looking for the chair at the head table. it's near the door.
Figure Qualitative results of both the success cases and failure cases of LESS.
14

---

C Time Consumption Comparison
we evaluate the training and inference time both of T3 and X- X-ReiSeg 29 All experiments are conducted on an NVIDIA 4090 GPU and the number of batch sizes and epoch of three methods are kept the same. For the two-stage training and inference of TGNN and X-RefSeg. we followed the settings in their open source codes. We can find that our LESS consumes less time than both of TGNN and X-RefSeg in both training and inference.
As shown in Tab 7
Table The comparison of inference time and training time with previous work.
Inference
Inference (Per Scan) (ms)
Training (Stage 1) (h)
Training (Stage 2) (h)
Training (All) (h)
Method
(Whole Dataset) (min)
TGNN
27.98
176.57 126.23
156.02 156.02
8.53 7.59
164.55
X-RefSeg
20.00
163.61
Ours
7.09
44.76
40.89
D More Quantitative Results
concurrent work 3D-STMN 335 utilizes pre-trained 3D feature extractor 31 perform referring 3D segmentation. Given that their backbone is pre-trained on an instance segmentation task with semantic and instance label, it is reasonable conclude that their approach cannot be considered a label-efficient and single-stage method. For fair comparison, we follow the settings in their open source code and train their network from scratch, except for BERT [8] module. As shown in Tab our method overtakes 3D-STMN by 11.34%, 18.27% and 12.65% on mloU, Acc@0. and Acc@0.5 respectively.
Table 8: Quantitative results of 3D-STMN 33 and ours.
Method
mIoU
Acc@0.25
A.c@0.5
3D-STMN (from scratch)
22.40
34.96 53.23
17.23
Ours
33.74
29.88
E
More Ablation Results
E.1
Query Number
further conduct more ablation studies in terms of the number of queries on ScanRefer. As shown in Ta we ablate the number of queries as 5, 15, 20, it can be observed that keeping 20 queries brings higher accuracy while lower accuracy when using fewer queries. We suppose that fewer queries not help to learn comprehensive feature patterns while appropriate number of queries is enough to cover the needed feature patterns.
Table 9: The impact of the number of queries.
Num queries
mloU
Acc A.25
Aco@0.5
5
32.75
51.81
28.68
15
33.27
52.56
28.97
20
33.74
53.23
29.88
E.2
Mask Selection Strategy
We also conduct the ablation study which selects the mask with the highest score during QSA process shown in Tab I0 The results demonstrate that the aggregation of multiple masks yields superior
15

---

performance compared to the selection of single, highest-ranker mask. This approach facilitates the capture of subtle nuances and intricate details that may overlooked when relying on single mask alone. The incorporation of multiple masks offers more comprehensive and precise representation, ultimately enhancing the accuracy of the final model prediction.
Table 10: The impact of mask selection strategy.
Num of queries
mloU
Acc@0.25
A.c@0.5
Top-1 Ours
33.18 33.74
52.93 53.23
28.65
29.88
F Potential Negative Societal Impacts
Our method has ethical risk on datasets usage and privacy violation because all the datasets and tools are publicly available and transparent.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see the Abstract and Introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see the Limitations subsection precede the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please the Method section and Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [No]
Justification:
Once the paper is accepted, we will release our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
crsiiinnooccycl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.cc/pulli///uiii wwwpc//ggue////eeeeeoooc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see the Experiments section. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The experiments are conducted under fixed seed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please the Appendix section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //t::://neuisu.eeeeee https//cccccccccccccccccoooaaad
Answer: [Yes]
Justification: Our research is with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please the Appendix section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see the Experiment and Reference sections.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.reewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
21

---

Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "KbItwrpjco", "reviewer": "Reviewer_NB5W", "summary": "This paper addresses the problem of referring 3D Instance Segmentation, which segments all points belonging to an object in a 3D point cloud described by a query prompt. Previous methods use a two-stage approach requiring both instance and semantic labels, which is labor-intensive. The authors propose LESS (Label-Efficient and Single-Stage), a new pipeline that only requires binary mask supervision, reducing annotation costs. Key innovations include a Point-Word Cross-Modal Alignment module, a Query Mask Predictor module, a Sentence Query Alignment module, and two new losses for weaker supervision. LESS significantly outperforms existing methods on the ScanRefer dataset.", "questions": "1. How does the loss coefficient affect the results?\n2. What is the impact if there are less than 20 queries in Table 5?\n3. Are there any benchmarks other than ScanRefer? The main paper only shows the result on one benchmark, which may be not sufficient. How about Sr3D and Nr3D", "limitations": "Yes, this paper has a session discussing its limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clear and easy to understand. The proposed method is straightforward and well-explained\n2. Strong quantitative results are reported on the ScanRefer dataset.", "weaknesses": "1. No substantial novelty in module design or architecture, they are already proposed in other contexts:\n   + Query Mask Predictor: similar to Mask2Former: M2F3D: Mask2Former for 3D Instance Segmentation\n   + Point-Word Cross-Modal Alignment: similar to cross-attention\n   + Query-Sentence Alignment: cosine CLIP score\n2. The paper lacks explanations of why TGNN and X-RefSeg perform worse, and in which case these methods fail.\n3. The area regularization loss brings the most significant gain, but is not explored in detail, i.e. ablation of the loss coefficient.\n4. Running time compared to the two-stage approaches? If it is not significantly reduced, there is no advantage compared to the two-stage approaches. \n5. The results with Acc@0.5 are worse than those of TGNN and X-RefSeg (Table 1) while the results with Acc@0.25 are opposite indicating that the proposed method is better locate coarse location rather than exact matching as in TGNN and X-RefSeg.\n\nComment: TGNN and X-RefSeg use a pretrained 3DIS network, when training on this Referring 3DIS task, only need binary mask supervision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and constructive comments. Next we address to your concerns as follows:\n## Q1: For the novelty of the module design\nThe core motivation of our approach is to explore a novel single-stage R3DIS to fully embrace semantic concepts from text query and visual cues into a unified paradigm, while leveraging an efficient binary supervision. Most of existing works focus on two-stage pipeline to reach R3DIS: first instance proposal network and then a new network to classify these proposal into the corresponding text query, which separate the interaction between text query and visual cues. To investigate label-efficient and single-stage pipeline in R3DIS task, we delve into fusion mechanism for better aligning visual features with textual embeddings within a cross-attention manner. Specially, we design coarse-grained feature alignment and fine-grained feature alignment strategies, corresponding to our QMP and SQA, and PWCA, respectively. Besides, the utility of area regularization and point-to-point contrastive learning are effective to eliminate ambiguities among multiple objects and backgrounds, since single-stage approach possibly struggles with positive false and negative true samples without a well-trained proposal generator network. All of the proposed components in our paper lead to our competitive SoTA and pioneering single-stage R3DIS LESS. Moreover, our methods needs only binary mask annotations to train the model while the previous ones need heavy both instance labels and semantic labels, which strikes to label-efficient and single-stage approach for this task. \n## Q2: For the explanations of why TGNN and X-RefSeg perform worse and the cases they fail\nFirstly, as mentioned in line 31-34 in the paper, owing to accuracy of mode and the lacking of linguistic guidance, TGNN and X-RefSeg  may miss the target in the pre-segmentation stage. In this case, it is impossible to provide language-related high-quality instance proposals for the second stage. Besides, compared to our fine-grained and coarse-grained alignment, their visual-language alignment is weak, usually applying language guidance at the second stage only. We deduce these are the main reasons why both of them performed poorly. Moreover, if the targets are severely missed in the first stage or queries are complex, TGNN and X-RefSeg will perform worse.\n## Q3: For the loss coefficient of area regularization\nWe conduct further ablations in Q7 together.\n## Q4: For the running time compared to pervious two-stage approaches\nWe also conduct comparisons to address this concern, please kindly refer to the table of our **General Response to all Reviewers**.\n## Q5: For the results with Acc@0.5 are worse than those of TGNN and X-RefSeg while the results with Acc@0.25 are opposite\nSince TGNN and X-RefSeg are all two-stage methods, that their 3D instance semantic segmentation networks in the first stage are first well-trained by both instance labels and semantic labels to support higher precision instance proposals for the second stage than ours. This benefits for the Acc@0.5 metric. However, as shown in Q7, we can still find that our LESS also surpasses both TGNN and X-RefSeg in Acc@50, which indicates our model is better on not only locating coarse location but also exact matching, thanks to our elaborate alignment strategies for training between language information and visual cues.\n## Q6: For the comment of detailed setting of TGNN and X-RefSeg\nIn our main paper for the R3DIS task, TGNN and X-RefSeg both leverage a two-stage paradigm, which consists of a 3D instance semantic network and a instance matching network. Obviously, TGNN and X-RefSeg rely on instance labels, semantic labels and binary masks to finalize referring 3DIS task rather than \"only binary masks\", the efficient way we used. Noting that, we trained 3D spares U-Net’s architecture from scratch without its pretrained weights, which is different from previous methods.\n## Q7: For how does the loss coefficient affect the result\nWe further conduct more ablation studies of λp2p, λseg and λarea. It can be seen that λp2p presents marginal influence in terms of final performance and we tend to set with default 0.05. And reducing λseg is advantageous for obtaining decent performance. We analyze that this is due to the reduction of λseg has the effect of amplifying the impact of the area regularization loss, thereby facilitating a stronger discrimination between backgrounds and foregrounds, leading to local mask predictions. We still observe improvements when λarea increases, especially for Acc@50 metric, which indicates our area regularization loss can effectively exclude backgrounds and get more precise masks.  We have uploaded the corresponding results on the **PDF**, please kindly refer to that due to the Rebuttal Words Limitation.\n## Q8: For the ablation studies of the number of queries\nWe further conduct more ablation studies in terms of the number of queries on ScanRefer. As shown in the following table, we ablate the number of queries as {5, 15, 20}, it can be observed that keeping more queries brings higher accuracy while lower accuracy when using fewer queries. We suppose that fewer queries can not help to learn comprehensive feature patterns while an appropriate number of queries is enough to cover the needed feature patterns. \n| Num of queries |   mIoU    |  Acc@25   |  Acc@50   |\n| :------------: | :-------: | :-------: | :-------: |\n|       5        |   32.75   |   51.81   |   28.68   |\n|       15       |   33.27   |   52.46   |   28.97   |\n| 20 (Baseline)  | **33.74** | **53.23** | **29.88** |\n## Q9: For the validation on other dataset\nWe also conduct the experiments on the Nr3d and Sr3d datasets to address this concern, please kindly refer to the table of our **General Response to all Reviewers**.\n\n**Thanks again for these constructive comments, we will take all these experimental results into consideration into our revised version.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o3twNu9gKC", "reviewer": "Reviewer_6zjS", "summary": "This paper proposes a label-efficient single-stage method for referring 3D instance segmentation. Specifically, this method enhances feature extraction by integrating multi-modal features, using only binary labels for supervision. It achieves fine-grained alignment between points and words, distinguishing points or objects with similar features.", "questions": "See Weaknesses part.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces the single-stage method pioneered by R3DIS, which I find very appealing.\n\nThe structure of the article is clear and the writing is fluent.\n\nIt achieves state-of-the-art (SOTA) performance on the ScanRefer dataset.", "weaknesses": "I agree with the authors' point that annotating precise instance-level labels in tasks like open-vocabulary or 3D visual grounding is time-consuming and labor-intensive. Therefore, it is impressive that this method reduces label effort by approximately 10 times while maintaining SOTA performance. However, the authors only used mIoU and Acc as metrics for semantic segmentation in the experiments, without using any 3D instance segmentation metrics (e.g., AP). I hope the authors can explain in detail why they did not use instance segmentation metrics. If no instance segmentation metrics were used, why is this task named 3D Instance Segmentation?\n\nI hope the authors can validate their method's robustness across multiple datasets, as using only the ScanRefer dataset provides very limited persuasiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and constructive comments. Next we address your questions.\n\n## Q1: For the metrics of Referring 3D Instance Segmentation\n\nThanks for pointing out this question. For the metrics of Referring 3D Instance Segmentation task, our experimental metrics are constantly followed by the previous works[1,2], called Acc@kIOU. Acc@kIOU measures the accuracy of test samples with an IoU higher than specific threshold k, where k ∈ {0.25, 0.5}. And we can find that the calculation method is much similar with the APx metric for common instance segmentation task. Hence, we choose the used Acc@kIOU in our main paper as the previous methods for the same Referring 3D Instance Segmentation task. We hope this explanation can help to clarify your confusion.\n\n[1]. Text-guided graph neural networks for referring 3d instance segmentation.\n\n[2]. X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks.\n\n## Q2: For the validation on other dataset\n\nThanks for your detailed reading and pointing out this. We also conduct the experiments on the Nr3d and Sr3d datasets, as shown at General Response and in the following table. We can find that the performance of our LESS surpass the performance of TGNN by 3.6% and 1.8% on Nr3d and Sr3d respectively.  Such results present promising potential for the future works and shed new light on label-efficient and single-stage exploration for R3DIS task.\n\n|      | Method | Overall  |   Easy   |   Hard   | View-dep. | View-indep. |\n| :--: | :----: | :------: | :------: | :------: | :-------: | :---------: |\n| Nr3D |  TGNN  |   37.3   |   44.2   |   30.6   |   35.8    |    38.0     |\n|      |  Ours  | **40.9** | **47.4** | **35.2** | **39.7**  |  **41.9**   |\n| Sr3D |  TGNN  |   45.0   |   48.5   |   36.9   |   45.8    |    45.0     |\n|      |  Ours  | **46.8** | **50.5** | **37.8** | **46.6**  |  **46.3**   |\n\nThanks again for these constructive comments, we will take all these experimental results or further discussions into consideration into our revised version. If any questions, please kindly let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0sq19XxTzF", "reviewer": "Reviewer_pLC3", "summary": "The paper proposes LESS, a single-stage, label-efficient pipeline for referring 3D instance segmentation. It introduces fine-grained and coarse-grained cross-modal alignment to improve feature matching and employs additional losses to reduce irrelevant predictions. Experiments are conducted on the ScanRefer dataset.", "questions": "See weaknesses", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of this paper is meaningful. It proposes a label-efficient method for referring 3D instance segmentation, which can significantly reduce annotation costs.\n2. The paper is well-written and easy to follow.\n3. The experiments on the ScanRefer dataset show promising results.", "weaknesses": "1. The experiments are only conducted on the ScanRefer dataset, lacking validation on additional datasets like Nr3D/Sr3D used in TGNN[1].\n2. In the Query-Sentence Alignment module, the final mask prediction is produced by a weighted sum of the similarity score and mask prediction. The rationale for using a weighted sum is not intuitive; why not select the mask with the highest score?\n3. The paper lacks references to related literature. The Query Mask Predictor module is similar to the framework used in Mask3D[2], but Mask3D is not cited.\n\n[1]. Text-guided graph neural networks for referring 3d instance segmentation, AAAI 2021.\n[2]. Mask3d: Mask transformer for 3d semantic instance segmentation, ICRA 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you very much for your detailed and constructive comments.\n\n## Q1: For the validation on other dataset\nThanks for your detailed reading and pointing out this. We also conduct the experiments on the Nr3d and Sr3d datasets, as shown at General Response and in the following table. We can find that the performance of our LESS surpass the performance of TGNN by 3.6% and 1.8% on Nr3d and Sr3d respectively.  Such results present promising potential for the future works and shed new light on label-efficient and single-stage exploration for R3DIS task.\n\n|      | Method | Overall  |   Easy   |   Hard   | View-dep. | View-indep. |\n| :--: | :----: | :------: | :------: | :------: | :-------: | :---------: |\n| Nr3D |  TGNN  |   37.3   |   44.2   |   30.6   |   35.8    |    38.0     |\n|      |  Ours  | **40.9** | **47.4** | **35.2** | **39.7**  |  **41.9**   |\n| Sr3D |  TGNN  |   45.0   |   48.5   |   36.9   |   45.8    |    45.0     |\n|      |  Ours  | **46.8** | **50.5** | **37.8** | **46.6**  |  **46.3**   |\n\n## Q2: For whether not select the mask with the highest score\nThanks for your detailed reading and pointing out this. \n\nFirstly, using a weighted sum enables a soft decision-making process rather than a hard and binary choice. This can help in cases where multiple masks have similar scores, allowing the final prediction to be an aggregation of these maks, which can be more accurate than any single one.\n\nBesides, we also conduct the ablation study which selects the mask with the highest score in the ScanRefer dataset, as shown in the following table.  It can be shown that aggregating multiply masks leads to better performance than selecting only the highest one. This approach can help in capturing nuances and fine details that might be missed when relying on a solely single mask, which provide a more robust and accurate final prediction for the final model performance.\n\n|          |   mIoU    |  Acc@25   |  Acc@50   |\n| :------: | :-------: | :-------: | :-------: |\n|  Top-1   |   33.18   |   52.93   |   28.65   |\n| Baseline | **33.74** | **53.23** | **29.88** |\n\n\n## Q3: For the cite of Mask3D\n\nThanks for reminding us with this. We will add Mask3D[1] to our related literature and extend further discussions with Mask3D.\n\n[1]. Mask3d: Mask transformer for 3d semantic instance segmentation, ICRA 2023.\n\nThanks again for these constructive comments, we will take all these experimental results or further discussions into consideration into our revised version. If any questions, please kindly let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vjYtMouQOW", "reviewer": "Reviewer_Monv", "summary": "In this paper, the authors propose a Label-Efficient and Single-Stage referring 3D instance segmentation method, which is under the supervision of binary mask. They propose Point-Word Cross-Modal Alignment, Query Mask Predictor and Query Alignment modules to achieve cross-modal alignment. Besides, the area regularization loss and the point-to-point contrastive loss are introduced to eliminate interference caused by multiple objects and backgrounds. Experimental results show that the proposed method outperforms the existing state-of-the-art method on ScanRefer dataset with only the supervision of binary. This is a nice paper dealing an interesting research problem, presenting insights, and being well-written.", "questions": "1.\tWhy the proposed alignment modules work well? \n2.\tHow about the training time and inference time comparison?", "limitations": "No negative societal impact", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper firstly investigated the single stage referring 3D instance segmentation task to bridge the gap between detection and matching, only under the supervision of binary mask. The proposed method can provide valuable insights to the further research in multi-modal 3D scene understanding.\n2.\tTo align features among different modalities, such as fined-grained alignment between point and word features, coarse-grained alignment between masks and sentences, several alignments modules are well designed, which exhibit a certain level of novelty.\n3.\tExperimental results show that the proposed method outperforms the existing state-of-the-art method on ScanRefer dataset with only the supervision of binary label.\n4.\tThe paper is well-written and is easy to follow.", "weaknesses": "1.\tMethods TGNN and X-RefSeg are not evaluated with the RoBERTa backbone.\n2.\tThe explanation of why more queries and layers as shown in Tab. 5 and 6 do not bring performance gains are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that you recognize the significance of our work. We will respond to your concerns in the following:\n\n## Q1: For the validation of TGNN and X-RefSeg with the RoBERTa backbone\n\nThanks for your detailed reading and pointing out this. We reimplement the TGNN and X-RefSeg with the same RoBERTa backbone as ours to conduct experiments on ScanRefer benchmark, since these exisitng works have not applied RoBERTa backbone as language module for their experiments. We then follow their settings to train and validate the performances. As shown in the table below, we can obtain their newly corresponding results, 28.83% mIoU for TGNN (RoBERTa) and 30.72% mIoU for X-RefSeg (RoBERTa), which still shows inferior performance to ours 33.74%, which further validates the superiority of our single-stage prardigm. Besides, we observe that when the loss coefficient is set to 10, our LESS outperforms the performance of TGNN and X-RefSeg on all the metrics. More discussion will be added in our paper.\n\n|               Method               |   mIoU    |  Acc@25   |  Acc@50   |\n| :--------------------------------: | :-------: | :-------: | :-------: |\n|             TGNN (GRU)             |   26.10   |   35.00   |   29.00   |\n|            TGNN (BERT)             |   27.80   |   37.50   |   31.40   |\n|           TGNN (RoBERTa)           |   28.83   |   39.15   |   32.50   |\n|           X-RefSeg (GRU)           |   29.77   |   39.85   |   33.52   |\n|          X-RefSeg (BERT)           |   29.94   |   40.33   |   33.77   |\n|         X-RefSeg (RoBERTa)         |   30.72   |   41.54   |   34.42   |\n| Ours (RoBERTa) $\\lambda_{area}=1$  |   33.74   | **53.23** |   29.88   |\n| Ours (RoBERTa) $\\lambda_{area}=10$ | **35.08** |   52.30   | **34.43** |\n\n## Q2: For the more querie and layers do not bring performance gains\n\nThanks for pointing out this. With an excessive number of queries, too many queries become redundant, leading to overlapping or duplicate proposal masks. This redundancy does not contribute to better segmentation and can instead introduce noise, even degrading performance improvements. Besides, Increasing the number of layers can lead to overfitting, especially with the limited supervisory signals available in our label-efficient setting, binay mask supervision. \n\n## Q3: For reasons for good performance of proposed aligment modules\n\nThanks for your detailed reading and pointing out this. Point-Word Cross-Modal Alignment module aligns fined-grained point and word features. Also Query Mask Predictor module and Sentence Query Alignment modules make the coarse-grained alignment between masks and sentences. The coarse-grained  sentences-masks and fine-grained words-points effectively coupled to enhance the capability in capturing multi-modal context, which is helpful for our aligment modules. \n\n## Q4: For the training time and inference time\n\nThanks for your detailed reading and pointing out this.  As shown at General Respons and in the following table, we evaluate the training and inference time both of TGNN and X-RefSeg.  All experiments are conducted on an NVIDIA 4090 GPU and the batch sizes of three methods are kept the same. For the two-stage training and inference of TGNN and X-RefSeg, we followed the settings in their open source codes. We can find that our LESS consumes less time than both of TGNN and X-RefSeg in both training and inference. \n\n|  Method  | Inference (Whole Dataset) (min) | Inference (Per Scan) (ms) | Training (Stage 1) (h) | Training (Stage 2) (h) | Training (All) (h) |\n| :------: | :-----------------------------: | :-----------------------: | :--------------------: | :--------------------: | :----------------: |\n|   TGNN   |              27.98              |          176.57           |         156.02         |          8.53          |       164.55       |\n| X-RefSeg |              20.00              |          126.23           |         156.02         |          7.59          |       163.61       |\n|   Ours   |            **7.09**             |         **44.76**         |           -            |           -            |     **40.89**      |\n\nThanks again for these constructive comments, we will take all these experimental results or further discussions into consideration into our revised version. If any questions, please kindly let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response to all Reviewers:

Dear all Reviewers,

We would like to express our great thank and appreciation to all the reviewers for their constructive and thoughtful comments on our submission. In this rebuttal response, we address the questions raised by the reviewers and clarify how we will revise our manuscript in repsonse to the offered comments. Our paper aims at proposing a novel single-stage Referring 3D Instance Segmentation to fully embrace semantic concepts from text query and visual cues into a unified paradigm, which differs from most exisitng two-stage works necessitating both instance and semantic labels for each object first and using a separate interaction between text and visual cues during the training. We are motivated and glad that our proposed method is well-explained, appealing and exhibiting a certain level of novelty (Reviewer NB5W, 6zjS, pLC3 and Monv), is of clear writing and structure for understanding (Reviewer NB5W, 6zjS, pLC3 and Monv), demonstrates competitive SoTA performance on ScanRefer benchmark (Reviewer NB5W, 6zjS, pLC3 and Monv) and sheding meaningful research for the future to enable label-efficient learning for this community (Reviewer pLC3 and Monv). 

Here we answer some general questions of reviewers:

## Q1. For the evaluation on more benchmarks, especially on Nr3D and Sr3D
Thanks for the constructive suggestions to conduct more validations on other benchmarks. 

We further employ our method on extra Nr3D and Sr3D as shown in table below. Following the evaluation setting in TGNN, we can observe that the performances of our LESS consistently surpasses the performances of TGNN by 3.6% and 1.8% on Nr3d and Sr3d, respectively. This further demonstrates the effiectiveness of our method, still achieving competitive performances on additional Nr3D and Sr3D, which presents promising potential for the future works. We will add these new experimental results to our revised manuscript.

|      | Method | Overall  |   Easy   |   Hard   | View-dep. | View-indep. |
| :--: | :----: | :------: | :------: | :------: | :-------: | :---------: |
| Nr3D |  TGNN  |   37.3   |   44.2   |   30.6   |   35.8    |    38.0     |
|      |  Ours  | **40.9** | **47.4** | **35.2** | **39.7**  |  **41.9**   |
| Sr3D |  TGNN  |   45.0   |   48.5   |   36.9   |   45.8    |    45.0     |
|      |  Ours  | **46.8** | **50.5** | **37.8** | **46.6**  |  **46.3**   |

## Q2. For the comparisons with pervious two-stage approaches, especially with TGNN and X-RefSeg

Thanks for the constructive suggestions regarding more comprehensive comparisons with two-stage methods. 

Firstly, we compare the training and inference time of our method with both TGNN and X-RefSeg, shown in the following table. All comparative experiments are conducted on a NVIDIA 4090 GPU machine and the batch sizes of three methods are kept the same. For the two-stage training and inference of TGNN and X-RefSeg, we strictly follow the settings in their open source codes. From the table, we can find that, our LESS consumes less time than both of TGNN and X-RefSeg in terms of training and inference cost, which showcases the promising potential of single-stage approach, our proposed LESS.  


|  Method  | Inference (Whole Dataset) (min) | Inference (Per Scan) (ms) | Training (Stage 1) (h) | Training (Stage 2) (h) | Training (All) (h) |
| :------: | :-----------------------------: | :-----------------------: | :--------------------: | :--------------------: | :----------------: |
|   TGNN   |              27.98              |          176.57           |         156.02         |          8.53          |       164.55       |
| X-RefSeg |              20.00              |          126.23           |         156.02         |          7.59          |       163.61       |
|   Ours   |            **7.09**             |         **44.76**         |           -            |           -            |     **40.89**      |


Secondly, we reimplement the TGNN and X-RefSeg with the same RoBERTa backbone as ours to conduct experiments on ScanRefer benchmark, since these exisitng works have not applied RoBERTa backbone as language module for their experiments. We then follow their settings to train and validate the performances. As shown in table below, we can obtain their newly corresponding results, 28.83% mIoU for TGNN (RoBERTa) and 30.72% mIoU for X-RefSeg (RoBERTa), which still shows inferior performance to ours 33.74%, which further validates the superiority of our single-stage prardigm.

|               Method               |   mIoU    |  Acc@25   |  Acc@50   |
| :--------------------------------: | :-------: | :-------: | :-------: |
|             TGNN (GRU)             |   26.10   |   35.00   |   29.00   |
|            TGNN (BERT)             |   27.80   |   37.50   |   31.40   |
|           TGNN (RoBERTa)           |   28.83   |   39.15   |   32.50   |
|           X-RefSeg (GRU)           |   29.77   |   39.85   |   33.52   |
|          X-RefSeg (BERT)           |   29.94   |   40.33   |   33.77   |
|         X-RefSeg (RoBERTa)         |   30.72   |   41.54   |  **34.42**   |
| Ours (RoBERTa)   |   **33.74**   | **53.23** |   29.88   |


Overall, we can find that our approach can not only exhibits much more efficient training/inference cost but also superior performances when equiping TGNN and X-RefSeg with the same RoBERTa backbone. Thanks for the thoughful comments from all reviewers, we will take these comparisons into consideration by adding these experimental results in our revised version. Next, for the specific questions of each kind reviewer, we answer them in the reply to each one.

The following PDF document contains tables for three ablation experiments for Reviewer NB5W, the inclusion of which is due to space limitations.

Meta-review: The reviewers all agree on the proposed label efficient single stage 3d referring expression modeling being either reasonable and straightforward or a sufficiently novel contribution. The rebuttal and discussion clarified some question about evaluation and setup, and overall the authors are encouraged to take into account the reviewer comments.
Decision: Accept (poster)