Title: LESS: Label-Efficient and Single-Stage Referring 3D Segmentation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Referring 3d segmentation

label-efficient

single-stage

cross-modal
Abstract: Referring 3D Segmentation is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. Previous works perform a two-stage paradigm, first conducting language-agnostic instance segmentation then matching with given text query. However, the semantic concepts from text query and visual cues are separately interacted during the training, and both instance and semantic labels for each object are required, which is time consuming and human-labor intensive. To mitigate these issues, we propose a novel Referring 3D Segmentation pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. Specifically, we design a Point-Word Cross-Modal Alignment module for aligning the fine-grained features of points and textual embedding. Query Mask Predictor module and Query-Sentence Alignment module are introduced for coarse-grained alignment between masks and query. Furthermore, we propose an area regularization loss, which coarsely reduces irrelevant background predictions on a large scale. Besides, a point-to-point contrastive loss is proposed concentrating on distinguishing points with subtly similar features. Through extensive experiments, we achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels. Code is available at https://github.com/mellody11/LESS.

LESS: Label-Efficient and Single-Stage Referring 3D Segmentation

Abstract

Referring 3D Segmentation is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. Previous works perform a two-stage paradigm, first conducting language-agnostic instance segmentation then matching with given text query. However, the semantic concepts from text query and visual cues are separately interacted during the training, and both instance and semantic labels for each object are required, which is time consuming and human-labor intensive. To mitigate these issues, we propose a novel Referring 3D Segmentation pipeline, L abelE fficient and S ingleS tage, dubbed LESS , which is only under the supervision of efficient binary mask. Specifically, we design a Point-Word Cross-Modal Alignment module for aligning the fine-grained features of points and textual embedding. Query Mask Predictor module and Query-Sentence Alignment module are introduced for coarse-grained alignment between masks and query. Furthermore, we propose an area regularization loss, which coarsely reduces irrelevant background predictions on a large scale. Besides, a point-to-point contrastive loss is proposed concentrating on distinguishing points with subtly similar features. Through extensive experiments, we achieve stateof-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels. Code is available at https://github. com/mellody11/LESS .

1 Introduction

Referring 3D Segmentation task aims to segment the specific object from a 3D point cloud scene with a free-form natural language expression. It allows users to interact and analyze 3D data through verbal instructions or queries. This approach is particularly beneficial in applications that necessitate direct interaction with 3D environments, such as augmented reality (AR) systems, embodied-AI, robotics, virtual reality (VR) environments, and in fields like architecture and medical imaging, where precise identification and segmentation of objects based on descriptive queries can significantly enhance user experience and operational efficiency.

Previous Referring 3D Segmentation [13; 29] mainly leverage a two-stage workaround, as shown in Fig.1 (a). They typically adopt a 3D instance semantic segmentation network to get the instance proposals at the first stage. The predicted instance proposals will be utilized to match with the queries and finally get the final prediction mask according to the matching score. Although those method have achieved remarkable performance, there still exist some problems. First, owing to the large-scale and

Pre-segmentation Stage

U-Net

Instance label

Semantic label

Instance proposals

The chair is to

the right of the

round table.

Query

Instance Predictor

Predicted Score

Ground Truth

Loss

Instance Matching Stage

U-Net

3D

The chair is to the right

of the round table.

Query

Text Encoder

Mask Predictor

Single Stage

Binary Label

Predicted Mask

Loss

Loss

Loss

(a) Previous two-stage method

(b)  Our single-stage method

Sentence

Word

irregular 3D point clouds, some instance proposals may leave out the target in the pre-segmentation stage. Besides, lacking of linguistic guidance in the segmentation stage fails to focus on the objects that are more essential to the referring task. Moreover, existing Referring 3D Segmentation utilizes both instance labels and semantic labels to segment target proposal rather than binary mask used in referring image segmentation, which is more time consuming and labor intensive.

To address the aforementioned problems, we propose a L abelE fficient and S ingleS tage Referring 3D Segmentation method, namely LESS , which is under the supervision of binary mask, as shown in Fig.1 (b). We first process the query with text encoder to get the word-level feature and sentence-level feature. Then we extract the multi-modal feature with the guidance of text feature through a 3D sparse U-Net. Finally, the mask predictor aligns multi-modal features with language features, and directly predicts the mask of the described object. Here only object mask serves as the label to supervise the whole training procedure.

However, 3D point cloud inherently provide a higher level of complexity and a large scale. There exists numerous different objects in a single 3D scene compared to the referring image segmentation task. Besides, binary mask has less semantic meanings compared to instance labels and semantic labels. These challenges make it difficult to supervise our model to localize and segment target objects with only binary mask. Therefore, we propose to alleviate these problems by some ways, as shown in Fig.2. Firstly, to facilitate fine-grained alignment between points and words, we propose Point-Word Cross-Modal Alignment module. The PWCA module utilizes cross-modal attention in the multi-modal context to align textual features extracted by the text encoder with point cloud features, followed by further extraction of useful multi-modal information using robust 3D sparse convolutional layers. Thus we can extract a more semantic meaning fused feature. Meanwhile, we employ Query Mask Predictor, which utilizes the extracted multi-modal features to decode learnable query embeddings, generating candidate masks. By introducing the Query-Sentence Alignment modules, we can compute similarity between the decoded query embeddings and sentence features, using the similarity as weights to perform weighted summation of the candidate masks to produce the final mask. To address the significant interference caused by multiple objects and backgrounds, we propose an area regularization loss and a point-to-point contrastive loss. Area regularization loss reduces irrelevant background predictions by constraining the probabilities of the predicted mask, while point-to-point contrastive loss constrains the distances between positive and negative points in the latent space to achieve better segmentation.

To summarize, our contributions are as follows:

We propose a new Referring 3D Segmentation method LESS, which directly performs Referring 3D Segmentation at a single stage to bridge the gap between detection and matching under the supervision of binary mask. To the best of our knowledge, LESS is the first work investigating label-efficient and single-stage in Referring 3D Segmentation task.

Our LESS utilize a Point-Word Cross-Modal Alignment module to align fined-grained point and word features. Besides, we employed Query Mask Predictor module and QuerySentence Alignment modules for coarse-grained alignment between masks and sentences. Moreover, the area regularization loss and the point-to-point contrastive loss are introduced to better support to eliminate interference caused by multiple objects and backgrounds.

Extensive experiments confirm the effectiveness of our method. Our method outperforms the existing state-of-the-art method on ScanRefer dataset with only the supervision of binary mask. Our LESS and its results provide valuable insights to improve further research of label-efficient and single-stage Referring 3D Segmentation.

2 Related Works

2.1 Referring 3D Segmentation

Referring 3D Segmentation has previously received limited exploration, however, with the advancements in multi-modal learning and embodied AI, it is set to attract increasing interest in the future. TGNN [13] is the first to introduce Referring 3D Segmentation task. They initially trained an instance segmentation network, followed by a Graph Neural Network (GNN) to learn features of instances and their relationships guided by linguistic information. Building on TGNN, X-RefSeg [29] developed a cross-modal graph. They employed an GNN to model the texture and spatial relationships of instances, and refining the results through inference and matching processes.

2.2 3D Visual Grounding

3D visual grounding aims to locate the object within point clouds mentioned by free-form natural language descriptions. Most methods follow a two-stage detection-then-matching pipeline. Initially, they utilize a pre-trained 3D detector [28; 23] or segmenter [15; 33] to extract object representations. Subsequently, these methods align text features with object features to identify the best-matched object. Researchers primarily concentrate on the second stage which involves modeling object relationships and exploring feature fusion between language and objects. Methods employed include multi-modal feature concatenation [3; 1], attention-based multi-modal feature alignment [44; 11], graph neural network-based reasoning [13; 41], and the alignment of visual and language features aided by 2D images [40; 42; 38; 37].Other researchers have investigated single-stage approaches for 3D visual grounding. 3D-SPS [25] views the task as key point selection, progressively identifying keypoints with the guidance of language and directly locates the target. BUTD-DETR [14] employs a transformer decoder [2] to identify described objects using language cues and proposal boxes. Building on this, EDA [36] enhances dense alignment between objects and point clouds by explicitly decoupling textual attributes from sentences.

The distinction between 3D visual grounding and Referring 3D Segmentation lies in the latter's enhanced localization precision, offering significant value in applications like robotic grasping. While traditional single-stage 3D visual grounding methods regress a 3D bounding box, our singlestage approach decodes a binary mask for the entire point cloud scene, presenting a more complex challenge.

2.3 Referring Image Segmentation

Referring image segmentation is a visual task that involves pixel-level segmentation of an image's target object based on a referring expression. Early approaches [12; 20; 18; 4; 17] utilize CNNs and RNNs to extract features and then perform simple concatenation for multi-modal feature fusion. Subsequent research [16; 19; 9] adopt transformer models for more effective feature extraction and fusion. Recent studies [39; 10; 43] focus on identifying optimal positions for language-vision alignment. Additionally, some methods [34; 5] have enhanced alignment between language and pixel features by designing specialized loss functions. Other approaches [21; 45] treat referring image

PWCA

Text

Encoder

1

2

3

3

2

1

Sparse 3D Feature Extractor

Query

Mask

Predictor

k

Queries

QSA

PWCA

Point-Word Cross-Modal Alignment

Cross-Attention

Q

MLP

K

V

Multi-Modal Features

Query-Sentence Alignment

Similarity

Score

Sentence

Features

Element-wise addition

Matrix multiplication

Seg Loss

Contrastive Loss

The chair is to the right

of the round table.

Queries

Embedding

σ

Sigmoid Function

Area Loss

Query Mask Predictor

Query Decoder

Q

K

V

Shared

MLP

Mask Predictor

Queries  Embedding

σ

Attention Mask

Proposal

Masks

0.2

m

×

m

×

PWCA

Point Features

Word Features

Fused Features

k

Queries

segmentation as an auto-regressive vector generation problem, creating masks from generated closed vectors.

Methods for referring image segmentation have been extensively explored, yet they cannot be directly applied to Referring 3D Segmentation due to the inherent challenges of point cloud scenes. Additionally, unlike referring image segmentation methods that typically employ pre-trained visual encoders, Referring 3D Segmentation lacks such resources, compelling reliance on limited supervisory signals for model training.

3 Method

The overall framework of our proposed LESS is shown in Fig.2, which leverages Point-Word CrossModal Alignment and Query-Sentence Alignment to facilitate multi-modal interaction. In this section, we start by introducing our visual and text feature extractor in Sec.3.1. Then Query Mask Predictor and Query-Sentence Alignment is detailed in Sec.3.2 and Sec.3.3. Finally in Sec.3.4, we introduce our area regularization loss and point-to-point contrastive loss.

3.1 Visual and Text Feature Extractor

Sparse 3D Feature Extractor. The point cloud scene P ∈ R N × 6 contains N points in the scene, and each point is represented with six dimensions of RGBXYZ attributes. We first voxelize points into regular voxels and adopt a sparse 3D U-Net [31; 15] to extract point-wise fused feature F ∈ R N × C . Here the encoder part of the U-Net has 5 stages and the feature from the i -th encoder stage is denoted as V i .

Text Encoder. Given the query sentence T with L words, a text encoder is used to embed the query into C -dimensional feature vectors. In this paper we choose GRU [6], BETR [8] and RoBERTa [22] as our text encoder respectively and fine-tune the BERT or RoBERTa during training. Finally, we can get both word features W ∈ R L × C and sentence features S ∈ R C after the text encoder.

Point-Word Cross-Modal Alignment. Due to the lack of such rich annotations as [13; 29], it is crucial for our model to learn the relationship between fine-grained word-level features and pointlevel features in such a point-level segmentation task. Meanwhile, we notice that leveraging the rich convolutional layers in the encoder to excavate multi-modal context is effective way to extract language-aware visual feature. Therefore, we design Point-Word Cross-Modal Alignment (PWCA) module, as shown in the lower left part in Fig.2, which contains a standard cross-attention module [32] and an MLP-Tanh gate. Cross-attention module aligns point-wise and word-wise feature to get the language-aware visual feature. A nonlinear tanh gate is adopted to prevent the fused signal from overwhelming the original signal. Given the multi-scale point cloud features { V i ∈ R N i × C i } 5 i =1 , we simply project the word feature W ∈ R L × C to W i ∈ R L × C i . PWCA can be formulated as follow:

where i indicates the i -th stage of the encoder part of our sparse 3D feature extractor. Here we use V i as the query and W i as the key and value for cross attention.

3.2 Query Mask Predictor

Inspired by [31; 30], Query Mask Predictor (QMP) module, as shown in Fig.2, takes fused feature F ∈ R N × C and learnable queries Q 0 ∈ R K × C as input and progressively distinguishes the referring target by multi-layer cross-modal transformers. Finally, we extract the proposal masks M m ∈ R K × N based on queries embeddings Q m ∈ R K × C and fused feature F .

Query Decoder. As illustrated in the lower middle section of Fig.2. Here query decoder is comprised m -layer masked cross-attention modules [32], where the fused features F are served as keys and values and A j -1 ∈ { 0 , 1 } K × N is utilized as the attention mask. Therefore, in the j -th layer, the learnable queries Q j -1 capture multi-modal contextual information from fused feature F via a query decoder to obtain the query embeddings Q j ∈ R K × C .

Mask Predictor. First the fused feature F ∈ R N × C is processed by a Shared MLP, which indicates each Query Mask Predictor layer shares the same MLP. We perform the matrix multiplication on the new F and Q j ∈ R K × C to generate proposal mask predictions M j ∈ R K × N . After applying a sigmoid function whose threshold of 0.5, we can get the new binary attention mask A j ∈ { 0 , 1 } K × N .

Finally, the Query Mask Predictor is formally described as follows:

where ⊤ denotes the transpose operation, ⊗ represents matrix multiplication, and σ refers to the sigmoid function. Here we need to note that the Q 0 is randomly initialized. Therefore we can leverage the function (3-4) to initialize the attention mask A 0 .

3.3 Query-Sentence Alignment

The Referring 3D Segmentation task involves the segmentation of a solitary target object according to the query. Previous modules focus on extracting language-aware visual feature from aligning point-wise feature and word-wise feature, which lacks a comprehensive perception of the entire query sentence. Therefore, we adopt Query-Sentence Alignment (QSA) to better align the query feature with sentence-level feature. We perform the matrix multiplication on Q m ∈ R K × C and S ∈ R C to get the their similarity score R ∈ [0 , 1] K . Formally, Query-Sentence Alignment can be represented as:

The final mask prediction ˆ M ∈ R N is produced by weighted sum of similarity score R and proposed mask prediction M m ∈ R K × N . Finally, we use a sigmoid function and a threshold of 0 . 5 to produce the final predicted binary mask ˆ Y ∈ { 0 , 1 } N :

3.4 Loss Function

Segmentation Loss. Different from previous work [13; 29], we take the Referring 3D Segmentation task as segmentation task with only binary mask Y ∈ { 0 , 1 } N . Here we utilize the Binary CrossEntropy (BCE) loss function to compute the segmentation loss, which can be formulated as:

Area Regularization Loss. For Referring 3D Segmentation task, each query always corresponds to one target object in the point cloud scene. The target objects occupy a smaller area in the large scale of 3D point cloud. As a result, the predicted mask often includes backgrounds or other objects. To address this, we propose a region regularization loss, which promotes the network to predict a smallest mask by minimize the output probability of each point, formulated as:

By combining with the segmentation loss, we intend to segment only the most probable regions while reducing segmentation of large-scale irrelevant background areas.

Point-to-Point Contrastive Loss. Area regularization loss uniformly penalizes the predicted probabilities of all points, which can reduce the majority of the background points. However, the network struggles to differentiate between objects that possess characteristics similar to those described target object in the latent space. Therefore, we propose a point-to-point contrastive loss [26] that pull the points from the described object together and push away the rest points:

where P is the positive point set from the described object, N is the negative point set from the background, P i denotes the L2-normalized feature vector of i -th positive points from F , while N j denotes the L2-normalized feature vector of j -th negative points from F , P avg is the average feature vector of positive point P avg = 1 P ∑ |P| i =1 P i , and τ is the hyper-parameter. The contrastive loss promotes the network to distinguish the described object from the adjacent background points in a fined-grained manner.

The overall loss function is the weighted sum of the aforementioned three loss functions:

where { λ seg , λ area , λ p2p } is set to { 1 , 1 , 0 . 05 } in practice to balance the contrastive loss because of the large amount of points.

4 Experiments

4.1 Dataset and Experiment Settings

ScanRefer. ScanRefer [3] is a dataset for 3D referring expression comprehension tasks such as 3D visual grounding and 3D referring instance segmentation. It contains 51 , 583 queries of 11 , 046 objects from 800 ScanNet [7] scenes. Each scene contains 13 . 81 objects and 64 . 48 queries on average.

‡ The evaluate of label effort is base on a single sample.

Evaluation Metric. Following previous work [13; 29], we use the mean intersection-over-union ( mIoU ), and Acc@kIOU as the evaluation metrics. The mIoU is the average of the IoU over all test samples and the Acc@kIOU measures the accuracy of test samples with an IoU higher than the threshold k , where k ∈ { 0 . 25 , 0 . 5 } . We use A@25 and A@50 for brevity in some of the following tables.

Implementation Details. We adopt the 3D spares U-Net [31] as our 3D feature extractor. we explore multiple text encoders, i.e., GRU [6], BERT [8] and RoBERTa [22] in our experiments for comparative analysis. For BERT and RoBERTa, we use the official pre-trained weights and fine-tune them during training. We set the number of queries K to 20 and use a single layer for QMP module. We set an initial learning rate of 2e-5 for the text encoder and 1e-4 for the others. We reduce the learning rate by a multiplicative factor of 0.95 each epoch and adopt Adam [24] as our optimizer. The weights of our loss function { λ seg , λ area , λ p2p } is set to { 1 , 1 , 0 . 05 } . We train for 64 epochs with a batchsize of 14, and all experiments are implemented on PyTorch [27].

4.2 Comparison with State-of-the-Art Methods

(c)

this is a brown

chair facing to the

right. it is sturdy

and made of a

brown material

(d)

this is a tan table.

it is in between

trash cans.

(a)

this is a tan

cabinet. it is

to the left of a

chair.

(b)

the lamp has a

white shade. the

lamp is behind a

black chair.

GT

Input

Query

(e)

this lamp is on

the top left. it is

white.

十 Lseg

+Lp2p

J+

area

4.3 Ablation Studies.

We conducted several ablation studies to evaluate the effectiveness of the key components in our proposed network. In these studies, all other components and hyper-parameter settings are kept consistent with the aforementioned experiments, except for the component being ablated.

Module Ablation. In this ablation study, we evaluate the effectiveness of the QSA and PWCA modules. As indicated in Table2, the ablation model (a) only retain the sparse 3D feature extractor and query mask predictor. Here we perform element-wise addition of the word features W to the multiscale features { V i } 5 i =1 instead of PWCA and project the proposed mask prediction M m ∈ R K × N to the final mask prediction ˆ M ∈ R N via an MLP instead of QSA. We set model (a) as the baseline of our experiment. Compared to model (a), model (b) adopts PWCA module and we find that the model performance increase greatly from 32.66% to 33.44%. This observation proves that PWCA facilitate fine-grained cross-modal alignment between points and words, which is more effectively to leverage the rich convolutional layers in the encoder to excavate multi-modal context. When we introduce QSA to model (b), as shown in model (c), we can find that the performance of mIoU is improved from 33.44% to 33.74%. Such results indicates that the fine-grained point-word alignment of PWCA and the coarse-grained query-sentence alignment of QSA effectively coupled to enhance the capability in capturing multi-modal context.

Loss Ablation. As demonstrated in 3.4, we refine predicted masks from coarse to fine by introducing two loss functions, i.e., area regularization loss L area and point-to-point contrastive loss L p 2 p . We successively add the loss functions and the results are shown in Tab.3. The model (a) is only supervised by segmentation loss L seg . When we introduce the L area into model (a), as shown in model (b), the performance greatly increase from 25.86% to 31.04% on mIoU, which indicates our

area loss can exclude a significant number of irrelevant background points. Moreover, we successively add the contrastive loss L p 2 p , as shown in the model (c). The performance is improved from 31.04% to 33.74%, which proves that the contrastive loss can make the model more focused on the target area rather than others.

Qualitative results are shown in Fig.3, it indicates that: i) When only the segmentation loss L seg is applied (column 3) , the predicted masks include many points from other regions. ii) After adding the L area , most of the irrelevant points disappear (row a, row d) . iii) After incorporating the L p 2 p , objects that were previously difficult to distinguish due to their similarity are successfully separated, and the predictions are close to the ground-truth.

Both quantitative and qualitative experiment demonstrate that our proposed loss function effectively reduces large-scale background misclassifications and distinguish objects or points with the similar characteristics.

4.4 Extension Experiments

Linguistic Features at Different Levels of Granularity. In this section, we will investigate the impact of linguistic features at different levels of granularity, as shown in Table.4. The first row represent that we leverage word-level features in both PWCA module and QSA module as text features. The second row indicates we utilize sentence-level features in both modules. We can find that sentence-only method even outperforms the word-only one. Referring 3D Segmentation task involves the segmentation of a solitary target object. This mandates a more profound and thorough comprehension of the semantic information conveyed by the sentence, extending beyond a mere focus on its individual words. The last row indicates the word-level features are utilized in PWCA module and sentence-level features are leveraged in QSA module. We can find that the performance of it outperforms two introduced above, which proves that fine-grained word-level feature is also helpful in extracting the 3D language-aware visual feature.

Layer and Query Number of QMP. We also investigate the performance on different query and layer numbers in the QMP module. As shown in Tab.5 and Tab.6, we find that too many queries and layers do not bring performance gains for Referring 3D Segmentation task. As a result, we set 20 queries and 1 QMP layer as the default configuration after balancing performance and efficiency.

4.5 Limitations

The limitations of LESS due to the inherent complexity of 3D point clouds and the ambiguous queries, although we have made significant improvements on previous methods. The scarcity of detailed semantic annotations are still challenging our model from distinguishing multiple similar objects. These limitations could guide our future work.

5 Conclusions

In this paper, we propose LESS, a label-efficient and single-stage approach for Referring 3D Segmentation. Specifically, our LESS enhances feature extraction by integrating multi-modal features and employs progressive constraints on predicted masks, achieving fine-grained alignment between points and words and distinguishing between points or objects with similar characteristics. Comprehensive experiments demonstrate that our single-stage method outperforms existing two-stage approaches on ScanRefer dataset, using only the binary labels as supervision. Though our framework still has some limitations, we believe that solving the Referring 3D Segmentation task only using the binary labels is a new and promising research, and we hope that LESS can serve as a simple but powerful baseline to inspire future research on Referring 3D Segmentation.

References

Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 , pages 422-440. Springer, 2020.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020.

Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision , pages 202-221. Springer, 2020.

Ding-Jie Chen, Songhao Jia, Yi-Chen Lo, Hwann-Tzong Chen, and Tyng-Luh Liu. See-throughtext grouping for referring image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7454-7463, 2019.

Yong Xien Chng, Henry Zheng, Yizeng Han, Xuchong Qiu, and Gao Huang. Mask grounding for referring image segmentation. arXiv preprint arXiv:2312.12198 , 2023.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5828-5839, 2017.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16321-16330, 2021.

Guang Feng, Zhiwei Hu, Lihe Zhang, and Huchuan Lu. Encoder fusion network with coattention embedding for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15506-15515, 2021.

Dailan He, Yusheng Zhao, Junyu Luo, Tianrui Hui, Shaofei Huang, Aixi Zhang, and Si Liu. Transrefer3d: Entity-and-relation aware transformer for fine-grained 3d visual grounding. In Proceedings of the 29th ACM International Conference on Multimedia , pages 2344-2352, 2021.

Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14 , pages 108-124. Springer, 2016.

Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 1610-1618, 2021.

Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In European Conference on Computer Vision , pages 417-433. Springer, 2022.

Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and Pattern recognition , pages 4867-4876, 2020.

Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Restr: Convolutionfree referring image segmentation using transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18145-18154, 2022.

Jinlong Li, Zequn Jie, Xu Wang, Xiaolin Wei, and Lin Ma. Expansion and shrinkage of localization for weakly-supervised semantic segmentation. In Advances in Neural Information Processing Systems , volume 35, pages 16037-16051, 2022.

Jinlong Li, Zequn Jie, Xu Wang, Yu Zhou, Xiaolin Wei, and Lin Ma. Weakly supervised semantic segmentation via progressive patch learning. IEEE Transactions on multimedia , 2022.

Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. Advances in neural information processing systems , 34:19652-19664, 2021.

Ruiyu Li, Kaican Li, Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Referring image segmentation via recurrent refinement networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5745-5753, 2018.

Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18653-18663, 2023.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2949-2958, 2021.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16454-16463, 2022.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9277-9286, 2019.

Zhipeng Qian, Yiwei Ma, Jiayi Ji, and Xiaoshuai Sun. X-refseg3d: Enhancing referring 3d instance segmentation via structured cross-modal graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4551-4559, 2024.

Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 8216-8223. IEEE, 2023.

Jiahao Sun, Chunmei Qing, Junpeng Tan, and Xiangmin Xu. Superpoint transformer for 3d scene instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 2393-2401, 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang D Yoo. Softgroup for 3d instance segmentation on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2708-2717, 2022.

Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11686-11695, 2022.

Changli Wu, Yiwei Ma, Qi Chen, Haowei Wang, Gen Luo, Jiayi Ji, and Xiaoshuai Sun. 3d-stmn: Dependency-driven superpoint-text matching network for end-to-end 3d referring expression segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 5940-5948, 2024.

Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. Eda: Explicit text-decoupling and dense alignment for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19231-19242, 2023.

Xiaoxu Xu, Yitian Yuan, Jinlong Li, Qiudan Zhang, Zequn Jie, Lin Ma, Hao Tang, Nicu Sebe, and Xu Wang. 3d weakly supervised semantic segmentation with 2d vision-language guidance. arXiv preprint arXiv:2407.09826 , 2024.

Xiaoxu Xu, Yitian Yuan, Qiudan Zhang, Wenhui Wu, Zequn Jie, Lin Ma, and Xu Wang. Weakly-supervised 3d visual grounding based on visual linguistic alignment. arXiv preprint arXiv:2312.09625 , 2023.

Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18155-18165, 2022.

Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: 2d semantics assisted training for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1856-1866, 2021.

Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1791-1800, 2021.

Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15225-15236, 2023.

Zicheng Zhang, Yi Zhu, Jianzhuang Liu, Xiaodan Liang, and Wei Ke. Coupalign: Coupling word-pixel with sentence-mask alignments for referring image segmentation. Advances in Neural Information Processing Systems , 35:14729-14742, 2022.

Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2928-2937, 2021.

Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. In European Conference on Computer Vision , pages 598-615. Springer, 2022.

Appendix

A Semantic and Instance Labels vs. Binary Labels

The differences between semantic labels, instance labels, and binary labels are shown in Fig.4.

Semantic labels assign a category ID to each point in a scene, with points belonging to the same category (e.g., chairs) sharing the same ID. Instance labels assign a object ID to each point within the same object (e.g., a specific chair), distinguishing different objects by assigning different instance IDs. Binary labels assign a Boolean value to each point in the scene; points that are part of the described object are assigned a value of 1, and points that are not are assigned a value of 0.

Therefore, compared to binary labels, the annotations of semantic labels and instance labels are more time consuming and labor intensive.

Semantic Label

Instance Label

Binary Label

{0,1}Nx1

B More Qualitative Results

We present our success cases and failure cases in Fig.5. Our method accurately segments objects with clear queries. However, ambiguous descriptions can still confuse our model and leads to segment both the referred object and other similar objects.

GT

Query

Ours

(a)

a cushion sofa in

brown color. it is

near the wall.

(c)

the coffee table is

between orange

couches. the coffee

(d)

this is a small

round trash can.

it is to the right

of a toilet.

(e)

you are looking

for the chair at the

head of the table.

it's near the door.

(f)

the bookshelf is left

of my sofa chair.

the bookshelf has a

cream color with

four sections.

Success cases

Failure cases

Ours

GT

(b)

the chair is black

and in the corner.

the chair left of

the others.

Query

C Time Consumption Comparison

As shown in Tab.7, we evaluate the training and inference time both of TGNN[13] and X-RefSeg[29]. All experiments are conducted on an NVIDIA 4090 GPU and the number of batch sizes and epoch of three methods are kept the same. For the two-stage training and inference of TGNN and X-RefSeg, we followed the settings in their open source codes. We can find that our LESS consumes less time than both of TGNN and X-RefSeg in both training and inference.

D More Quantitative Results

E More Ablation Results

E.1 Query Number

We further conduct more ablation studies in terms of the number of queries on ScanRefer. As shown in Tab.9, we ablate the number of queries as 5, 15, 20, it can be observed that keeping 20 queries brings higher accuracy while lower accuracy when using fewer queries. We suppose that fewer queries can not help to learn comprehensive feature patterns while an appropriate number of queries is enough to cover the needed feature patterns.

E.2 Mask Selection Strategy

We also conduct the ablation study which selects the mask with the highest score during QSA process as shown in Tab.10. The results demonstrate that the aggregation of multiple masks yields superior

performance compared to the selection of a single, highest-ranked mask. This approach facilitates the capture of subtle nuances and intricate details that may be overlooked when relying on a single mask alone. The incorporation of multiple masks offers a more comprehensive and precise representation, ultimately enhancing the accuracy of the final model prediction.

F Potential Negative Societal Impacts

Our method has no ethical risk on datasets usage and privacy violation because all the datasets and tools are publicly available and transparent.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please see the Abstract and Introduction sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see the Limitations subsection precede the Conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please see the Method section and Experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Once the paper is accepted , we will release our code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see the Experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The experiments are conducted under a fixed seed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see the Appendix section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research is with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please see the Appendix section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Please see the Experiment and Reference sections.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a Label-Efficient and Single-Stage referring 3D instance segmentation method, which is under the supervision of binary mask. They propose Point-Word Cross-Modal Alignment, Query Mask Predictor and Query Alignment modules to achieve cross-modal alignment. Besides, the area regularization loss and the point-to-point contrastive loss are introduced to eliminate interference caused by multiple objects and backgrounds. Experimental results show that the proposed method outperforms the existing state-of-the-art method on ScanRefer dataset with only the supervision of binary. This is a nice paper dealing an interesting research problem, presenting insights, and being well-written.\n\nSTRENGTHS:\n1.\tThis paper firstly investigated the single stage referring 3D instance segmentation task to bridge the gap between detection and matching, only under the supervision of binary mask. The proposed method can provide valuable insights to the further research in multi-modal 3D scene understanding.\n2.\tTo align features among different modalities, such as fined-grained alignment between point and word features, coarse-grained alignment between masks and sentences, several alignments modules are well designed, which exhibit a certain level of novelty.\n3.\tExperimental results show that the proposed method outperforms the existing state-of-the-art method on ScanRefer dataset with only the supervision of binary label.\n4.\tThe paper is well-written and is easy to follow.\n\nWEAKNESSES:\n1.\tMethods TGNN and X-RefSeg are not evaluated with the RoBERTa backbone.\n2.\tThe explanation of why more queries and layers as shown in Tab. 5 and 6 do not bring performance gains are missing.\n\nQUESTIONS:\n1.\tWhy the proposed alignment modules work well? \n2.\tHow about the training time and inference time comparison?\n\nLIMITATIONS:\nNo negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

comments: SUMMARY:
The paper proposes LESS, a single-stage, label-efficient pipeline for referring 3D instance segmentation. It introduces fine-grained and coarse-grained cross-modal alignment to improve feature matching and employs additional losses to reduce irrelevant predictions. Experiments are conducted on the ScanRefer dataset.

STRENGTHS:
1. The motivation of this paper is meaningful. It proposes a label-efficient method for referring 3D instance segmentation, which can significantly reduce annotation costs.
2. The paper is well-written and easy to follow.
3. The experiments on the ScanRefer dataset show promising results.

WEAKNESSES:
1. The experiments are only conducted on the ScanRefer dataset, lacking validation on additional datasets like Nr3D/Sr3D used in TGNN[1].
2. In the Query-Sentence Alignment module, the final mask prediction is produced by a weighted sum of the similarity score and mask prediction. The rationale for using a weighted sum is not intuitive; why not select the mask with the highest score?
3. The paper lacks references to related literature. The Query Mask Predictor module is similar to the framework used in Mask3D[2], but Mask3D is not cited.

[1]. Text-guided graph neural networks for referring 3d instance segmentation, AAAI 2021.
[2]. Mask3d: Mask transformer for 3d semantic instance segmentation, ICRA 2023.

QUESTIONS:
See weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-07

comments: SUMMARY:
This paper proposes a label-efficient single-stage method for referring 3D instance segmentation. Specifically, this method enhances feature extraction by integrating multi-modal features, using only binary labels for supervision. It achieves fine-grained alignment between points and words, distinguishing points or objects with similar features.

STRENGTHS:
This paper introduces the single-stage method pioneered by R3DIS, which I find very appealing.

The structure of the article is clear and the writing is fluent.

It achieves state-of-the-art (SOTA) performance on the ScanRefer dataset.

WEAKNESSES:
I agree with the authors' point that annotating precise instance-level labels in tasks like open-vocabulary or 3D visual grounding is time-consuming and labor-intensive. Therefore, it is impressive that this method reduces label effort by approximately 10 times while maintaining SOTA performance. However, the authors only used mIoU and Acc as metrics for semantic segmentation in the experiments, without using any 3D instance segmentation metrics (e.g., AP). I hope the authors can explain in detail why they did not use instance segmentation metrics. If no instance segmentation metrics were used, why is this task named 3D Instance Segmentation?

I hope the authors can validate their method's robustness across multiple datasets, as using only the ScanRefer dataset provides very limited persuasiveness.

QUESTIONS:
See Weaknesses part.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of referring 3D Instance Segmentation, which segments all points belonging to an object in a 3D point cloud described by a query prompt. Previous methods use a two-stage approach requiring both instance and semantic labels, which is labor-intensive. The authors propose LESS (Label-Efficient and Single-Stage), a new pipeline that only requires binary mask supervision, reducing annotation costs. Key innovations include a Point-Word Cross-Modal Alignment module, a Query Mask Predictor module, a Sentence Query Alignment module, and two new losses for weaker supervision. LESS significantly outperforms existing methods on the ScanRefer dataset.\n\nSTRENGTHS:\n1. The paper is clear and easy to understand. The proposed method is straightforward and well-explained\n2. Strong quantitative results are reported on the ScanRefer dataset.\n\nWEAKNESSES:\n1. No substantial novelty in module design or architecture, they are already proposed in other contexts:\n   + Query Mask Predictor: similar to Mask2Former: M2F3D: Mask2Former for 3D Instance Segmentation\n   + Point-Word Cross-Modal Alignment: similar to cross-attention\n   + Query-Sentence Alignment: cosine CLIP score\n2. The paper lacks explanations of why TGNN and X-RefSeg perform worse, and in which case these methods fail.\n3. The area regularization loss brings the most significant gain, but is not explored in detail, i.e. ablation of the loss coefficient.\n4. Running time compared to the two-stage approaches? If it is not significantly reduced, there is no advantage compared to the two-stage approaches. \n5. The results with Acc@0.5 are worse than those of TGNN and X-RefSeg (Table 1) while the results with Acc@0.25 are opposite indicating that the proposed method is better locate coarse location rather than exact matching as in TGNN and X-RefSeg.\n\nComment: TGNN and X-RefSeg use a pretrained 3DIS network, when training on this Referring 3DIS task, only need binary mask supervision.\n\nQUESTIONS:\n1. How does the loss coefficient affect the results?\n2. What is the impact if there are less than 20 queries in Table 5?\n3. Are there any benchmarks other than ScanRefer? The main paper only shows the result on one benchmark, which may be not sufficient. How about Sr3D and Nr3D\n\nLIMITATIONS:\nYes, this paper has a session discussing its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}