LESS: Label-Efficient and Single-Stage Referring 3D Segmentation

Authors: xuexun  liu
xiaoxu  xu
jinlong  li
qiudan  zhang
xu  wang
nicu  sebe
lin  ma
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:57:19

Abstract: Referring 3D Segmentation is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. Previous works perform a two-stage paradigm, first conducting language-agnostic instance segmentation then matching with given text query. However, the semantic concepts from text query and visual cues are separately interacted during the training, and both instance and semantic labels for each object are required, which is time consuming and human-labor intensive. To mitigate these issues, we propose a novel Referring 3D Segmentation pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. Specifically, we design a Point-Word Cross-Modal Alignment module for aligning the fine-grained features of points and textual embedding. Query Mask Predictor module and Query-Sentence Alignment module are introduced for coarse-grained alignment between masks and query. Furthermore, we propose an area regularization loss, which coarsely reduces irrelevant background predictions on a large scale. Besides, a point-to-point contrastive loss is proposed concentrating on distinguishing points with subtly similar features. Through extensive experiments, we achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels. Code is available at https://github.com/mellody11/LESS.

Main paper: 
 introduction: 
 Referring 3D Segmentation task aims to segment the specific object from a 3D point cloud scene with a free-form natural language expression. It allows users to interact and analyze 3D data through verbal instructions or queries. This approach is particularly beneficial in applications that necessitate direct interaction with 3D environments, such as augmented reality (AR) systems, embodied-AI, robotics, virtual reality (VR) environments, and in fields like architecture and medical imaging, where precise identification and segmentation of objects based on descriptive queries can significantly enhance user experience and operational efficiency.Previous Referring 3D Segmentation [13; 29] mainly leverage a two-stage workaround, as shown in Fig. 1 (a) . They typically adopt a 3D instance semantic segmentation network to get the instance proposals at the first stage. The predicted instance proposals will be utilized to match with the queries and finally get the final prediction mask according to the matching score. Although those method have achieved remarkable performance, there still exist some problems. First, owing to the large-scale and irregular 3D point clouds, some instance proposals may leave out the target in the pre-segmentation stage. Besides, lacking of linguistic guidance in the segmentation stage fails to focus on the objects that are more essential to the referring task. Moreover, existing Referring 3D Segmentation utilizes both instance labels and semantic labels to segment target proposal rather than binary mask used in referring image segmentation, which is more time consuming and labor intensive.To address the aforementioned problems, we propose a Label-Efficient and Single-Stage Referring 3D Segmentation method, namely LESS, which is under the supervision of binary mask, as shown in Fig. 1 (b ). We first process the query with text encoder to get the word-level feature and sentence-level feature. Then we extract the multi-modal feature with the guidance of text feature through a 3D sparse U-Net. Finally, the mask predictor aligns multi-modal features with language features, and directly predicts the mask of the described object. Here only object mask serves as the label to supervise the whole training procedure.However, 3D point cloud inherently provide a higher level of complexity and a large scale. There exists numerous different objects in a single 3D scene compared to the referring image segmentation task. Besides, binary mask has less semantic meanings compared to instance labels and semantic labels. These challenges make it difficult to supervise our model to localize and segment target objects with only binary mask. Therefore, we propose to alleviate these problems by some ways, as shown in Fig. 2 . Firstly, to facilitate fine-grained alignment between points and words, we propose Point-Word Cross-Modal Alignment module. The PWCA module utilizes cross-modal attention in the multi-modal context to align textual features extracted by the text encoder with point cloud features, followed by further extraction of useful multi-modal information using robust 3D sparse convolutional layers. Thus we can extract a more semantic meaning fused feature. Meanwhile, we employ Query Mask Predictor, which utilizes the extracted multi-modal features to decode learnable query embeddings, generating candidate masks. By introducing the Query-Sentence Alignment modules, we can compute similarity between the decoded query embeddings and sentence features, using the similarity as weights to perform weighted summation of the candidate masks to produce the final mask. To address the significant interference caused by multiple objects and backgrounds, we propose an area regularization loss and a point-to-point contrastive loss. Area regularization loss reduces irrelevant background predictions by constraining the probabilities of the predicted mask, while point-to-point contrastive loss constrains the distances between positive and negative points in the latent space to achieve better segmentation.To summarize, our contributions are as follows:• We propose a new Referring 3D Segmentation method LESS, which directly performs Referring 3D Segmentation at a single stage to bridge the gap between detection and matching under the supervision of binary mask. To the best of our knowledge, LESS is the first work investigating label-efficient and single-stage in Referring 3D Segmentation task.• Our LESS utilize a Point-Word Cross-Modal Alignment module to align fined-grained point and word features. Besides, we employed Query Mask Predictor module and Query-Sentence Alignment modules for coarse-grained alignment between masks and sentences. Moreover, the area regularization loss and the point-to-point contrastive loss are introduced to better support to eliminate interference caused by multiple objects and backgrounds.• Extensive experiments confirm the effectiveness of our method. Our method outperforms the existing state-of-the-art method on ScanRefer dataset with only the supervision of binary mask. Our LESS and its results provide valuable insights to improve further research of label-efficient and single-stage Referring 3D Segmentation.2 Related Works 
referring 3d segmentation: 
 Referring 3D Segmentation has previously received limited exploration, however, with the advancements in multi-modal learning and embodied AI, it is set to attract increasing interest in the future. TGNN [13] is the first to introduce Referring 3D Segmentation task. They initially trained an instance segmentation network, followed by a Graph Neural Network (GNN) to learn features of instances and their relationships guided by linguistic information. Building on TGNN, X-RefSeg [29] developed a cross-modal graph. They employed an GNN to model the texture and spatial relationships of instances, and refining the results through inference and matching processes.3D visual grounding aims to locate the object within point clouds mentioned by free-form natural language descriptions. Most methods follow a two-stage detection-then-matching pipeline. Initially, they utilize a pre-trained 3D detector [28; 23] or segmenter [15; 33] to extract object representations. Subsequently, these methods align text features with object features to identify the best-matched object. Researchers primarily concentrate on the second stage which involves modeling object relationships and exploring feature fusion between language and objects. Methods employed include multi-modal feature concatenation [3; 1] , attention-based multi-modal feature alignment [44; 11] , graph neural network-based reasoning [13; 41] , and the alignment of visual and language features aided by 2D images [40; 42; 38; 37] .Other researchers have investigated single-stage approaches for 3D visual grounding. 3D-SPS [25] views the task as key point selection, progressively identifying keypoints with the guidance of language and directly locates the target. BUTD-DETR [14] employs a transformer decoder [2] to identify described objects using language cues and proposal boxes. Building on this, EDA [36] enhances dense alignment between objects and point clouds by explicitly decoupling textual attributes from sentences.The distinction between 3D visual grounding and Referring 3D Segmentation lies in the latter's enhanced localization precision, offering significant value in applications like robotic grasping. While traditional single-stage 3D visual grounding methods regress a 3D bounding box, our singlestage approach decodes a binary mask for the entire point cloud scene, presenting a more complex challenge.Referring image segmentation is a visual task that involves pixel-level segmentation of an image's target object based on a referring expression. Early approaches [12; 20; 18; 4; 17] utilize CNNs and RNNs to extract features and then perform simple concatenation for multi-modal feature fusion. Subsequent research [16; 19; 9] adopt transformer models for more effective feature extraction and fusion. Recent studies [39; 10; 43] focus on identifying optimal positions for language-vision alignment. Additionally, some methods [34; 5] have enhanced alignment between language and pixel features by designing specialized loss functions. Other approaches [21; 45] segmentation as an auto-regressive vector generation problem, creating masks from generated closed vectors.Methods for referring image segmentation have been extensively explored, yet they cannot be directly applied to Referring 3D Segmentation due to the inherent challenges of point cloud scenes. Additionally, unlike referring image segmentation methods that typically employ pre-trained visual encoders, Referring 3D Segmentation lacks such resources, compelling reliance on limited supervisory signals for model training.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please see the Limitations subsection precede the Conclusion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 The overall framework of our proposed LESS is shown in Fig. 2 , which leverages Point-Word Cross-Modal Alignment and Query-Sentence Alignment to facilitate multi-modal interaction. In this section, we start by introducing our visual and text feature extractor in Sec.3.1. Then Query Mask Predictor and Query-Sentence Alignment is detailed in Sec.3.2 and Sec.3.3. Finally in Sec.3.4, we introduce our area regularization loss and point-to-point contrastive loss.Sparse 3D Feature Extractor. The point cloud scene P ∈ R N ×6 contains N points in the scene, and each point is represented with six dimensions of RGBXYZ attributes. We first voxelize points into regular voxels and adopt a sparse 3D U-Net [31; 15] to extract point-wise fused feature F ∈ R N ×C .Here the encoder part of the U-Net has 5 stages and the feature from the i-th encoder stage is denoted as V i .Text Encoder. Given the query sentence T with L words, a text encoder is used to embed the query into C-dimensional feature vectors. In this paper we choose GRU [6] , BETR [8] and RoBERTa [22] as our text encoder respectively and fine-tune the BERT or RoBERTa during training. Finally, we can get both word features W ∈ R L×C and sentence features S ∈ R C after the text encoder.Point-Word Cross-Modal Alignment. Due to the lack of such rich annotations as [13; 29] , it is crucial for our model to learn the relationship between fine-grained word-level features and pointlevel features in such a point-level segmentation task. Meanwhile, we notice that leveraging the rich convolutional layers in the encoder to excavate multi-modal context is effective way to extract language-aware visual feature. Therefore, we design Point-Word Cross-Modal Alignment (PWCA) module, as shown in the lower left part in Fig. 2 , which contains a standard cross-attention module [32] and an MLP-Tanh gate. Cross-attention module aligns point-wise and word-wise feature to get the language-aware visual feature. A nonlinear tanh gate is adopted to prevent the fused signal from overwhelming the original signal. Given the multi-scale point cloud features {V i ∈ R Ni×Ci } 5 i=1 , we simply project the word feature W ∈ R L×C to W i ∈ R L×Ci . PWCA can be formulated as follow:EQUATIONwhere i indicates the i-th stage of the encoder part of our sparse 3D feature extractor. Here we use V i as the query and W i as the key and value for cross attention.Inspired by [31; 30] , Query Mask Predictor (QMP) module, as shown in Fig. 2 , takes fused feature F ∈ R N ×C and learnable queries Q 0 ∈ R K×C as input and progressively distinguishes the referring target by multi-layer cross-modal transformers. Finally, we extract the proposal masks M m ∈ R K×N based on queries embeddings Q m ∈ R K×C and fused feature F . Query Decoder. As illustrated in the lower middle section of Fig. 2 . Here query decoder is comprised m-layer masked cross-attention modules [32] , where the fused features F are served as keys and values and A j-1 ∈ {0, 1} K×N is utilized as the attention mask. Therefore, in the j-th layer, the learnable queries Q j-1 capture multi-modal contextual information from fused feature F via a query decoder to obtain the query embeddings Q j ∈ R K×C .Mask Predictor. First the fused feature F ∈ R N ×C is processed by a Shared MLP, which indicates each Query Mask Predictor layer shares the same MLP. We perform the matrix multiplication on the new F and Q j ∈ R K×C to generate proposal mask predictions M j ∈ R K×N . After applying a sigmoid function whose threshold of 0.5, we can get the new binary attention mask A j ∈ {0, 1} K×N . Finally, the Query Mask Predictor is formally described as follows:EQUATIONEQUATIONEQUATIONwhere ⊤ denotes the transpose operation, ⊗ represents matrix multiplication, and σ refers to the sigmoid function. Here we need to note that the Q 0 is randomly initialized. Therefore we can leverage the function (3) (4) to initialize the attention mask A 0 .The Referring 3D Segmentation task involves the segmentation of a solitary target object according to the query. Previous modules focus on extracting language-aware visual feature from aligning point-wise feature and word-wise feature, which lacks a comprehensive perception of the entire query sentence. Therefore, we adopt Query-Sentence Alignment (QSA) to better align the query feature with sentence-level feature. We perform the matrix multiplication on Q m ∈ R K×C and S ∈ R C to get the their similarity score R ∈ [0, 1] K . Formally, Query-Sentence Alignment can be represented as:EQUATIONThe final mask prediction M ∈ R N is produced by weighted sum of similarity score R and proposed mask prediction M m ∈ R K×N . Finally, we use a sigmoid function and a threshold of 0.5 to produce the final predicted binary mask Ŷ ∈ {0, 1} N :EQUATIONSegmentation Loss. Different from previous work [13; 29] , we take the Referring 3D Segmentation task as segmentation task with only binary mask Y ∈ {0, 1} N . Here we utilize the Binary Cross-Entropy (BCE) loss function to compute the segmentation loss, which can be formulated as:EQUATIONArea Regularization Loss. For Referring 3D Segmentation task, each query always corresponds to one target object in the point cloud scene. The target objects occupy a smaller area in the large scale of 3D point cloud. As a result, the predicted mask often includes backgrounds or other objects.To address this, we propose a region regularization loss, which promotes the network to predict a smallest mask by minimize the output probability of each point, formulated as:EQUATIONBy combining with the segmentation loss, we intend to segment only the most probable regions while reducing segmentation of large-scale irrelevant background areas.Point-to-Point Contrastive Loss. Area regularization loss uniformly penalizes the predicted probabilities of all points, which can reduce the majority of the background points. However, the network struggles to differentiate between objects that possess characteristics similar to those described target object in the latent space. Therefore, we propose a point-to-point contrastive loss [26] that pull the points from the described object together and push away the rest points:EQUATION)where P is the positive point set from the described object, N is the negative point set from the background, P i denotes the L2-normalized feature vector of i -th positive points from F , while N j denotes the L2-normalized feature vector of j-th negative points from F , P avg is the average feature vector of positive point P avg = 1 P |P| i=1 P i , and τ is the hyper-parameter. The contrastive loss promotes the network to distinguish the described object from the adjacent background points in a fined-grained manner.The overall loss function is the weighted sum of the aforementioned three loss functions:EQUATIONwhere {λ seg , λ area , λ p2p } is set to {1, 1, 0.05} in practice to balance the contrastive loss because of the large amount of points. Evaluation Metric. Following previous work [13; 29] , we use the mean intersection-over-union (mIoU), and Acc@kIOU as the evaluation metrics. The mIoU is the average of the IoU over all test samples and the Acc@kIOU measures the accuracy of test samples with an IoU higher than the threshold k, where k ∈ {0.25, 0.5}. We use A@25 and A@50 for brevity in some of the following tables.Implementation Details. We adopt the 3D spares U-Net [31] as our 3D feature extractor. we explore multiple text encoders, i.e., GRU [6] , BERT [8] and RoBERTa [22] in our experiments for comparative analysis. For BERT and RoBERTa, we use the official pre-trained weights and fine-tune them during training. We set the number of queries K to 20 and use a single layer for QMP module.We set an initial learning rate of 2e-5 for the text encoder and 1e-4 for the others. We reduce the learning rate by a multiplicative factor of 0.95 each epoch and adopt Adam [24] as our optimizer. The weights of our loss function {λ seg , λ area , λ p2p } is set to {1, 1, 0.05}. We train for 64 epochs with a batchsize of 14, and all experiments are implemented on PyTorch [27] .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
comparison with state-of-the-art methods: 
 As shown in Tab.1, we evaluate our LESS against the previous two-stage methods. LESS outperforms the previous SOTA method using GRU [6] and BERT [8] by an impressive progress of 2.42%, and 2.50% on mIoU, and 11.15% and 11.08% on Acc@0.25 respectively. Moreover, we conduct an extra experiment using RoBERTa [22] , outperforming the best method of 3.8% and 12.9% on mIoU and Acc@0.25 respectively. Such results demonstrate the potential of our method. We also reported the comparison between our method and the two-stage approach in terms of label effort. We find that our label-efficient method saves more than 90% label effort compared to existing methods.However we find that the performance of our LESS has a gap on Acc@0.5 compared to previous methods. Previous methods employed a segmentation-matching strategy. Once matching successfully, the IoU between predicted mask and ground truth is mostly higher than 0.5, which is beneficial for the Acc@0.5. In contrast, our single-stage method without instance labels and semantic labels can not extract the more accurate instance candidates as prior knowledge to assist referring 3D segmentation task. Therefore, it is acceptable for our method to perform lower than previous methods on Acc@0.5 with fewer supervisory signals. it is in between trash cans. We conducted several ablation studies to evaluate the effectiveness of the key components in our proposed network. In these studies, all other components and hyper-parameter settings are kept consistent with the aforementioned experiments, except for the component being ablated.Module Ablation. In this ablation study, we evaluate the effectiveness of the QSA and PWCA modules. As indicated in Table2, the ablation model (a) only retain the sparse 3D feature extractor and query mask predictor. Here we perform element-wise addition of the word features W to the multiscale features {V i } 5 i=1 instead of PWCA and project the proposed mask prediction M m ∈ R K×N to the final mask prediction M ∈ R N via an MLP instead of QSA. We set model (a) as the baseline of our experiment. Compared to model (a), model (b) adopts PWCA module and we find that the model performance increase greatly from 32.66% to 33.44%. This observation proves that PWCA facilitate fine-grained cross-modal alignment between points and words, which is more effectively to leverage the rich convolutional layers in the encoder to excavate multi-modal context. When we introduce QSA to model (b), as shown in model (c), we can find that the performance of mIoU is improved from 33.44% to 33.74%. Such results indicates that the fine-grained point-word alignment of PWCA and the coarse-grained query-sentence alignment of QSA effectively coupled to enhance the capability in capturing multi-modal context.Loss Ablation. As demonstrated in 3.4, we refine predicted masks from coarse to fine by introducing two loss functions, i.e., area regularization loss L area and point-to-point contrastive loss L p2p . We successively add the loss functions and the results are shown in Tab. Qualitative results are shown in Fig. 3 , it indicates that: i) When only the segmentation loss L seg is applied (column 3), the predicted masks include many points from other regions. ii) After adding the L area , most of the irrelevant points disappear (row a, row d). iii) After incorporating the L p2p , objects that were previously difficult to distinguish due to their similarity are successfully separated, and the predictions are close to the ground-truth.Both quantitative and qualitative experiment demonstrate that our proposed loss function effectively reduces large-scale background misclassifications and distinguish objects or points with the similar characteristics.Linguistic Features at Different Levels of Granularity. In this section, we will investigate the impact of linguistic features at different levels of granularity, as shown in Table .4. The first row represent that we leverage word-level features in both PWCA module and QSA module as text features. The second row indicates we utilize sentence-level features in both modules. We can find that sentence-only method even outperforms the word-only one. Referring 3D Segmentation task involves the segmentation of a solitary target object. This mandates a more profound and thorough comprehension of the semantic information conveyed by the sentence, extending beyond a mere focus on its individual words. The last row indicates the word-level features are utilized in PWCA module and sentence-level features are leveraged in QSA module. We can find that the performance of it outperforms two introduced above, which proves that fine-grained word-level feature is also helpful in extracting the 3D language-aware visual feature.Layer and Query Number of QMP. We also investigate the performance on different query and layer numbers in the QMP module. As shown in Tab.5 and Tab.6, we find that too many queries and layers do not bring performance gains for Referring 3D Segmentation task. As a result, we set 20 queries and 1 QMP layer as the default configuration after balancing performance and efficiency.The limitations of LESS due to the inherent complexity of 3D point clouds and the ambiguous queries, although we have made significant improvements on previous methods. The scarcity of detailed semantic annotations are still challenging our model from distinguishing multiple similar objects. These limitations could guide our future work. 
conclusions: 
 In this paper, we propose LESS, a label-efficient and single-stage approach for Referring 3D Segmentation. Specifically, our LESS enhances feature extraction by integrating multi-modal features and employs progressive constraints on predicted masks, achieving fine-grained alignment between points and words and distinguishing between points or objects with similar characteristics. Comprehensive experiments demonstrate that our single-stage method outperforms existing two-stage approaches on ScanRefer dataset, using only the binary labels as supervision. Though our framework still has some limitations, we believe that solving the Referring 3D Segmentation task only using the binary labels is a new and promising research, and we hope that LESS can serve as a simple but powerful baseline to inspire future research on Referring 3D Segmentation. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Please see the Experiment and Reference sections. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A Semantic and Instance Labels vs. Binary LabelsThe differences between semantic labels, instance labels, and binary labels are shown in Fig. 4 .Semantic labels assign a category ID to each point in a scene, with points belonging to the same category (e.g., chairs) sharing the same ID. Instance labels assign a object ID to each point within the same object (e.g., a specific chair), distinguishing different objects by assigning different instance IDs. Binary labels assign a Boolean value to each point in the scene; points that are part of the described object are assigned a value of 1, and points that are not are assigned a value of 0.Therefore, compared to binary labels, the annotations of semantic labels and instance labels are more time consuming and labor intensive. We present our success cases and failure cases in Fig. 5 . Our method accurately segments objects with clear queries. However, ambiguous descriptions can still confuse our model and leads to segment both the referred object and other similar objects. Ours GT (b) the chair is black and in the corner.the chair left of the others.Figure 5 : Qualitative results of both the success cases and failure cases of our LESS.As shown in Tab.7, we evaluate the training and inference time both of TGNN [13] and X-RefSeg [29] .All experiments are conducted on an NVIDIA 4090 GPU and the number of batch sizes and epoch of three methods are kept the same. For the two-stage training and inference of TGNN and X-RefSeg, we followed the settings in their open source codes. We can find that our LESS consumes less time than both of TGNN and X-RefSeg in both training and inference. A concurrent work 3D-STMN [35] utilizes a pre-trained 3D feature extractor [31] to perform referring 3D segmentation. Given that their backbone is pre-trained on an instance segmentation task with semantic and instance label, it is reasonable to conclude that their approach cannot be considered a label-efficient and single-stage method. For fair comparison, we follow the settings in their open source code and train their network from scratch, except for BERT [8] module. As shown in Tab.8, our method overtakes 3D-STMN by 11.34%, 18.27% and 12.65% on mIoU, Acc@0.25 and Acc@0.5 respectively.Table 8 : Quantitative results of 3D-STMN [35] and ours.Method mIoU Acc@0.25 Acc@0. We further conduct more ablation studies in terms of the number of queries on ScanRefer. As shown in Tab.9, we ablate the number of queries as 5, 15, 20, it can be observed that keeping 20 queries brings higher accuracy while lower accuracy when using fewer queries. We suppose that fewer queries can not help to learn comprehensive feature patterns while an appropriate number of queries is enough to cover the needed feature patterns.Table 9 : The impact of the number of queries.Num of queries mIoU Acc@0.25 Acc@0.5 We also conduct the ablation study which selects the mask with the highest score during QSA process as shown in Tab.10. The results demonstrate that the aggregation of multiple masks yields superior performance compared to the selection of a single, highest-ranked mask. This approach facilitates the capture of subtle nuances and intricate details that may be overlooked when relying on a single mask alone. The incorporation of multiple masks offers a more comprehensive and precise representation, ultimately enhancing the accuracy of the final model prediction.Table 10 : The impact of mask selection strategy.Num of queries mIoU Acc@0.25 Acc@0. Our method has no ethical risk on datasets usage and privacy violation because all the datasets and tools are publicly available and transparent.NeurIPS Paper Checklist Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The reviewers all agree on the proposed label efficient single stage 3d referring expression modeling being either reasonable and straightforward or a sufficiently novel contribution. The rebuttal and discussion clarified some question about evaluation and setup, and overall the authors are encouraged to take into account the reviewer comments.

Reviews: review: paper_summary: paper_summary: This paper addresses the problem of referring 3D Instance Segmentation, which segments all points belonging to an object in a 3D point cloud described by a query prompt. Previous methods use a two-stage approach requiring both instance and semantic labels, which is labor-intensive. The authors propose LESS (Label-Efficient and Single-Stage), a new pipeline that only requires binary mask supervision, reducing annotation costs. Key innovations include a Point-Word Cross-Modal Alignment module, a Query Mask Predictor module, a Sentence Query Alignment module, and two new losses for weaker supervision. LESS significantly outperforms existing methods on the ScanRefer dataset.
strength_weakness: strength_weakness: 1. No substantial novelty in module design or architecture, they are already proposed in other contexts:
   + Query Mask Predictor: similar to Mask2Former: M2F3D: Mask2Former for 3D Instance Segmentation
   + Point-Word Cross-Modal Alignment: similar to cross-attention
   + Query-Sentence Alignment: cosine CLIP score
2. The paper lacks explanations of why TGNN and X-RefSeg perform worse, and in which case these methods fail.
3. The area regularization loss brings the most significant gain, but is not explored in detail, i.e. ablation of the loss coefficient.
4. Running time compared to the two-stage approaches? If it is not significantly reduced, there is no advantage compared to the two-stage approaches. 
5. The results with Acc@0.5 are worse than those of TGNN and X-RefSeg (Table 1) while the results with Acc@0.25 are opposite indicating that the proposed method is better locate coarse location rather than exact matching as in TGNN and X-RefSeg.

Comment: TGNN and X-RefSeg use a pretrained 3DIS network, when training on this Referring 3DIS task, only need binary mask supervision.
questions: questions: 1. How does the loss coefficient affect the results?
2. What is the impact if there are less than 20 queries in Table 5?
3. Are there any benchmarks other than ScanRefer? The main paper only shows the result on one benchmark, which may be not sufficient. How about Sr3D and Nr3D
limitations: limitations: Yes, this paper has a session discussing its limitations.
paper_summary: paper_summary: This paper proposes a label-efficient single-stage method for referring 3D instance segmentation. Specifically, this method enhances feature extraction by integrating multi-modal features, using only binary labels for supervision. It achieves fine-grained alignment between points and words, distinguishing points or objects with similar features.
strength_weakness: strength_weakness: I agree with the authors' point that annotating precise instance-level labels in tasks like open-vocabulary or 3D visual grounding is time-consuming and labor-intensive. Therefore, it is impressive that this method reduces label effort by approximately 10 times while maintaining SOTA performance. However, the authors only used mIoU and Acc as metrics for semantic segmentation in the experiments, without using any 3D instance segmentation metrics (e.g., AP). I hope the authors can explain in detail why they did not use instance segmentation metrics. If no instance segmentation metrics were used, why is this task named 3D Instance Segmentation?

I hope the authors can validate their method's robustness across multiple datasets, as using only the ScanRefer dataset provides very limited persuasiveness.
questions: questions: See Weaknesses part.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper proposes LESS, a single-stage, label-efficient pipeline for referring 3D instance segmentation. It introduces fine-grained and coarse-grained cross-modal alignment to improve feature matching and employs additional losses to reduce irrelevant predictions. Experiments are conducted on the ScanRefer dataset.
strength_weakness: strength_weakness: 1. The experiments are only conducted on the ScanRefer dataset, lacking validation on additional datasets like Nr3D/Sr3D used in TGNN[1].
2. In the Query-Sentence Alignment module, the final mask prediction is produced by a weighted sum of the similarity score and mask prediction. The rationale for using a weighted sum is not intuitive; why not select the mask with the highest score?
3. The paper lacks references to related literature. The Query Mask Predictor module is similar to the framework used in Mask3D[2], but Mask3D is not cited.

[1]. Text-guided graph neural networks for referring 3d instance segmentation, AAAI 2021.
[2]. Mask3d: Mask transformer for 3d semantic instance segmentation, ICRA 2023.
questions: questions: See weaknesses
limitations: limitations: Yes
paper_summary: paper_summary: In this paper, the authors propose a Label-Efficient and Single-Stage referring 3D instance segmentation method, which is under the supervision of binary mask. They propose Point-Word Cross-Modal Alignment, Query Mask Predictor and Query Alignment modules to achieve cross-modal alignment. Besides, the area regularization loss and the point-to-point contrastive loss are introduced to eliminate interference caused by multiple objects and backgrounds. Experimental results show that the proposed method outperforms the existing state-of-the-art method on ScanRefer dataset with only the supervision of binary. This is a nice paper dealing an interesting research problem, presenting insights, and being well-written.
strength_weakness: strength_weakness: 1.	Methods TGNN and X-RefSeg are not evaluated with the RoBERTa backbone.
2.	The explanation of why more queries and layers as shown in Tab. 5 and 6 do not bring performance gains are missing.
questions: questions: 1.	Why the proposed alignment modules work well? 
2.	How about the training time and inference time comparison?
limitations: limitations: No negative societal impact

Comments: title: Response to Reviewer Monv
Response to Reviewer pLC3
Response to Reviewer NB5W
Response to Reviewer 6zjS
Response to Reviewer NB5W
Response to Reviewer NB5W
Response to Reviewer 6zjS
Response to Reviewer NB5W
Additional concerns to label efficiency
comment: We are grateful for your comprehensive and encouraging review and acknowledgement. We are pleased that you appreciate the technical contributions of our work, which could motivate us to explore more interesting works in the future. 

Thank you again for your support!
We are grateful for your comprehensive and encouraging review. Thank you.
Thank you for your questions and suggestions. A recent work, OpenScene[1] has established a straightforward and efficient baseline network in the field of open vocabulary 3D segmentation using a simple cross modal distillation method based on primary cosine smilarity driven, but also with a pioneering explroation. The objective of our work is to present a straightforward and effective single-stage baseline for the task of 3D referring segmentation. In addition to a single stage architecture design, our work makes a further contribution in the form of a single-stage coarse-to-fine cross-modal alignment network and a coarse-to-fine background point filtering loss function. These have been developed with the aim of achieving state-of-the-art performance under efficient binary mask supervision, which have been demonstrated with extensive ablations mentioned before.

Thank you again for your suggestions. We sincerely hope that you will be able to evaluate the contribution of our work based on its overall design and the differences that it presents in comparison to previous works [2][3], and also the hitting cornerstone of single-stage pipeline of this task.

[1]. Openscene: 3d scene understanding with open vocabularies, CVPR 2023.

[2]. Text-guided graph neural networks for referring 3d instance segmentation, AAAI 2021

[3]. X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks, AAAI 2024
Thanks for the authors' response. I have read all the contents, and most of my concerns have been addressed.
However, I have to admit that I'm not an expert in this field, so please consider more about the review opinions of reviewers with higher confidence.
Thank the authors for the rebuttal. 

First, my comments were clearly addressed. Specifically, the validation of TGNN and X-RefSeg with the RoBERTa backbone, and the comparisons of the training time and inference time demonstrate the proposed method achieves the best performance. Besides, the explanations on the performance gains and motivation of the proposed alignment module makes the contributions of the proposed method clearer.

Second, I read the other reviewers’ comments as well as the rebuttal information. For the raised questions, such as the validation on other dataset, ablation studies on the mask selection strategy, explanation of the significant differences between the proposed modules and existing techniques and concerns to label efficiency, the authors also responded accordingly.

In summary, this work is relatively novel, which investigated the single stage referring 3D instance segmentation task to bridge the gap between detection and matching, only under the supervision of binary mask. The proposed method can provide valuable insights to the further research in multi-modal 3D scene understanding. Therefore, I am maintaining my original score.
Thank you for your response. However, based on your clarification, I still don’t see a significant contribution. The differences appear to be mainly in the implementation details, with only minor modifications to make it work.
Dear reviewer,

It appears that no email reminders have been sent since our last reply. In our most recent responses, we have addressed the concerns and questions you raised. Should you have further inquiries or suggestions, we would be pleased to engage in further discussion.
Dear reviewer,

It appears that no email reminders have been sent since our last reply. In our most recent responses, we have addressed the concerns and questions you raised. Should you have further inquiries or suggestions, we would be pleased to engage in further discussion.
Here are the differences. We will point them out one by one:

- **Query Mask Predictor:** We adopted an architecture similar to Mask3D [1] and Mask2Former [2], but since our single-stage model does not require semantic labels for supervision, we removed the semantic branch; And since only specific masks need to be predicted, the process of complex Hungarian Matching  can be omitted.

- **Point-Word Cross-Modal Alignment:** The distinctions are we remove the position encoding in order to address the disorder property of 3D point clouds, and we incorporate an MLP Tanh gate following the cross attention to regulate the information flow between language and visual features.

- **Query-Sentence Alignment:** We use cosine similarity to calculate the similarity score R between sentence features and object queries. The calculation formula is provided in equation (5) of the paper. Cosine similarity is a technique widely used in common machine learning to measure the similarity between two vectors, in particularly for cross-modal vectors. Besides, similarity calculation is only a part of Query-Sentence Alignment, and the rest involves weighted summation to select the final prediction mask.

In our previous response, we provided a detailed introduction of our design concept about the entire framework and the loss function, which demonstrates the effectiveness and efficacy of our proposed single-stage pipeline, and superb even SoTA performances can be obtained within an label-efficient training. We do believe this line of works will raise increaing attentions to explore better training strategies or model architectures based on single-stage pipeline, instead of separate cross-modal interaction and multi-stage training stages.

[1]. Mask3d: Mask transformer for 3d semantic instance segmentation, ICRA 2023.

[2]. Masked-attention Mask Transformer for Universal Image Segmentation, CVPR, 2022.
Thanks for your thoughtful reflections on this question and pointing out the confusion about the calculation and comparation of label efficiency. We will discuss these questions more deeply and below are our analysis and explanation:

## Q1: Rethinking whether not using the AP metric in instance segmentation

- **Why the previous works[1,2] are called "instance segmentation" :** As we mentioned in the paper, the previous works adopt two-stage workaround. The first stage utilizes a 3D instance semantic segmentation network to get instance proposals. And the second stage leverages a network to match the query with corresponding instance proposals. Because the first stage is of great importance to this task, previous works simply named this task with  "instance segmentation", while we followed the previous works to do that and will extand more discussions into our revised version to rethink how to strictly name this task.

- **Why not use AP metric in the R3DIS task:**  Thanks for this detailed reading and question. The target of referring 3D instance segmentation about a corresponding query is only one. Therefore the IoU metric is sufficient to measure this task. In addition, the Acc@x metrics also is used to measure the accuracy of the segmentation mask.  We will consider to add more comparative definition between Acc@x and AP into our revised implementation details.

- **Rethinking about our LESS:**  We will take necessary in-depth discusions to explain and compare referring 3D instance segmentation with conventional instance segmentation task. Since our method is one-stage and lacks a first instance proposal stage, the title with "instance segmentation" may be not strictly align with the common instance segmentation task. However, the core contribution of our paper is to develop a simple yet effective single-stage for this community, shifting from complicated two-stage pipeline into single-stage one, which paves more spaces for future works. We will add more in-depth discussions to the final version of the paper and consider whether to revise the confusing title.

## Q2: For the labelling time on the whole dataset

- **The number of scenes needed to annotate:**  Previous works [1,2] contain a 3D instance segmentation and a matching network. Based on their open source codes, we find that training a typical instance segmentation network requires annotations for **1513** scenes (train+val) rather than **800** scenes (which is the number of ScanRefer). Therefore, in light of the aforementioned assumption that the time required for the annotation of a single scene is 25 minutes, the overall time required for the training process should be 25 * 1,513 = 37,825 minutes, rather than the 20,000 minutes as estimated before. 

- **The number of objects needed to annotate:** In order to make a fairer and more reasonable comparison of label-efficiency between our LESS and previous works, we compare the number of objects required labelling. Firstly, we follow the open source codes of the previous works to process the dataset. From the processed dataset we find that the number of annotated objects under 1513 scenes is about **45,711**, even though excluding walls and floors. As for our LESS, we only need to annotate **11,046** objects, which is 4 times fewer compared to previous works. Here we need to note that the objects referred by ScanRefer is the subset of the objects in ScanNetv2.

- **For the comparative method of time in the paper:** As we mentioned above, the objects referred by ScanRefer are a subset of all objects among all scenes. In the context of the time comparison presented in the paper, the comparison of labeling effort is based on a single sample.  This is because that, in consideration of reality scenarios,  it is often the case that the model is adapted to incorporate new data in order to align with evolving referential requirements.  Therefore, in our method, there only necessitate just a single or a few objects needed to be labelled, rather than the entire scene. When scaling up the dataset scale for this task, our advantages will become much more significant. As a result, we compared the label effort of a single sample in the paper. Thank you for pointing out this issue, and we will make further discussion on this in our paper to reduce any potential ambiguity.

In summary, our method takes less time than the previous works in different comparison methods, which indicates our method is more label-efficient. Also We would like to extend our gratitude once more for your insightful suggestion. It is our sincere hope that the responses above could address your concerns. If you have any additional questions or suggestions, please let us know.

[1]. Text-guided graph neural networks for referring 3d instance segmentation, AAAI 2021

[2]. X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks, AAAI 2024
It seems there may have been a misunderstanding. I was expecting a detailed explanation of the significant differences between the proposed modules and existing techniques. Specifically:
* Query Mask Predictor: How does it differ from the approach used in Mask2Former, specifically in M2F3D for 3D Instance Segmentation?
* Point-Word Cross-Modal Alignment: This appears similar to cross-attention. Could you clarify the distinctions?
* Query-Sentence Alignment: How does your method compare to using the cosine CLIP score?
Thank you for your response! We are grateful and pleased that our response can address most of your concerns. Regarding the novelty of our work, we would like to answer from the following points and discuss the differences from previous works.

- The novelty of the Query Mask Predictor and Query-Sentence Alignment. As we explained in the paper, QMP can provide candidates of the target in the scene. However, different candidates have different importance to the target and most of them contains unrelated background points. To address this issue, we propose the QSA module to weight the mask and query by calculating the relationship between the candidate and the query through the cosine similarity. Moreover, **different from CLIP[1]** use cosine, which selects the mask with the highest score as the final result, our final mask prediction is facilitated by a weighted sum of the similarity score and mask prediction. Using a weighted sum enables a soft decision-making process rather than a hard and binary choice. This can help in cases where multiple masks have similar scores, allowing the final prediction to be an aggregation of these maks, which can be more accurate than any single one. As shown in Tab.x, we also conduct related experiments about this, which also confirm the advantages of our method.   **In comparison to Mask3D [2]**, our QMP and SQA generate possible mask proposals guided by the language caption, whereas Mask3D's decoder enumerates all possible proposals in a scene without any guidance. 

|          |   mIoU    |  Acc@25   |  Acc@50   |
| :------: | :-------: | :-------: | :-------: |
|  Top-1   |   33.18   |   52.93   |   28.65   |
| Baseline | **33.74** | **53.23** | **29.88** |

- The novelty of the Point-Word Cross-Modal Alignment. The PWCA is based on the cross-attention structure, which is a prevalent approach in cross-modal fusion networks. **In comparison to the original cross attention model[3]**, we have removed the position encoding in order to address the disorder of 3D point clouds. Furthermore, we have incorporated an MLP Tanh gate on top of the original model in order to regulate the information flow between language and visual features.

- The novelty of module design.  In contrast to previous research, which solely aligned word features with proposal features, we introduce a novel one-stage coarse-to-fine point cloud language cross-modal alignment paradigm. PWCA, QMP and QSA are design to align  word-point and sentence-mask, respectively. Furthermore, another significant innovation is the introduction of a coarse-to-fine background point filtering strategy, comprising Area Regularization Loss and Point-to-Point Contrastive Loss, which enables the effective processing of high-complexity, large-scale 3D scenes.

- Finally, related experiments shown that compared to previous works, our LESS is simple yet effective not only in SoTA performance but also in efficient time consumption with only binary mask supervision. As mentioned in conclusion of our paper, we aims to construct a straightforward and universal single-stage baseline to establish a solid foundation for this task, rather than focusing solely on the performance enhancement offered by a single novel model architecture, which sheds new lights on future exploration on how to better align the visual feature with language feature (cross-modal) or propose different training strategies.

We would like to express our gratitude for your valuable suggestions and for highlighting the existing issues. Hope that our response could provide the necessary clarification to address your concerns. If you have any additional questions or suggestions, we are very glad to have a further discussion.

[1]. Learning Transferable Visual Models From Natural Language Supervision, ICML 2021.

[2]. Mask3d: Mask transformer for 3d semantic instance segmentation, ICRA 2023.

[3]. Attention is All You Need, NeurIPS 2017.
Thank you to the authors for addressing my questions one by one. However, I believe the first question was not adequately answered. While I understand that you have adopted the same metrics as in previous methods, I would like a clearer understanding of the deeper reason for not using the AP metric in instance segmentation. My guess is that it is because each prediction involves only one instance, so measuring accuracy based on IoU alone suffices.

Additionally, there is an important issue I would like the authors to clarify. A core aspect of the paper is its emphasis on being label-efficient, and Table 1 compares the label effort of two-stage methods with that of the method proposed by the authors. To my knowledge, it takes approximately 25 minutes to semantically label an entire indoor scene on a per-instance basis, and about 2 minutes to label a single instance mask, which aligns with the time mentioned by the authors. However, training a typical instance segmentation network requires annotations for about 800 scenes from ScanNet, which totals approximately 25 * 800 = 20,000 minutes. On the other hand, as mentioned in the section "Dataset and Experiment Settings," training a rendering-based instance segmentation network requires 51,583 queries of 11,046 objects. Without considering the time for matching queries with objects, the total annotation time would be at least 2 * 11,046 = 22,092 minutes, which is more than the total time required for standard instance segmentation annotations. Therefore, based on this analysis, I find it unfair and unreasonable for the authors to conclude that their method is label-efficient by comparing only the annotation time for a single sample without considering the total number of samples required for training.

I look forward to a more detailed explanation of the above questions. Thank you very much!
Thank you for your detailed response. However, my concerns about the novelty of your work remain unaddressed, particularly regarding how your proposed modules differ from previous methods. I was looking for a clearer explanation of how your approach offers significant advancements rather than just adopting existing techniques. Therefore, I will maintain my original rating.