RefMask3D: Language-Guided Transformer for 3D Referring Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 17:59:22

Abstract: 3D referring segmentation is an emerging and challenging vision-language task that aims to segment the object described by a natural language expression in a point cloud scene. The key challenge behind this task is vision-language feature fusion and alignment. In this work, we propose RefMask3D to explore the comprehensive multi-modal feature interaction and understanding. First, we propose a Geometry-Enhanced Group-Word Attention to integrate language with geometrically coherent sub-clouds through cross-modal group-word attention, which effectively addresses the challenges posed by the sparse and irregular nature of point clouds. Then, we introduce a Linguistic Primitives Construction to produce semantic primitives representing distinct semantic attributes, which greatly enhance the vision-language understanding at the decoding stage. Furthermore, we introduce an Object Cluster Module that analyzes the interrelationships among linguistic primitives to consolidate their insights and pinpoint common characteristics, helping to capture holistic information and enhance the precision of target identification. The proposed RefMask3D achieves new state-of-the-art performance on 3D referring segmentation, 3D visual grounding, and also 2D referring image segmentation. Especially, RefMask3D outperforms previous state-of-the-art method by a large margin of 5.36% mIoU on the challenging ScanRefer dataset.

Main paper: 
 related works 2.1 3d instance and referring segmentation: 
 3D Instance Segmentation aims to detect and segment instances in the sparse point clouds. The success of the transformer has permeated the 3D modeling domain. In 3D object detection, methods [29, 33] employing the transformer have achieved state-ofthe-art results. Inspired by this, transformer-based methods like OneFormer3D [23] , Mask3D [37] , SPFormer [38] , MAFT [24] for instance segmentation have subsequently been developed. Mask3D [37] treats each object as an instance query. Through Transformer decoders, it learns these queries by attending to multi-scale point cloud features and the queries concurrently produce all instance masks integrating with point features.Influenced by advancements in 2D multimodality, 3D referring segmentation is increasingly gaining attention. This field focuses on segmenting a target instance based on a given language expression. TGNN [16] is the first work to solve this challenging problem and introduces aggregating textual features by considering the neighboring local structure of each instance but also emphasizes capturing spatial interactions centered around each object. X-RefSeg3D [35] follows the paradigm of TGNN and builds a multimodal graph for the given 3D environment, integrating textual and spatial connections to facilitate reasoning through the use of graph neural networks. 3D-STMN [40] is proposed to construct dense superpoint-text matching which is enhanced through dependencydriven insights in the end-to-end paradigm. Despite the notable achievements of existing methods, they fall short in facilitating comprehensive multi-modal feature interaction and understanding. This paper is dedicated to addressing this challenge.3D visual grounding aims to identify and locate the referred object in a 3D scene based on the language expression. Three datasets including ScanRefer [5] , ReferIt3D [2] and Multi3DRefer [47] are proposed to facilitate the research which contains object-expression pairs based on ScanNet [9] . Previous methods [1, 3, 12-15, 17, 36, 44, 44, 46, 48] mostly employ a two-stage pipeline. They first exploit either a 3D object detector [20, 22, 29] or ground truth information to generate object proposals. Subsequently, they utilize a text encoder [10, 27] to extract linguistic features and then identify ! ! ! " ! # ! $ ! % ! ! & ! " & ! # & ! $ & ! % & Point EncoderText Encoder The above operations are all performed on the dim N t .With Â, each object query effectively accumulates the 307 word embeddings corresponding to its designated group and 308 obtains language-cluster embeddings, namely W = ÂV .To ensure that these language-cluster embeddings appropri-310 ately complement rather than dominate the object queries, Finally, the language-cluster object query is formulated 320 by: Specifically, within a mini-batch, we organize itive samples into a set denoted as P, which con samples with different textual descriptions for the ject. Other samples not categorized as positive ar bled under the negative sample set N. A limitati due to the finite number of negative samples, co by the batch size. Additionally, the quality of pos negative samples for contrastive learning cannot be under such a random mini-batch. To address thes we introduce a memory bank and hard sample sel refine the composition of P and N for effective lea • Momory Bank Construction. Building upon from recent studies [15, 41, 44] , which emphasiz portance of a large pool of negative samples f tive contrastive learning, we introduce two distin ory banks: a visual memory bank M v 2 R Ns⇥ textual memory bank M t 2 R Ns⇥D to address tations imposed by mini-batch sizes. These memo are structured to store object embeddings and text respectively. The term N s denotes the total count ing samples, while D represents the dimensionali feature vectors. The content of M v and M t is dyn updated throughout the training process with featu all training samples. This approach allows us to ef access the entire dataset's sample pool.EQUATION• Hard Sample Selection: With the extensive sam lection in our memory bank, a key question emerg do we select the most appropriate positive and samples? Drawing from previous studies [22, 23, it's evident that hard samples play a crucial role trastive learning by aiding the development of ro resentations. Therefore, for a given target objec textual description, we specifically select objects long to the same class and are located in the same hard negative samples. This approach effectively a the issue of distractions caused by multiple instanc same class in a scene. To ensure a stable training the rest of the negative samples are acquired via sampling. For the positive sample selection, given object, we include all its corresponding textual desc Utilizing the constructed positive and negativ sets, we develop the inter-sample loss L inter using positive supervised contrastive approach, formularL inter = 1 |P| X a + 2P log exp(< a, a + > /⌧ P a 0 2P,N exp(< a, a 0where a represents the anchor item, which can be object or a language feature. When a is an objec ding, a + and a 0 are text embeddings, and vice ve primary aim of L inter is to refine the embedding that embeddings of the same object from differen ties (e.g., text and point) are brought closer, while dings of different objects are distanced. This app 5 the referred object through feature fusion and language feature matching. In this context, InstanceRefer [46] streamlines the task by approaching it as an instance-matching challenge. LanguageRefer [36] turns the multi-modal problem into a language modeling scenario, achieved by substituting 3D features with predicted object labels. SAT [44] employs additional 2D semantics to boost multi-modal alignment and achieve superior performance.EQUATIONIn contrast to the two-stage methods, 3D-SPS [31] adopts a progressive keypoint selection strategy guided by language and locates the target directly within a single-stage network for the first time. BUTD-DETR [18] calculates the correlation between each word and object, subsequently selecting the word features that align with the object's name to match the corresponding object. EDA [41] builds upon BUTD-DETR and further proposes text decoupling to separate text components by grammatical analysis. Single-stage methods have taken a dominant role and both BUTD-DETR and EDA have demonstrated superior performance. However, these methods either need to train an additional text span predictor network or employ external tools to decouple text which is time-consuming and complicated. Besides, they are designed for visual grounding and cannot perfectly adapt to referring segmentation. To this end, we propose our method to simplify the image-text matching and apply it to both 3D referring segmentation and visual grounding. 
architecture overview: 
 Figure 2 shows the overall architecture of our proposed end-toend 3D referring segmentation approach RefMask3D. RefMask3D takes an input pair comprising a point cloud scene and a textual description, and produces a point-wise mask for the target object indicated by the textual description. The point cloud scene, denoted as 𝑃 𝑐𝑙𝑜𝑢𝑑 ∈ R 𝑁 × (3+𝐹 ) , consists of a total of 𝑁 points, with each point containing 3D coordinate information ∈ R 3 and an auxiliary feature ∈ R 𝐹 like color.First, a text encoder is employed to embed the text description into language features, denoted as 𝑭 𝑡 ∈ R 𝑁 𝑡 ×𝐷 , where 𝐷 and 𝑁 𝑡 represent the number of channels and words, respectively. We extract point features from the point encoder which builds deep interaction between vision and language through Geometry-Enhanced Group-Word Attention. The point encoder is a 3D U-Net-like backbone. In the vanilla U-Net architecture, the feature map at the 𝑖-th layer in the upsampling path is obtained by combining the features from the corresponding 𝑖-th layer in the downsampling path with the features from the (𝑖 + 1)-th layer which is illustrated in the gray line in Figure 2 . The vanilla output features denote as 𝑭 𝑖 ∈ R 𝑁 𝑖 ×𝐷 , 𝑖 = {1, 2, 3, 4}. Therefore, the multi-scale visionlanguage features derived from the point encoder are denoted as 𝑭 ′ 𝑖 ∈ R 𝑁 𝑖 ×𝐷 , 𝑖 = {1, 2, 3, 4}. The full-resolution feature map 𝑭 ′ 0 ∈ R 𝑁 ×𝐷 is used as the mask feature for mask predictions. Then, Linguistic Primitives Construction produces primitives O ′ to represent distinct semantic attributes by informative language cues, enhancing the model's capability to accurately localize and identify targets through interaction with specific linguistic information. The linguistic primitives O ′ , the multi-scale point features{𝑭 ′ 1 , 𝑭 ′ 2 , 𝑭 ′ 3 , 𝑭 ′ 4 }, and language features 𝑭 𝑡 collectively serve as input to a 4-layer cross-modal transformer decoder, which is applied 𝐿 times, leading to an overall transformer decoder with 4𝐿 layers.Next, the output of linguistic primitives through transformer decoder and object queries O 𝑐 are fed into the Object Cluster Module to analyze the interrelationships among linguistic primitives to unify their insights and pinpoint common characteristics. Finally, we obtain masks by performing a multiplication operation between the last-layer object embedding O ′ 𝑐 and mask feature 𝑭 ′ 0 . Meanwhile, we employ a MLP head to predict confidence score. The mask with the highest confidence score is selected as the output. Different from models [16, 18, 31, 41] that stack a modality fusion module on top of the vision or language backbones, our approach integrates multi-modal fusion within the point encoder, leveraging the advantages of an end-to-end paradigm. Early-stage fusion of cross-modal features is believed to enhance the effectiveness of the fusion process in 2D-RES tasks [43] . However, it remains unexplored within the 3D field. The 3D domain, characterized by the sparse and irregular distribution of point cloud data, presents unique challenges to cross-modal fusion. The conventional 2D method of directly establishing relationships between each point and linguistic terms introduces excessive noise, potentially impairing performance due to the complexity and variability of point cloud structures. Addressing this challenge, we introduce Geometry-Enhanced Group-Word Attention mechanism. Unlike traditional methods that calculate cross-modal relationships at the individual point level, our approach innovatively processes local groups (subclouds) with geometrically adjacent points. This methodology not only mitigates the noise associated with direct point-to-word correlations but also capitalizes on the inherent geometric relationships within point clouds, enhancing the model's ability to accurately integrate linguistic and 3D structure. Firstly, we employ farthest point sampling (FPS) to downsample the point number of features 𝑭 𝑖 from 𝑁 𝑖 to 𝑁 𝑔 , which serves as a set of local centroids. Following the identification of these local centroids, we proceed to gather their neighboring points. This is achieved through the application of the 𝑘-nearest Neighbor (𝑘-NN) algorithm, which allows us to systematically associate each local centroid with its adjacent points based on its geometric proximity. The aforementioned process is formulated as:EQUATIONAs such, we obtain 𝑁 𝑔 local groups, each of which consists of 𝑘 points. Next, we perform group-word cross-modal attention between 𝑁 𝑔 local groups and language features 𝑭 𝑡 , We formulate it as:EQUATIONwhere S 𝑔 𝑖 denotes the relationship between local groups 𝑭 𝑔 𝑖 and 𝑭 𝑡 and we sum along 𝑘 nearest neighbors dimentation to obtain the relationship between each local centroid with 𝑭 𝑡 denoted as S 𝑐 𝑖 ∈ R 𝑁 𝑔 ×𝑁 𝑡 . Subsequently, we extract linguistic features 𝑭 𝑐 𝑖𝑡 related to the local centroids as,EQUATIONFinally, inspired by PointNet++ [34] , we propagate 𝑭 𝑐 𝑖𝑡 from each local centroid to its corresponding original point number with a weighted summation and obtain linguistic features related to the whole point feature map 𝑭 𝑖𝑡 ∈ R 𝑁 𝑖 ×𝐷 which have the same shape as 𝑭 𝑖 . We combine them to generate multi-modal feature maps 𝑭 ′ 𝑖 via element-wise multiplication.EQUATIONThrough the above process, we have implemented cross-modal fusion based on the sparse and irregular characteristics of 3D data in the point feature extraction stage.Besides, in vanilla cross-modal attention, the challenge arises when dealing with situations where a point has no corresponding words related to it. To tackle this, we introduce learnable background embeddings for language features, denoted as 𝑻 𝑏𝑔 ∈ R 𝐷 . This strategy is designed to allow points without corresponding text information to focus on a generic background text embedding, 𝑻 𝑏𝑔 , reducing the potential distortion caused by unrelated text on the point feature. Specifically, at each feature encoding layer 𝑖, we perform a concatenation operation on the language feature 𝑭 𝑡 with the background feature 𝑻 𝑏𝑔 . Subsequently, the above relationship calculation process becomes:EQUATIONwhere [; ] represents the concatenation operation, and 𝐷 is the channel dimension. S 𝑔 ′𝑖 enables point features without language element correspondence to emphasize the background feature, reducing irrelevant feature impact. The operation R (•) is used to remove the last column from a given matrix, which prevents the integration of the background feature into the final fused feature. As such, we obtain 𝑭 𝑐 ′ 𝑖𝑡 which represent refined linguistic features related to the local centroids and not influenced by irrelevant words. The background embedding is a learnable parameter, designed to capture the overall data distribution of the dataset and represent background information effectively. This embedding is only used during the attention calculation process. By incorporating the background embedding, we facilitate more accurate cross-modal interactions, which are not adversely influenced by irrelevant words.In the 3D domain, the existing approaches typically initialize queries with point coordinates sampled from the input point cloud [29, 33, 37] . These sampling strategies are based on either the farthest point sampling method [33, 37] or the 𝑘-closest points sampling technique [18, 29] . Such methods are prevalent in tasks like visual grounding [18, 29, 41] and instance segmentation [37] . However, a key limitation of these methods is their neglect of language information, which is vital for precise referring segmentation. Relying solely on farthest point sampling often results in predictions that stray from the objects of interest, especially in sparse scenes, thereby hindering convergence or resulting in the loss of objects. Although 3D-SPS [31] attempts to address this issue by introducing a language-aware query selection that selects the top-𝑘 query points close to the given language description. Yet, this approach is manually tailored and susceptible to noise. This becomes particularly problematic when the selected points don't accurately reflect the target object or when they are all anchored to a single word. In response to these challenges, we propose a Linguistic Primitives Construction to incorporate semantic content from language that learn diverse linguistic primitives to locate objects related to corresponding semantic properties separately.In the process outlined in Figure 3 , we begin by initializing 𝑁 𝑜 learnable primitives by sampling them from different Gaussian distributions, each specifying a distinct semantic property to be he background embedding is a learnable parameter, deed to capture the overall data distribution of the dataset represent background information effectively. This eming is only used during the attention calculation pro-. By incorporating the background embedding, we faate more accurate cross-modal interactions, which are adversely influenced by irrelevant words.Language-Cluster Query Generation e 3D domain, the existing approaches typically initialueries with point coordinates sampled from the input t cloud [31, 35, 39] . These sampling strategies are d on either the farthest point sampling method [35, 39] e k-closest points sampling technique [19, 31] . Such ods are prevalent in tasks like visual grounding [19, 43] and instance segmentation [39] . However, a key tation of these methods is their neglect of language ination, which is vital for precise referring segmentation. Finally, the language-cluster object query is formulated 320 by:EQUATIONwhere ! 1 is the linear projection layer. O 0 represents the 323 language-cluster object queries, with distinct queries con-324 centrating on various clustered language semantics. The EQUATIONby the batch size. Additionally, the quality of positive and negative samples for contrastive learning cannot be ensured under such a random mini-batch. To address these issues we introduce a memory bank and hard sample selection to refine the composition of P and N for effective learning.• Momory Bank Construction. Building upon insights from recent studies [15, 41, 44] , which emphasize the im portance of a large pool of negative samples for effec tive contrastive learning, we introduce two distinct mem ory banks: a visual memory bank M v 2 R Ns⇥D , and a textual memory bank M t 2 R Ns⇥D to address the limi tations imposed by mini-batch sizes. These memory banks are structured to store object embeddings and text features respectively. The term N s denotes the total count of train ing samples, while D represents the dimensionality of the feature vectors. The content of M v and M t is dynamically updated throughout the training process with features from all training samples. This approach allows us to effectively access the entire dataset's sample pool.• Hard Sample Selection: With the extensive sample col lection in our memory bank, a key question emerges: how do we select the most appropriate positive and negative samples? Drawing from previous studies [22, 23, 37, 41] it's evident that hard samples play a crucial role in con trastive learning by aiding the development of robust rep resentations. Therefore, for a given target object and its textual description, we specifically select objects that be long to the same class and are located in the same scene as hard negative samples. This approach effectively addresses the issue of distractions caused by multiple instances of the same class in a scene. To ensure a stable training process the rest of the negative samples are acquired via random sampling. For the positive sample selection, given the targe object, we include all its corresponding textual descriptions Utilizing the constructed positive and negative sample sets, we develop the inter-sample loss L inter using a multi positive supervised contrastive approach, formularted as:L inter = 1 |P| X a + 2Plog exp(< a, a + > /⌧ ) P a 0 2P,N exp(< a, a > /⌧ ) , (7 where a represents the anchor item, which can be either an object or a language feature. When a is an object embed ding, a + and a 0 are text embeddings, and vice versa. The primary aim of L inter is to refine the embedding space so that embeddings of the same object from different modali ties (e.g., text and point) are brought closer, while embed dings of different objects are distanced. This approach ef Finally, the language-cluster object query is formulated 320 by:EQUATION)322where ! 1 is the linear projection layer. O 0 represents the 323 language-cluster object queries, with distinct queries con-324 centrating on various clustered language semantics. The object, we include all its corresponding textual descriptions.Utilizing the constructed positive and negative sample 397 sets, we develop the inter-sample loss L inter using a multi-398 positive supervised contrastive approach, formularted as:EQUATIONwhere a represents the anchor item, which can be either an 401 object or a language feature. When a is an object embed- EQUATIONwhere 𝑁 𝑜 is the number of primitives, 𝜇 𝑖 and 𝜎 𝑖 are learnable parameters of the Gaussian distribution. These primitives are assumed to contain diverse semantics e.g., shape, color, size, material, relationship, and location. Then, we let each primitive aggregate language of distinct linguistic entities and extract corresponding information from the given language. For example, in the description in Figure 2 , the primitive responsible for color concentrates for "white", and the primitive responsible for location highlight "above". To achieve this, three distinct linear layers, 𝜔 𝑄 , 𝜔 𝐾 , and 𝜔 𝑉 , are employed to project the primitives and the language word feature 𝑭 𝑡 into a shared feature space, which are represented as 𝑄, 𝐾, and 𝑉 , respectively. Subsequently, we calculate the similarity between each pair of primitive and word feature, formulated as:EQUATIONAfter computing the similarity matrix A, unlike the conventional cross-attention that performs a softmax operation to normalize the correlation of all words corresponding to a specific primitive, we aim to enable each primitive to concentrate on the most related semantic. e.g. "white". Therefore, we employ the differentiable Gumbel-Softmax approach [19, 42] to assign the word features to the primitives 𝑄 in a differentiable way and generate the corresponding clustering matrix Â between 𝑄 and 𝐾,EQUATIONWith Â, each primitive effectively chooses the word embeddings corresponding to its semantic and then obtains linguistic primitives O ′ , which are formulated by:EQUATIONwhere 𝜔 1 is the linear projection layer. The linguistic primitives are designed to exhibit semantic patterns. Feeding such primitives into the transformer decoder enables it to emphasize diverse language information, which contributes to accurately identifying target objects in the later stage.Each linguistic primitive focuses on different semantic patterns in the given point cloud that correlate with their linguistic attributes. However, our ultimate goal, based on the given text, is to identify a unique target object, which requires a comprehensive understanding of language. For example, the primitive responsible for "white" will focus on all the white parts of the picture, but we need to find the one above the cabinets. To this end, we employ an Object Cluster Module to explore the relationships among the linguistic primitives to identify commonalities and differences in their focus areas. Specifically, we initialize 𝑁 𝑐 potential object queries O 𝑐 ∈ R 𝑁 𝑐 ×𝐷 , leveraging sentence-level linguistic features. This initialization facilitates a deeper grasp of object descriptions. We conduct a self-attention mechanism to extract these common characteristics from linguistic primitives. In the decoding process, we input these object queries, treating them as queries, with the common characteristics enriched linguistic primitives acting as keys/values. This setup enables the decoder to merge linguistic insights from linguistic primitives into object queries, efficiently identifying and grouping the referred object's queries into O ′ 𝑐 , thereby achieving precise object identification.While the proposed Object Cluster Module aids in identifying target objects, it does not effectively mitigate ambiguities arising from other embeddings. These ambiguities could lead to false positives during the inference stage. We employ contrastive learning to differentiate the target token from others. This is achieved by maximizing the similarity between the target token and the corresponding expression while minimizing similarities with non-target (negative) pairs. The process can be formulated as,EQUATIONwhere <, > denotes cosine similarity, 𝜏 is the temperature parameter, 𝑡 ∈ R 𝐷 represents the text embedding obtained by pooling language features 𝑭 𝑡 . The set {𝑜 𝑘 } 𝑁 𝑜 𝑘=1 comprises object embeddings derived from the output of the object cluster decoder O ′ 𝑐 , with 𝑜 + being positive object embedding that matches with ground-truth mask.During the training phase, we use Hungarian matching to select an object embedding with the lowest cost to the ground-truth object as our output. The matching losses include single-class crossentropy loss on the output score, binary cross-entropy loss, and dice loss on mask prediction,EQUATIONThe total training loss is calculated as: L = L match + 𝜆 con L con , where the weights 𝜆 cls , 𝜆 bce , 𝜆 dice , and 𝜆 con are used for balancing different loss terms.  
experiments 4.1 datasets and evaluation metrics: 
 The ScanRefer dataset [5] is based on the 800 ScanNet scenes [9] and comprises 51, 583 linguistic descriptions. On average, each scene contains 13.81 objects and 64.48 descriptions. The dataset's performance is evaluated using the Acc@𝑚IoU metric. It represents the proportion of descriptions where the predicted box or mask aligns with the ground truth, having an IoU > 𝑚, where 𝑚 ∈ {0.25, 0.5}. The results are categorized into unique and multiple.An object is regarded as unique if it's the sole entity of its class in a scene, and multiple otherwise. For 3D referring segmentation, there is another metric mean intersection-over-union (mIoU) which calculates the IoU between the prediction and ground truth masks averaged across all the test samples.Our experimental setup generally adheres to the default configurations of Mask3D [37] , except where explicitly noted. For the point encoder, we utilize Minkowski Res16UNet34C [8] . For language embeddings, we utilize BERT [10] , following its proven effectiveness in capturing linguistic nuances. We employ the AdamW optimizer [21] with an initial learning rate of 4×10 -5 , accompanied by a cosine decay schedule to adjust the learning rate progressively. We configure the training process to span a maximum of 20 epochs with a batch size of 16. Following Mask3D, we set the coefficients for different loss functions 𝜆 𝑐𝑙𝑠 , 𝜆 𝑏𝑐𝑒 , and 𝜆 𝑑𝑖𝑐𝑒 to 2, 5, and 5, respectively. For hyper-parameter optimization, we meticulously choose values that best suit our model's architecture and the task's complexity. Specifically, we set the contrastive loss coefficient 𝜆 con to 0.3, the temperature parameter 𝜏 to 0.05 to manage the softmax distribution's sharpness, and the layer count 𝐿 to 3. Additionally, • Component Analysis. We conduct detailed experiments to demonstrate the effectiveness of each component in the proposed method. As shown in Table 1 , our vanilla baseline integrating Mask3D with language input only in the transformer decoder achieves a mIoU of 39.24%, establishing a strong pipeline. Then, the introduction of the Geometry-Enhanced Group-Word Attention (GEGWA) enhances performance by 2.92% mIoU, highlighting its effectiveness in fusing linguistic and visual features in the point encoder and minimizing the influence of irrelevant points and words for improved cross-modal interaction. Furthermore, incorporating the Linguistic Primitives Construction (LPC) boosts mIoU by 2.34%, underscoring its importance in clustering diverse semantic language information for target identification in 3D referring segmentation. The addition of the Object Cluster Module (OCM) further enhances our model, contributing a 1.57% increase in mIoU by constructing a discriminative identification for the target. By combining all these components in RefMask3D, we achieve new state-of-the-art performance with a remarkable mIoU of 44.86% and an Acc@0.5 of 49.24% on the challenging ScanRefer dataset, demonstrating the effectiveness of our proposed approach.• Different Fusion Strategies for GEGWA. In Table 2 , we evaluate the impact of different fusion designs. The absence of visionlanguage fusion within the encoder results in a decrease of 2.92% in mIoU, confirming the effectiveness of early-stage fusion. The w/o bg configuration uses a standard cross-attention module without background embeddings to filter out irrelevant information. This approach underperforms our method by 0.91% in mIoU, highlighting the importance of eliminating irrelevant data in cross-modal interactions. The point-level fusion setup directly conducts fusion in the individual point without considering the neighbors and geometric information, leading to a 1.83% reduction in mIoU. This outcome demonstrates that processes local groups with geometrically adjacent points are beneficial for overall performance.• Different Group and Neighbors Numbers for GEGWA. In Table 3 , we assess the impact of varying the number of groups 𝑁 𝑔 and the number of neighbors 𝑘. By holding 𝑁 𝑔 constant at 64 and incrementally increasing 𝑘 from 8 to 32, we observed a rapid improvement in performance followed by a plateau. Similarly, fixing 𝑘 at 16 and elevating 𝑁 𝑔 from 32 to 128 also resulted in a swift rise in outcomes, which then stabilized. Considering computational efficiency, we ultimately opted for 𝑁 𝑔 = 64 and 𝑘 = 16 as the configuration that yields optimal results.• Comparison with Other Designs for LPC. In Table 4 , we examine the effect of input queries fed into the Transformer Decoder. The FPS method, widely used in point sampling and described in Point-Net++ [34] , covers the entire scene but lacks focus on languagerelevant points. The Top-𝑘 approach [31] selects the top 𝑘 points most relevant to a given text based on object confidence. We use 𝑘 = 100 in our experiments. According to the results in Table 4 , the proposed Linguistic Primitives Construction outperforms other query inputs, achieving promising results. Besides, we observed a notable point when not utilizing Gaussian initialization for the primitives, opting instead for learnable parameters directly, which resulted in a performance decrease of 1.07% mIoU. This finding There is the brown chair nearest the TV. underscores the significance of employing Gaussian initialization to ensure independence among multiple primitives, playing a crucial role in effectively capturing diverse semantic information for the final result. Additionally, the employment of Gumbel-softmax enhances the overall performance which is in line with our analysis.• 3D Referring Segmentation Benchmark Results. We evaluate RefMask3D on ScanRefer [5] and report the mIoU and Acc@𝑚IoU performance in Table 5 . For a fair comparison with SOTA methods, we implement BUTD-DETR [18] and EDA [41] with an auxiliary mask head [41] . The results show that RefMask3D outperforms existing methods, achieving the highest scores across all metrics with a significant improvement of +5.36% in mIoU. This demonstrates the effectiveness of RefMask3D and highlights its exceptional visionlanguage understanding capabilities.• 3D Visual Grounding Benchmark Results. Instance segmentation predictions can be readily converted into bounding box predictions by determining the minimum and maximum coordinates of the masked instances. With this capability, we extend our experimentation to the realm of 3D visual grounding. Table 6 shows the results on ScanRefer dataset [5] , with Acc@0.5 as the metric. Our findings show that RefMask3D surpasses previous methods, achieving an improvement of +3.36% in Acc@0.5 compared to previous state-of-the-art methods. It is worth noting that this achievement [39] 70.5 62.3 59.9 RefTR [25] 70.6 --LAVT [43] 72.7 62.1 61.2 VLT [11] 73.0 63.5 63.5 GRES [26] 73.8 66.0 65.0 RefMask3D (ours) 75.3 66.9 66.3is made using only the bounding boxes derived from instance segmentation predictions, without dedicated designs for 3D visual grounding. This highlights the effectiveness of our RefMask3D in extracting and understanding more discriminative and effective vision-language joint representations.• 2D Referring Image Segmentation Results. In Table 7 , we apply RefMask3D to 2D referring image segmentation by using Mask2Former [7] framework based on Swin-B [28] backbone. We remove GEGWA component which is specifically designed for 3D realm. Our results are compared with the leading methods on Re-fCOCO/+/g [32, 45] . We train a single RefMask3D model on Ref-COCO/+/g without the need for extensive pre-training. RefMask3D outperforms previous methods on all three benchmarks, underscoring its versatility and effectiveness in referring segmentation.In Figure 4 , we visualize the primitive heatmap on the given point cloud. Different primitives represent distinct semantic attributes like "color" (a), "relationship" (b), "name" (c). The above qualitative results highlight our Linguistic Primitive Construction is capable of acquiring corresponding attribute values which improves the ability to accurately localize and identify the target object.In Figure 5 , we visualize the grounding truth masks, predictions of TGNN [16] , and predictions of RefMask3D in each column from top to bottom. RefMask3D effectively interprets complex language phrases like "soap dispenser" (a) and accurately segments the target object, even among items of the same category (b)-(f). In contrast, TGNN often misinterprets sentences and is easily misled by distractors. These qualitative results highlight RefMask3D's superior ability to fuse and align vision-language features, enabling a deeper understanding of complex language descriptions. 
conclusion: 
 In this work, we present an effective single-stage approach Ref-Mask3D to address the challenging 3D referring segmentation and grounding. In the proposed RefMask3D, first, a Geometry-Enhanced Group-Word Attention (GEGWA) is introduced for continuous and contextual fusion of language and vision features at the encoding stage. Then, a Linguistic Primitives Construction (LPC) is utilized to learn semantic primitives for representing distinct semantic attributes, enhancing fine-grained vision-language understanding at decoding stage. Moreover, an Object Cluster Module (OCM) is designed to capture holistic information and produce object embedding for generating the final segmentation prediction. The proposed RefMask3D consistently achieves new state-of-the-art performance on 3D referring segmentation, 3D visual grounding, and 2D referring image segmentation. Especially, RefMask3D surpasses the previous 3D referring segmentation methods by a large margin of 5.36% mIoU on ScanRefer dataset, demonstrating its effectiveness.  
matching (b) refmask3d: 
 Instance Segmentation Geometry-Enhanced Group-Word Attention Object Cluster (a) Two-stage Framework "This is a white kitchen counter.Group A "This is a white kitchen counter.Contrastive LossFigure 1 : (a) Two-stage framework, fusing language features in the later matching stage, exhibit limited interactions and weak alignment between vision and language features. In contrast, (b) our RefMask3D conducts comprehensive visionlanguage fusion in both the early feature encoding stage and decoding stage. Combined with contrastive learning, our model learns a well-structured vision-language joint feature space than two-stage methods.The prevailing methods [2, 5, 16, 35] in 3D referring segmentation, as shown in Figure 1 (a), typically adopt a two-stage segmentationthen-matching pipeline that firstly segments redundant objects using a pre-trained instance segmentation model, and then selects the target object by matching visual and linguistic features [16, 35] . However, this pipeline exhibits inherent limitations. Given that the matching phase relies on segmentation predictions from the first phase, any omissions or inaccuracies at the first phase inevitably significantly weaken the accuracy of the following matching phase. Furthermore, simplistic incorporation of textual information during matching fails to analyze the semantic impact of individual words, thus overlooking complex semantic nuances and leading to matching errors. These issues underscore the necessity for a unified approach that integrates both segmentation and matching processes with a deeper linguistic understanding. In response, our work introduces a streamlined, effective end-to-end pipeline that comprehensively leverages language information. The proposed method, RefMask3D, enhances the interaction and understanding between vision and language, aiming to enhance the performance of 3D referring segmentation.First, different from previous two-stage works [2, 5, 16, 35 ] that adopt segmentation-then-matching pipeline, we propose to fully harness the early feature encoding layers to extract rich multi-modal context. To this end, we introduce a Geometry-Enhanced Group-Word Attention, which conducts cross-modal attention between language and local groups (sub-clouds) with geometrically adjacent points at each stage of the Point Encoder. This integration not only reduces the noise from direct point-to-word correlations, which is common due to the sparse and irregular nature of point clouds, but also leverages the intrinsic geometric relationships and fine-grained 3D structure within the point clouds. The proposed Geometry-Enhanced Group-Word Attention significantly improves the model's ability to interact with and understand linguistic and geometric data. Additionally, we incorporate a learnable "background" language token to prevent the entanglement of irrelevant language features with local group features. For example, as shown in Figure 1 (b), "Group A" as a non-target group exhibits low responses to these irrelevant words. This approach ensures that point features are enriched with semantic-related linguistic information, maintaining a continuous and context-aware awareness of the relevant language context at each group or point in the network.Second, with the fused vision-language features, we further design how to effectively identify the target object in decoder. Effectively localizing the target significantly hinges on the cues provided by the query input to the cross-modal decoder [11, 29, 31] . A majority of existing works [18, 33, 37, 41] initialize non-parametric query with point positions sampled with furthest point sampling (FPS) or parametric learnable query. Such strategies typically aim to scan the whole point cloud scene to identify all possible objects as candidates. However, it inadvertently sidesteps an essential aspect: the fact that only one language-referred target object is present, potentially leading to insufficient training or posing challenges in optimization. Although 3D-SPS [31] proposes to select the top-𝑘 query points close to the given language description to alleviate this issue, the selection process is susceptible to noise, particularly in scenarios where the chosen points do not encompass the languagerelated target. To address these challenges, we introduce a strategy termed Linguistic Primitives Construction (LPC). We initialize a diverse set of primitives, each designed to represent distinct semantic attributes such as shape, color, size, relationships, location, and so on. By engaging in interactions with specific linguistic information, these primitives are capable of acquiring corresponding attributes. Feeding these semantically enriched primitives into the decoder enhances the network's focus on diverse semantics within the point cloud, thereby significantly improving the model's ability to accurately localize and identify the target object.Furthermore, an Object Cluster Module is proposed to capture holistic information and generate the object embedding for segmenting the target object. Linguistic primitives are designed to focus on specific parts of a point cloud that correlate with their semantic attributes. However, our ultimate goal is to identify a unique target object based on the given text, which requires a holistic understanding of language. To achieve this, we propose an Object Cluster Module. This module first explores the relationships among the linguistic primitives to discern commonalities and differences in their focus areas. Utilizing this insight, we initialize language-based queries to capture these common characteristics, which form the final object embedding crucial for identifying the target object. The proposed Object Cluster Module greatly help the model to deepen the holistic understanding of linguistic and visual information.Our main contributions are summarised as follows:• We propose a Geometry-Enhanced Group-Word Attention, which enhances cross-modal interactions by integrating language with geometrically coherent sub-clouds, effectively addressing the challenges posed by the sparse and irregular nature of point clouds. • We design a Linguistic Primitives Construction, a strategy that learns primitives to represent distinct semantic attributes, enhancing the model's capability to accurately identify targets through interaction with specific linguistic information. • We introduce an Object Cluster Module that analyzes the interrelationships among linguistic primitives to unify their insights and pinpoint common characteristics to deepen the holistic understanding of linguistic and visual information. • We achieve new state-of-the-art performance on 3D referring segmentation and visual grounding, and significantly outperform previous methods by a large margin, e.g., 5.36% mIoU on the challenging ScanRefer dataset. 
 
 Appendix: & ! $ & 


Decision: After considering all reviews, the rebuttal, and the subsequent discussion, the consensus is to accept the paper.

Reviews: review: paper_summary: paper_summary: The paper presents RefMask3D, an innovative model for 3D referring segmentation. RefMask3D integrates language with geometrically coherent sub-clouds using a Geometry-Enhanced Group-Word Attention mechanism and introduces a Linguistic Primitives Construction and Object Cluster Module. The paper provides impressive experimental results that fully validate the effectiveness of the proposed method, achieving SOTA performance. However, the paper lacks some experiments that I would have liked to see, and some of the proposed structures do not demonstrate significant novelty, leaning more towards engineering and the integration of existing methods. Nonetheless, I will still give it a borderline accept score.
strength_weakness: strength_weakness: 1. The proposed Geometry-Enhanced Group-Word Attention mechanism and Linguistic Primitives Construction module offer effective methods for vision-language feature fusion and alignment.
2. The paper provides detailed experiments and ablation studies that validate the contributions of each component of the model.
limitations: limitations: My first concern is whether the proposed method will significantly increase the computational burden? Beside, for the 3D visual grounding task, the paper lacks comparisons with some advanced methods, such as 3D-VisTA, which shows better performance across multiple metrics compared to the proposed method. For the 2D tasks, the paper does not include comparisons with advanced methods based on SAM. Additionally, I am curious about the potential effectiveness of using advanced 3D visual grounding methods to detect the bounding box of the referring object and then finding the point cloud within that box.
paper_summary: paper_summary: Aiming to handle the 3D referring segmentation task, this paper proposes RefMask3D to explore the comprehensive multi-modal feature interaction and understanding. It contains 3 key points: (1) Geometry-Enhanced Group-Word Attention; (2)  Linguistic Primitives Construction; (3) Object Cluster Module. From the experimental results, it shows that RefMask3D achieves SOTA performance on multiple tasks, including: 3D referring segmentation, 3D visual grounding, and 2D referring image segmentation.
strength_weakness: strength_weakness: - This paper has good organization and clear figures. It is easy for the readers to follow.
 - RefMask3D achieves SOTA performance on multiple tasks: 3D referring segmentation, 3D visual grounding, and 2D referring image segmentation. Furthermore, it outperforms previous SOTA by a large margin of 5.36% mIoU on ScanRefer benchmark.
limitations: limitations: - The primitives contain common attributes such as location, color, and relationship. In the description, some of the primitives may be related to referred objects, rather than the target. Does the primitives of the reference mislead the learning of the target mask?
 - For the one-stage methods, different segmentation backbones may affect the final metrics on ScanRefer. It is suggested that the authors also conduct experiments on segmentation-independent datasets such as ReferIt3D to verify the cross-modal learning ability of the proposed method.
paper_summary: paper_summary: The main content of this paper is the introduction of a novel approach called RefMask3D for3D referring segmentation and grounding. The paper addresses the challenges of fusing and aligning vision-language features, enhancing feature interaction, and understanding complex language descriptions in the context of3D visual scenes.
strength_weakness: strength_weakness: It is very novelty. The paper proposes several novel contributions.  It introduces a Geometry-Enhanced Group-Word Attention mechanism that enhances cross-modal interactions by integrating language with geometrically coherent sub-clouds.  It also presents a Linguistic Primitives Construction strategy to learn primitives for representing distinct semantic attributes.  Additionally, an Object Cluster Module is introduced to analyze the interrelationships among linguistic primitives.  These novel ideas contribute to the improvement of 3D referring segmentation and visual grounding tasks.
limitations: limitations: The paper briefly mentions the performance of the proposed method (RefMask3D) in terms of mIoU (mean Intersection over Union) and Acc@0.5 (accuracy at an IoU threshold of 0.5) on the ScanRefer dataset.  While achieving state-of-the-art performance is mentioned, the paper does not provide a comprehensive evaluation or comparison with other existing methods. For example, 2D-based methods.
Besides, the paper evaluates the proposed method on several benchmark datasets, including ScanRefer.  It compares the performance of RefMask3D with state-of-the-art methods, demonstrating superior results in terms of mIoU and Acc@IoU metrics. The evaluation is comprehensive and provides a fair comparison with existing approaches.