XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation

Authors: ziyi  wang
yanbo  wang
xumin  yu
jie  zhou
jiwen  lu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-25 05:19:39

Abstract: Existing methodologies in open vocabulary 3D semantic segmentation primarily concentrate on establishing a unified feature space encompassing 3D, 2D, and textual modalities. Nevertheless, traditional techniques such as global feature alignment or vision-language model distillation tend to impose only approximate correspondence, struggling notably with delineating fine-grained segmentation boundaries. To address this gap, we propose a more meticulous mask-level alignment between 3D features and the 2D-text embedding space through a cross-modal mask reasoning framework, XMask3D. In our approach, we developed a mask generator based on the denoising UNet from a pre-trained diffusion model, leveraging its capability for precise textual control over dense pixel representations and enhancing the open-world adaptability of the generated masks. We further integrate 3D global features as implicit conditions into the pre-trained 2D denoising UNet, enabling the generation of segmentation masks with additional 3D geometry awareness. Subsequently, the generated 2D masks are employed to align mask-level 3D representations with the vision-language feature space, thereby augmenting the open vocabulary capability of 3D geometry embeddings. Finally, we fuse complementary 2D and 3D mask features, resulting in competitive performance across multiple benchmarks for 3D open vocabulary semantic segmentation. Code is available at https://github.com/wangzy22/XMask3D.

Main paper: 
 introduction: 
 As the integration of vision and language in deep learning continues to expand, text descriptions are increasingly utilized in visual generation [37, 50, 39, 35, 26, 40] and perception [36, 22, 23, 13, 45, 12] tasks. This integration enhances the adaptability of models in real-world applications and improves user experiences in customized artificial intelligence systems. Open vocabulary 3D semantic segmentation exemplifies a perception task that is trained on base categories and demands robust extrapolation capabilities to discriminate fine-grained geometry in novel categories that are invisible during training. The base and novel classes are only linked by the shared open vocabulary within the language space. However, constructing a shared 3D-text space while maintaining precise, modality-specific representation remains a significant challenge. Successfully addressing this issue would advance open vocabulary 3D semantic segmentation, facilitating virtual reality interactions and manipulations, and thereby contributing to the development of user-friendly robotics and autonomous driving technologies.Existing approaches for open vocabulary 3D perception predominantly aim to bridge the gap between 3D and text representations by using the 2D modality as an intermediary. One line of research [34, 11, 47, 16] suggests aligning 3D embeddings with vision-language embedding spaces through global, patch-wise, or point-wise contrastive loss, while another line [49, 28, 44] investigates distilling open vocabulary knowledge from foundational vision-language models into 3D models. However, global feature alignment and model distillation techniques tend to overlook fine-grained 3D geometric details and produce coarse results. Conversely, point-wise contrastive learning is prone to noise and outliers.Although patch-wise feature alignment offers a compromise, a single patch in 2D may correspond to multiple unrelated and discontinuous regions in 3D, which can be misleading and ineffective.In this paper, we propose a more precise and consistent mask-level alignment between 3D features and the 2D-text embedding space, achieved through our proposed cross-modal mask reasoning method. The proposed XMask3D model comprises a 3D branch, a 2D branch, and a fusion block. The 3D branch, adaptable as any point cloud segmentation model, excels in capturing geometric features but struggles with novel category extrapolation. Conversely, the 2D branch serves as a mask generator, which predicts masks with embeddings aligned to the vision-language feature space but lacks 3D spatial perception capabilities. The fusion block allows these two branches to complement each other. Specifically, we propose utilizing the denoising UNet of a pre-trained text-to-image diffusion model with advanced vision-language modeling capabilities as the 2D mask generator. Diffusion model's exceptional control over text-driven image generation demonstrates strong potential for creating fine-grained segmentation masks of novel categories. To promote thorough interaction between the two modality branches, we propose three mask-level techniques: (1) 3D-to-2D Mask Generation.We condition the denoising mask generator on global point cloud features, producing geometry-aware masks that are better suited for transferring to the 3D modality. (2) 2D-to-3D Mask Regularization.We apply mask-level regularization on 3D features to align with the vision-language embedding space.This enhances the open vocabulary capability of 3D features for novel categories while preserving fine-grained geometric information. (3) 3D-2D Mask Feature Fusion. We merge mask features from both modalities in the fusion block, enhancing the synergy between 2D and 3D features.We conduct extensive experiments on multiple benchmarks of different datasets, including Scan-Net20 [9] , ScanNet200 [38] , and S3DIS [1] datasets, to evaluate the effectiveness of our proposed method. XMask3D demonstrates competitive performance across all benchmarks. Additionally, we perform thorough ablation studies and provide intuitive visualizations to showcase the contribution of each proposed mask-level technique. In conclusion, the contributions of this paper can be summarized as follows:• We propose a novel XMask3D framework that, for the first time, leverages the denoising UNet of a generative text-to-image diffusion model for open vocabulary 3D perception. • We introduce 3D-to-2D mask generation, 2D-to-3D mask regularization, and 3D-2D mask feature fusion techniques to enhance meticulous mask-level 3D-2D-text feature alignment and strengthen cross-modal feature synergy. • We demonstrate the effectiveness of XMask3D on multiple benchmarks of various datasets and show outstanding performance. Ablation studies further convince the contribution of each proposed mask-level technique.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The main claims in the abstract and introduction is that we propose an XMask3D framework for open vocabulary 3D semantic segmentation, and the paper is all about introducing its design and effectiveness. 
2d open vocabulary segmentation: 
 Open vocabulary perception is a recently emerged research problem that focuses on enabling perception models to recognize novel categories that are invisible during supervised training, relying solely on a shared language vocabulary with the base categories. The key to addressing this problem lies in the Vision-Language model, which creates a shared embedding space for images and texts. Based on the types of Vision-Language models, previous literature on 2D open vocabulary segmentation can be broadly divided into two approaches: utilizing Vision-Language perception models like CLIP [36] or leveraging Vision-Language generation models like the diffusion model [18, 37] .Although traditional Vision-Language perception models like CLIP are primarily designed for classification, open vocabulary segmentation can also be viewed as a dense classification task.Consequently, several studies have proposed aligning dense image features with 2D-text embeddings, a concept pioneered by LSeg [23] . Successor models have introduced various techniques to enhance the feature alignment process, including attention-based combinations [30, 12] , mask embedding decoupling [10] , side network injection [46, 24] , and cross-modal aggregation [6] .Generative-based methods utilize Vision-Language generation models, such as diffusion models, to produce segmentation masks that can be extrapolated to open vocabulary categories. Since the diffusion model can generate semantically meaningful images based on text conditions, its intermediate features effectively represent the vision-language embedding space. ODISE [45] first proposed using the intermediate features from the denoising UNet of a pre-trained diffusion model as input to a mask generator for segmentation. Other works [25, 21, 41] leverage the strong generative capabilities of the diffusion model to create prototypes or augmented image-mask pairs, thereby enhancing the open vocabulary capacity of the segmentation model from a data perspective.In 3D vision, Semantic Abstraction [15] opens up the avenue to leverage Vision-Language models for open-world 3D scene understanding. Subsequent studies have primarily developed two types of methods to address 3D open vocabulary segmentation: feature alignment and model distillation.The principle of feature alignment methods is to explicitly pull 3D representations towards the visionlanguage embedding space, using the 2D modality as a mediator to establish 3D-text relationships.OpenScene [34] employs a cosine similarity loss between point cloud features and image CLIP features, integrating them for open vocabulary perception. PLA [11] introduces hierarchical 3D caption pairs to progressively align scene-level, view-level, and entity-level features with the CLIP feature space in a coarse-to-fine manner. Its successor, RegionPLC [47] , further introduces regionlevel captions with sliding windows and object bounding boxes, while CLIP-FO3D [49] similarly divides super-pixels for finer feature alignment. UniM-OV3D [16] utilizes a pre-trained point-text model, PointBind [14] , to enforce uni-modality representation learning of point clouds, images, depth maps, and texts. OV3D [20] proposes leveraging foundation models to establish point-entityText associations through pixel, thereby enhancing open-vocabulary recognition within the 3D domain.Model distillation methods typically involve selecting a foundational Vision-Language model and transferring its knowledge to a 3D network using paired point clouds and image data. Seal [28] introduces spatial contrastive learning and temporal consistency regularization to distill vision foundation models for point cloud sequence segmentation in an open vocabulary setting. 3D-OVS [27] aims to distill CLIP [36] and DINO [2, 33] into a neural radiance field [32] using novel alignment losses for 3D perception. Xiao et al. [44] propose object-level and voxel-level distillation losses for fine-grained 3D open vocabulary panoptic segmentation.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please refer to Section 4.4.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 The detailed architecture of XMask3D is depicted in Figure implemented using any off-the-shelf architectures such as sparse convolution networks [7, 8] or Transformer-based networks [51, 43, 42] . These frameworks are specifically designed to extract geometric features F 3d ∈ R N ×C from 3D point clouds, where N and C represent the numbers of points and feature dimensions, respectively. However, F 3d is discriminative only on base categories trained with supervision, relying on a pre-defined classifier implemented as a fully connected layer or a multi-layer perceptron. Therefore, it's essential for F 3d to be aligned with the vision-language feature space to enable unbounded open vocabulary segmentation, which is performed via perpoint similarity comparisons between F 3d and CLIP [36] text embeddings of given category names: From the 2D mask generator, we obtain the 2D mask embeddings G 2d ∈ R M ×C and binary mask maps M 2d ∈ R H×W ×M , where M represents the number of candidate masks, and H and W denote the height and width of the input images. Using the camera intrinsic matrix K ∈ R 3×3 and the view projection matrix V ∈ R 4×4 , we can establish associations between pixels in image I and surface points in the corresponding point cloud P . This allows us to derive back-projected 3D binary masksF text = E CLIPT (C name ) ∈ R L×C ,M 3d ∈ R N ′ ×M from M 2d .Here, N ′ signifies the number of points that correspond to image pixels, following the relationship N ′ < N, N ′ < H × W . Subsequently, we derive a pseudo mask featureF 2d = M 3d • G 2d , F 2d ∈ R N ′ ×C, with the subscript 2d indicating its origin from the 2D branch.Having obtained F 3d from the 3D geometry extraction branch and F 2d from the 2D mask generation branch, we implement a 3D-2D fusion block to combine these cross-modal and complementary features, resulting in F fuse ∈ R N ×C . Detailed information on this process can be found in Section 3. 4 .The final open vocabulary 3D semantic segmentation output O ∈ R N ×L is then calculated via:EQUATIONDetailed training objectives can be found in Section 3.5.Design Insights. An optimal 2D branch is expected to exhibit robust open vocabulary capabilities, enabling it to predict accurate masks for novel categories. To this end, we employ the denoising UNet of the renowned text-to-image diffusion model [37] to extract features from the well-established text-2D embedding space, followed by a mask generator to convert features into segmentation masks.We favor the generative diffusion model over the commonly adopted CLIP [36] model for two primary reasons. Indicated by Prompt-to-Prompt [17] , the cross-attention maps from the intermediate layers of the diffusion model exhibit a high correlation with text concepts. Consequently, a welltrained diffusion model constructs a superior vision-language feature space, which can be effectively leveraged for open vocabulary perception. Moreover, the diffusion model provides precise text control over dense pixel generation, demonstrating a higher potential for generating fine-grained segmentation masks compared to the CLIP model, which relies on coarse global feature alignment.Preliminary: Open Vocabulary Mask Generation with diffusion. The inference process of a text-to-image diffusion model [37] begins with a random Gaussian noise and a conditional text embedding, generating high-quality images through iterative denoising steps. ODISE [45] was the first to propose the use of the denoising UNet of a pre-trained diffusion model for open vocabulary 2D segmentation. Given an input image x, a noisy image x t is first sampled at time step t:EQUATIONwhere ᾱt = t k=1 α k , and α 1 , . . . , α T are pre-defined noise schedule. Then the diffusion model's visual representation f can be computed via the denoising step:EQUATIONwhere the denoising UNet is the building block of the diffusion model, MLP stands for multi-layer perceptron, and V is a frozen CLIP image encoder to encode x into the vision-language embedding space. V(x) is the implicit caption embedding which serves as the text condition input to the diffusion model. Subsequently, the mask generator, implemented with Mask2Former [4, 3] , uses the feature f to produce M class-agnostic binary masks M 2d and their corresponding mask embedding features F mask . Since f is highly representative of the vision-language feature space, the model is inherently capable of generating open vocabulary segmentation masks and embeddings. The experimental results of ODISE strongly confirm this hypothesis.Geometry-aware Mask Generation. In XMask3D, the 2D mask generation branch is implemented by a variant of the ODISE model. We propose an Implicit 3D Captioner that takes the global 3D feature f 3d ∈ R 1×Cg from the 3D encoder as input, and predicts the implicit condition embedding to be injected into the diffusion model. Then Equation 3 can be replaced by:EQUATIONwhere E represents the encoder of the point cloud segmentation model, and C g denotes the feature dimension of the global point cloud feature. The rationale behind this design is twofold. First, since MLP • f 3d serves as the text condition for the pre-trained denoising UNet with frozen weights, the training objective of the 2D mask generation branch implicitly pushes f 3d closer to the text-2D feature space. If f 3d does not align with this space, the pre-trained denoising UNet will not recognize the condition, resulting in a high loss of the 2D branch. Through gradient descent, the point cloud encoder gradually distills some vision-language knowledge from the pre-trained and frozen denoising UNet. Second, f 3d encapsulates rich 3D geometric information that the 2D branch lacks due to occlusion and dimensional compression issues inherent in images. Using f 3d as the condition for the 2D branch encourages the model to produce geometry-aware mask outlines and embeddings, facilitating the back-projection of 2D masks into 3D space. The effectiveness of the proposed geometry-aware mask generation will be validated through quantitative ablation comparisons in Section 4.3.Although some vision-language knowledge from the diffusion model is distilled to the point cloud encoder E via the proposed Implicit 3D Captioner, the 3D feature f 3d still deviates from the 2D-text embedding space. This is because there is no constraint on the point cloud decoder D, and the encoder distillation via gradient descent is inherently weak. Consequently, it is crucial to introduce contrastive regularization in the training pipeline to explicitly align 3D features with the shared 2D-text embedding space. Existing contrastive learning methods [34, 11] between 3D features and 2D-text features typically explore global, patch-wise, or point-wise relations. However, global contrastive learning is too coarse, and point-wise feature alignment is prone to noise. While patchwise contrastive learning is more fine-grained and robust, it still lacks semantic clarity. A patch in a 2D image may correspond to multiple irrelevant and discontinuous regions in a 3D point cloud due to depth compression, resulting in ambiguous and less representative local features in 3D.To this end, we propose an explicit 2D-to-3D mask regularization term for fine-grained and consistent feature space alignment between the 3D and 2D-text modalities. Specifically, we extract 3D mask embeddings G 3d ∈ R M ×C from 3D features F 3d using the back-projected 3D binary mask M 3d :EQUATIONwhere G i 3d ∈ R 1×C represents the i th mask embedding, and avgpool signifies the average pooling operation. F i 3d is sampled from F 3d at indices where M i 3d = 1, with M i 3d being the i th binary mask. Given the 2D binary mask M 2d and the input image I, we can also derive a ground truth mask CLIP feature G CLIP via a pre-trained CLIP model [36] . For detailed information on obtaining the ground truth G CLIP , please refer to MaskCLIP [12] or Section A.1. Subsequently, the 2D-to-3D regularization term can be computed using a classical cosine contrastive loss:EQUATIONAs each mask region ideally corresponds to a distinct category, the pooled mask embedding achieves semantic consistency and representativeness. Consequently, contrastive learning at the mask level offers finer granularity than global contrast, greater robustness than point-wise contrast, and clearer distinction than patch-wise contrast. Through our proposed 2D-to-3D mask regularization, the 3D features are explicitly aligned with the 2D-text feature space, enhancing the performance of the 3D branch in open vocabulary segmentation. This progress is further substantiated by the ablation studies outlined in Section 4.3.The 3D-2D mask feature fusion block is devised to merge 3D features F 3d ∈ R N ×C with the pseudo mask feature F 2d ∈ R N ′ ×C derived from the 2D branch. It is noteworthy that each element in F 3d possesses unique and distinguishing embeddings, whereas elements in F 2d pertaining to the same mask share identical mask embeddings. Consequently, F 3d offers detailed geometric structural information, while F 2d provides semantic features with robust open vocabulary capabilities. Our approach combines features from these two modalities to leverage their complementary insights, resulting in F fuse which excels in both precise geometry delineation and expansive semantic extrapolation. Concretely, given that N ′ < N , we selectively merge F 2d and F 3d solely on the N ′ points where correspondences exist:EQUATIONwhere cat represents concatenation. Ablation studies in Section 4.3 and visualization results in Section 4.2 will demonstrate that F fuse effectively integrates the strengths of both F 3d and F 2d .In XMask3D, our training strategy encompasses supervised segmentation loss (L seg ) computed from 3D (L 3d seg ), 2D (L 2d seg ), and fusion (L fuse seg ) modalities. We employ Cross Entropy loss for 3D and fusion segments, and for 2D, we adopt multi-head losses including Cross Entropy, Dice, and Focal Loss, following ODISE [45] and Mask2Former [4, 3] guidelines. Additionally, we follow PLA [11] to introduce a binary head and view-level contrastive loss. The binary head is optimized with Binary Cross Entropy loss (L bi ) to differentiate between base and novel categories. The view-level contrastive loss (L view ) is calculated between the view global feature and text embedding of the view image caption, weighted by respective coefficients (ω 3d view , ω 2d view , ω fuse view ). Detailed information about the binary head and the view-level regularization can be found in the PLA paper or in Section A.3 and A.2. In conclusion, the overall training objective can be adjusted by:EQUATIONwhere ω seg , ω mask , ω 3d view , ω 2d view , ω fuse view , ω bi , are loss weight hyperparameters. Category Partition. In alignment with previous research, we exclude the otherfurniture class and partition the remaining classes into three benchmarks: B15/N4, B12/N7, and B10/N9. Here, B15 signifies the 15 fundamental categories that remain visible and supervised during the training process, while N4 denotes the presence of 4 novel categories introduced during evaluation. For the ScanNet variant featuring 200 classes [38] , we adopt a similar approach, dividing the dataset into B170/N30 and B150/N50 benchmarks, each representing a distinct configuration of base and novel categories.Similarly, in the case of S3DIS, comprising 13 classes, we disregard the clutter class and organize the dataset into B8/N4 and B6/N6 benchmarks. Detailed information can be found in Section A.4.Following PLA [11] , we present the mean Intersection over Union (mIoU) scores separately for both base and novel categories to assess open vocabulary segmentation performance. Additionally, to provide a comprehensive evaluation of the segmentation capability, we report the harmonic mean IoU (hIoU) derived from the mIoU scores of base and novel categories. This holistic metric offers insights into the overall segmentation efficacy across the dataset.Implementation Details. Our implementation incorporates MinkUNet [7] as the 3D branch and ODISE [45] as the 2D branch within the architecture. For the vision-language model, we opt for CLIP-L [36] . The training regimen for the XMask3D model involves utilizing the AdamW optimizer [29] with a Cosine learning rate scheduler. We train the model for 150 epochs on 4 NVIDIA A800 GPUs, employing a batch size of 64. Notably, we introduce mask-level regularization to the training pipeline after the initial 50 epochs. This decision is motivated by the observation that the quality of mask prediction at the onset of training may be suboptimal, making the mask-level contrastive loss ineffective and potentially misleading. Detailed information regarding hyperparameter selections is provided in Section A.4.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Please refer to the supplemental zip file for our code and we will officially release the code upon the acceptance of the paper. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
main results: 
 Quantitative Comparisons. From Table 70.5% and 72.7%. On the S3DIS dataset, XMask3D shows boosts of 51.8% and 33.0% over PLA. Among these benchmarks, XMask3D achieves the highest improvements on the long-tail ScanNet200 dataset, primarily due to the introduction of the denoising UNet from the pre-trained diffusion model, which constructs a comprehensive text-2D embedding space with unlimited text descriptions.It is noteworthy that the OpenScene results are derived from the implementation of UniM-OV3D [16] , but we do not compare XMask3D with UniM-OV3D in the tables. This is because UniM-OV3D employs PointBIND [14] and CLIP2Point [19] with an already aligned 3D-text embedding space, whereas XMask3D only integrates the commonly used 2D-text space and introduces techniques to pull 3D features from any point cloud models towards the shared embedding space. Another outstanding concurrent method is OV3D [20] , which primarily focuses on EntityText extraction and Point-EntityText association, while XMask3D concentrates on enhancing mask-level interaction between 2D and 3D modalities. Therefore, the contributions of OV3D and XMask3D are orthogonal and could complement each other in future explorations. Additionally, OV3D only provides experiment results on the ScanNet20 benchmarks, while its effectiveness on the ScanNet200 dataset that is more challenging and on the S3DIS dataset with different data distribution remains unexplored.Visualization Results. In Figure 3 , we present a comparative visualization of novel categories between XMask3D and previous methods [11, 34] . XMask3D demonstrates superior accuracy in category predictions, produces finer segmentation boundaries, and generates more cohesive mask regions. Notably, the missegmented region on the bookshelf in the second row of XMask3D is classified as picture, which appears reasonable given the corresponding part in the view image.In this section, we comprehensively discuss the design choices of XMask3D through extensive ablation studies on the ScanNet B12/N7 benchmark. The results are presented in Table 3 .Mask Generation Condition. In Section 3.2, we introduce an Implicit 3D Captioner to convert global 3D features into implicit condition embeddings for the diffusion model. In Table 3a , we compare this implicit 3D condition with the vanilla text condition and implicit 2D condition. The text condition is generated by a ViT-GPT2 [48] captioning model and encoded via the frozen CLIP text encoder. The 2D condition is generated by the frozen CLIP image encoder and a learnable MLP, following the design in ODISE [45] (Equation 3). Our proposed implicit 3D condition outperforms the others in novel category segmentation, demonstrating that integrating the 3D global feature with the diffusion model produces the most compatible open vocabulary masks with the 3D branch.Mask Regularization. In Section 3.3, we propose a fine-grained mask regularization term to align 3D features with the 2D-text embedding space. In the first line of Table 3b , we remove the fine-grained mask-level loss L mask from the training pipeline. Besides the final results from the fusion block, we also report the intermediate results from the 2D and 3D branches. The inclusion of the mask loss results in a significant improvement of 11.7 in 3D performance on novel categories, demonstrating that our fine-grained mask-level regularization effectively brings 3D features closer to the 2D-text embedding space. Additionally, the performance gain on the fused output is 1.5, highlighting the positive impact of this regularization from 3D to fusion features. Table Desk View Image Bed Chair Bed Table Table Chair Sofa Bookshelf Picture Cabinet Figure 3: Visualization Comparisons between XMask3D and Previous Methods. We compare XMask3D with PLA [11] and OpenScene [34] on the novel categories table, bookshelf, chair and bed. The regions corresponding to the novel categories are highlighted in red boxes.We also analyze the effects of mask regularization through visualizations in Figure 4 . Comparing the first and second rows, the 3D segmentation results without the mask loss show inconsistency in the local region for the novel category chair, with most points misclassified as bookshelf. When the mask-level regularization is added, more points are correctly classified, resulting in a clearer segmentation mask. The visualizations of the fused outputs also demonstrate consistent enhancement in mask regularization within the final column. This is evident from the rectification of misclassified points previously labeled as table on the armrest of the right chair in the last column.Modality Fusion. When comparing the 2D, 3D, and fused metrics within the same line in Table 3b , we empirically find that the 2D branch performs relatively better on novel category segmentation, while the 3D branch excels at base category segmentation. This quantitative observation aligns with our design intention: to exploit geometric knowledge via the 3D branch and to enhance the model's open vocabulary capability via the 2D branch. More importantly, the fused output outperforms both the 2D and 3D intermediate results on both base and novel splits. These results strongly support the effectiveness of our fusion design in merging the complementary knowledge from both modalities.We also present visualization evidence regarding modal complementarity and fusion effects in Figure 4 . In the first group, the 2D branch exhibits unsatisfactory results of the base category wall around the whiteboard and behind the right chair, whereas effectively segmenting the novel category chair with high quality despite minor artifacts. Conversely, the 3D branch produces an unsatisfactory mask for the chair but excels in segmenting the wall based on geometric information. The fusion block leverages the strengths of both branches, mitigates their weaknesses, and yields satisfactory outcomes. Moreover, the fusion block accurately delineates regions such as the left foot of the chair on the right, where both the 2D and 3D branches falter, highlighting its potential for integrating cross-modality knowledge. The second group displays similar results that convince the effectiveness of the proposed mask loss and cross-modality fusion design.Due to resource constraints, we only evaluate the performance of XMask3D on semantic segmentation in this study. However, the XMask3D pipeline has the potential to be extended to the instance and panoptic perception by replacing the 3D backbone with an instance or panoptic segmentation model. This adaptability arises from the auxiliary 2D branch, which is mask-based and capable of generating instance-level masks given appropriate annotations. Furthermore, since the XMask3D pipeline is agnostic to the 3D model used, it would be interesting to conduct experiments with more advanced point cloud models to compare performance. We hope we can exploit the strength of XMask3D with high-standard 3D backbones and a wider range of dense perception tasks in future research. Computational cost is another limitation of XMask3D, as we implement the Denoising UNet which has numerous parameters and relatively slow latency. In our future work, we plan to address this limitation by replacing the 2D branch with a more lightweight 2D open vocabulary mask generator. 
conclusion: 
 In this paper, we present XMask3D, designed for open vocabulary 3D semantic segmentation. We propose the integration of the denoising UNet of a pre-trained diffusion model to produce geometryaware segmentation masks conditioned on learnable implicit 3D embeddings. These binary 2D masks filter mask-level embeddings of 3D representations and apply mask regularization to enhance the open vocabulary capacity of 3D features. By fusing the 2D mask embeddings with fine-grained 3D features, we leverage the complementary knowledge from both modalities, achieving competitive performance across various benchmarks and datasets. Ablation studies and visualization comparisons further validate the effectiveness of the proposed cross-modal mask reasoning method. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have cited the original paper of the pre-trained models and data we used. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Mask-level regularization facilitates local and finely-grained alignment between features extracted from the 3D branch of XMask3D and the 2D-text embedding space. The ground truth for mask-level loss is computed using MaskCLIP [12] , utilizing the predicted segmentation mask M 2d ∈ R M ×H×W and the view image I ∈ R 3×H×W :EQUATIONwhere M represents the number of masks, and H and W denote the height and width of the image, respectively. V refers to the pre-trained CLIP [36] image encoder.Specifically, the view image I is encoded into image tokens T I ∈ R N ×C with the pre-trained CLIP image model, where N is the number of image tokens and C is the dimension of the CLIP embeddings.The class token T C ∈ R 1×C is duplicated M times as the mask class tokens T M ∈ R M ×C . Then T I , T C , T M are concatenated together to perform masked attention with frozen weights from the pre-trained CLIP image model. The attention mask is designed asEQUATION)where T m×n is an m × n True matrix, F m×n is an m × n False matrix and P is defined as:EQUATION)where True means that this position is masked out i.e. not allowed to attend and False otherwise.Then the updated mask class tokens T ′ M from the masked attention layers can be regarded as the CLIP embedding of the masked regions, serving as the ground truth G CLIP for mask-level regularization.The view-level regularization facilitates coarse and high-level alignment between features extracted from the 3D branch and the 2D-text embedding space. To elaborate, upon receiving an image I, we initially generate its text caption using a pre-trained captioning model, ViT-GPT2 [48] . Subsequently, we employ the pre-trained CLIP [36] text encoder to encode the text caption into the 2D-text embedding space, yielding f T view , which serves as the ground truth for view-level regularization. We perform average pooling operation on dense 3D point cloud features, 2D image features and fused features to obtain their global embeddings f 3d view , f 2d view , f fuse view . Then we implement contrastive loss between global features and the ground truth text embeddings:EQUATIONwhere m = 3d, 2d, fuse denotes different modalities.Following PLA [11] , we include a binary head to predict whether the points belong to base or novel categories. We implement a small 3D model as the binary head with minimum computation cost. The prediction s b is utilized to modulate the over-confident semantic score s:EQUATION)where s B is the semantic score computed solely on base classes with novel class scores set to zero. Similarly, s N is computed only for novel classes, setting base class scores to zero. Training. The supervised segmentation loss L m seg is calculated via the per-point classification Cross Entropy Loss on N points:EQUATIONEQUATIONwhere m = 3d, 2D, fuse denotes different modalities, y i is the ground truth for base categories, Fm is the normalized feature, and τ is a learnable temperature parameter.Inference. We follow ODISE [45] to combine the predicted logits p with the prediction from a text-image discriminative model to enhance the open vocabulary classification capacity of the model. Specifically, we leverage the mask-level regularization ground truth feature G CLIP from Section A.1 to modulate the segmentation logits:EQUATIONEQUATION)where ḠCLIP is the normalized feature of G CLIP , λ ∈ [0, 1] is the fixed balancing factor.Hyper-parameters. For all benchmarks, we set the same ω seg = 4, ω 3d view = 1, ω 2d view = 4, ω fuse view = 1.5 as the hyper-parameter choices. The ω mask and ω bi are set differently across benchmarks. We We present supplementary illustrations of ablation studies pertaining to mask regularization and cross-modality fusion in Figure 6 , which corresponds to the sample in Figure 1 from the ScanNet B15/N4 benchmark.Comparing the first and second rows of the third column reveals that the proposed mask regularization loss yields significant improvements in the segmentation of novel categories by the 3D branch. Further comparison of the last three columns in the second row demonstrates that the output from the fusion block effectively leverages the strengths of both modality branches. Specifically, while the 2D branch excels at segmenting novel categories, it tends to produce discontinuous masks for base categories with larger regions (e.g., wall). Conversely, the 3D branch may perform relatively poorly in novel category segmentation but provides consistent geometric knowledge as a complementary aspect. Consequently, the fused output achieves superior performance in open vocabulary segmentation.In Figure 7 , we display some failure cases of XMask3D. The first sample shows a bathtub with a shower curtain in the bathroom. However, the 2D/3D branch and the fused output of XMask3D all misclassify the shower curtain as curtain. This may be because these two categories are similar in object shape and texture, with the only difference being the surrounding environment. Since XMask3D only takes a corner of the room as input instead of the entire scene, the global environmental information is insufficient for making the correct category prediction.The second sample shows a large area of picture on the wall. The 2D, 3D branches and the fused output of XMask3D all misclassified it as wall, due to their similar geometry. In most cases, a picture is a small region on the wall, and this picture, as large as the wall, is a typical corner case. This failure case may reveal XMask3D's over-reliance on geometric knowledge and lesser consideration of texture information when encountering out-of-distribution samples.The third sample shows a sink on a counter. Due to the occlusion problem, the sink point cloud is incomplete, negatively affecting the prediction of segmentation boundaries between the sink and the counter. This occurs because they are geometrically similar when the sinking-down part of the sink is missing.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper introduces XMask3D, an approach to open-vocabulary 3D semantic segmentation that addresses key limitations in existing methods by integrating 3D geometric features with 2D and text-based embeddings. The authors build on the ODISE method, extending it to the 3D domain and introducing a novel implicit captioner that generates geometry-aware 2D masks. This enhances the alignment between 3D and 2D-text embedding spaces, resulting in improved segmentation accuracy, particularly for novel categories. The method employs a design incorporating distillation losses and mask-level regularization, which boosts its open-vocabulary capabilities.

XMask3D is evaluated on several benchmarks, including ScanNet, ScanNet200, and S3DIS, where it demonstrates superior performance compared to existing methods. These results are further supported by a comprehensive ablation study. All reviewers have recommended acceptance of this paper, and the AC finds no significant reasons to overturn these recommendations. Therefore, the paper is recommended for acceptance.

Reviews: review: paper_summary: paper_summary: This paper addresses the challenge of open-vocabulary 3D semantic segmentation by utilizing 3D geometric features, 2D semantic embeddings, and text modality. The proposed approach adapts the ODISE method to the 3D domain, aiming to distill open-vocabulary semantic segmentation knowledge from a pre-trained text-to-image denoising diffusion model to a 3D segmentation model. Initially, an input point cloud is fed into a 3D encoder-decoder segmentation network, producing point-wise geometric features. Simultaneously, a pre-trained visual-language diffusion model generates 2D masks and embeddings from posed images of the same scene, conditioned on the 3D global feature of the 3D branch’s encoder. Unlike the ODISE method, an implicit $3D$ captioner is introduced to produce geometry-aware 2D masks while also distilling information from the 2D branch network to the 3D encoder. To further regularize the 3D network, a distillation loss ($\mathcal{L}_{mask}$) is applied to the 3D mask embeddings, derived from the per-point features and the 2D masks back-projected to the point cloud as 3D binary masks. By obtaining ground truth mask features from a pre-trained CLIP model, the 3D masked embeddings are aligned with the image-text joint embedding space, through a cosine similarity loss. This alignment leads to more coherent segmentation results and enhances the model's open-vocabulary capabilities. Finally, the per-point features are combined with the pseudo mask 2D features (formed by the back-projected 3D mask and 2D mask embeddings), resulting in a fused per-point representation that incorporates the geometric information from the 3D segmentation network and the semantic open-vocabulary capabilities of the 2D branch. The approach is evaluated on three semantic segmentation benchmarks (ScanNet, ScanNet200, and S3DIS) and demonstrates superior performance compared to competing methods.
strength_weakness: strength_weakness: While the method exhibits superior performance w.r.t. competing methods, it seems that the output fused embeddings yields to geometric inconsistent features for semantic classes that cover large areas of the point cloud such as wall, ceiling and floor. This is evident in both partitioning settings when the class is either base or novel (Table 5 (a) and (b) in supp.).
questions: questions: Following the weaknesses section, do the authors have any additional insights into why this phenomenon occurs with the fused embeddings? Could the 2D regularization terms ($\mathcal{L}_{seg}^{2D}$, $\mathcal{L}\_{view}^{2D}$) be introducing too much bias towards the 2D visual modality, thereby causing geometric discontinuities in the output fused features?
limitations: limitations: Yes, the authors have discussed the method's limitations in detail in Section 4.4.
paper_summary: paper_summary: The paper addresses the limitations of current open vocabulary 3D semantic segmentation methods, which primarily focus on creating a unified feature space for 3D, 2D, and textual modalities but struggle with fine-grained segmentation boundaries. To overcome these limitations, the authors propose XMask3D, a cross-modal mask reasoning framework that achieves more precise mask-level alignment between 3D features and the 2D-text embedding space.
strength_weakness: strength_weakness: 1. The paper evaluates the proposed method on a limited set of benchmarks (ScanNet20, ScanNet200, S3DIS), all of which are indoor scene datasets. Authors could discuss how the method might perform on outdoor datasets. Additionally, the authors could provide a qualitative analysis of the model's potential limitations when applied to different environments.
2. The reliance on the denoising UNet from a pre-trained diffusion model could be seen as a potential weakness or limitation, especially given the computational resources required for training and inference.
questions: questions: 1. The paper could benefit from a more detailed error analysis to understand the failure modes of XMask3D, especially in novel category segmentation.
limitations: limitations: The authors have addressed limitations.
paper_summary: paper_summary: The paper proposes a precise and consistent mask-level alignment between 3D features and the 2D-text embedding space through a method called cross-modal mask reasoning. The proposed XMask3D model includes a 3D branch for capturing geometric features, a 2D branch for generating vision-language aligned masks, and a fusion block to combine 3D with 2D. Using a pre-trained text-to-image diffusion model as the 2D mask generator, the model leverages three techniques: 3D-to-2D mask generation, 2D-to-3D mask regularization, and 3D-2D mask feature fusion.
strength_weakness: strength_weakness: 1- The authors don't compare with state-of-the-art 3D semantic segmentation OV3D[1]

2- The authors highlighed fututre work in the limitation, it would be good if you can expand it with some limitation on the technical side or some failure cases.

[1] Jiang, Li, Shaoshuai Shi, and Bernt Schiele. "Open-Vocabulary 3D Semantic Segmentation with Foundation Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
questions: questions: Please compare to OV3D mentioned in the weaknesses
limitations: limitations: Needs to be expanded
paper_summary: paper_summary: This paper introduces XMask3D, a framework developed for open vocabulary 3D semantic segmentation. They propose the integration of the denoising UNet, derived from a pre-trained diffusion model, to generate geometry-aware segmentation masks conditioned on learnable implicit 3D embeddings. These binary 2D masks are used to filter mask-level embeddings of 3D representations and apply mask regularization, thereby improving the open vocabulary capacity of 3D features.
strength_weakness: strength_weakness: 1.	The organization should be improved. Section 3.1 provides an overview, while section 3.2 includes design insights and preliminary findings. The flow of these writings has puzzled me, making it difficult to grasp your key contribution.
questions: questions: 1.	Please provide further clarification on how mask-level alignment between 3D features and 2D embedding space can address the limitations of traditional techniques, such as global feature alignment or vision-language model distillation. Additionally, if texts (Category Labels) are concatenated with fused features, will it still create a unified feature space that encompasses 3D, 2D, and textual modalities?
2.	Could you please provide further clarification on the main contributions of your research compared to PLA and OpenScene? Although the 3D caption process shares similarities with PLA, the overall pipeline resembles OpenScene, with the exception of the diffusion model and mask generator, which differ from the Multi-view Feature Fusion in OpenScene.
3.	Does the Implicit 3D Captioner effectively work with your 3D features? From my understanding, the most reliable 3D captioner currently available is Cap3D, which generates captions for 3D objects by rendering multi-view images and utilizing BLIP2 and LLM for assistance. In the context of indoor-scenes, can we consider the Implicit 3D Captioner to be equally robust? It would be beneficial to present additional evidence to support this claim.
4.	Can your text-to-image diffusion model effectively generalize to your datasets? If not, please provide examples of failure cases. Additionally, is the diffusion model fine-tuned during the training process or is it frozen? If not, please present additional results to demonstrate the robustness of your diffusion model in generating high-quality images within your datasets.
5.	What is view-level contrastive loss? Why this loss is calculated between the view global feature and text embedding of the view image caption but have three coefficients?
6.	It is recommended to show your 2D Mask and 3D Mask in Figure 3 to provide more visual evidence.
7.	The authors should provide results on ScanNet++ (CVPR’23), which is a up-to-date dataset compared with ScanNet.
8.	Since diffusion models are utilized, it is recommended to compared the model parameters and FLOPs compared with PLA and OpenScene.
limitations: limitations: 1.	The authors have addressed the limitations and potential negative societal impact of their work.

Comments: title: Response to Reviewer 4VWC
Response to Reviewer AnYT
Response to Reviewer Xo8p
Response to Reviewer tdkT
Reviewer response
Discussions and finalize rating
comment: We greatly appreciate your response and valuable suggestions, which improved the quality and comprehensiveness of our paper! We will update our revised paper according to our discussions.
Thank you for the clarifications and for addressing my concerns. I will maintain my positive rating
We deeply appreciate the time and effort you dedicated to the careful review and insightful feedback on our paper! We will update our revised paper and add OV3D comparison according to our discussions.
Thanks a lot for clarifying these points. The authors addressed my comments. Thus, I am happy to keep my score, and it would be great to report the results of OV3D and XMask3D together in the revised version.
We greatly appreciate your response and valuable suggestions, which improved the quality and comprehensiveness of our paper! We will update our revised paper according to our discussions.
Thanks for upgrading your score and providing valuable feedback. We will update our revised paper according to our discussions and release our code and checkpoints after the conference decision. Thank you again for your insightful and constructive suggestions that improve paper quality!
Thanks for your responses, most of my concerns have been solved. I will change my rating to positive. Please carefully release your code and checkpoints in your future version.
Dear Reviewers,

Thank you for your efforts. Please review the rebuttals, engage in the discussions, and provide your final ratings.

Thank you again for your valuable contributions.

AC
I've thoroughly reviewed the authors' responses and appreciate their thoughtful engagement. Most of my concerns have been addressed. I will stay in touch for further discussion as we approach the final rating.