Title: XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D open vocabulary

semantic segmentation

cross-modal learning
Abstract: Existing methodologies in open vocabulary 3D semantic segmentation primarily concentrate on establishing a unified feature space encompassing 3D, 2D, and textual modalities. Nevertheless, traditional techniques such as global feature alignment or vision-language model distillation tend to impose only approximate correspondence, struggling notably with delineating fine-grained segmentation boundaries. To address this gap, we propose a more meticulous mask-level alignment between 3D features and the 2D-text embedding space through a cross-modal mask reasoning framework, XMask3D. In our approach, we developed a mask generator based on the denoising UNet from a pre-trained diffusion model, leveraging its capability for precise textual control over dense pixel representations and enhancing the open-world adaptability of the generated masks. We further integrate 3D global features as implicit conditions into the pre-trained 2D denoising UNet, enabling the generation of segmentation masks with additional 3D geometry awareness. Subsequently, the generated 2D masks are employed to align mask-level 3D representations with the vision-language feature space, thereby augmenting the open vocabulary capability of 3D geometry embeddings. Finally, we fuse complementary 2D and 3D mask features, resulting in competitive performance across multiple benchmarks for 3D open vocabulary semantic segmentation. Code is available at https://github.com/wangzy22/XMask3D.

XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation

Abstract

1 Introduction

As the integration of vision and language in deep learning continues to expand, text descriptions are increasingly utilized in visual generation [37, 50, 39, 35, 26, 40] and perception [36, 22, 23, 13, 45, 12] tasks. This integration enhances the adaptability of models in real-world applications and improves user experiences in customized artificial intelligence systems. Open vocabulary 3D semantic segmentation exemplifies a perception task that is trained on base categories and demands robust extrapolation capabilities to discriminate fine-grained geometry in novel categories that are invisible during training. The base and novel classes are only linked by the shared open vocabulary within the language space. However, constructing a shared 3D-text space while maintaining precise, modality-specific representation remains a significant challenge. Successfully addressing this issue would advance open vocabulary 3D semantic segmentation, facilitating virtual reality interactions and manipulations, and thereby contributing to the development of user-friendly robotics and autonomous driving technologies.

Existing approaches for open vocabulary 3D perception predominantly aim to bridge the gap between 3D and text representations by using the 2D modality as an intermediary. One line of research [34,

Generator

Extractor

Condition

Toilet

Wall

Desk

Bathtub

Cross-Modal

Fusion Block

Ground Truth

Novel Category

Base Category

3D Model with

Global

3D-2D-Text Alignment

3D Result

Fused Result

2D Result

11, 47, 16] suggests aligning 3D embeddings with vision-language embedding spaces through global, patch-wise, or point-wise contrastive loss, while another line [49, 28, 44] investigates distilling open vocabulary knowledge from foundational vision-language models into 3D models. However, global feature alignment and model distillation techniques tend to overlook fine-grained 3D geometric details and produce coarse results. Conversely, point-wise contrastive learning is prone to noise and outliers. Although patch-wise feature alignment offers a compromise, a single patch in 2D may correspond to multiple unrelated and discontinuous regions in 3D, which can be misleading and ineffective.

In this paper, we propose a more precise and consistent mask-level alignment between 3D features and the 2D-text embedding space, achieved through our proposed cross-modal mask reasoning method. The proposed XMask3D model comprises a 3D branch, a 2D branch, and a fusion block. The 3D branch, adaptable as any point cloud segmentation model, excels in capturing geometric features but struggles with novel category extrapolation. Conversely, the 2D branch serves as a mask generator, which predicts masks with embeddings aligned to the vision-language feature space but lacks 3D spatial perception capabilities. The fusion block allows these two branches to complement each other. Specifically, we propose utilizing the denoising UNet of a pre-trained text-to-image diffusion model with advanced vision-language modeling capabilities as the 2D mask generator. Diffusion model's exceptional control over text-driven image generation demonstrates strong potential for creating fine-grained segmentation masks of novel categories. To promote thorough interaction between the two modality branches, we propose three mask-level techniques: (1) 3D-to-2D Mask Generation. We condition the denoising mask generator on global point cloud features, producing geometry-aware masks that are better suited for transferring to the 3D modality. (2) 2D-to-3D Mask Regularization. We apply mask-level regularization on 3D features to align with the vision-language embedding space. This enhances the open vocabulary capability of 3D features for novel categories while preserving fine-grained geometric information. (3) 3D-2D Mask Feature Fusion. We merge mask features from both modalities in the fusion block, enhancing the synergy between 2D and 3D features.

We conduct extensive experiments on multiple benchmarks of different datasets, including ScanNet20 [9], ScanNet200 [38], and S3DIS [1] datasets, to evaluate the effectiveness of our proposed method. XMask3D demonstrates competitive performance across all benchmarks. Additionally, we perform thorough ablation studies and provide intuitive visualizations to showcase the contribution of each proposed mask-level technique. In conclusion, the contributions of this paper can be summarized as follows:

We propose a novel XMask3D framework that, for the first time, leverages the denoising UNet of a generative text-to-image diffusion model for open vocabulary 3D perception.

We introduce 3D-to-2D mask generation, 2D-to-3D mask regularization, and 3D-2D mask feature fusion techniques to enhance meticulous mask-level 3D-2D-text feature alignment and strengthen cross-modal feature synergy.

We demonstrate the effectiveness of XMask3D on multiple benchmarks of various datasets and show outstanding performance. Ablation studies further convince the contribution of each proposed mask-level technique.

2 Related Work

2.1 2D Open Vocabulary Segmentation

Open vocabulary perception is a recently emerged research problem that focuses on enabling perception models to recognize novel categories that are invisible during supervised training, relying solely on a shared language vocabulary with the base categories. The key to addressing this problem lies in the Vision-Language model, which creates a shared embedding space for images and texts. Based on the types of Vision-Language models, previous literature on 2D open vocabulary segmentation can be broadly divided into two approaches: utilizing Vision-Language perception models like CLIP [36] or leveraging Vision-Language generation models like the diffusion model [18, 37].

Although traditional Vision-Language perception models like CLIP are primarily designed for classification, open vocabulary segmentation can also be viewed as a dense classification task. Consequently, several studies have proposed aligning dense image features with 2D-text embeddings, a concept pioneered by LSeg [23]. Successor models have introduced various techniques to enhance the feature alignment process, including attention-based combinations [30, 12], mask embedding decoupling [10], side network injection [46, 24], and cross-modal aggregation [6].

Generative-based methods utilize Vision-Language generation models, such as diffusion models, to produce segmentation masks that can be extrapolated to open vocabulary categories. Since the diffusion model can generate semantically meaningful images based on text conditions, its intermediate features effectively represent the vision-language embedding space. ODISE [45] first proposed using the intermediate features from the denoising UNet of a pre-trained diffusion model as input to a mask generator for segmentation. Other works [25, 21, 41] leverage the strong generative capabilities of the diffusion model to create prototypes or augmented image-mask pairs, thereby enhancing the open vocabulary capacity of the segmentation model from a data perspective.

2.2 3D Open Vocabulary Segmentation

In 3D vision, Semantic Abstraction [15] opens up the avenue to leverage Vision-Language models for open-world 3D scene understanding. Subsequent studies have primarily developed two types of methods to address 3D open vocabulary segmentation: feature alignment and model distillation.

The principle of feature alignment methods is to explicitly pull 3D representations towards the visionlanguage embedding space, using the 2D modality as a mediator to establish 3D-text relationships. OpenScene [34] employs a cosine similarity loss between point cloud features and image CLIP features, integrating them for open vocabulary perception. PLA [11] introduces hierarchical 3D caption pairs to progressively align scene-level, view-level, and entity-level features with the CLIP feature space in a coarse-to-fine manner. Its successor, RegionPLC [47], further introduces regionlevel captions with sliding windows and object bounding boxes, while CLIP-FO3D [49] similarly divides super-pixels for finer feature alignment. UniM-OV3D [16] utilizes a pre-trained point-text model, PointBind [14], to enforce uni-modality representation learning of point clouds, images, depth maps, and texts. OV3D [20] proposes leveraging foundation models to establish point-entityText associations through pixel, thereby enhancing open-vocabulary recognition within the 3D domain.

Model distillation methods typically involve selecting a foundational Vision-Language model and transferring its knowledge to a 3D network using paired point clouds and image data. Seal [28] introduces spatial contrastive learning and temporal consistency regularization to distill vision foundation models for point cloud sequence segmentation in an open vocabulary setting. 3DOVS [27] aims to distill CLIP [36] and DINO [2, 33] into a neural radiance field [32] using novel alignment losses for 3D perception. Xiao et al. [44] propose object-level and voxel-level distillation losses for fine-grained 3D open vocabulary panoptic segmentation.

3 Approach

3.1 Overview

The detailed architecture of XMask3D is depicted in Figure 2. It consists of three components: a 3D geometry extraction branch, a 2D mask generation branch, and a 3D-2D feature fusion module. The 3D geometry extraction branch is an encoder-decoder segmentation network, which can be

Encoder

Diffusion UNet

Mask Generator

Decoder

Image

Mask Loss

Fusion Block

CLIP Text Encoder

Results

CLIP Image Encoder

Mask Feat.

CLIP I/T Feat.

Mask Contour

Condition

implemented using any off-the-shelf architectures such as sparse convolution networks [7, 8] or Transformer-based networks [51, 43, 42]. These frameworks are specifically designed to extract geometric features F 3d ∈ R N × C from 3D point clouds, where N and C represent the numbers of points and feature dimensions, respectively. However, F 3d is discriminative only on base categories trained with supervision, relying on a pre-defined classifier implemented as a fully connected layer or a multi-layer perceptron. Therefore, it's essential for F 3d to be aligned with the vision-language feature space to enable unbounded open vocabulary segmentation, which is performed via perpoint similarity comparisons between F 3d and CLIP [36] text embeddings of given category names: F text = E CLIP T ( C name ) ∈ R L × C , where L represents the number of categories, E CLIP T denotes the CLIP text encoder, and C name stands for the category name such as Table, Chair .

Previous literature on feature alignment techniques varies from global or point-level contrastive learning to vision-language model distillation. However, none of these methods simultaneously achieves fine-grained precision and robustness to outliers. To address this limitation, we introduce a cross-modal mask reasoning method that performs mask-level feature alignment. Specifically, we propose a 2D mask generation branch in conjunction with the 3D branch to generate 2D masks with open vocabulary capability and use these masks to regularize F 3d . Detailed information on mask generation and mask regularization can be found in Section 3.2 and Section 3.3, respectively.

From the 2D mask generator, we obtain the 2D mask embeddings G 2d ∈ R M × C and binary mask maps M 2d ∈ R H × W × M , where M represents the number of candidate masks, and H and W denote the height and width of the input images. Using the camera intrinsic matrix K ∈ R 3 × 3 and the view projection matrix V ∈ R 4 × 4 , we can establish associations between pixels in image I and surface points in the corresponding point cloud P . This allows us to derive back-projected 3D binary masks M 3d ∈ R N ′ × M from M 2d . Here, N ′ signifies the number of points that correspond to image pixels, following the relationship N ′ < N,N ′ < H × W . Subsequently, we derive a pseudo mask feature F 2d = M 3d · G 2d , F 2d ∈ R N ′ × C , with the subscript 2d indicating its origin from the 2D branch.

Having obtained F 3d from the 3D geometry extraction branch and F 2d from the 2D mask generation branch, we implement a 3D-2D fusion block to combine these cross-modal and complementary features, resulting in F fuse ∈ R N × C . Detailed information on this process can be found in Section 3.4. The final open vocabulary 3D semantic segmentation output O ∈ R N × L is then calculated via:

Detailed training objectives can be found in Section 3.5.

3.2 3D-to-2D Mask Generation

Design Insights. An optimal 2D branch is expected to exhibit robust open vocabulary capabilities, enabling it to predict accurate masks for novel categories. To this end, we employ the denoising UNet of the renowned text-to-image diffusion model [37] to extract features from the well-established text-2D embedding space, followed by a mask generator to convert features into segmentation masks.

We favor the generative diffusion model over the commonly adopted CLIP [36] model for two primary reasons. Indicated by Prompt-to-Prompt [17], the cross-attention maps from the intermediate layers of the diffusion model exhibit a high correlation with text concepts. Consequently, a welltrained diffusion model constructs a superior vision-language feature space, which can be effectively leveraged for open vocabulary perception. Moreover, the diffusion model provides precise text control over dense pixel generation, demonstrating a higher potential for generating fine-grained segmentation masks compared to the CLIP model, which relies on coarse global feature alignment.

Preliminary: Open Vocabulary Mask Generation with diffusion. The inference process of a text-to-image diffusion model [37] begins with a random Gaussian noise and a conditional text embedding, generating high-quality images through iterative denoising steps. ODISE [45] was the first to propose the use of the denoising UNet of a pre-trained diffusion model for open vocabulary 2D segmentation. Given an input image x , a noisy image x t is first sampled at time step t :

where ¯ α t = ∏ t k =1 α k , and α 1 , . . . , α T are pre-defined noise schedule. Then the diffusion model's visual representation f can be computed via the denoising step:

where the denoising UNet is the building block of the diffusion model, MLP stands for multi-layer perceptron, and V is a frozen CLIP image encoder to encode x into the vision-language embedding space. V ( x ) is the implicit caption embedding which serves as the text condition input to the diffusion model. Subsequently, the mask generator, implemented with Mask2Former [4, 3], uses the feature f to produce M class-agnostic binary masks M 2d and their corresponding mask embedding features F mask . Since f is highly representative of the vision-language feature space, the model is inherently capable of generating open vocabulary segmentation masks and embeddings. The experimental results of ODISE strongly confirm this hypothesis.

Geometry-aware Mask Generation. In XMask3D, the 2D mask generation branch is implemented by a variant of the ODISE model. We propose an Implicit 3D Captioner that takes the global 3D feature f 3d ∈ R 1 × C g from the 3D encoder as input, and predicts the implicit condition embedding to be injected into the diffusion model. Then Equation 3 can be replaced by:

where E represents the encoder of the point cloud segmentation model, and C g denotes the feature dimension of the global point cloud feature. The rationale behind this design is twofold. First, since MLP ◦ f 3d serves as the text condition for the pre-trained denoising UNet with frozen weights, the training objective of the 2D mask generation branch implicitly pushes f 3d closer to the text-2D feature space. If f 3 d does not align with this space, the pre-trained denoising UNet will not recognize the condition, resulting in a high loss of the 2D branch. Through gradient descent, the point cloud encoder gradually distills some vision-language knowledge from the pre-trained and frozen denoising UNet. Second, f 3d encapsulates rich 3D geometric information that the 2D branch lacks due to occlusion and dimensional compression issues inherent in images. Using f 3d as the condition for the 2D branch encourages the model to produce geometry-aware mask outlines and embeddings, facilitating the back-projection of 2D masks into 3D space. The effectiveness of the proposed geometry-aware mask generation will be validated through quantitative ablation comparisons in Section 4.3.

3.3 2D-to-3D Mask Regularization

Although some vision-language knowledge from the diffusion model is distilled to the point cloud encoder E via the proposed Implicit 3D Captioner, the 3D feature f 3d still deviates from the 2D-text embedding space. This is because there is no constraint on the point cloud decoder D , and the encoder distillation via gradient descent is inherently weak. Consequently, it is crucial to introduce contrastive regularization in the training pipeline to explicitly align 3D features with the shared 2D-text embedding space. Existing contrastive learning methods [34, 11] between 3D features and 2D-text features typically explore global, patch-wise, or point-wise relations. However, global contrastive learning is too coarse, and point-wise feature alignment is prone to noise. While patchwise contrastive learning is more fine-grained and robust, it still lacks semantic clarity. A patch in a 2D image may correspond to multiple irrelevant and discontinuous regions in a 3D point cloud due to depth compression, resulting in ambiguous and less representative local features in 3D.

To this end, we propose an explicit 2D-to-3D mask regularization term for fine-grained and consistent feature space alignment between the 3D and 2D-text modalities. Specifically, we extract 3D mask embeddings G 3d ∈ R M × C from 3D features F 3d using the back-projected 3D binary mask M 3d :

where G i 3d ∈ R 1 × C represents the i th mask embedding, and avgpool signifies the average pooling operation. ˜ F i 3d is sampled from F 3 d at indices where M i 3d = 1 , with M i 3d being the i th binary mask.

Given the 2D binary mask M 2d and the input image I , we can also derive a ground truth mask CLIP feature G CLIP via a pre-trained CLIP model [36]. For detailed information on obtaining the ground truth G CLIP , please refer to MaskCLIP [12] or Section A.1. Subsequently, the 2D-to-3D regularization term can be computed using a classical cosine contrastive loss:

As each mask region ideally corresponds to a distinct category, the pooled mask embedding achieves semantic consistency and representativeness. Consequently, contrastive learning at the mask level offers finer granularity than global contrast, greater robustness than point-wise contrast, and clearer distinction than patch-wise contrast. Through our proposed 2D-to-3D mask regularization, the 3D features are explicitly aligned with the 2D-text feature space, enhancing the performance of the 3D branch in open vocabulary segmentation. This progress is further substantiated by the ablation studies outlined in Section 4.3.

3.4 3D-2D Mask Feature Fusion

The 3D-2D mask feature fusion block is devised to merge 3D features F 3d ∈ R N × C with the pseudo mask feature F 2d ∈ R N ′ × C derived from the 2D branch. It is noteworthy that each element in F 3d possesses unique and distinguishing embeddings, whereas elements in F 2d pertaining to the same mask share identical mask embeddings. Consequently, F 3d offers detailed geometric structural information, while F 2d provides semantic features with robust open vocabulary capabilities. Our approach combines features from these two modalities to leverage their complementary insights, resulting in F fuse which excels in both precise geometry delineation and expansive semantic extrapolation. Concretely, given that N ′ < N , we selectively merge F 2d and F 3d solely on the N ′ points where correspondences exist:

where cat represents concatenation. Ablation studies in Section 4.3 and visualization results in Section 4.2 will demonstrate that F fuse effectively integrates the strengths of both F 3d and F 2d .

3.5 Training Objectives

In XMask3D, our training strategy encompasses supervised segmentation loss ( L seg ) computed from 3D ( L 3d seg ), 2D ( L 2d seg ), and fusion ( L fuse seg ) modalities. We employ Cross Entropy loss for 3D and fusion segments, and for 2D, we adopt multi-head losses including Cross Entropy, Dice, and Focal Loss, following ODISE [45] and Mask2Former [4, 3] guidelines. Additionally, we follow PLA [11] to introduce a binary head and view-level contrastive loss. The binary head is optimized with Binary Cross Entropy loss ( L bi ) to differentiate between base and novel categories. The view-level contrastive loss ( L view ) is calculated between the view global feature and text embedding of the view image caption, weighted by respective coefficients ( ω 3d view , ω 2d view , ω fuse view ). Detailed information about the binary head and the view-level regularization can be found in the PLA paper or in Section A.3 and A.2. In conclusion, the overall training objective can be adjusted by:

where ω seg , ω mask , ω 3d view , ω 2d view , ω fuse view , ω bi , are loss weight hyperparameters.

4 Experiments

4.1 Experiment Settings

Datasets. In accordance with prior literature, our research conducts experimentation on two prominent indoor scene datasets: ScanNet [9] and S3DIS [1]. ScanNet, a foundational dataset in this domain, comprises 1201 scenes allocated for training and 312 scenes designated for validation. Each scene within ScanNet furnishes point cloud data, multi-view images, and corresponding camera pose matrices. Similarly, S3DIS offers analogous data modalities, encompassing 271 rooms across six distinct indoor environments. Conforming to established conventions, we reserve Area 5 of S3DIS for validation purposes, ensuring consistency with prior methodologies.

Category Partition. In alignment with previous research, we exclude the otherfurniture class and partition the remaining classes into three benchmarks: B15/N4, B12/N7, and B10/N9. Here, B15 signifies the 15 fundamental categories that remain visible and supervised during the training process, while N4 denotes the presence of 4 novel categories introduced during evaluation. For the ScanNet variant featuring 200 classes [38], we adopt a similar approach, dividing the dataset into B170/N30 and B150/N50 benchmarks, each representing a distinct configuration of base and novel categories. Similarly, in the case of S3DIS, comprising 13 classes, we disregard the clutter class and organize the dataset into B8/N4 and B6/N6 benchmarks. Detailed information can be found in Section A.4.

Metrics. Following PLA [11], we present the mean Intersection over Union (mIoU) scores separately for both base and novel categories to assess open vocabulary segmentation performance. Additionally, to provide a comprehensive evaluation of the segmentation capability, we report the harmonic mean IoU (hIoU) derived from the mIoU scores of base and novel categories. This holistic metric offers insights into the overall segmentation efficacy across the dataset.

Implementation Details. Our implementation incorporates MinkUNet [7] as the 3D branch and ODISE [45] as the 2D branch within the architecture. For the vision-language model, we opt for CLIPL [36]. The training regimen for the XMask3D model involves utilizing the AdamW optimizer [29] with a Cosine learning rate scheduler. We train the model for 150 epochs on 4 NVIDIA A800 GPUs, employing a batch size of 64. Notably, we introduce mask-level regularization to the training pipeline after the initial 50 epochs. This decision is motivated by the observation that the quality of mask prediction at the onset of training may be suboptimal, making the mask-level contrastive loss ineffective and potentially misleading. Detailed information regarding hyperparameter selections is provided in Section A.4.

4.2 Main Results

Quantitative Comparisons. From Table 1 and Table 2, our proposed XMask3D outperforms previous methods across most benchmarks, irrespective of the novel category proportion or dataset sources. The performance indicates that XMask3D is a robust and generalizable method for open vocabulary 3D semantic segmentation. Notably, we compare XMask3D with our baseline method, PLA, on novel category performance. On the ScanNet dataset, XMask3D demonstrates improvements ranging from 7.4% to 20.0% over PLA. On the ScanNet200 dataset, XMask3D surpasses PLA by an impressive

70.5% and 72.7%. On the S3DIS dataset, XMask3D shows boosts of 51.8% and 33.0% over PLA. Among these benchmarks, XMask3D achieves the highest improvements on the long-tail ScanNet200 dataset, primarily due to the introduction of the denoising UNet from the pre-trained diffusion model, which constructs a comprehensive text-2D embedding space with unlimited text descriptions.

It is noteworthy that the OpenScene results are derived from the implementation of UniM-OV3D [16], but we do not compare XMask3D with UniM-OV3D in the tables. This is because UniM-OV3D employs PointBIND [14] and CLIP2Point [19] with an already aligned 3D-text embedding space, whereas XMask3D only integrates the commonly used 2D-text space and introduces techniques to pull 3D features from any point cloud models towards the shared embedding space. Another outstanding concurrent method is OV3D [20], which primarily focuses on EntityText extraction and PointEntityText association, while XMask3D concentrates on enhancing mask-level interaction between 2D and 3D modalities. Therefore, the contributions of OV3D and XMask3D are orthogonal and could complement each other in future explorations. Additionally, OV3D only provides experiment results on the ScanNet20 benchmarks, while its effectiveness on the ScanNet200 dataset that is more challenging and on the S3DIS dataset with different data distribution remains unexplored.

Visualization Results. In Figure 3, we present a comparative visualization of novel categories between XMask3D and previous methods [11, 34]. XMask3D demonstrates superior accuracy in category predictions, produces finer segmentation boundaries, and generates more cohesive mask regions. Notably, the missegmented region on the bookshelf in the second row of XMask3D is classified as picture, which appears reasonable given the corresponding part in the view image.

4.3 Ablation Studies

In this section, we comprehensively discuss the design choices of XMask3D through extensive ablation studies on the ScanNet B12/N7 benchmark. The results are presented in Table 3.

Mask Generation Condition. In Section 3.2, we introduce an Implicit 3D Captioner to convert global 3D features into implicit condition embeddings for the diffusion model. In Table 3a, we compare this implicit 3D condition with the vanilla text condition and implicit 2D condition. The text condition is generated by a ViT-GPT2 [48] captioning model and encoded via the frozen CLIP text encoder. The 2D condition is generated by the frozen CLIP image encoder and a learnable MLP, following the design in ODISE [45] (Equation 3). Our proposed implicit 3D condition outperforms the others in novel category segmentation, demonstrating that integrating the 3D global feature with the diffusion model produces the most compatible open vocabulary masks with the 3D branch.

Mask Regularization. In Section 3.3, we propose a fine-grained mask regularization term to align 3D features with the 2D-text embedding space. In the first line of Table 3b, we remove the fine-grained mask-level loss L mask from the training pipeline. Besides the final results from the fusion block, we also report the intermediate results from the 2D and 3D branches. The inclusion of the mask loss results in a significant improvement of 11.7 in 3D performance on novel categories, demonstrating that our fine-grained mask-level regularization effectively brings 3D features closer to the 2D-text embedding space. Additionally, the performance gain on the fused output is 1.5, highlighting the positive impact of this regularization from 3D to fusion features.

Ground Truth

PLA

OpenScene

Table

Desk

View Image

Bed

Chair

Bed

Table

Table

Chair

Sofa

Bookshelf

Picture

Cabinet

We also analyze the effects of mask regularization through visualizations in Figure 4. Comparing the first and second rows, the 3D segmentation results without the mask loss show inconsistency in the local region for the novel category chair, with most points misclassified as bookshelf. When the mask-level regularization is added, more points are correctly classified, resulting in a clearer segmentation mask. The visualizations of the fused outputs also demonstrate consistent enhancement in mask regularization within the final column. This is evident from the rectification of misclassified points previously labeled as table on the armrest of the right chair in the last column.

Modality Fusion. When comparing the 2D, 3D, and fused metrics within the same line in Table 3b, we empirically find that the 2D branch performs relatively better on novel category segmentation, while the 3D branch excels at base category segmentation. This quantitative observation aligns with our design intention: to exploit geometric knowledge via the 3D branch and to enhance the model's open vocabulary capability via the 2D branch. More importantly, the fused output outperforms both the 2D and 3D intermediate results on both base and novel splits. These results strongly support the effectiveness of our fusion design in merging the complementary knowledge from both modalities.

We also present visualization evidence regarding modal complementarity and fusion effects in Figure 4. In the first group, the 2D branch exhibits unsatisfactory results of the base category wall around the whiteboard and behind the right chair, whereas effectively segmenting the novel category chair with high quality despite minor artifacts. Conversely, the 3D branch produces an unsatisfactory mask for the chair but excels in segmenting the wall based on geometric information. The fusion block leverages the strengths of both branches, mitigates their weaknesses, and yields satisfactory outcomes. Moreover, the fusion block accurately delineates regions such as the left foot of the chair on the right, where both the 2D and 3D branches falter, highlighting its potential for integrating cross-modality knowledge. The second group displays similar results that convince the effectiveness of the proposed mask loss and cross-modality fusion design.

4.4 Limitations

Due to resource constraints, we only evaluate the performance of XMask3D on semantic segmentation in this study. However, the XMask3D pipeline has the potential to be extended to the instance and panoptic perception by replacing the 3D backbone with an instance or panoptic segmentation model.

without

Mask

Loss

with

Mask

Loss

Ground Truth

2D Branch Output

3D Branch Output

Fusion Block Output

Table

Chair

Novel

Bookshelf

Wall

Floor

Base

Desk

Picture

without

Mask

Loss

with

Mask

Loss

Ground Truth

2D Branch Output

3D Branch Output

Fusion Block Output

Table

Chair

Novel

Bookshelf

Wall

Floor

Base

Desk

Curtain

This adaptability arises from the auxiliary 2D branch, which is mask-based and capable of generating instance-level masks given appropriate annotations. Furthermore, since the XMask3D pipeline is agnostic to the 3D model used, it would be interesting to conduct experiments with more advanced point cloud models to compare performance. We hope we can exploit the strength of XMask3D with high-standard 3D backbones and a wider range of dense perception tasks in future research. Computational cost is another limitation of XMask3D, as we implement the Denoising UNet which has numerous parameters and relatively slow latency. In our future work, we plan to address this limitation by replacing the 2D branch with a more lightweight 2D open vocabulary mask generator.

5 Conclusion

In this paper, we present XMask3D, designed for open vocabulary 3D semantic segmentation. We propose the integration of the denoising UNet of a pre-trained diffusion model to produce geometryaware segmentation masks conditioned on learnable implicit 3D embeddings. These binary 2D masks filter mask-level embeddings of 3D representations and apply mask regularization to enhance the open vocabulary capacity of 3D features. By fusing the 2D mask embeddings with fine-grained 3D features, we leverage the complementary knowledge from both modalities, achieving competitive performance across various benchmarks and datasets. Ablation studies and visualization comparisons further validate the effectiveness of the proposed cross-modal mask reasoning method.

Acknowledgments and Disclosure of Funding

This work was supported in part by the National Natural Science Foundation of China under Grant 623B2063, Grant 62321005, Grant 62336004, and Grant 62125603.

References

Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In CVPR , 2016.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV , 2021.

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR , 2022.

Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurIPS , 2021.

Ali Cheraghian, Shafin Rahman, Dylan Campbell, and Lars Petersson. Transductive zero-shot learning for 3d point cloud classification. In WACV , 2020.

Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. arXiv preprint arXiv:2303.11797 , 2023.

Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR , 2019.

Spconv Contributors. Spconv: Spatially sparse convolution library. https://github.com/ traveller59/spconv , 2022.

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR , 2017.

Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In CVPR , pages 11583-11592, 2022.

Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open-vocabulary 3d scene understanding. In CVPR , 2023.

Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskclip. In ICML , 2023.

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In ECCV , 2022.

Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.00615 , 2023.

Huy Ha and Shuran Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. arXiv preprint arXiv:2207.11514 , 2022.

Qingdong He, Jinlong Peng, Zhengkai Jiang, Kai Wu, Xiaozhong Ji, Jiangning Zhang, Yabiao Wang, Chengjie Wang, Mingang Chen, and Yunsheng Wu. Unim-ov3d: Uni-modality openvocabulary 3d scene understanding with fine-grained feature representation. arXiv preprint arXiv:2401.11395 , 2024.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS , 33, 2020.

Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In ICCV , 2023.

Li Jiang, Shaoshuai Shi, and Bernt Schiele. Open-vocabulary 3d semantic segmentation with foundation models. In CVPR , pages 21284-21294, 2024.

Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot open-vocabulary segmentation. arXiv preprint arXiv:2306.09316 , 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV , 2023.

Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Languagedriven semantic segmentation. arXiv preprint arXiv:2201.03546 , 2022.

Jingyao Li, Pengguang Chen, Shengju Qian, and Jiaya Jia. Tagclip: Improving discrimination ability of open-vocabulary semantic segmentation. arXiv preprint arXiv:2304.07547 , 2023.

Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Guiding text-to-image diffusion model towards grounded generation. In ICCV , 2023.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR , 2023.

Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. 3d open-vocabulary segmentation with foundation models. arXiv preprint arXiv:2305.14093 , 2023.

Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. NeurIPS , 36, 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Chaofan Ma, Yuhuan Yang, Yanfeng Wang, Ya Zhang, and Weidi Xie. Open-vocabulary semantic segmentation with frozen vision-language models. arXiv preprint arXiv:2210.15138 , 2022.

Björn Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet. Generative zero-shot learning for semantic segmentation of 3d point clouds. In 3DV , 2021.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 2021.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR , 2023.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

David Rozenberszki, Or Litany, and Angela Dai. Language-grounded indoor 3d semantic segmentation in the wild. In ECCV , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS , 35, 2022.

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR , 2023.

Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In ICCV , 2023.

Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster, stronger. arXiv preprint arXiv:2312.10035 , 2023.

Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-based pooling. NeurIPS , 2022.

Zihao Xiao, Longlong Jing, Shangxuan Wu, Alex Zihao Zhu, Jingwei Ji, Chiyu Max Jiang, WeiChih Hung, Thomas Funkhouser, Weicheng Kuo, Anelia Angelova, et al. 3d open-vocabulary panoptic segmentation with 2d-3d vision-language distillation. arXiv preprint arXiv:2401.02402 , 2024.

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR , 2023.

Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In CVPR , 2023.

Jihan Yang, Runyu Ding, Weipeng Deng, Zhe Wang, and Xiaojuan Qi. Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. arXiv preprint arXiv:2304.00962 , 2023.

ydshieh. Vit-gpt2 image captioning. https://huggingface.co/nlpconnect/ vit-gpt2-image-captioning .

Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. In ICCV , 2023.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV , 2023.

Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In ICCV , 2021.

A Additional Implementation Details

A.1 Mask-level Regularization

Mask-level regularization facilitates local and finely-grained alignment between features extracted from the 3D branch of XMask3D and the 2D-text embedding space. The ground truth for mask-level loss is computed using MaskCLIP [12], utilizing the predicted segmentation mask M 2d ∈ R M × H × W and the view image I ∈ R 3 × H × W :

where M represents the number of masks, and H and W denote the height and width of the image, respectively. V refers to the pre-trained CLIP [36] image encoder.

Specifically, the view image I is encoded into image tokens T I ∈ R N × C with the pre-trained CLIP image model, where N is the number of image tokens and C is the dimension of the CLIP embeddings. The class token T C ∈ R 1 × C is duplicated M times as the mask class tokens T M ∈ R M × C . Then T I , T C , T M are concatenated together to perform masked attention with frozen weights from the pre-trained CLIP image model. The attention mask is designed as

where T m × n is an m × n True matrix, F m × n is an m × n False matrix and P is defined as:

where True means that this position is masked out i.e. not allowed to attend and False otherwise. Then the updated mask class tokens T ′ M from the masked attention layers can be regarded as the CLIP embedding of the masked regions, serving as the ground truth G CLIP for mask-level regularization.

A.2 View-level Regularization

The view-level regularization facilitates coarse and high-level alignment between features extracted from the 3D branch and the 2D-text embedding space. To elaborate, upon receiving an image I , we initially generate its text caption using a pre-trained captioning model, ViT-GPT2 [48]. Subsequently, we employ the pre-trained CLIP [36] text encoder to encode the text caption into the 2D-text embedding space, yielding f T view , which serves as the ground truth for view-level regularization.

We perform average pooling operation on dense 3D point cloud features, 2D image features and fused features to obtain their global embeddings f 3d view , f 2d view , f fuse view . Then we implement contrastive loss between global features and the ground truth text embeddings:

where m = 3d , 2d , fuse denotes different modalities.

A.3 Binary Head

Following PLA [11], we include a binary head to predict whether the points belong to base or novel categories. We implement a small 3D model as the binary head with minimum computation cost. The prediction s b is utilized to modulate the over-confident semantic score s :

where s B is the semantic score computed solely on base classes with novel class scores set to zero. Similarly, s N is computed only for novel classes, setting base class scores to zero.

(a) ScanNet dataset.

(a) ScanNet dataset. Shower c. is short for shower curtain.

(b) S3DIS dataset.

A.4 Training and Inference Settings

Training. The supervised segmentation loss L m seg is calculated via the per-point classification Cross Entropy Loss on N points:

where m = 3d , 2D , fuse denotes different modalities, y i is the ground truth for base categories, ¯ F m is the normalized feature, and τ is a learnable temperature parameter.

Inference. We follow ODISE [45] to combine the predicted logits p with the prediction from a text-image discriminative model to enhance the open vocabulary classification capacity of the model. Specifically, we leverage the mask-level regularization ground truth feature G CLIP from Section A.1 to modulate the segmentation logits:

where ¯ G CLIP is the normalized feature of G CLIP , λ ∈ [0 , 1] is the fixed balancing factor.

Hyper-parameters. For all benchmarks, we set the same ω seg = 4 , ω 3d view = 1 , ω 2d view = 4 , ω fuse view = 1 . 5 as the hyper-parameter choices. The ω mask and ω bi are set differently across benchmarks. We

Ground Truth

PLA

OpenScene

XMask3D

(Fused)

View Image

XMask3D

(2D Branch)

XMask3D

(3D Branch)

Ground Truth

2D Branch Output

3D Branch Output

Fusion Block Output

without

Mask

Loss

with

Mask

Loss

Desk

Toilet

Novel

Bookshelf

Wall

Floor

Base

Bathtub

set ω mask = 0 . 5 / 0 . 5 / 1 / 2 / 2 / 1 / 1 , ω bi = 16 / 12 / 8 / 48 / 32 / 20 / 15 for ScanNet B15/N4, B12/N7, B10/N9, ScanNet200 B170/N30, B150/N50, S3DIS B8/N4, B6/N6 benchmarks, respectively.

Category Partitions. We follow PLA [11] to partition the ScanNet [9] and S3DIS [1] datasets into several different benchmarks. Here we list the detailed partition principle in Table 4 for reference. We do not list splits for the ScanNet200 dataset since there are too many categories.

B Additional Experimental Results

B.1 Per-class Results Comparison

We contrast per-class segmentation mean Intersection over Union (mIoU) with PLA [11] across the ScanNet and S3DIS datasets in Table 5. XMask3D significantly outperforms PLA, particularly in novel categories, thereby showcasing the effectiveness of our proposed cross-modal mask reasoning approach on open vocabulary 3D semantic segmentation.

B.2 Visualization Comparison

In Figure 5, we show 2D and 3D branch outputs from XMask3D in addition to Figure 3 in the main paper. The 2D branch misclassifies the novel categories, while the 3D branch produces discontinuous segmentation masks. Neither of them outperforms PLA or OpenScene. However, when their features are fused together, the output becomes superior, demonstrating the complementary knowledge across different modalities.

2D Branch Output

3D Branch Output

Fused Output

Ground Truth

Curtain

Shower

Curtain

Bathtub

Wall

Picture

Door

Counter

Sink

Cabinet

B.3 Illustrations of Ablations

We present supplementary illustrations of ablation studies pertaining to mask regularization and cross-modality fusion in Figure 6, which corresponds to the sample in Figure 1 from the ScanNet B15/N4 benchmark.

Comparing the first and second rows of the third column reveals that the proposed mask regularization loss yields significant improvements in the segmentation of novel categories by the 3D branch. Further comparison of the last three columns in the second row demonstrates that the output from the fusion block effectively leverages the strengths of both modality branches. Specifically, while the 2D branch excels at segmenting novel categories, it tends to produce discontinuous masks for base categories with larger regions (e.g., wall). Conversely, the 3D branch may perform relatively poorly in novel category segmentation but provides consistent geometric knowledge as a complementary aspect. Consequently, the fused output achieves superior performance in open vocabulary segmentation.

C Failure Cases Analysis

In Figure 7, we display some failure cases of XMask3D. The first sample shows a bathtub with a shower curtain in the bathroom. However, the 2D/3D branch and the fused output of XMask3D all misclassify the shower curtain as curtain. This may be because these two categories are similar in object shape and texture, with the only difference being the surrounding environment. Since XMask3D only takes a corner of the room as input instead of the entire scene, the global environmental information is insufficient for making the correct category prediction.

The second sample shows a large area of picture on the wall. The 2D, 3D branches and the fused output of XMask3D all misclassified it as wall, due to their similar geometry. In most cases, a picture is a small region on the wall, and this picture, as large as the wall, is a typical corner case. This failure case may reveal XMask3D's over-reliance on geometric knowledge and lesser consideration of texture information when encountering out-of-distribution samples.

The third sample shows a sink on a counter. Due to the occlusion problem, the sink point cloud is incomplete, negatively affecting the prediction of segmentation boundaries between the sink and the counter. This occurs because they are geometrically similar when the sinking-down part of the sink is missing.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and introduction is that we propose an XMask3D framework for open vocabulary 3D semantic segmentation, and the paper is all about introducing its design and effectiveness.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Section 4.4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not propose new theorem. We conduct experiments and show illustrations to evaluate our model design.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Section 4.1 and Section A for experiment settings and implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to the supplemental zip file for our code and we will officially release the code upon the acceptance of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Section 4.1 and Section A for experiment settings and implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We do not report error bars since previous literature do not report them either and the experiments on scene-level dataset is resource consuming. However, we provide extensive experiment results on multiple benchmarks of various datasets and show consistent performance gain over previous methods. Therefore, the statistical significance is convinced by the variety of datasets and benchmarks.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Section 4.1 for experiments compute resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully read the Ethics Guidelines and strictly followed them.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper of the pre-trained models and data we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces XMask3D, a framework developed for open vocabulary 3D semantic segmentation. They propose the integration of the denoising UNet, derived from a pre-trained diffusion model, to generate geometry-aware segmentation masks conditioned on learnable implicit 3D embeddings. These binary 2D masks are used to filter mask-level embeddings of 3D representations and apply mask regularization, thereby improving the open vocabulary capacity of 3D features.\n\nSTRENGTHS:\n1.\tThe motivation is clear.\n2.\tThe proposed method is intuitive, and the experiments have validated their contributions.\n\nWEAKNESSES:\n1.\tThe organization should be improved. Section 3.1 provides an overview, while section 3.2 includes design insights and preliminary findings. The flow of these writings has puzzled me, making it difficult to grasp your key contribution.\n\nQUESTIONS:\n1.\tPlease provide further clarification on how mask-level alignment between 3D features and 2D embedding space can address the limitations of traditional techniques, such as global feature alignment or vision-language model distillation. Additionally, if texts (Category Labels) are concatenated with fused features, will it still create a unified feature space that encompasses 3D, 2D, and textual modalities?\n2.\tCould you please provide further clarification on the main contributions of your research compared to PLA and OpenScene? Although the 3D caption process shares similarities with PLA, the overall pipeline resembles OpenScene, with the exception of the diffusion model and mask generator, which differ from the Multi-view Feature Fusion in OpenScene.\n3.\tDoes the Implicit 3D Captioner effectively work with your 3D features? From my understanding, the most reliable 3D captioner currently available is Cap3D, which generates captions for 3D objects by rendering multi-view images and utilizing BLIP2 and LLM for assistance. In the context of indoor-scenes, can we consider the Implicit 3D Captioner to be equally robust? It would be beneficial to present additional evidence to support this claim.\n4.\tCan your text-to-image diffusion model effectively generalize to your datasets? If not, please provide examples of failure cases. Additionally, is the diffusion model fine-tuned during the training process or is it frozen? If not, please present additional results to demonstrate the robustness of your diffusion model in generating high-quality images within your datasets.\n5.\tWhat is view-level contrastive loss? Why this loss is calculated between the view global feature and text embedding of the view image caption but have three coefficients?\n6.\tIt is recommended to show your 2D Mask and 3D Mask in Figure 3 to provide more visual evidence.\n7.\tThe authors should provide results on ScanNet++ (CVPR’23), which is a up-to-date dataset compared with ScanNet.\n8.\tSince diffusion models are utilized, it is recommended to compared the model parameters and FLOPs compared with PLA and OpenScene.\n\nLIMITATIONS:\n1.\tThe authors have addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

comments: SUMMARY:
The paper proposes a precise and consistent mask-level alignment between 3D features and the 2D-text embedding space through a method called cross-modal mask reasoning. The proposed XMask3D model includes a 3D branch for capturing geometric features, a 2D branch for generating vision-language aligned masks, and a fusion block to combine 3D with 2D. Using a pre-trained text-to-image diffusion model as the 2D mask generator, the model leverages three techniques: 3D-to-2D mask generation, 2D-to-3D mask regularization, and 3D-2D mask feature fusion.

STRENGTHS:
1- The idea is novel, the author propose to merge 2D which provides high OV capabilities, with 3D features shich endoces 3D geometry. 

2- The method performs remarkably better than the reported models, namely OpenScene. The experiments are also well structure

WEAKNESSES:
1- The authors don't compare with state-of-the-art 3D semantic segmentation OV3D[1]

2- The authors highlighed fututre work in the limitation, it would be good if you can expand it with some limitation on the technical side or some failure cases.

[1] Jiang, Li, Shaoshuai Shi, and Bernt Schiele. "Open-Vocabulary 3D Semantic Segmentation with Foundation Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

QUESTIONS:
Please compare to OV3D mentioned in the weaknesses

LIMITATIONS:
Needs to be expanded
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
The paper addresses the limitations of current open vocabulary 3D semantic segmentation methods, which primarily focus on creating a unified feature space for 3D, 2D, and textual modalities but struggle with fine-grained segmentation boundaries. To overcome these limitations, the authors propose XMask3D, a cross-modal mask reasoning framework that achieves more precise mask-level alignment between 3D features and the 2D-text embedding space.

STRENGTHS:
1. The part "incorporating a 2D mask generator to create geometry-aware open masks and apply fine-grained mask-level regularization on 3D features" seems reasonable and novel.
2. The paper is well-structured and easy to follow.
3. Analysis is thorough and insightful.

WEAKNESSES:
1. The paper evaluates the proposed method on a limited set of benchmarks (ScanNet20, ScanNet200, S3DIS), all of which are indoor scene datasets. Authors could discuss how the method might perform on outdoor datasets. Additionally, the authors could provide a qualitative analysis of the model's potential limitations when applied to different environments.
2. The reliance on the denoising UNet from a pre-trained diffusion model could be seen as a potential weakness or limitation, especially given the computational resources required for training and inference.

QUESTIONS:
1. The paper could benefit from a more detailed error analysis to understand the failure modes of XMask3D, especially in novel category segmentation.

LIMITATIONS:
The authors have addressed limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of open-vocabulary 3D semantic segmentation by utilizing 3D geometric features, 2D semantic embeddings, and text modality. The proposed approach adapts the ODISE method to the 3D domain, aiming to distill open-vocabulary semantic segmentation knowledge from a pre-trained text-to-image denoising diffusion model to a 3D segmentation model. Initially, an input point cloud is fed into a 3D encoder-decoder segmentation network, producing point-wise geometric features. Simultaneously, a pre-trained visual-language diffusion model generates 2D masks and embeddings from posed images of the same scene, conditioned on the 3D global feature of the 3D branch’s encoder. Unlike the ODISE method, an implicit $3D$ captioner is introduced to produce geometry-aware 2D masks while also distilling information from the 2D branch network to the 3D encoder. To further regularize the 3D network, a distillation loss ($\\mathcal{L}_{mask}$) is applied to the 3D mask embeddings, derived from the per-point features and the 2D masks back-projected to the point cloud as 3D binary masks. By obtaining ground truth mask features from a pre-trained CLIP model, the 3D masked embeddings are aligned with the image-text joint embedding space, through a cosine similarity loss. This alignment leads to more coherent segmentation results and enhances the model's open-vocabulary capabilities. Finally, the per-point features are combined with the pseudo mask 2D features (formed by the back-projected 3D mask and 2D mask embeddings), resulting in a fused per-point representation that incorporates the geometric information from the 3D segmentation network and the semantic open-vocabulary capabilities of the 2D branch. The approach is evaluated on three semantic segmentation benchmarks (ScanNet, ScanNet200, and S3DIS) and demonstrates superior performance compared to competing methods.\n\nSTRENGTHS:\nThe XMask3D effectively aligns 3D geometric features with 2D and textual modailities through knowledge distillation from visual-text joint embedding spaces inherent in the pre-trained 2D denoising UNet and the CLIP model. As evident by the ablation, the implicit 3D captioner is a crucial step in the overall pipeline, and it outperforms vanilla text conditioning or the implicit 2D captioner of ODISE, in both base and novel semantic categories. Moreover, the 2D-to-3D mask regularization is also essential, since it significantly improves the accuracy of the proposed method esp. in novel categories. This justifies the need for this additional distillation step from the CLIP joint space, to further enhance the open-vocabulary capabilities of the XMask3D method. Finally, the discussion on modality fusion, both in the main paper and supplementary, is highly appreciated. By dissecting the method and providing qualitative and quantitative results for each step, the authors make it easier for readers to understand and gain intuition about the presented approach.\n\nWEAKNESSES:\nWhile the method exhibits superior performance w.r.t. competing methods, it seems that the output fused embeddings yields to geometric inconsistent features for semantic classes that cover large areas of the point cloud such as wall, ceiling and floor. This is evident in both partitioning settings when the class is either base or novel (Table 5 (a) and (b) in supp.).\n\nQUESTIONS:\nFollowing the weaknesses section, do the authors have any additional insights into why this phenomenon occurs with the fused embeddings? Could the 2D regularization terms ($\\mathcal{L}_{seg}^{2D}$, $\\mathcal{L}\\_{view}^{2D}$) be introducing too much bias towards the 2D visual modality, thereby causing geometric discontinuities in the output fused features?\n\nLIMITATIONS:\nYes, the authors have discussed the method's limitations in detail in Section 4.4.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}