XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation
Authors: Ziyi Wang
Yanbo Wang
Xumin Yu
Jie Zhou
Jiwen Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D open vocabulary
semantic segmentation
cross-modal learning
Primary area: machine_vision

Abstract: Existing methodologies in open vocabulary 3D semantic segmentation primarily concentrate on establishing a unified feature space encompassing 3D, 2D, and textual modalities. Nevertheless, traditional techniques such as global feature alignment or vision-language model distillation tend to impose only approximate correspondence, struggling notably with delineating fine-grained segmentation boundaries. To address this gap, we propose a more meticulous mask-level alignment between 3D features and the 2D-text embedding space through a cross-modal mask reasoning framework, XMask3D. In our approach, we developed a mask generator based on the denoising UNet from a pre-trained diffusion model, leveraging its capability for precise textual control over dense pixel representations and enhancing the open-world adaptability of the generated masks. We further integrate 3D global features as implicit conditions into the pre-trained 2D denoising UNet, enabling the generation of segmentation masks with additional 3D geometry awareness. Subsequently, the generated 2D masks are employed to align mask-level 3D representations with the vision-language feature space, thereby augmenting the open vocabulary capability of 3D geometry embeddings. Finally, we fuse complementary 2D and 3D mask features, resulting in competitive performance across multiple benchmarks for 3D open vocabulary semantic segmentation. Code is available at https://github.com/wangzy22/XMask3D.

XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation
Ziyi Wang" Yanbo Wang" Xumin Yu Jie Zhou Jiwen Lu Department of Automation, Tsinghua University, China {wziyi22, wyb23, yuxm20)0mmails tsinghua. en; {jzhou, lujiven)@tsinghua. edu.
Abstract
Existing methodologies in open vocabulary 3D semantic segmentation primarily concentrate on establishing a unified feature space encompassing 3D, 2D, and textual modalities. Nevertheless, traditional techniques such as global feature aligmment or vision-language model distillation tend to impose only approximate correspondence, struggling notably with delineating fine-grained segmentation boundaries. To address this gap, we propose a more meticulous mask-level align- ment between 3D features and the 2D>ttxtt emhedding space through cross-modal mask reasoning framework, XMask3D. In our approach, we developed a mask generator based on the denoising UNet from pre-trained diffusion model, leverag- ing its capability for precise textual control over dense pixel representations and enhancing the open-wort world adaptability of the generated masks. We further integrate 3D global features as implicit conditions into the pre- vre-trained 2D denoising UNet, enabling the generation of segmentation masks with additional 3D geometry aware- ness. Subsequently, the generated 2D masks are employed to align mask-level 3D representations with the vision-language feature space, thereby augmenting the open vocabulary capability of 3D geometry embeddings. Finally, we fuse complementary 2D and 3D mask features, resulting in competitive performance across multiple benchmarks for 3D open vocabulary semantic segmentation. Code is available at hhttts:///// //tg://ggttttt con//wangzy22/XXass33
Introduction
As the integration vision and language deep learning continues to expand, text descriptions are increasingly utilized in visual generation [37 50, 39 35
26 40 and perception [36] 22 23
13 45 [2]
tasks. This integration enhances the adaptability of models in real-world applications and improves user experiences in customized artificial intelligence systems. Open vocabulary 3D semantic segmentation exemplifies a perception task that trained on base categories and demands robust extrapolation capabilities discriminate fine-gra mne-grained geometry in novel categories that invisible during training. The base and novel classes are only linked by the shared open vocabulary within the language space. However, constructing shared 3D-text space while maintaining precise, modality-speciff representation remains significant challenge. Successfully addressing this issue would advance open vocabulary 3D semantic segmentation, facilitating virtual reality interactions and manipulations and thereby contributing to the development of user-friendly robotics and autonomous driving technologies.
Existing approaches for open vocabulary 3D perception predominantly aim to bridge the gap between 3D and text representations by using the 2D modality an intermediary. One line of research 34]
Equal contribution. 'Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Model with Globol 3D-20: Text Alignment
+
Cross Modal
3D Geometry Extractor
Fusion Block
20-to-3D Mask Loss
Fused Result
3D-to to 20 Mask Condition
Catapory
Toilet
Disk
2D Open Mask Generator
Base Categony
Will
Bathtub
20 Result
Ground Trush
The overall framework of XMask3D. The 3D model with only coarse 3D-2D-text alignment struggles to segment novel categories with accurate boundaries. We propose to incorporate 2D open mask generator conditioned on global 3D geometry features to create geometry aware -ware segmentation masks of novel categories. Then we apply fine ahe-grained mask-level regularization on 3D features, thereby enhancing the dense open vocabulary capability of the 3D model. The cross-modal fusion block leverages the strengths of both branches to achieve optimal results.
Figure 1:
16 suggests aligning 3D embeddings with vision-language embedding spaces through global, investigates distilling open vocabulary knowledge from foundational vision- laiiuge models into 3D models. However, global feature alignment and model distillation techniques tend to overlook fine-grained 3D) geometric details and produce coarse results. Conversely, point-wise contrastive learning is prone to noise and outliers. Although patch-wise feature alignment offers compromise, single patch in 2D may correspond to multiple unrelated and discontinuous regions in 3D, which can misleading and ineffective.
patch-wise, or point-wise contrastive loss, while another line 49
28
44
In this paper, we propose a more precise and consistent mask-level alignment between 3D features and the 2D-text embedding space, achieved through our proposed cross-modal mask reasoning method. The proposed XMask3D model comprises 3D branch, a 2D branch, and fusion block. The 3D branch, adaptable as any point cloud segmentation model, excels in capturing geometrie features but struggles with novel category extrapolation. Conversely, the 2D branch serves as mask generator, which predicts masks with embeddings aligned to the vision-language feature space but lacks 31) spatial perception capabilities. The fusion block allows these two branches to complement each other. Specifically, we propose utilizing the denoising UNet of a pre-trained eek-to-immge diffusion model with advanced vision- vision-language modeling capabilities the 2D mask generator. Diffusion model's exceptional control over text-driven image generation demonstrates strong potential for creating fine-grained segmentation masks of novel categories. To promote thorough interaction between the two modality branches, we propose three mask-level techniques: (1) 3D-to-28 2D Mask Generation. We condition the denoising mask generator global point cloud features, producing geometry- masks that are better suited for transferring to the 3D modality. (2) 2D-to-3D Mask Regularization. We apply mask-level regularization on 3D features align with the vision-language embedding space. This enhances the open vocabulary capability of 3D features for novel categories while preserving fine-grained geometric information. (3) 3D-2D Mask Feature Fusion. We merge mask features from both modalities in the fusion block, enhancing the synergy between 2D and 3D features.
We conduct extensive experiments multiple benchmarks of different datasets, including Scan- Net20 [9]. ScanNet200 [38]. and S3DIS [ datasets, to evaluate the effectiveness of our proposed method. XMask3D demonstrates competitive performance across all benchmarks. Additionally, perform thorough ablation studies and provide intuitive visualizations to showcase the contribution of each proposed mask-level technique. In conclusion, the contributions of this paper be summarized as follows:
We propose a novel XMask3D framework that, for the first time, leverages the denoising UNet of a generative text-to-image diffusion model for open vocabulary 3D perception.
We introduce 3D-to-2D mask generation, 2D-to-3D mask regularization, and 3D-2D mask feature fusion techniques to enhance meticulous mask-level 3D-2D-text feature alignment and strengthen cross-modal feature synergy.
We demonstrate the effectiveness of XMask3D multiple benchmarks of various datasets and show outstanding performance. Ablation studies further convince the contribution of each proposed mask-level technique.

---

2 Related Work
2.1 2D Open Vocabulary Segmentation
Open vocabulary perception is a recently emerged research problem that focuses on enabling percep- tion models recognize novel categories that invisible during supervised training, relying solely on a shared language vocabulary with the base categories. The key to addressing this problem lies in the Vision-Language model, which creates a shared embedding space for images and texts. Based on the types of Vision-Language models, previous literature on 2D open vocabulary segmentation can be broadly divided into two approaches: utiliziig Vision- Language perception models like CLIP [36 or leveraging Vision-Languee generation models like the diffusion model [18 37].
Although traditional Vision-Language perception models like CLIP are primarily designed for classification, open vocabulary segmentation can also be viewed as a dense classification task. Consequently, several studies have proposed aligning dense image features with 2D-text embeddings, a concept pioneered by LSeg (23). Successor models have introduced various techniques to enhance
the feature alignment process, including iitention-based combinations 30 [2], mask embedding
decoupling [10) side network injection [46] 22], and cross-modal aggregation [6]
Generative based methods utilize Vision-Language generation models, such diffusion models, to produce segmentation masks that can be extrapolated to open vocabulary categories. Since the diffusion model can generate semantically meaningful images based on text conditions, its intermediate features effectively represent the vision- Aision-language embedding space. ODISE [45] first proposed using the intermediate features from the denoising UNet of pre-trained diffusion model as input to a mask generator for segmentation. Other works [25 21 leverage the strong generative capabilities of the diffusion model create prototypes augmented image-mask pairs, thereby enhancing the open vocabulary capacity of the segmentation model from data perspective.
2.2 3D Open Vocabulary Segmentation
In 3D vision, Semantic Abstraction [5 opens up the avenue to leverage Vision-Language models for open-world 3D scene understanding. Subsequent studies have primarily developed two types of methods to address 3D open vocabulary segmentation: feature alignment and model distillation.
The principle of feature alignment methods is to explicitly pull 3D representations towards the vision- language embedding space, using the 2D modality as mediator to establish 3D-text relationships. OpenScene (34] employs a cosine similarity loss between point cloud features and image CLIP features, integrating them for open vocabulary perception. PLA [11] introduces hierarchical 3D caption pairs to progressively align scene-level, view-level, and entity-level features with the CLIP feature space in coarse-to-fine manner. Its successor, RegionPLC [47]. further introduces region- level captions with sliding windows and object bounding boxes, while CLIP-FO3D (49] similarly divides super-pixels for finer feature alignment. UniM-OV3D [16) utilizes a pre-trained point-text model, PointBind [14] to enforce uni-modality representation learning of point clouds, images, depth maps, and texts. OV3D (20] proposes leveraging foundation models to establish point-entity Text associations through pixel, thereby enhancing open open-vocabulary recognition within the 3D domain.
Model distillation methods typically involve selecting foundational Vision-Language model and transferring its knowledge introduces spatial contrastive learning and temporal consistency regularization to distill vision foundation models for point cloud sequence segmentation
3D network using paired point clouds and image data. Seal [2X]
an open vocabulary setting. 3D-
into neural radiance field (32) using novel alignment losses for 3D perception. Xiao et al. 44 propose object-level and voxel-level distillation losses for fine-grained 3D open vocabulary panoptic segmentation.
OVS 27 aims to distill CL.P R6 and DINO
2
33
3
Approach
3.1
Overview
The detailed architecture of XMask3D is depicted in Figure 2
It consists of three components: 3D geometry extraction branch, a 2D mask generation branch, and 3D-2D feature fusion module. The 3D geometry extraction branch is an encoder-decoder segmentation network, which can be
3

---

Point Cloud
Encoder
Decoder
Fusion Black
Mask Loss
Implicit 3D Captioner
CLIP Image Encoder
CLIP Text Encoder
Condition
Category Labels
Results
Image
Mask Generator
Mask Feat.
Mask Contour
3D Feat.
Diffusion UNet
Fused Feat.
CLIP I/T Feat.
Figure 2: The detailed architecture of X.Mssk3D We introduce an auxiliary 2D branch, which utilizes global point cloud features as conditional input to generate open vocabulary masks. The contour of the mask is utilized for regularization at the mask level on 3D features, and the embeddings of the mask are fused with the 3D features to enhance cross-modal complementarity.
implemented using any off-the-shelf architectures such as sparse convolution networks
or Transformer-basee networks [5]] 43 42]. These frameworks are specifically designed to extract geometric features F'3a E R from 3D point clouds, where N and C represent the numbers of points and feature dimensions, respectively. However, F33 discriminative only on base categories trained with supervision, relying on pre-defined classifier implemented or a multi-layer perceptron. Therefore, it's essential for Fsa to be aligned with the vision-lange laniune feature space enable unbounded open vocabulary segmentation, which is performed via per- point similarity comparisons between Faa and CLIP 36 text embeddings of given category names: (CLap,CCname) (Cname) E TR. LR where I. represents the number of categories, ECLIP, denotes the CLIP text encoder, and Cuame stands for the category name such as Table, Chair.
fully connected layer
Fiext ECLIPT
Previous literature feature alignment techniques varies from global point-level contrastive learning to vision-language model distillation. However, none of these methods simultaneously achieves fine-grained precision and robustness to outliers. To address this limitation, we introduce cross-modal mask reasoning method that performs mask-level feature alignment. Specifically, we propose a 2D mask generation branch in conjunction with the 3D branch generate 2D masks with open vocabulary capability and use these masks to regularize Fsa- Detailed information on mask generation and mask regularization can be found in Section 3.2 and Section 3.3 respectively.
From the 2D mask generator, we obtain the 2D mask embeddings Gza E MCC and binary mask maps M2a E IR xxxW>xM where M represents the number of candidate masks, and H and W denote the height and width of the input images. Using the camera intrinsic matrix K R3x3 and the view projection matrix V E R4 we can establish associations between pixels in image / and surface points in the corresponding point cloud P. This allows us to derive back-projected 3D binary masks M3d E N xM from M2a- Here, N' significs the number of points that correspond to image pixels, following the relationship N' < N' < H x W. Subsequently, we derive pseudo mask feature Fea Msa Gza, F2a NCC, with the subseript 2d indicating its origin from the 2D branch.
Having obtained Far from the 3D geometry extraction branch and Fza from the 2D mask generation branch, we implement a 3D-2D fusion block to combine these cross-modal and complementary features, resulting in Fruse NCC Detailed information on this process can be found in Section 3.4 The final open vocabulary 3D semantic segmentation output O E N is then calculated via:
Fluse
FT
= argmaxN F
text
Frest
Detailed training objectives can be found in Section 3.5
3.2 3D-to-2D Mask Generation
Design Insights. An optimal 2D branch expected to exhibit robust open vocabulary capabilities, enabling it to predict accurate masks for novel categories. To this end, we employ the denoising UNet of the renowned text-to-image diffusion model (37 to extract features from the well-established text-2D embedding space, followed by a mask generator to convert features into segmentation masks.

---

We favor the generative diffusion model over the commonly adopted CLIP [6] model for two primary reasons. Indicated by Compp-to-Prompy (17), the cross-attention maps from the intermediate layers of the diffusion model exhibit high correlation with text concepts. Consequently, well- trained diffusion model constructs superior vision-language feature space, which can be effectively leveraged for open vocabulary perception. Moreover, the diffusion model provides precise text control over dense pixel generation, demonstrating a higher potential for generating fine-grained segmentation masks compared to the CLIP model, which relies on coarse global feature alignment.
Open Vocabulary Mask Generation with diffusion. The inference process of a rext-to-image -ingg diffusion model [37 begins with a random Gaussian noise and a conditional text embedding, generating nigh-quality images through iterative denoising steps. ODISE [45] was the first to propose the use of the denoising UNet of pre-traate vee-trained diffusion model for open vocabulary 2D segmentation. Given an input image x, a noisy image Ze first sampled at time step f:
Preliminary:
Ft Vā,z vI dec, (~,,
where a₂ IT= org, and at..... QT pre-defined noise schedule. Then the diffusion model's visual representation f can be computed via the denoising step:
UNet(E+, MLP o V(r)
where the denoising UNet is the building block of the diffusion model, MLP stands for multi-layer perceptron, and V is a frozen CLIP image encoder to encode æ into the vision vision-language embedding space. ((x) the implicit caption embedding which serves as the text condition input to the diffusion uses the feature s'
model. Subsequently, the mask generator, implemented with Mask2Former 4 R to produce M class-agnostic binary masks M2a and their corresponding mask embedding features Fiask. Since f is highly representative of the vision-language feature space, the model is inherently capable of generating open vocabulary segmentation masks and embeddings. The experimental results of ODISE strongly confirm this hypothesis.
Geometry-aware Mask Generation. In XMask3D, the 2D mask generation branch is implemented by variant of the ODISE model. We propose an Implicit 3D Captioner that takes the global 3D feature faa E Ix from the 3D encoder as input, and predicts the implicit condition embedding to be injected into the diffusion model. Then Equation
3 can be replaced by:
f UNet(z;, MLP faa), fsa E(P)
where & represents the encoder of the point cloud segmentation model, and Cg denotes the feature dimension of the global point cloud feature. The rationale behind this design is twofold. First, since MLP o faa serves as the text condition for the pre-trained denoising UNet with frozen weights, the training objective of the 2D mask generation branch implicitly pushes faa closer to the text-2D feature space. If fad does not align with this space, the pre-trained denoising UNet will not recognize the condition, resulting in high loss of the 2D branch. Through gradient descent, the point cloud encoder gradually distills some vision- lision-language knowledge from the pre- trained and frozen denoising UNet. Second. faa encapsulates rich 3D geometric information that the 2D branch lacks due to occlusion and dimensional compression issues inherent in images. Using fsa as the condition for the 2D branch encourages the model to produce geometry-aware mask outlines and embeddings, facilitating the back- back-projectts projectioo of 2D masks into 3D space. The effectiveness of the proposed geometry-awan mask generation will be validated through quantitative ablation comparisons in Section 4.3
3.3 2D-to-3D Mask Regularization
Although some vision-language knowledge from the diffusion model is distilled to the point cloud encoder E via the proposed Implicit 3D Captioner, the 3D feature faa still deviates from the 2D-text embedding space. This is because there is no constraint on the point cloud decoder D, and the encoder distillation via gradient descent is inherently weak. Consequently, is crucial to introduce contrastive regularization in the training pipeline to explicitly align 3D features with the shared 2D-text embedding space. Existing contrastive learning methods [34 I1 between 3D features and 2D-text features typically explore global, patch-wise, or wise relations. However, global contrastive learning is too coarse, and point-wise feature alignment is prone to noise. While patch- wise contrastive learning is more fine-grained and robust, it still lacks semantic clarity. A patch in 2D image may correspond to multiple irrelevant and discontinuous regions in 3D point cloud due to depth compression, resulting in ambiguous and less representative local features in 3D.
5

---

To this end, we propose an explicit 2D-to-3D mask regularization term for fine- tine-grained and consistent feature space aligmment between the 3D and 2D-text modalities. Specifically, we extract 3D mask embeddings G3d E RB from 3D features F34 using the back-projected 3D binary mask Maa:
G avgpool(Fad)
IsiS M
(5)
where G'dd RIICC represents the 7th mask embedding, and avgpool signifies the average pooling operation. Fā is sampled from F' at indices where M = 1, with Maa being the 7th binary mask.
Given the 2D binary mask M2d and the input image I, we can also derive ground truth mask CLIP feature GCLIP via pre-trained CLIP model [36]. For detailed information on obtaining the ground truth GCLIP, please refer to MaskCLIP [12 or Section A.1 Subsequently, the 2D-to-3D
regularization term can computed using a classical cosine contrastive loss:
Gidd (GCLIP)" ||Gaall|G
maskk
Σ
1
(6)
M
As each mask region ideally corresponds to a distinct category, the pooled mask embedding achieves semantic consistency and representativennss. Consequently, contrastive learning at the mask level offers finer granularity than global contrast, greater robustness than point-wise contrast, and clearer distinction than patch-wise contrast. Through our proposed 2D-to-3D mask regularization, the 3D features are explicitly aligned with the 2D-text feature space, enhancing the performance the 3D branch in open vocabulary segmentation. This progress is further substantiated by the ablation studies outlined in Section 4.3
3.4 3D-2D Mask Feature Fusion
The 3D-2D mask feature fusion block is devised to merge 3D features F34 E NR with the pseudo mask feature F24 E R N" xC derived from the 2D branch. It is noteworthy that each element in Fs possesses unique and distinguishing embeddings, whereas elements in F2a pertaining to the same mask share identical mask embeddings. Consequently, F3a offers detailed geometric structural infor- mation, while F2a provides semantic features with robust open vocabulary capabilities. Our approach combines features from these two modalities to leverage their complementary insights, resulting in Fruse which excels in both precise geometry delineation and expansive semantic extrapolation. Concretely, given that N' < N, we selectively merge F2a and F3a solely on the N' points where correspondences exist:
MLP cat F2a) have correspondence no correspondence
F Fusee
Faa
where cat represents concatenation. Ablation studies in Section 4.3 and visualization results in Section 4.2 will demonstrate that Fruse effectively integrates the strengths of both Fsa and F2a-
3.5 Training Objectives
In XMask3D, our training strategy encompasses supervised segmentation loss (Ceeg) computed from 3D (C ). 2D (C2d)) ), and fusion (Cfuge) modalities. We employ Cross Entropy loss for 3D and fusion segments, and for 2D, we adopt multi-head losses including Cross Entropy, Dice, and Focal
Loss, following ODISE (55 and Mask2Former 4 3] guidelines. Additionally, we follow PLA (II] to introduce binary head and view-level contrastive loss. The binary head is optimized with Binary Cross Entropy loss (Lb)) to differentiate between base and novel categories. The view-level contrastive loss (Cview) is calculated between the view global feature and text embedding of the view image caption, weighted by respective coefficients the binary head and the view-level regularization can be found in the PLAA paper or in Section A.3 and A.2 In conclusion, the overall training objective can be adjusted by:
2d (w3d (uvvee: vieeww wieew). Detailed information about
= Waegfoeg Wmask mmack Liview Coi
(8)
3d 2d w Juse where Wseg Wmask, vieww w view view Wbi, loss weight hyperparameters.

---

Table Results for open-vocabulary 3D semantic segmentation on ScanNet dataset. We evaluate the performance with hloU, base and novel mloU on five benchmarks with different category splits.
Scannet
ScanNet200
Method
B15/N4
B12/N7
B10/99
7170/330
B150/N50
hloU Base
Novel
hloU
Base
Novel
hloU
Base
Novel
hloU
Base
Novel
hloU
Base Novel
LSeg-3D 23
0.0
64.4
0.0
0.9
55.7
0.1
1.8
68.4
0.9
1.5
21.1
0.8
3.0
20,6
1.6
3DGenZ. 31
20.6
56.0
12.6
19.8
35.5
13.3
12.0
63.6
6.6
2.6
15.8
1.4
3.3
14.1
1.9
3DT/SS. [
10.5
36.7
6.1
3.8
36.6
2.0
7.8
55.5
4.2
0.9
4.0
0.5
0.7
3.8
PLA [T]
65.3
68.3
62.4
55.3
69.5
45.9
53.1
76.2
40.8
11.4
20.9
7.8
10.1
20.9
6.6
OpenScene 33
65.7
68.8
62.8
56.8
61.5
51.7
54.3
71.8
43.6
14.2
22.5
10.4
15.2
23.5
11.2
OV3D 20
72.4
70.2
74.7
68.5
74.1
63.7
64.8
77.6
55.6
XMask3D
70.0
69.8
70.2
61.7
70.2
55.1
55.7
76.5
43.8
18.0
27.8
13.3
15.5
24.4
11.4
4 Experiments
4.1 Experiment Settings
Datasets. In accordance with prior literature, our research conducts experimentation on two prominent indoor scene datasets: ScanNet 99] and S3DIS []. ScanNet, foundational dataset this domain, comprises 1201 scenes allocated for training and 312 scenes designated for validation. Each scene within ScanNet furnishes point cloud data, multi-view images, and corresponding camera pose matrices. Similarly, S3DIS offers analogous data modalities, encompassing 271 rooms across six distinct indoor environments. Conforming to established conventions, we reserve Area of S3DIS for validation purposes, ensuring consistency with prior methodologies.
Category Partition. In alignment with previous research, we exclude the otherfurniture class and partition the remaining classes into three benchmarks: B B15/N4, B12/N7, and B10/N9. Here, B15 signifies the 15 fundamental categories that remain visible and supervised during the training process, while N4 denotes the presence of 4 novel categories introduced during evaluation. For the ScanNet variant featuring 200 classes 38 we adopt similar approach, dividing the dataset into B 70/N/30 and B B150/N50 benchmarks, each representing distinct configuration of base and novel categories. Similarly, in the case of S3DIS, comprising 13 classes, we disregard the clutter class and organize the dataset into B8/N4 and B6/N6 benchmarks. Detailed information can be found in Section
Metries. Following PLA we present the mean Intersection over Union (mloU) scores separately for both base and novel categories to assess open vocabulary segmentation performance. Additionally, to provide a comprehensive evaluation of the segmentation capability, we report the harmonic mean ToU (hloU) derived from the mIoU scores of base and novel categories. This holistic metric offers insights into the overall segmentation efficacy across the dataset.
Implementation Details. Our implementation incorporates MinkUNet [7] as the 3D branch and ODISE [45] as the 2D branch within the architecture. For the languaea model, we opt for CLIP- L [36]. The training regimen for the XMask3D model involves utilizing the Adam optimizer [29] with a Cosine learning rate scheduler. We train the model for 150 epochs on 4 NVIDIA A800 GPUs, employing a batch size of 64. Notably, we introduce mask-level regularization to the training pipeline after the initial 50 epochs. This decision motivated by the observation that the quality of mask prediction at the onset of training may be suboptimal, making the mask-level contrastive loss ineffective and potentially misleading. Detailed information regarding hyperparameter selections is provided in Section A.4
4.2 Main Results
Quantitative Comparisons. From Table Table our proposed XMask3D outperforms previous methods across most benchmarks, irrespective of the novel category proportion or dataset sources. The performance indicates that XMask3D is a robust and generalizable method for open vocabulary 3D semantic segmentation. Notably, we compare XMask3D with our baseline method, PLA, on novel category performance. On the ScanNet dataset, XMask3D demonstrates improvements ranging from 7.4% to 20.0% over PLA. On the ScanNer200 dataset, XMask3D surpasses PLA by an impressive
7

---

Table 2: Open-vocabulary 3D semantic seg- mentation results on S3DIS dataset. We report hloU, base mloU and novel mloU metrics. Best
Table 3: Ablations for XMask3D pipeline design.
We conduct experiments on the B B12/N7 benchmark.
(a) Ablation for implicit condition of the diffusion model.
open-vocabulary results are highlighted in bold.
Cond Base Novel bed chair table BKS pic sink
BT
S3DIS
Text 69.5 52.7
72.9 60.6 36.7 70.0 14.3 44,6 70.3
Method
B8/N4
B6/N6
2D 69.7
53.6
70.6 63.3
40.9
12,4 51.6 67.8 68.4
3D
70.2
55.1 72.5 62.7 37.3 70.6 18.6 51.2 73.0
hloU Base Novel
hloU
Base
Novel
LSeg-3D 26
0.1
49.0
0.1
0.0
30.1
0.0
(b) Ablation for mask regularization and fusion block.
3DTZSL [S]
8.4
43.1
4.7
3.5
28.2
1.9
Base
Novel
3DGenZ 31
8.8
50.3
4.8
9.4
20.3
6.1
Limak
PLA [1]
34.6
59.0
24.5
38.5
55.5
29.4
2D 3D Fuse 2D
3D (Asp) Fuse (Arws)
OpenScene [34]
42.4
58.6
33.2
44.2
56.2
36.4
40.1 63.9 70.0 30.9 14.0
53,5
XMask3D
46,8
63.1
37.2
44.9
52.8
39.1
40.6 64.3 70.2 30.8
25.7 (+11.7)
55.1 (+1.6)
70.5% and 72.7%. On the S3DIS dataset, XMask3D shows boosts of 51.8% and 33.0% over PLA. Among these benchmarks, XMask3D achieves the highest improvements on the long-tail ScanNer200 dataset, primarily due to the introduction of the denoising UNet from the pre-trained diffusion model, which constructs comprehensive text-2D embedding space with unlimited text descriptions.
It is noteworthy that the OpenScene results are derived from the implementation of UniM-OV3D (T6). but we do not compare XMask3D with UniM-OV3D in the tables. This is because UniM-OV3D employs PointBIND [14 and CLIP2Point [19 with an already aligned 3D-text embedding space, whereas 3Mask3D only integrates the commonly used 2D-text space and introduces techniques to pull 3D features from any point cloud models towards the shared embedding space. Another outstanding concurrent method is OV3D 20 which primarily focuses on Entity'Text extraction and Point- EntityTexx association, while XMask3D concentrates on enhancing mask-level interaction between 2D and 3D modalities. Therefore, the contributions of OV3D and XMask3D are orthogonal and could complement each other future explorations. Additionally, OV3D only provides experiment results on the ScanNet20 benchmarks, while its effectiveness on the ScanNer200 dataset that is more challenging and on the S3DIS dataset with different data distribution remains unexplored.
Visualization Results. In Figure 3 we present a comparative visualization of novel categories between XMask3D and previous methods [ 34 XMask3D demonstrates superior accuracy in category predictions, produces finer segmentation boundaries, and generates more cohesive mask regions. Notably, the missegmented region on the bookshelf in the second row of XMask3D is classified as picture, which appears reasonable given the corresponding part in the view image.
4.3 Ablation Studies
In this section, we comprehensively discuss the design choices of XMask3D through extensive ablation studies on the ScanNet B 12/N77 benchmark. The results are presented in Table
Mask Generation Condition. In Section 3.2] we introduce an Implicit 3D Captioner to convert global 3D features into implicit condition embeddings for the diffusion model. In Table 33 we compare this implicit 3D condition with the vanilla text condition and implicit 2D condition. The text condition is generated by ViT-GPT2 [48] captioning model and encoded via the frozen CLIP text encoder. The 2D condition is generated by the frozen CLIP image encoder and a learnable MLP, following the design in ODISE (45] (Fquation B}. Our proposed implicit 3D condition outperforms the others in novel category segmentation, demonstrating that integrating the 3D global feature with the diffusion model produces the most compatible open vocabulary masks with the 3D branch.
we propose a fine- fine-grained mask regularization term align 3D features with the 2D-text embedding space. In the first line of Table 3b) we remove the fine- tne-grained mask-level loss Lussk from the training pipeline. Besides the final results from the fusion block, we also report the intermediate results from the 2D and 3D branches. The inclusion of the mask loss results in a significant improvement of 11.7 in 3D performance on novel categories, demonstrating that our fine fine-grained mask-level regularization effectively brings 3D features closer to the 2Dxtextt embedding space. Additionally, the performance gain on the fused output 1.5, highlighting the positive impact of this regularization from 3D to fusion features.
Mask Regularization. In Section 3.3
8

---

Table
Chair
Sofa
Bookshelf
Cabinet
Picture
Chair
Bed
Table
Bed
Table
Image
Ground Truth
OpenScene
XMask3D (Ours)
Figure 3: Visualization Comparisons between XMask3D and Previous Methods. We compare XMask3D with PLA [I]] and OpenScene [4 on the novel categories table, bookshelf, chair and bed. The regions corresponding the novel categories are highlighted in red boxes.
also analyze effects of mask regularization through visualizations Figure Comparing the first and second rows, the 3D segmentation results without the mask loss show inconsistency the local region for novel category chair, with most points misclassified as bookshelf. When the mask-level regularization is added, more points correctly classified, resulting in clearer segmentation mask. The visualizations of the fused outputs also demonstrate consistent enhancement in mask regularization within the final column. This is evident from the rectification of misclassified points previously labeled as table on the armrest of the right chair in the last column.
Modality Fusion. When comparing the 2D, 3D, and fused metrics within the same line in Table 3b we empirically find that the 2D branch performs relatively better on novel category segmentation, while the 3D branch excels at base category segmentation. This quantitative observation aligns with our design intention: to exploit geometric knowledge via the 3D branch and to enhance the model's open vocabulary capability via the 2D branch. More importantly, the fused output outperforms both the 2D and 3D intermediate results on both base and novel splits. These results strongly support the effectiveness of our fusion design in merging the complementary knowledge from both modalities.
We also present visualization evidence regarding modal complementarity and fusion effects in Figure In the first group, the 2D branch exhibits unsatisfactory results of the base category wall around the whiteboard and behind the right chair, whereas effectively segmenting the novel category chair with high quality despite minor artifacts. Conversely, the 3D branch produces unsatisfactory mask for the chair but excels segmenting the wall based on geometric information. The fusion block leverages the strengths of both branches, mitigates their weaknesses, and yields satisfactory outcomes. Moreover, the fusion block accurately delineates regions such as the left foot of the chair on the right, where both the 2D and 3D branches falter, highlighting its potential for integrating cross-modality knowledge. The second group displays similar results that convince the effectiveness of the proposed mask loss and cross-modalle mross-modality fusion design.
4.4 Limitations
Due to resource constraints, we only evaluate the performance of XMask3D semantic segmentation in this study. However, the XMask3D pipeline has the potential be extended the instance and panoptic perception by replacing the 3D backbone with instance panoptic segmentation model.

---

Novel
Chair
without Mask Loss
Table
Bookshelf
Picture
with
Base
Mask
Wall
Loss
Floor
Desk
Ground Truth
2D Branch Output
3D Branch Output
Fusion Block Output
Novel
Chair
without
Mask Loss
Table
Bookshelf
Base
Wall
with
Mask
Floor
Loss
Desk
Curtain
Ground Truth
2D Branch Output
3D Branch Output
Fusion Block Output
Figure 4: Visualization Results of Ablations. The first and second groups show results from ScanNet B15/N4 and B B12/N7 benchmark, respectively. each group, the first and second rows display segmentation results without and with the proposed mask regularization. The last three columns compare the outputs from the intermediate 2D and 3D branches with the final fusion block.
This adaptability arises from the auxiliary 2D branch. which is mask-based and capable of generating instance-level masks given appropriate annotations. Furthermore, since the XMask3D pipeline is agnostic the 3D model used, it would be interesting conduct experiments with more advanced point cloud models to compare performance. We hope we can exploit the strength of XMask31) with high-standard 3D backbones and wider range of dense perception tasks in future research. Computational cost is another limitation of XMask3D, as we implement the Denoising UNet which has numerous parameters and relatively slow latency. In our future work, we plan address this limitation by replacing the 2D branch with a more lightweight 2D open vocabulary mask generator.
Conclusion
In this paper, we present XMask3D, designed for open vocabulary 3D semantic segmentation. We propose the integration of the denoising UNet of a pre-trained diffusion model to produce geometry- aware segmentation masks conditioned on learnable implicit 3D embeddings. These binary 2D masks filter mask-level embeddings of 3D representations and apply mask regularization to enhance the open vocabulary capacity 3D features. By fusing the 2D mask embeddings with fine-grained 31) features, leverage the complementary knowledge from both modalities, achieving competitive performance across various benchmarks and datasets. Ablation studies and visualization comparisons further validate the effectiveness of the proposed cross-modal mask reasoning method.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China under Grant 623B2063, Grant 62321005, Grant 62336004, and Grant 62125603.
10

---

References
[1] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In CVPR, 2016.
[2] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[3] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
[4] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurIPS, 2021.
[5] Ali Cheraghian, Shafin Rahman, Dylan Campbell, and Lars Petersson. Transductive zero-shot learning for 3d point cloud classification. In WACV, 2020.
[6] Seokju Cho, Heeseong Shin, Sunghwan Hong. Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. arXiv preprint arXiv:2303. //97, 2023.
[7] Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets:
Minkowski convolutional neural networks. In CVPR, 2019.
[8] Spconv Contributors. Spconv: Spatially sparse convolution library. https:///ithu/ com/ ttaaeller59
2022.
[9] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
[10] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmen- tation. In CVPR, pages 11883-15992, 2022.
[1I] Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open-vocabulary 3d scene understanding. In CVPR, 2023.
[12] Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskclip. In ICML, 2023.
[13] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-vocabulary image segmen- tation with image-level labels. In ECCV, 2022.
[14] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with modaliy for 3d understanding, generation, and instruction following. arXiv preprins arXiv:2309.0061 2023.
[15] Huy Ha and Shuran Song. Semantic abstraction: Open Open-world 3d scene understanding from 2d vision-language models. arXiv preprint arXiv:2207.11514 2022.
[16] Qingdong He, Jinlong Peng, Zhengkai Jiang, Kai Wu, Xiaozhong Ji, Jiangning Zhang, Yabiao Wang, Chengjie Wang, Mingang Chen, and Yunsheng Wu. Unim-ov3d: Uni-modality open- vocabulary 3d scene understanding with fine-grained feature representation. arXiv preprint arXiv:2401.11395. 2024.
[17] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.0166 2022.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33, 2020.
[19] Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In ICCV, 2023.

---

[20] Li Jiang, Shaoshuai Shi, and Bernt Schiele. Open-vocabulary 3d semantic segmentation with foundation models. In CVPR. pages 21284-21294, 2024.
[21] Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot vecen-ocayulary segmentation. arXiv preprint (rXiv:2306.09316, 2023.
[22] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything, In ICCV, 2023.
[23] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language- driven semantic segmentation. arXiv preprint arXiv:2201.03544 2022.
[24] Jingyao Li, Pengguang Chen, Shengju Qian, and Jiaya Jia. Tagelip: Improving discrimination ability of open-vocabulary semantic segmentation. arXiv preprint arXiv:2304.07544 2023.
[25] Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Guiding -ixttto-mmae diffusion model towards grounded generation. In ICCV, 2023.
[26] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
[27] Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. 3d open-vocabulary segmentation with foundation models. arXiv preprint arXiv:2305. 14093, 2023.
[28] Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. NeurlPS, 36, 2023.
[29] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05500,,22 2017.
[30] Chaofan Ma, Yuhuan Yang, Yanfeng Wang, Ya Zhang, and Weidi Xie. Open-vocabulary semantic segmentation with frozen vision-language models. arXiv preprint 100 2022.
[31] Bjöm Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet. Generative zero-shot learning for semantic segmentation of 3d point clouds. In 3DV, 2021.
[32] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021.
[33] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[34] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023.
[35] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2022.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[38] David Rozenberszki, Or Litany, and Angela Dai. Language--rounded indoor 3d semantic segmentation in the wild. In ECCV, 2022.
12

---

[39] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 35, 2022.
[40] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2023.
[41] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In ICCV, 2023.
[42] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster, stronger. arXiv preprint arXiv:2312.1000 10035, 2023.
[43] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-basee pooling. NeurIPS, 2022.
[44] Zihao Xiao, Longlong Jing, Shangxuan Wu, Alex Zihao Zhu, Jingwei Ji, Chiyu Max Jiang, Wei- Chih Hung, Thomas Funkhouser, Weicheng Kuo, Anelia Angelova, al. 3d open-vocabulary panoptic segmentation with 2d-3d vision- ssion-language distillation. arXiv preprint arXiv:2401.02400 024042, 2024.
[45] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023.
[46] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. CVPR, 2023.
[47] Jihan Yang, Runyu Ding, Weipeng Deng, Zhe Wang, and Xiaojuan Qi. Regionple: Regional point-language contrastive learning for open-world 3d scene understanding. arXiv preprint arXiv:2304.0096. 2023.
[48] ydshieh. Vit-gpt2 image captioning.
http:://uugiingfaa.ccoo/olpaaaee
vit-gpt2- v.t-got2-Immge-cpptiiee -apeemgatitciinnnggggggggngggggg
[49] Junbo Zhang, Rumpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. In ICCV, 2023.
[50] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to- text-to-image diffusion models. In ICCV, 2023.
[51] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021.
13

---

A Additional Implementation Details
A.1 Mask-level Regularization
Mask-level regularization facilitates local and finely-grainet alignment between features extracted from the 3D branch of XMask3D and the 2D-text embedding space. The ground truth for mask- loss is computed using MaskCLIP [2) utilizing the predicted segmentation mask M2a E RM and the view image E R3xHxW
GeLi Maskkoollng(((()).Ma
where M represents the number of masks, and H and W denote the height and width of the image, respectively. refers to the pre-trained CLIP 36 image encoder.
Specifically, the view image is encoded into image tokens Ti E RNNC with the pre-trained CLIP image model, where N is the number of image tokens and C the dimension of the CLIP embeddings. The class token Te E R RxxC is duplicated M times as the mask class tokens T& E R RMC. Then Tú, Tc. TM are concatenated together to perform masked attention with frozen weights from the pre-trained CLIP image model. The attention mask is designed as
((N+))x(N+))
T(N+I)M TMXM
Mi
(10)
PMXN
Fxx1
where Taxn is an m X n True matrix, FF Fxxn is an m x n False matrix and is defined as:
False
if mask, contains at least one pixel in patch,
Pay = True otherwise.
(11)
where True means that this position is masked out i.e. not allowed to attend and False otherwise. Then the updated mask class tokens Ty from the masked attention layers can be regarded as the CLIP embedding of the masked regions, serving as the ground truth GALP for mask-level regularization.
A.2 View-level Regularization
The view-level regularization facilitates coarse and high-level alignment between features extracted from the 3D branch and the 2D-text embedding space. To elaborate, upon receiving an image I, we initially generate its text caption using mrettrained captioning model, ViT-GPT2 [48). Subsequently, we employ the pre-trained CLIP [36] text encoder to encode the text caption into the 2D-text embedding space, yielding fTew which serves as the ground truth for view-level regularization.
We perform average pooling operation on dense 3D point cloud features, 2D image features and fused Then we implement contrastive loss
features to obtain their global embeddings p3d 22d view- between global features and the ground truth text embeddings:
vvew ( yT
cm view
(12)
I View
where m = 33d, 2d, fuse denotes different modalities.
A.3
Binary Head
Following PLA [1], we include a binary head to predict whether the points belong to base or novel categories. We implement small 3D model the binary head with minimum computation cost. The prediction " is utilized to modulate the over-confident semantic score s:
8 - -²) + SN
(13)
where 8n the semantic score computed solely on base classes with novel class scores set zero. Similarly, sy is computed only for novel classes, setting base class scores to zero.
14

---

Table
Category Partitions. We follow PLA to split ScanNet and S3DIS into several benchmarks.
(a) ScanNet dataset.
Partition
Base Calegories
Novel Categories
wall, floor, cabinet, bed, chair, table, door, window, picture. counter, curtain, refrigerator, showercurtain, sink, bathtub
B15/44
sofa, bookshelf, desk, toilet
wall, floor, cahinet, sofa, door, window, counter, desk, curtain, refrigerator, showercurtain, toilet
B12N77
bed, chair, table, bookshelf, picture, sink, bathtub
wall. floor. cabinet, bed, chair, sofa, table, door, window, showercurtain, curtain
bookshelf, picture, counter, desk, refrigerator. toilct. sink, bathtub
B10N9
(b) S3DIS dataset.
Partition
Base Categories
Novel Categories
B&/N4
ceiling, floor, wall, beam, column, door, chair, board ceiling, wall, beam, column, chair, bookcase
window, table, sofa, bookcase
B6/N6
floor, window, door, table, sofa, board
Table 5: Per-Class Results Comparison with PLA. We compare the per-class open vocabulary segmentation results with PLA. Novel Classes are marked in blue
(a) ScanNet dataset. Shower c, is short for shower curtain.
Partirion
Method
Base
Novel
wall
floor
cabioct
bed
chair
sofa
tbbil
door
window
booksthel
picture
Fridee
shower C.
100
sink
bachmb
B15/N4
XMask3D
PLA
68.3
62.4
84.6
95.0
64.9
81.1
87.9
75.9
72.2
61.9
62.1
69.5
30.9
60.1
46.5
70.7
50.5
66.1
56.8
59.0
81.7
B12N7
PLA
69.5
45.9
84.7
95.1
65.3
57.8
44.2
75.9
34.5
62,5
62.3
62.1
20.5
57.8
61.4
72.4
47.9
64.9
85.9
28.4 69.6
XMask3D
70.2
55.1
94.6
68.6
72.5
76.0
37.3
62.6
60.0
83.9
73.0
B10/N9
PLA
76.2
40.8
83.8
95.2
64.3
80.9
88.0
78.5
73.2
60.6
61.5
68.6
17.7
23,4
51.3
70.6
25.7
38.2
51.3
27.3
61.7
XMask3D
76.5
43.8
83.8
94.7
67.3
82.6
89.1
78.8
72.9
61.2
62.7
75.2
17.7
45.9
54.5
71.9
28.2
22.9
59.6
42.3
47.6
(b) S3DIS dataset.
Partition
Method
Novel
celling
Bloo
wall
column
window
door
rable
chuic
sofaa
bookcase
board
BR/N4
XMask3D
PLA
59.0
63.1
37.2
24.5
86.4
93.9
88.3
97.8
81.4
82.9
0.0
31.4
17.2
15.6
31.1
53.7
61.4
35.8
35.8
86.3
75.4
05.3
17.6
37.3
64.5
80.7
43.3
B6NN6
XMas33D
PLA
52.8
39.1
29.4
93.7
86.4
79.1
47.4
80.1
80.9
0.2
23.7
28.5
24.1
33.7
08.4
30.2
37.6
14.7
74.2
87.0
54.0
51.6
24.0
47.4
60.8
A.4 Training and Inference Settings
Training. The supervised segmentation loss Entropy Loss on N points:
seg is calculated via the per-point classification Cross
N
m
CrossEntropy (p. 9:)
(14)
N
p Softmax(F, /T)
(15)
where m = 3d, 2D, fuse denotes different modalities, y: the ground truth for base categories, F... is the normalized feature, and 7 is learnable temperature parameter.
combine the predicted logits p with the prediction from text-image discriminative model to enhance the open vocabulary classification capacity of the model. Specifically, we leverage the mask-leved regularization ground truth feature GeLiP from Section A.] to modulate the segmentation logits:
Inference. We follow ODISE 45
Prinal (1-A)
(16)
Paux Softmax (G F(Tx//T)
(17)
where GeLIP is the normalized feature of GeLIP, À [0, 1] is the fixed balancing factor.
Hyper-parameters. For all benchmarks, we set the same warg 4, 3d 1, 2 4, w view
fuse
1.5 as the hlper-parameed choices. The W mask and Wbi set differently across benchmarks. We
15

---

View Image
Ground Truth
PLA
OpenScene
XMask3D
XMask3D
XMMsk30 (3D Branch)
(Fused)
(2D Branch)
Figure 5: Complete Visualization Comparisons. We show comprehensive comparison between
XMask3D fused/2D branch/3D branch outputs and previous methods (PLA [[]/OpenSeene [3]]. The figure corresponds to Figure 3 in the main paper.
Novel
Toilet
without Loss
Bookshelf
Base
Wall
with Mask
Floor
Loss
Bathtub
Ground Truth
2D Branch Output
Branch Output
Fusion Block Output
Figure Visualization Results of Ablations. The first and second rows display segmentation results without and with the proposed mask regularization. The last three columns compare the outputs from intermediate 2D and 3D branches with the final fusion block.
set Wmask = 0.5/0.5/112/2/1LLL was = 16/12/8/48/32/1 10 for ScanNet B15/N4, B12/N7, B10/N9, ScanNer200 B170/N30, B150/N50, S3DIS B8/N4, B6/N6 benchmarks, respectively.
Category Partitions. We follow PLA [ to partition the ScanNet 9 and S3DIS datasets into several different benchmarks. Here list the detailed partition principle in Table for reference. We do not list splits for the ScanNet200 dataset since there are too many categories.
B Additional Experimental Results
B.1 Per-class Results Comparison
We contrast per-class segmentation mean Intersection over Union (mloU) with PLA I1 across the ScanNet and S3DIS datasets in 5 XMask3D significantly outperforms PLA, particularly in novel categories, thereby showcasing the effectiveness of proposed cross-modal mask reasoning approach on open vocabulary 3D semantic segmentation.
B.2 Visualization Comparison
In Figure 5 we show 2D and 3D branch outputs from XMask3D in addition to Figure in the main paper. The branch misclassifies the novel categories, while the branch produces discontinuous segmentation masks. Neither of them outperforms PL.A or OpenScene. However, when their features are fused together, the output becomes superior, demonstrating the complementary knowledge across different modalities.
16

---

Shower
Curtain
Curtain
Bathtub
Picture
Wall
Sink
Cabinet
Input Image
Ground Truth
2D Branch Output
3D Branch Output
Fused Output
Figure 7: Failure cases XMask3D. We focus on shower curtain, picture, and sink novel categories in each line, respectively.
B.3 Illustrations of Ablations
We present supplementary illustrations of ablation studies pertaining to mask regularization and cross-modality fusion in Figure which corresponds to the sample in Figure] from the ScanNet B15/N4 benchmark.
Comparing the first and second rows of the third column reveals that the proposed mask regularization loss yields significant improvements in the segmentation of novel categories by the 3D branch. Further comparison the last three columns in the second row demonstrates that the output from the fusion block effectively leverages the strengths of both modality branches. Specifically, while the 2D branch excels at segmenting novel categories, it tends to produce discontinuous masks for base categories with larger regions (e.g., wall). Conversely, the 3D branch may perform relatively poorly in novel category segmentation but provides consistent geometric knowledge as complementary aspect. Consequently, the fused output achieves superior performance in open vocabulary segmentation.
C Failure Cases Analysis
In Figure we display some failure cases XMask3D. The first sample shows a bathtub with a shower curtain in the bathroom. However, the 2D/3D branch and the fused output of XMask3D all misclassify the shower curtain as curtain. This may be because these two categories are similar in object shape and texture, with the only difference being the surrounding environment. Since XMask3D only takes a corner the room as input instead the entire scene, the global environmental information insufficient for making the correct category prediction.
The second sample shows a large area of picture on the wall. The 2D, 3D branches and fused output of XMask3D all misclassified it wall, due to their similar geometry. In most cases, picture a small region on the wall, and this picture, as large as the wall, is typical corner case. This failure may reveal XMask3D's over-reliance geometric knowledge and lesser consideration of texture information when encountering citto--distribution samples.
The third sample shows a sink counter. Due to the occlusion problem, the sink point cloud is incomplete, negatively affecting the prediction of segmentation boundaries between the sink and the counter. This occurs because they are geometrically similar when the sinking-down part of the sink is missing.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction is that propose an XMask3D framework for open vocabulary 3D semantic segmentation, and the paper is all about introducing its design and effectiveness.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [NA]
Justification: The paper does not propose new theorem. We conduct experiments and show illustrations to evaluate our model design.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Please refer to Section 4.1 and Section A for experiment settings and imple- mentation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Please refer the supplemental zip file for our code and we will officially release the code upon the acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
Sublic/guides httiii//uuides//oeeSuttiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
cttpss.cccpubblci gwwdes/ddeeeee wtt.oodeeSubmssssoooooooooo.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Section 4.1 and Section A for experiment settings and imple- mentation details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We do not report error bars since previous literature do not report them either and the experiments scene-level dataset is resource consuming. However, provide extensive experiment results on multiple benchmarks of various datasets and show consistent performance gain over previous methods. Therefore, the statistical significance is convinced by the variety of datasets and benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 4.1 for experiments compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have carefully read the Ethics Guidelines and strictly followed them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the original paper of the pre-trained models and data we used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
22

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "UeCtgjNXVU", "reviewer": "Reviewer_4VWC", "summary": "This paper addresses the challenge of open-vocabulary 3D semantic segmentation by utilizing 3D geometric features, 2D semantic embeddings, and text modality. The proposed approach adapts the ODISE method to the 3D domain, aiming to distill open-vocabulary semantic segmentation knowledge from a pre-trained text-to-image denoising diffusion model to a 3D segmentation model. Initially, an input point cloud is fed into a 3D encoder-decoder segmentation network, producing point-wise geometric features. Simultaneously, a pre-trained visual-language diffusion model generates 2D masks and embeddings from posed images of the same scene, conditioned on the 3D global feature of the 3D branch’s encoder. Unlike the ODISE method, an implicit $3D$ captioner is introduced to produce geometry-aware 2D masks while also distilling information from the 2D branch network to the 3D encoder. To further regularize the 3D network, a distillation loss ($\\mathcal{L}_{mask}$) is applied to the 3D mask embeddings, derived from the per-point features and the 2D masks back-projected to the point cloud as 3D binary masks. By obtaining ground truth mask features from a pre-trained CLIP model, the 3D masked embeddings are aligned with the image-text joint embedding space, through a cosine similarity loss. This alignment leads to more coherent segmentation results and enhances the model's open-vocabulary capabilities. Finally, the per-point features are combined with the pseudo mask 2D features (formed by the back-projected 3D mask and 2D mask embeddings), resulting in a fused per-point representation that incorporates the geometric information from the 3D segmentation network and the semantic open-vocabulary capabilities of the 2D branch. The approach is evaluated on three semantic segmentation benchmarks (ScanNet, ScanNet200, and S3DIS) and demonstrates superior performance compared to competing methods.", "questions": "Following the weaknesses section, do the authors have any additional insights into why this phenomenon occurs with the fused embeddings? Could the 2D regularization terms ($\\mathcal{L}_{seg}^{2D}$, $\\mathcal{L}\\_{view}^{2D}$) be introducing too much bias towards the 2D visual modality, thereby causing geometric discontinuities in the output fused features?", "limitations": "Yes, the authors have discussed the method's limitations in detail in Section 4.4.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The XMask3D effectively aligns 3D geometric features with 2D and textual modailities through knowledge distillation from visual-text joint embedding spaces inherent in the pre-trained 2D denoising UNet and the CLIP model. As evident by the ablation, the implicit 3D captioner is a crucial step in the overall pipeline, and it outperforms vanilla text conditioning or the implicit 2D captioner of ODISE, in both base and novel semantic categories. Moreover, the 2D-to-3D mask regularization is also essential, since it significantly improves the accuracy of the proposed method esp. in novel categories. This justifies the need for this additional distillation step from the CLIP joint space, to further enhance the open-vocabulary capabilities of the XMask3D method. Finally, the discussion on modality fusion, both in the main paper and supplementary, is highly appreciated. By dissecting the method and providing qualitative and quantitative results for each step, the authors make it easier for readers to understand and gain intuition about the presented approach.", "weaknesses": "While the method exhibits superior performance w.r.t. competing methods, it seems that the output fused embeddings yields to geometric inconsistent features for semantic classes that cover large areas of the point cloud such as wall, ceiling and floor. This is evident in both partitioning settings when the class is either base or novel (Table 5 (a) and (b) in supp.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and constructive comments! Hopefully the following response will address your concerns.\n\n### **1. About the less satisfactory results of classes that cover large areas.**\n> While the method exhibits superior performance w.r.t. competing methods, it seems that the output fused embeddings yields to geometric inconsistent features for semantic classes that cover large areas of the point cloud such as wall, ceiling and floor. This is evident in both partitioning settings when the class is either base or novel (Table 5 (a) and (b) in supp.).\n\nThe underlying reasons for the unsatisfactory performances of semantic classes on the ScanNet and S3DIS datasets are distinct. For the ScanNet dataset, XMask3D processes only a corner of the scene in each forward pass, whereas PLA directly feeds the entire scene into the model. This segmentation approach negatively impacts categories that cover large areas, such as `wall`, `ceiling`, and `floor`, because cutting these categories into smaller pieces inevitably undermines segmentation performance by losing long-range relational information.\n\nFor the S3DIS dataset, in addition to the aforementioned partial input issue, we encounter a trade-off between computational resource consumption and segmentation performance. The validation samples in S3DIS are densely populated, with an image view potentially corresponding to over 500,000 points, which causes out-of-memory errors on a 24GB NVIDIA 4090 device and slow inference speed. Consequently, we discard views exceeding a threshold of 260,000 points for higher efficiency. However, this view selection process results in hollow regions in the merged outputs, primarily in the corners of rooms, and these regions typically correspond to the categories `wall`, `ceiling`, or `floor`. We conducted an ablation study, increasing the selection threshold from 260,000 to 500,000 points, as shown in the following table. When the threshold is raised, the segmentation performance for `ceiling`, `floor`, and `wall` consistently improves.\n\n| Partition | Threshold | Ceiling | Floor | Wall |\n| :-------: | :-------: | :-----: | :---: | :--: |\n| B8/N4     | 260,000   | 86.4    | 88.3  | 81.4 |\n|           | 500,000   | 90.1    | 91.1  | 81.6 |\n| B6/N6     | 260,000   | 86.4    | 47.4  | 80.9 |\n|           | 500,000   | 88.7    | 51.1  | 81.6 |\n\n> Following the weaknesses section, do the authors have any additional insights into why this phenomenon occurs with the fused embeddings? Could the 2D regularization terms ($L_{seg}^{2D}$, $L_{view}^{2D}$) be introducing too much bias towards the 2D visual modality, thereby causing geometric discontinuities in the output fused features?\n\nAs discussed above, the primary reasons are the partial point cloud input and the trade-off with memory consumption. We believe that the 2D regularization term does not significantly bias the fused features towards geometric discontinuities. As illustrated in Figures 1, 4, and 6 of our main paper, the fused features of `wall` and `floor` fully leverage the continuous geometry from the 3D branch. Additionally, we provide a qualitative analysis in the following table. The 2D branch performance is significantly worse than the others. The fused features perform better than the 3D features for `wall` and only slightly worse for `floor`. The numerical results also demonstrate that the performance of fused features in segmenting `wall` and `floor` is not significantly affected by geometric discontinuities from the 2D regularization terms.\n\n| Partition | Method  | Branch | Wall | Floor |\n| :-------: | :-----: | :----: | :--: | :---: |\n| B10/N9    | PLA     | 3D     | 84.6 | 95.0  |\n|           | XMask3D | Fused  | 84.2 | 94.7  |\n|           | XMask3D | 3D     | 82.0 | 95.1  |\n|           | XMask3D | 2D     | 62.9 | 78.6  |\n| B12/N7    | PLA     | 3D     | 84.7 | 95.1  |\n|           | XMask3D | Fused  | 83.3 | 94.6  |\n|           | XMask3D | 3D     | 81.8 | 95.0  |\n|           | XMask3D | 2D     | 60.3 | 76.7  |\n| B10/N9    | PLA     | 3D     | 83.8 | 95.2  |\n|           | XMask3D | Fused  | 83.8 | 94.7  |\n|           | XMask3D | 3D     | 81.7 | 95.1  |\n|           | XMask3D | 2D     | 64.3 | 78.5  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I6EPZ3IczQ", "reviewer": "Reviewer_Xo8p", "summary": "The paper addresses the limitations of current open vocabulary 3D semantic segmentation methods, which primarily focus on creating a unified feature space for 3D, 2D, and textual modalities but struggle with fine-grained segmentation boundaries. To overcome these limitations, the authors propose XMask3D, a cross-modal mask reasoning framework that achieves more precise mask-level alignment between 3D features and the 2D-text embedding space.", "questions": "1. The paper could benefit from a more detailed error analysis to understand the failure modes of XMask3D, especially in novel category segmentation.", "limitations": "The authors have addressed limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The part \"incorporating a 2D mask generator to create geometry-aware open masks and apply fine-grained mask-level regularization on 3D features\" seems reasonable and novel.\n2. The paper is well-structured and easy to follow.\n3. Analysis is thorough and insightful.", "weaknesses": "1. The paper evaluates the proposed method on a limited set of benchmarks (ScanNet20, ScanNet200, S3DIS), all of which are indoor scene datasets. Authors could discuss how the method might perform on outdoor datasets. Additionally, the authors could provide a qualitative analysis of the model's potential limitations when applied to different environments.\n2. The reliance on the denoising UNet from a pre-trained diffusion model could be seen as a potential weakness or limitation, especially given the computational resources required for training and inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and constructive comments! Hopefully the following response will address your concerns.\n\n### **1. Application of XMask3D on other scenarios.**\n\n> The paper evaluates the proposed method on a limited set of benchmarks (ScanNet20, ScanNet200, S3DIS), all of which are indoor scene datasets. Authors could discuss how the method might perform on outdoor datasets. \n\nWe follow the experimental settings outlined in PLA for our experiments on indoor scene datasets. Based on your advice, we observed that other open vocabulary papers [1,2] also conduct experiments on the outdoor scene dataset nuScenes[3]. However, this dataset only provides point cloud data with annotations and raw image data without annotations. Without image-level segmentation ground truth, we cannot train the 2D mask generator. In our future work, we plan to replace the entire 2D branch with a pre-trained 2D open vocabulary segmentation model, enabling us to handle the nuScenes dataset without requiring 2D annotations.\n\nHowever, we have made efforts to provide some qualitative results on the nuScenes dataset. First, we project the 3D annotations onto 2D images using the camera's intrinsic and extrinsic parameters. Due to the sparsity of the 3D point cloud, we further employ the k-nearest-neighbor algorithm to fill in the blank regions without pixel-point correspondence, as shown in Figure 3 in the PDF attachment of the global response. The projected 2D label maps and the outputs from XMask3D on several nuScenes samples are shown in Figure 2, demonstrating the potential of XMask3D in handling outdoor scenarios.\n\n> Additionally, the authors could provide a qualitative analysis of the model's potential limitations when applied to different environments.\n\nAs discussed above, the current architecture of XMask3D requires full annotations for both 2D and 3D data. Therefore, when dealing with outdoor datasets that lack fine-grained 2D annotations, the performance of XMask3D may decrease. Additionally, we provide a qualitative analysis of failure cases in `3. Failure case analysis`, which reveals potential limitations of XMask3D under various circumstances.\n\n### **2. Computational costs.**\n\n> The reliance on the denoising UNet from a pre-trained diffusion model could be seen as a potential weakness or limitation, especially given the computational resources required for training and inference.\n\nWe acknowledge that the denoising UNet requires significant computational resources and slows down the inference speed. In our future work, we plan to address this limitation by replacing the 2D branch with a more lightweight 2D open vocabulary mask generator. Thank you for highlighting this potential weakness, which has inspired our future improvements.\n\n### **3. Failure case analysis.**\n\n> The paper could benefit from a more detailed error analysis to understand the failure modes of XMask3D, especially in novel category segmentation.\n\nThanks for your constructive suggestion! We provide three failure cases of XMask3D in Figure 1 in the PDF attachment of the global response. We will include this figure and the following discussions in the supplemental material of our revised paper.\n\nThe first sample shows a `bathtub` with a `shower curtain` in the bathroom. However, the 2D/3D branch and the fused output of XMask3D all misclassify the `shower curtain` as `curtain`. This may be because these two categories are similar in object shape and texture, with the only difference being the surrounding environment. Since XMask3D only takes a corner of the room as input instead of the entire scene, the global environmental information is insufficient for making the correct category prediction.\n\nThe second sample shows a large area of `picture` on the `wall`. The 2D/3D branch and the fused output of XMask3D all misclassified it as `wall`, due to their similar geometry. In most cases, a picture is a small region on the wall, and this picture, as large as the wall, is a typical corner case. This failure case may reveal XMask3D's over-reliance on geometric knowledge and lesser consideration of texture information when encountering out-of-distribution samples.\n\nThe third sample shows a `sink` on a `counter`. Due to the occlusion problem, the `sink` point cloud is incomplete, negatively affecting the prediction of segmentation boundaries between the `sink` and the `counter`. This occurs because they are geometrically similar when the sinking-down part of the sink is missing.\n\n### **References**\n[1] Jihan Yang, et al. \"RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding.\" CVPR. 2024.   \n[2] Qingdong, He, et al. \"UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with Fine-Grained Feature Representation.\" IJCAI. 2022.  \n[3] Holger Caesar, et al. \"nuscenes: A multimodal dataset for autonomous driving.\" CVPR. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m2utdEThw3", "reviewer": "Reviewer_AnYT", "summary": "The paper proposes a precise and consistent mask-level alignment between 3D features and the 2D-text embedding space through a method called cross-modal mask reasoning. The proposed XMask3D model includes a 3D branch for capturing geometric features, a 2D branch for generating vision-language aligned masks, and a fusion block to combine 3D with 2D. Using a pre-trained text-to-image diffusion model as the 2D mask generator, the model leverages three techniques: 3D-to-2D mask generation, 2D-to-3D mask regularization, and 3D-2D mask feature fusion.", "questions": "Please compare to OV3D mentioned in the weaknesses", "limitations": "Needs to be expanded", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1- The idea is novel, the author propose to merge 2D which provides high OV capabilities, with 3D features shich endoces 3D geometry. \n\n2- The method performs remarkably better than the reported models, namely OpenScene. The experiments are also well structure", "weaknesses": "1- The authors don't compare with state-of-the-art 3D semantic segmentation OV3D[1]\n\n2- The authors highlighed fututre work in the limitation, it would be good if you can expand it with some limitation on the technical side or some failure cases.\n\n[1] Jiang, Li, Shaoshuai Shi, and Bernt Schiele. \"Open-Vocabulary 3D Semantic Segmentation with Foundation Models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and constructive comments! Hopefully the following response will address your concerns.\n\n### **1. Results comparison.**\n\n> The authors don't compare with state-of-the-art 3D semantic segmentation OV3D.\n\nThanks for your suggestion! We will include this outstanding method in the ScanNet comparison in our revised paper. OV3D primarily focuses on EntityText extraction and Point-EntityText association, while XMask3D concentrates on enhancing mask-level interaction between 2D and 3D modalities. Therefore, the contributions of OV3D and XMask3D are orthogonal and could complement each other. Once the official code of OV3D is released, we plan to integrate our model with OV3D's advanced design in prompting LLM-powered LVLM models and Pixel-EntityText alignment to further enhance the performance of XMask3D.\n\n### **2. Expanded limitations.**\n\n> The authors highlighed fututre work in the limitation, it would be good if you can expand it with some limitation on the technical side or some failure cases.\n\n#### **(A) Technical limitations.**\n\nCurrently, the denoising UNet from the pre-trained diffusion model requires significant computational resources and impacts inference efficiency. In contrast to PLA, which consists solely of a 3D model, XMask3D could be further improved by replacing the 2D branch with a more lightweight and efficient 2D open vocabulary mask generator.\n\n#### **(B) Failure case analysis.**\n\nThanks for your constructive suggestion! We provide three failure cases of XMask3D in Figure 1 in the PDF attachment of the global response. We will include this figure and the following discussions in the supplemental material of our revised paper.\n\nThe first sample shows a `bathtub` with a `shower curtain` in the bathroom. However, the 2D/3D branch and the fused output of XMask3D all misclassify the `shower curtain` as `curtain`. This may be because these two categories are similar in object shape and texture, with the only difference being the surrounding environment. Since XMask3D only takes a corner of the room as input instead of the entire scene, the global environmental information is insufficient for making the correct category prediction.\n\nThe second sample shows a large area of `picture` on the `wall`. The 2D/3D branch and the fused output of XMask3D all misclassified it as `wall`, due to their similar geometry. In most cases, a picture is a small region on the wall, and this picture, as large as the wall, is a typical corner case. This failure case may reveal XMask3D's over-reliance on geometric knowledge and lesser consideration of texture information when encountering out-of-distribution samples.\n\nThe third sample shows a `sink` on a `counter`. Due to the occlusion problem, the `sink` point cloud is incomplete, negatively affecting the prediction of segmentation boundaries between the `sink` and the `counter`. This occurs because they are geometrically similar when the sinking-down part of the sink is missing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PxfWILE7VV", "reviewer": "Reviewer_tdkT", "summary": "This paper introduces XMask3D, a framework developed for open vocabulary 3D semantic segmentation. They propose the integration of the denoising UNet, derived from a pre-trained diffusion model, to generate geometry-aware segmentation masks conditioned on learnable implicit 3D embeddings. These binary 2D masks are used to filter mask-level embeddings of 3D representations and apply mask regularization, thereby improving the open vocabulary capacity of 3D features.", "questions": "1.\tPlease provide further clarification on how mask-level alignment between 3D features and 2D embedding space can address the limitations of traditional techniques, such as global feature alignment or vision-language model distillation. Additionally, if texts (Category Labels) are concatenated with fused features, will it still create a unified feature space that encompasses 3D, 2D, and textual modalities?\n2.\tCould you please provide further clarification on the main contributions of your research compared to PLA and OpenScene? Although the 3D caption process shares similarities with PLA, the overall pipeline resembles OpenScene, with the exception of the diffusion model and mask generator, which differ from the Multi-view Feature Fusion in OpenScene.\n3.\tDoes the Implicit 3D Captioner effectively work with your 3D features? From my understanding, the most reliable 3D captioner currently available is Cap3D, which generates captions for 3D objects by rendering multi-view images and utilizing BLIP2 and LLM for assistance. In the context of indoor-scenes, can we consider the Implicit 3D Captioner to be equally robust? It would be beneficial to present additional evidence to support this claim.\n4.\tCan your text-to-image diffusion model effectively generalize to your datasets? If not, please provide examples of failure cases. Additionally, is the diffusion model fine-tuned during the training process or is it frozen? If not, please present additional results to demonstrate the robustness of your diffusion model in generating high-quality images within your datasets.\n5.\tWhat is view-level contrastive loss? Why this loss is calculated between the view global feature and text embedding of the view image caption but have three coefficients?\n6.\tIt is recommended to show your 2D Mask and 3D Mask in Figure 3 to provide more visual evidence.\n7.\tThe authors should provide results on ScanNet++ (CVPR’23), which is a up-to-date dataset compared with ScanNet.\n8.\tSince diffusion models are utilized, it is recommended to compared the model parameters and FLOPs compared with PLA and OpenScene.", "limitations": "1.\tThe authors have addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe motivation is clear.\n2.\tThe proposed method is intuitive, and the experiments have validated their contributions.", "weaknesses": "1.\tThe organization should be improved. Section 3.1 provides an overview, while section 3.2 includes design insights and preliminary findings. The flow of these writings has puzzled me, making it difficult to grasp your key contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and constructive comments! Hopefully the following response will address your concerns.\n\n### **1. Paper organization.**\n\n> The organization should be improved ...\n\nThanks for your advice! We will reorganize the first two sections in our revised paper.\n\n### **2. Clarifications.**\n\n#### **(A) Mask-level alignment.**\n\n> ... clarification on how mask-level alignment ... \n\nThe most critical challenge in addressing the 3D open vocabulary problem is establishing 3D-text alignment. Previous approaches have tackled this by enforcing either global-wise or point-wise alignment between 3D features and 2D-text embeddings. \n\nGlobal feature alignment typically involves calculating a global feature of a scene or view through pooling and applying contrastive loss with 2D-text embeddings. However, pooling often result in the loss of detailed information, leading to unclear and imprecise boundaries on novel categories. \n\nPoint-wise feature alignment matches 3D points with 2D pixels using camera parameters, and performs dense contrastive learning between them. While this method preserves boundary details, it incurs a significantly larger computational burden and less stable training, as the loss is highly susceptible to outliers. \n\nOur proposed mask-level alignment presents an intermediate solution of these two approaches. Compared to global features, mask features retain more detailed information since there are dozens of masks within a single view. Additionally, mask features are more robust than point features, as the distraction from outliers is mitigated by the mask pooling operation.\n\n> ... unified feature space ...?\n\nThe 2D features with high open-vocabulary capacity are embedded in the 2D-text feature space. By applying 2D-to-3D mask regularization to the 3D features, these 3D features are drawn towards the 2D-text feature space. Consequently, the fused features, which integrate both 2D and 3D features, are expected to align well with the 2D-text embedding space. This alignment is further conveyed by the superior open-vocabulary performance of XMask3D compared to PLA and OpenScene. Thus, we believe that concatenating text and fused features will effectively create a unified feature space that integrates 3D, 2D, and textual modalities.\n\n#### **(B) Main contributions compared with OpenScene and PLA.**\n\n> ... main contributions compared to PLA and OpenScene ...\n\nSorry for not directly highlighting our contributions relative to OpenScene and PLA. \n\nOpenScene employs point-wise distillation and feature ensemble, while XMask3D utilizes mask-wise feature regularization. As discussed in *Section 2(A)*, mask-wise contrastive learning is more robust and less computationally demanding compared to point-wise counterpart. \n\nPLA proposes entity-level point-caption association through set difference and intersection operations, which is less precise and adaptable than our mask-level 3D-2D-text association. The mask for 3D-text alignment is adaptively predicted by the 2D mask generator, making XMask3D an end-to-end and integrated system. \n\nTherefore, the core contribution of our paper is the introduction of a more adaptive and robust mask-level alignment technique.\n\n#### **(C) Implicit 3D captioner.**\n\n> Does the Implicit 3D Captioner ...\n\nAccording to the ablation studies in Table 3(a), the proposed implicit 3D captioner yields better novel class results compared to using vanilla text embeddings or implicit 2D caption embeddings, which convey the effectiveness of the implicit 3D captioner. However, unlike Cap3D, the implicit 3D captioner does not directly generate text captions. It only produces conditional features that work effectively with the XMask3D pipeline and we cannot guarantee its robustness in other 3D-to-text generation scenarios.\n\n#### **(D) Text-to-image diffusion model.**\n\n> Can your text-to-image diffusion model ...\n\nIn our XMask3D pipeline, all weights of the denoising UNet are frozen. We do not generate images using the text-to-image diffusion model. Instead, we use the HxWxC features from the denoising UNet to generate 2D masks through the mask generator. Only the mask generator is trained on our datasets.\n\n#### **(E) View-level contrastive loss.**\n\n> What is view-level contrastive ...\n\nSorry for the unclear statement. The ground truth for the view-level contrastive loss is the text embedding of the view image caption. The predictions are derived from the average pooling results of the 3D branch features, 2D branch features and fused features respectively. Therefore, we have three view-level contrastive losses and design three separate coefficients for them.\n\n### **3. Additional visualizations.**\n\n> ... 2D Mask and 3D Mask in Figure 3 ...\n\nThanks for your insightful advice! In Figure 4 in the PDF attachment of the global response, we display the outputs from the 2D and 3D branches for the same samples shown in Figure 3 of our main paper.\n\n### **4. ScanNet++ dataset.**\n\n> ... results on ScanNet++ ...\n\nWe apologize for not providing results on ScanNet++, as we followed the protocols of previous papers (OpenScene and PLA). Following your advice, we have submitted a request to download ScanNet++, but unfortunately it is still pending, so we are unable to conduct experiments within the rebuttal period. We appreciate your suggestion regarding this advanced dataset and plan to add it to our future work. \n\n### **5. Resource comparisons.**\n\n> ... model parameters and FLOPs ...\n\nThanks for your suggestion! As the FLOPs of 3D models are conditioned on point cloud numbers, we only report the additional FLOPs of the 2D branch. In future work, we plan to replace the 2D branch with a more efficient and lightweight open vocabulary 2D mask generator.\n\n|Method|Trainable Params|Non-trainable Params|Extra FLOPs|\n|:-:|:-:|:-:|:-:|\n|PLA|11.0 M|--|--|\n|OpenScene|15.6 M|126.5 M|OpenSeg: 1.6 TFLOPs|\n|XMask3D|82.9 M|1493.8 M|Denoising UNet: 4.3 TFLOPs|\n||||Mask Generator: 93.5 GFLOPs|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for taking the time to review our submission and provide constructive feedback on our work. We are encouraged by the consensus among reviewers regarding the strengths of our approach, which aligns with our intentions and efforts:

1. **Novelty and Significance**: We appreciate that all reviewers recognize our approach as both novel and reasonable. It is encouraging to see comments such as those from Reviewer AnYT, who noted that "The idea is novel," and Reviewer Xo8p, who remarked that "the part 'incorporating a 2D mask generator to create geometry-aware open masks and apply fine-grained mask-level regularization on 3D features' seems reasonable and novel."
2. **Thorough Evaluation**: We are pleased that the reviewers acknowledged the comprehensiveness of our experiments and evaluations. Reviewer Xo8p commented that "Analysis is thorough and insightful," while Reviewer tdkT noted that "The proposed method is intuitive, and the experiments have validated their contributions."
3. **Clarity and Presentation**: We are gratified that our efforts to present our ideas clearly have been well-received. Reviewer 4VWC highlighted that "By dissecting the method and providing qualitative and quantitative results for each step, the authors make it easier for readers to understand and gain intuition about the presented approach." Reviewer Xo8p also mentioned that "The paper is well-structured and easy to follow."

We also appreciate the suggestions for further improving our work. In response to the specific concerns and recommendations raised by each reviewer, we have provided detailed discussions in our rebuttal and will update them in our revised paper. Additionally, we have prepared a PDF with additional illustrations to offer a more comprehensive visualization of our results and reinforce the validity of our work.

Best regards,

Submission 637 Authors

Meta-review: This paper introduces XMask3D, an approach to open-vocabulary 3D semantic segmentation that addresses key limitations in existing methods by integrating 3D geometric features with 2D and text-based embeddings. The authors build on the ODISE method, extending it to the 3D domain and introducing a novel implicit captioner that generates geometry-aware 2D masks. This enhances the alignment between 3D and 2D-text embedding spaces, resulting in improved segmentation accuracy, particularly for novel categories. The method employs a design incorporating distillation losses and mask-level regularization, which boosts its open-vocabulary capabilities.

XMask3D is evaluated on several benchmarks, including ScanNet, ScanNet200, and S3DIS, where it demonstrates superior performance compared to existing methods. These results are further supported by a comprehensive ablation study. All reviewers have recommended acceptance of this paper, and the AC finds no significant reasons to overturn these recommendations. Therefore, the paper is recommended for acceptance.
Decision: Accept (poster)