A Unified Framework for 3D Scene Understanding
Authors: Wei Xu
Chunsheng Shi
Sifan Tu
Xin Zhou
Dingkang Liang
Xiang Bai
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D scene understanding; Unified model; 3D segmentation
Primary area: machine_vision

Abstract: We propose UniSeg3D, a unified 3D scene understanding framework that achieves panoptic, semantic, instance, interactive, referring, and open-vocabulary segmentation tasks within a single model. Most previous 3D segmentation approaches are typically tailored to a specific task, limiting their understanding of 3D scenes to a task-specific perspective. In contrast, the proposed method unifies six tasks into unified representations processed by the same Transformer. It facilitates inter-task knowledge sharing, thereby promoting comprehensive 3D scene understanding. To take advantage of multi-task unification, we enhance performance by establishing explicit inter-task associations. Specifically, we design knowledge distillation and contrastive learning to transfer task-specific knowledge across different tasks. Experiments on three benchmarks, including ScanNet20, ScanRefer, and ScanNet200, demonstrate that the UniSeg3D consistently outperforms current SOTA methods, even those specialized for individual tasks. We hope UniSeg3D can serve as a solid unified baseline and inspire future work. Code and models are available at \url{https://dk-liang.github.io/UniSeg3D/}.

A Unified Framework for 3D Scene Understanding
Wei Xu', Chunsheng Shi", Sifan Tu, Xin Zhou, Dingkang Liang, Xiang Bai! Huazhong University of Science and Technology {wxu2023, cashi, dkliang, xbai] edu. cn
Paneptic Liting (CYPR
AGILE30 (CLR 24)
Prnoppic Seg.
Ieterntive
Panoptic
Interactive Seg.
OutFormer Fermer RAGGRAPH 23)
X-RofSag
OneFurmer (CVPR 24
Referring Seg.
Refeemin Seg.
Poormer
Open3DDS PU 24)
(CCV
Seg
Inatass Seg.
Segi
(a) Separated specialized SOTA solutions
(b) Existing unified method, supporting three tasks
VemeeeD LB Ours
Ours
712
71.3
766
76.9
Panoptic
Inttnnetive Sog.
Uniteg (Durs)
70
Pazupti: Seg.
Semantic Seg.
Instence Seg.
ACCIL
X-RofSog33
OperaniS
Semanti Seg.
21
$4.5
31
28.6
18.7
53.5
25.5
Seg.
Inleractive Seg.
Selertin Seg.
OV Seg.
Proposed unified method, supporting six tasks
(d) Comparisons with previous SOTA
Figure I: Comparisons between the proposed method and current SOTA approaches specialized for specific tasks. (a) Representative specialized approaches six tasks. (b) OneFormer3ID, recent unified framework, achieves SOTA performance on three generic segmentation tasks in one inference. (c) The proposed unified framework achieves six tasks in inference. (d) Our method outperforms current SOTA approaches across six tasks involving two modalities using single model.
Abstract
We propose UniSeg3D, a unified 3D scene understanding framework that achieves panoptic, semantic, instance, interactive, referring, and open-vocabulary segmenta- tion tasks within a single model. Most previous 3D segmentation approaches are typically tailored to a specific task, limiting their understanding of 3D scenes a task-specifit perspective. In contrast, the proposed method unifies six tasks into unified representations processed by the same Transformer. It facilitates inter-task knowledge sharing, thereby promoting comprehensive 3D scene understanding. To take advantage of multi-task unification, we enhance performance by establishing explicit inter-task associations. Specifically, we design knowledge distillation and contrastive learning to transfer task apecifcciiiic knowledge across different tasks. Ex-
"Equal contribution. *Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

periments on three benchmarks, including ScanNet20, ScanRefer, and ScanNet200, demonstrate that the UniSeg3D consistently outperforms current SOTA methods, even those specialized for individual tasks. We hope UniSeg3D can serve as a solid unified baseline and inspire future work. Code and models are available at https::/ //tk::///k-aa liang. gi gitubb. 10/ ho/UniSegDD/.
Introduction
3D scene understanding has been foundational aspect of various real-world applications [3, 13, 72, 71, 19], including robotics, autonomous navigation, and mixed reality. Among 3D scene un- derstanding tasks, 3D point cloud segmentation is crucial component. Generie 3D point cloud segmentation contains panoptic, semantic, and instance segmentation tasks [38, 42, 61, 69, 22], which segment classes annotated in the training set. As complement, 3D open-vocabulary (OV) segmentation task [41, 54, 16] segments open-vocabulary classes of interest. Another group of works study to utilize user priors. In particular, 3D interactive segmentation task [24, 70] segments instances specified by users. 3D referring segmentation task [15, 45, 59, 58] segments instances described by textual expressions. The above-mentioned tasks are core tasks in 3D scene understanding, drawing significant interest from researchers and achieving great success.
Previous studies [53, 8, 75, 30, 20] in the 3D scene understanding area focus separated solutions specialized for specific tasks, as shown in Fig. I(a). These approaches overlook intrinsic connections across different tasks, such as geometric and semantic consistency of objects. They also fail to share knowledge biased toward other tasks, limiting their understanding of 3D scenes to task-specific perspectives. It poses significant challenges for achieving comprehensive and in-depth 3D scene understanding. A recent exploration [23] named OneFormer3D designs an architecture to unify the 3D generic segmentation tasks, shown in Fig. 1(b). This architecture inputs instance and semantic queries to simultancously predict the 3D instance and semantic segmentation results. And the 3D panoptic segmentation is subsequently achieved by post-processing these predictions. It is simple yet effective. However, this architecture fails to support the 3D interactive, referring, and open-vo vocabulary segmentation tasks, which provide complementary scene information, including user priors and open-set classes, should be equally crucial in achieving 3D scene understanding as the generic segmentation tasks. This leads to a natural consideration that if these 3D scene understanding tasks can be unified in single framework?
A direct solution is integrate separated methods into single architecture. However, it faces challenges balancing customized optimizations specialized for specific tasks involved in these meth- ods. Therefore, we aim design simple and elegant framework without task-specific customized modules. This inspires us to design UniSeg3D, a unified framework processing six 3D segmentation tasks in parallel. Specifically, we use queries to unify representations of six tasks. The 3D generic and open- vocabulayy segmentation tasks, only input point clouds without human knowledge, can be processed by sharing the same workflow without worrying about prior knowledge leakage. We use a unified set of queries to extract features of these four tasks for simplification. The interactive segmentation inputs visual point prompts to condition the segmentation. We represent the point prompt information by sampling point cloud features as vision prompt queries, thereby avoiding repeated point feature extraction. The referring segmentation inputs textual expressions persist in modality gap with point clouds and are hard to unify in previous workflows. To reduce time consumption, we employ a parallel text prompt encoder to extract text features and regard them as text prompt queries. All these queries are decoded using the same mask decoder and share the same output head without the design of task-specific customized structures.
We further enhance performance by taking advantage of the multi-task design. In particular, we empirically find that the interactive segmentation outperforms the rest of the tasks in mask predictions, which is attributable to reliable vision priors. Hence, we design knowledge distillation to distill knowledge from the interactive segmentation the other tasks. Then, we build contrastive learning between interactive segmentation and referring segmentation to connect these two tasks. The proposed knowledge distillation and contrastive learning promote knowledge sharing across six tasks, effectively establishing inter-task associations. There are three significant strengths of the UniSeg3D: (1) It unifies six 3D scene understanding tasks in single framework, as shown in Fig. (cc). (2) is flexible because it can be casily extended to more tasks by simply feeding additional task-specific

---

queries. (3) The designed knowledge distillation and contrastive learning are only used in the training phase, optimizing performance with no extra inference cost.
We compare the proposed method with task-specific specialized SOTA approaches [50, 56, 36, 70, 45, 40] across six tasks evaluate its performance. As shown in Fig. ((d), the UniSeg3D demonstrates superior performance on all the tasks. It is worth noting that our performance on different tasks is achieved by single model, which is more efficient than running separate task-specific approaches individually. Furthermore, the structure of UniSeg3D is simple and elegant, containing no task- customized modules, while consistently outperforming specialized SOTA solutions, demonstrating a desirable potential to be solid unified baseline.
Our contributions can be summarized as follows: First, propose unified framework named UniSeg3D, offering a flexible and efficient solution for 3D scene understanding. It achieves six 31) segmentation tasks in one inference by single model. To our knowledge, this is the first work to unify six 3D segmentation tasks. Second, specialized approaches limit their 3D scene understanding to task-specific perspectives. We facilitate inter-task knowledge sharing to promote comprehensive 3D scene understanding. Specifically, we take ad advantage of the multi-task unification design, employing knowledge distillation and contrastive learning establish explicit inter-task associations.
2 Related Work
3D segmentation. The generic segmentation consists of panoptic, semantic, and instance segmenta- tion. The panoptic segmentation [38, 60] is the union of instance segmentation [9, 33, 2, 64, 55] and semantic segmentation [44, 42, 5, 73]. It contains instance masks from the instance segmentation and stuff masks from the semantic segmentation. These 3D segmentation tasks rely annotations, segmenting classes labeled in the training set. The open-vocabulary segmentation [40, 54] extends 3D segmentation to novel classes. Another group of works explores 3D segmentation conditioned by human knowledge. Specifically, the interactive segmentation [24, 70] segments instances specified by the point clicks. The referring segmentation [15, 45, 57] segments objects described by textual ex- pressions. Most previous researches [65, 4, 25, 32] focus on specific 3D segmentation tasks, limiting their efficiency in multi-task scenarios, such the domoties, that require multiple task-specific 3D segmentation approaches to be applied simultaneously. This work proposes a framework to achieve the six above-mentioned tasks in one inference.
Unified vision models. Unified research supports multiple tasks in a single model, facilitating efficiency and attracting extensive attention in the 2D area [43, 37, 29, 18]. However, rare works study the unified 3D segmentation architecture. It might be attributed to the higher dimension of the 3D data, which leads to big solution space, making it challenging for sufficient unification across multiple 3D tasks. Recent works 35] explore outdoor unified 3D segmentation architectures, and some others [76, 14, 17] delve into unified 3D representations. So far, only method, OneFormer3D [23]. focuses on indoor unified 3D segmentation. It extends the motivation proposed in OneFormer [18] to the 3DD area and proposes architecture to achieve three 3D generic segmentation tasks in a single model. We note that the supported tasks in OneFormer3D can be achieved one inference through post-processing predictions of a panoptic segmentation model. In contrast, we propose a simple framework that unifies six tasks, including not only generic segmentation but also interactive, referring, and open-vocabulary segmentation, into single model. Additionally, we establish explicit associations between these unified tasks to promote knowledge sharing, contributing to effective multi-tas, task unification.
3 Methodology
The framework of UniSeg3D is depicted in Fig. 2. It mainly consists of three modules: point cloud backbone, prompt encoders, and mask decoder. We illustrate their structures in the following.
3.1 Point Cloud Backbone and Prompt Encoders
Point cloud backbone. We represent a set of N input points P E Nx6 where each point is characterized by three-dimensional coordinates v, y. and three-channel colors r, g, b. These input points are then fed into sparse 3D U-Net, serving the point cloud backbone, to obtain point-wise
3

---

Mask generation
Supported tasks
q
Gu
4u
k.
(fi)
Panoplic Seg.
Point Backbone
Select
Self- attention
Semantic Seg.
Instance Seg.
OV Seg.
Ingut point cloud
Ev
Distilation
bufpe
Click
Sampler
Cross- attenticn
Interactive Seg.
Vision prompt Click
Distillation
Contrastive
"... the table
4:
under a monitor."
Text Encoder
Referning Seg.
"... the first chair from left."
Mask Decoder
Text prompts
Feature
Task-specific embelldngg
Pixel-wise add
Eimbeddeed query
The framework of UniSeg3D. This a simple framework handling six tasks in parallel without any modules specialized for specific tasks. We take advantage of the multi-task unification design and enhance performance by building associations between the supported tasks. Specifically, knowledge distillation transfers insights from interactive segmentation to the other tasks, while con- trastive learning establishes connections between interactive segmentation and referring segmentation.
Figure 2:
features F Nxddn, where di denotes the feature dimension. Processing dense points individually in 3D scene understanding can be into M superpoints and pool the point features within each superpoint to form superpoint features F. {f,],= where each E Rd and F, E IR xdis, This procedure exhibits awareness of edge textures [27] while reducing computation cost.
Therefore, we downsample the 3D scenario
Vision prompt encoder. Click is clear and convenient visual interaction condition widely employed in previous works [21, 24, 70J. We formulate clicks as vision prompts, as illustrated in Fig. 2. In practice, click is first indicated by the spatially nearest point. Then, we sample superpoint containing this point and employ its superpoint feature as vision prompt feature f, E RR to represent point prompt information, thus avoiding repeated feature extraction and maintaining feature consistency with the point clouds.
Text prompt encoder. UniSeg3D able segment instances described by textual expressions. The initial step of processing a text prompt involves tokenizing the text sentence to obtain its string tokens T E tixe, where is the sentence length, and c represents the token dimension. These tokens are then fed into a frozen CLIP [46] text encoder to produce C-dimensional text feature f; R This feature is subsequently projected into dimension of din using two linear layers, obtaining Rdd., aligning the dimension of the point features for subsequent processing.
3.2 Mask Generation
We employ a single mask decoder to output predictions six 3D scene understanding tasks. The generic and open- vocabulayy segmentation share the same input data, i.e., the point cloud without user knowledge. Therefore, we randomly select m features from M superpoint features to serve as unified queries q E mm for both the generic and vocabulcyy segmentation tasks. During training. we set m < to reduce computational costs, while for inference, we set m M to enable the segmentation of every region.
The prompt information is encoded into prompt features as discussed in Sec. 3.1. We employ the prompt features as prompt queries, which can be written as q k of = R where q. E R. xddn, q E K.dmn, K, and K, are the number of the point and text prompts, respectively, du. 9 q are three types of queries containing information from various aspects. Feeding them forward indiscriminately would confuse the mask decoder for digging task-specific information. Thus, we add task- ssec-fpecfiic embeddings C., C., and e₁ before further processing:
qu = =q' Cur q =q Co.
q: = 9 + Cr
(1)

---

Matched
Ranking
Text prompts
Text
Text
Text
Text
Distilaaion]
Distillation
ei
e
Matching
The annchairrae is to the right of the other chair. is facing the table.
S11
S12
Siz
Syx
Contrastive
NS
S11
Sa2
Sea
Szs
Output head
S11
S32
Sa2
The amochair' is facing the lable. here another chair right it.
NS
Sa1
S.p
S.x
S..
A8) challenging case
Ranking- baseed contrastive leemiing
(c) Explicit inter-task associations
Figure Illustration of the inter-task association. (a) challenging case requiring distinction
of position information within textual descriptions. (b) A contrastive learning matrix for paired
vision-text features, where a ranking rule is employed to suppress incorrect pairings. (c) Knowledge distillation across multiple tasks.
where E 1ddi. ey E Rdin E Rdi., and are broadcasted into RR Je PKKxxii. and R xdia. respectively. The mask decoder comprises L mask decoder layers, which contain self-attention layers integrating information among queries. Prompt priors involving human knowledge are unavailable for generie segmentation during inference. Therefore, in the training phase, we should prevent human knowledge from leaking to the generic segmentation. In practice, the prompt queries are exclusively fed into the crossiatiention layers. Output queries of the last mask decoder layer are sent into an output head consisting of MLP layers project dimensions of the output queries from dir to dout- In general, the mask generation process can be formally defined as:
F Fout MLP (MaskDeooder Decoder (q Concal (qus qu, q:): ;k FA;v= F.)),
(2)
where oout = { ) mm I K.+ X.)xu
represents output features, with four, Iedout and F Fout E
Subsequently,
process the output features obtain class and mask predictions. For class predictions, common practice involves replacing class names with class IDs [23]. However, for method support referring segmentation, the class names are crucial information that should not be overlooked. Hence, we encode the class names into text features Cdta E TR K. xdout using a frozen CLIP text encoder and propose regress the class name features instead, where Ke denotes the number of categories. Specifically, we formulate the mask predictions maskpred and class predictions elspred as follows:
(3) where maskpred (mask;) | m and clspred = {cls; | m. with mask [mm+ K. K.X and clspred E
mask pred = F Fout MLP (F.)T, clspred = Softmax (F (Fout ecls)
mask; E RM and cls, E RK represent the mask outcome and category probability predicted by the i-th query, respectively. The MLP projects Rdi.. into Rdos, Given that maskpred and clspred are derived from superpoints, we map the segmentation outputs for each superpoint back to the input point cloud generate point-wise mask and class predictions.
3.3 Explicit Inter-task Association
Previous studies have overlooked connections among 3D scene understanding tasks, resulting in task sesk-specffic approaches that fail leverage cross-task knowledge. This limitation restricts the under- standing of 3D scenes to task-specific perspective, hindering comprehensive 3D scene understanding. We establish explicit inter-task associations overcome these constraints.
Specifically, on the one hand, as shown in Fig. 3(a), the referring segmentation is challenging when multiple individuals of identical shapes are arranged adjacently. It requires the method distinguish the location variations inserted in the text prompts, such as "right of the other chair" vs. "another chair the right of it." However, the modality gap between 3D points and linguistic obstructions. We propose ranking- based contrastive learning between the vision and text features to reduce the modality gap and optimize the referring segmentation.
significant

---

On the other hand, as shown in Tab. 1, we evaluate our baseline framework built in Sec. 3.1 and Sec. 3.2 on instance and interactive segmentation tasks. Essentially, the main difference between the instance and interactive segmentation w/o or w/ vision prompts. The mloU metric, which directly measures the quality of mask pre- dictions, indicates that the interactive segmentation surpasses the instance segmentation by notable margin of 7.9%. It suggests that vision prompts provide reliable position priors, boosting the interac- tive segmentation to perform superior mask prediction performance. We design a knowledge distillation to share insights from interactive segmentation across unified tasks. The core idea of this approach is to leverage the task of predicting best-quality masks to guide the other tasks, iee, using a teacher to guide students.
Table 1: Mask prediction per- formance of instance and inter- active segmentation.
Tasks
mloli
Instance Seg.
68.1
Interactive Seg.
76.0 (+7.9)
3.3.1 Ranking-bassee based Contrastive Learning
We set the vision and text prompts specifying the same individual instances into pairs and align their pairwise features by employing contrastive learning.
Assuming B vision-text pairs within a training mini-batch, the corresponding vision and text output } B features are { [ i=t and (fout, ) i=="" 1" fout, Rdduu and f out, E Rdout are selected from output +Kk
features {fout, m |=mm1 and {fout, (mmm))))=KKKK+1 respectively. We normalize these selected features
obtain vision and text metric embeddings [e]),== and {e]],=1, where ef RRdou and e E RRdett Then, the contrastive learning can be formulated as Leon = Lv + Lr, with:
(4) where 7 is a learnable temperature parameter. The pairwise similarity is illustrated in Fig. 3(b), where we denote ef ee, as Si,j for simplification. To distinguish the target instances from adjacent ones with identical shapes, we introduce a ranking rule inspired by the CrowdCLIP [31] that the diagonal elements are greater than the off- off-diagonal elements, which can be described as:
B Σ log B 1=1
cxp (c cf/T)
1
Σ
B
log exp c./
E cxp (c c//7)²
B
2=1 Ej= exp (c) c///T)
Frank
1
ΣΣ max (0, S.j s1,))
(5)
B
1=
3.3.2 Knowledge Distillation
As shown in Fig. 3(c), we transfer knowledge from the interactive segmentation task to the generic and referring segmentation tasks to guide their training phases.
To generic segmentation. Define predictions decoded from the unified queries Pred, [mask;, c We employ the Hungarian algorithm, utilizing the Dice and cross-entropy
metrics as matching cost criteria, assign Predy with interactive segmentation labels GT, fmask,,,1. c)s ==1 The matched predictions are selected as positive samples Pos,
K.
(maskpos.t ) We denote mask predictions among the positive samples maskpoe (maskpos, ) with maskpos E R K The predicted masks of interactive segmentation can be formulated as mask, (mask, /im++1 where mask, E KRK.MM We select the pixels with top 1% scores of mask, as learning region R. and depict the knowledge transfer process from the interactive segmentation to the generic segmentation tasks as:
(6) where maskpos pos (R) and mask, (R) represent the predicted mask values within the region R. gathering from the positive samples and the interactive segmentation predictions, respectively.
Lu-g = LBCE (maskpos (R), mask, (R)),
To referring segmentation. Define pairwise class probabilities predicted by the vision and
text prompt queries as cls, E Bx.k and cls, E BBXK. selected from {cls; ) K, and KK respectively. We formulate a knowledge transfer process from the interactive [cls;).nt (=m K,+ segmentation the referring segmentation task as:
Ly LBCE (Sigmoid (cls;). Sigmoid (cls.))

---

3.4 Training Objectives
Open-set pseudo mask labels. For open-vocabulae tasks, we train models on close-set data. To enhance segmentation performance on open-set data, we use SAM3D [67] to generate segmentation masks with undetermined categories as pseudo mask labels (open-set masks). While training, we assign predictions of the unified queries with ground-truth masks selose-set masks). The assigned and misss-ssiigned predictions are divided into positive and negative samples. The positive samples supervised to regress the close-set masks. We match the negative samples with the pseudo mask labels and supervise the matched ones to regress the open-set masks. Note that the SAM3D is an unsupervised method and does not rely on ground-truth annotations, eliminating worries of label leakage. This process is exclusively applied in the training phase, incurring no extra inference cost.
Loss function. Training losses contain two components: (1) Basic losses, formulated as Lbase = Lyaakk + L.s. mmskk stands for pixel-wise wise mask loss, comprising the BCE and Dice losses. Lats indicates the classification loss, where we use the cross-entropy loss. (2) Losses used to build inter-task associations, summarized as Linter = Le-ig fy + Leon + frank- The final loss function is f = Libase XLinter, where À is balance weight set as 0.1.
4 Experiments
Datasets. We evaluate the UniSeg3D on three benchmarks: ScanNet20 [6], ScanNet200 [47], and ScanRefer [1]. ScanNer20 provides RGB-D images and 3D point clouds of 1, 613 scenes, including 18 instance categories and 2 semantic categories. ScanNet200 uses the same source data as ScanNet20, while it is more challenging for up to 198 instance categories and 2 semantic categories. ScanRefer contains 51, 583 natural language expressions referring to 11, 046 objects selected from 800 scenes.
Experimental setups. We train our method the ScanNet20 training split, and referring texts collected from the ScanRefer. din and dout are set as 32 and 256, respectively. m ranges [50, 100] percent of M with an upper limit of 3, 500. We set as 10 and L as For data augmentations, input point clouds are randomly rotated around the z-axis, clastic distorted, and scaled; the referring texts are augmented using public GPT tools following [63, 7]. We adopt the AdamW optimizer with the polynomial schedule, setting an initial learning rate as 0.0001 and the weight decay as 0.05. All models are trained for 512 epochs on a single NVIDIA RTX 4090 GPU and evaluated per 16 epochs on the validation set to find the best-performed model. To stimulate models' performance, we propose a two-stage fine-tuning trick, which fine-tunes the sest-performed model, setting the learning rate and weight decay 0.001 times the initial values for 40 epochs. The proposed framework achieves end-to-end generic, interactive, and referring segmentation tasks. We divide the upen-vocabulary employ the proposed
segmentation task into mask prediction and class prediction. Specifically, UniSeg3D to predict masks and then follow the Open3DIS [40] to generate class predictions.
We use PQ, mloU, and mAP metrics to evaluate performance on the generic segmentation tasks following [38, 42, 69]. Then, we use AP and mIoU metrics for the interactive and referring segmen- tation tasks, respectively. following [70, 45]. For the vocabblcyyy segmentation task, we train our model on the ScanNet20 and evaluate it on the ScanNet200 using AP metric, following [54]. The Overall metric represents the average performance across six tasks intended to reflect the model's s unified capability.
4.1 Comparison to SOTA Methods
The proposed method achieves six 3D scene understanding tasks in a single model. We demonstrate the effectiveness of our method by comparing it with SOTA approaches specialized for specific tasks. As shown in Tab. 2, the proposed method outperforms the specialized SOTA methods Panoptic- NDT [49]. OctFormer [56], MAFT [26], AGILE3D [70], X-RefSeg3D [45], and Open3DIS [40] on the panoptic, semantic, instance, interactive, referring, and open-vocabulary (OV) segmentation tasks by 12.1 PQ. 1.2 mloU, 0.9 mAP, 1.0 AP, 4.1 mloU, 0.7 AP, respectively. Even when compared with the competitive 3D unified method, i.e., OneFormer31) [23], the proposed UniSeg3D achieves 0.1 PQ improvement on the panoptic segmentation task, and 0.3 mloU improvement on the semantic seg- mentation task. More importantly, the OneFormer3D focuses on three generic segmentation tasks. fails to understand user prompts, which limits its application prospects. In contrast, UniSeg3D unifies six tasks and presents desirable performance, demonstrating UniSeg3D a powerful architecture
7

---

Table 2: Comparisons ScanNet20 [6], ScanRefer [1], and ScanNer200 [47]. The best results are highlighted in bold, and the second-best results are underscored. "*" indicates the use of the two-stage fine-tuning trick. "-/-" denotes training filtered or complete ScanRefer datasets.
Datasets
ScanNer20
ScanRefer
ScanNer200
3D scene understanding tasks
Pan.
Sem.
Inst.
Inter.
Ref.
OV
Method
Reference
PQ
mloU
MAP
AP
mloU
AP
SceneGraphFusionn [60]
CVPR 21
31.5
TUPPer-Map [68]
IROS 21
50.2
Panoptic Lifting [50]
CVPR 23
58.9
PanopticNDT [49]
IROS 23
59.2
PointNeXt-XI.
NeurIPS 22
71.5
PoinMMctaBase-XX1 [34]
CVPR
72.8
MM-3DScene [66]
CVPR 23
72.8
Point DainTraanSommmrV22 [62]
NeurIPS 22
75.4
ADS [10]
ICCV 23
75.6
OctFormer [56]
SIGGRAPH 23
75.7
SoftGroup [55]
CVPR 22
45.8
PBNet [74]
ICCV
ISBNet [39]
CVPR 23
54.5
SPFormer [52]
AAAI 23
56.3
Mask3D [48]
ICRA 23
55.2
MAFT [26]
ICCV
58.4
QueryFormer [36]
ICCV 23
56.5
OneFormer3D [23]
CVPR 24
71.2
76.6
59.3
InterObject3D [24]
ICRA 23
20.9
AGILE3D [70]
ICLR 24
53.5
TGNN [15]
AAAT 21
24.9/27.8
X-RefSeg3D [45]
AAAI 24
25.5/29.9
OpenScene [41] with [48]
CVPR 23
8.5
OpenMask3D [54]
NeurIPS 23
12.6
SOLE [28]
CVPR 24
18.7
Open3DIS [40]
CVPR 24
19.0
Un (ours)
71.3
76.3
59.1
54.1
29.5/-
19.6
UniSeg3D (ours)
71.3
76.9
59.3
54.5
29.6/-
19.7
The proposed method achieves six tasks in one training, which is elegant while facing an is-
Table 3: Ablation on task unification.
sue for fair comparison. Specifically, partial labels in the referring segmentation benchmark (10, 115 objects, 27. 27.6% of the complete Scan- Refer training set) annotate novel classes of the open-vocabulary segmentation task. Obviously, these labels should not be used for training to avoid label leakage. Thus, we filter out these labels and only employ the filtered ScanRefer training set to train our model. As shown in Tab. 2, model uses 72.4% training data to achieve closing performance with X-RefSeg3D [45] (29.6 vs. 29.9). Moreover, while reproducing the X- X-RefSeg3D using official code our filtered training data, the performance drops to 4.1 mloU lower than UniSeg3D, demonstrating our model's effectiveness.
ScanNet200 ScanRefer
ScanNet20
Ref.
Inter.
Pan.
Sem.
mloU
AP
PQ
mloU
mAP
X
X
71.0
76.2
59.0
X
56.8
71.0
76.4
58.7
X 19.7
29.1
56.0
70.3
76.3
58.4
29.1
54.5
70.4
76.2
58.0
4.2 Analysis and Ablation
We conduct ablation studies and analyze key insights of our designs. All models are evaluated on multiple tasks show the effectiveness of the proposed components on a broad scope.

---

Table 4: Ablation components. "Distillation", "Rank-Contrastive"" and "Trick" denote the knowl- edge distillation, manking-based contrastive learning, and two-stage fine-tuning trick, respectively.
Datasets
ScanNer20
ScanRefer
ScamNet200
Components
Pan.
Sem.
Inst.
Inter.
Ref.
OV
Overall
Distillation Conk-Contrastive
Trick
PQ
mioU
mAP
AP
mloU
AP
70.4
76.2
58.0
54.5
29.1
19.7
51.3
70.9
76.2
58.6
55.3
29.2
19.6
51.6
70.8
76.4
58.4
54.1
19.9
51.5
71.3
76.3
59.1
54.1
29.5
19.6
51.7
71.3
76.9
59.3
54.5
29.6
19.7
51.9
Table Ablation on different designs proposed components. " and t" denote knowledge distillation from the interactive segmentation to the generic segmentation and the referring segmentation, respectively. "Contrastive" and "Rank" denote the contrastive learning and the ranking rule, respectively.
(a) Ablation on designs for knowledge distillation.
Datasets
ScanNet20
ScanRefer
ScanNer200
Components
Pan.
Sem.
Inst.
Inter.
Ref.
OV
Overall
>
>
PQ
mloU
mAP
AP
mloU
AP
70.8
76.4
58.4
54.1
29.6
19.9
51.5
71.2
76.3
59.0
54.0
29.5
19.8
51.6
70.7
76.2
58.6
54.1
29.7
20.0
51.6
71.3
76.3
59.1
54.1
29.5
19.6
51.7
(b) Ablation on designs for ranking- based contrastive learning.
Datasets
ScanNer20
ScanRefer
ScanNer200
Components
Pan.
Sem.
Inst.
Inter.
Ref.
OV
Overall
Contrastive
Rank
PQ
mloU
mAP
AP
mloU
AP
70.9
76.2
55.3
29.2
19.6
51.6
71.0
76.3
59.0
54.5
29.4
19.7
51.7
v
71.0
76.2
58.7
54.6
29.5
19.8
$1.6
v
71.3
76.3
59.1
54.1
29.5
19.6
51.7
The challenge of' multi-task unification. We discuss the challenge of unifying multiple tasks in single model. Specifically, simply add interactive, referring, and open-vocabulary segmentation into our framework to build a unification baseline, as shown in Tab. 3. We observe a continuous performance decline on the panoptic, instance, and interactive segmentation tasks, indicating significant obstacle in balancing different tasks. Even so, we believe that unifying multiple tasks within single model is worth exploring, as it can reduce computation consumption and benefit real-world applications. Thus, this paper proposes to eliminate performance decline by delivering inter-task associations, and the following experiments demonstrate that this could be a valuable step.
Design of inter-task associations. Our approach uses knowledge distillation and contrastive learning connect supported tasks. As shown Tab. 4, when applying the knowledge distillation, i.e. row 2. the performance of instance and interactive segmentation increase to 58.6 mAP and 55.3 AP, respectively. We believe the improvement on the instance task because of the reliable knowledge distilled from the interactive segmentation, and the improvement on the interactive segmentation task attributed to the intrinsic connections between the two tasks. Then, we ablate the ranking- based contrastive learning, i.e. row 3. We observe improvements on five tasks, including the generic segmentation and the referring segmentation, while slight performance drop on the interactive segmentation. This phenomenon suggests that contrastive learning is effective on most tasks, but there a huge struggle align point and text modalities, which weakens the interactive segmentation performance. Overall metric measures multi-task unification performance. We choose models and

---

Table 6: Ablation hyper-parameter X.
Datasets Hyper-parameter
ScanNet20
ScanRefer
ScanNer200
Pan.
Sem.
Inst.
Inter.
Ref.
OV
Overall
PQ
mloU
mAP
AP
mloU
0.05
70.7
76.2
54.4
29.5
19,6
51.6
0.1
71.3
76.3
59.1
54.1
29.5
19.6
51.7
0.2
70.8
76.6
58.6
52.3
29.8
19.5
51.3
0.3
70.6
75.7
58.4
51.6
29.6
19.3
50.9
checkpoints with higher Overalls our experiments. In practical applications. checkpoints can be chosen based on preferred tasks while maintaining good performance across other tasks. Applying knowledge distillation and ranking-based contrastive learning obtains comparable performance on most tasks, performing higher Overall than rows 2 and 3, indicating complementarity of the two components. We employ the two-stage fine-tuning trick, consistently improving various tasks.
Detailed ablation on the components is shown in Tab. 5. It is observed that knowledge distillation to various tasks brings respective improvements. As for contrastive learning, comparing row and row 4 in Tab. 5(b), the ranking rule suppresses confusing point-text pairs, boosting contrastive learning to be more effective. controls the strength of the explicit inter-task associations. We empirically find that setting À to 0.1 obtains the best performance, as shown in Tab. 6.
Influence of vision prompts. We empirically find that vision prompts affect the interactive segmentation perfor-
Table 7: Ablation on vision prompts.
mance. To ensure fair comparison, adopt the vi-
sion prompts generation strategy designed in AGILE3D [70]
Strategy
mloU
AP
APau
AP2s
evaluate our interactive segmentation performance.
From [70]
78.8
54.5
79.4
93.2
We ablate 3D spatial distances between the vision prompts and instance centers. Specifically, assuming an instance con- these
Instance center
79.6
56.6
82.1
Fa = 0.1
79.1
55.9
81.1
94.4
taining n points, we denote the mean coordinate points the instance center and order the points based on their distances to the instance center. Then, we evaluate the interactive segmentation performance while employing the Ira x -th nearest point as the vision prompt, as shown in Tab. 7. When the vision prompt positioned at the instance center, the interactive segmentation achieves an upper-bound performance of 56.6 AP, exhibiting a substan- tial performance gap of up to 20.2 AP compared to when the vision prompt is located at the object's edge (ra = 1.0). illustrating considerable room for improvement. We also observe an unusual performance decline while increasing ra from 0.9 to 1.0, which we attribute to
= 0.2
78.7
55.1
80.0
93.4
0.3
78.0
53.8
78.5
92.4
Fa 0.4
77.5
53.0
77.4
91.7
Fa = 0.5
76.6
52.1
76.2
90.6
= 0.6
75.9
51.2
74.6
90.0
0.7
74.9
50.1
72.9
88.1
0.8
73.4
48.2
71.1
86.5
Ta = 0.9
71.0
45.3
66.6
82.1
1.0
62.7
36.4
54.8
70.2
Random
76.0
51.3
75.2
89.6
the ambiguity in distinguishing the edge points of adjacent instances. As we know, this is the first work ablating the influence of vision prompts, and we will explore this issue depth in future work.
Conclusion and Discussion
We propose unified framework named UniSeg3D, which provides flexible and efficient solution for 3D scene understanding, supporting six tasks within a single model. Previous task-specific approaches fail to leverage cross-task information, limiting their understanding of 3D scenes to task task-specific perspectives. In contrast, we take advantage of the multi-task design and enhance performance by building inter-task associations. Specifically, employ knowledge distillation and ranking- contrastive learning to facilitate cross-task knowledge sharing. Experiments demonstrate that the proposed framework is powerful method, achieving SOTA performance across six unified tasks.
Limitation. UniSeg3D aims to achieve unified 3D scene understanding. However, it works on indoor tasks and lacks explorations in outdoor scenes. Additionally, we observe that UniSeg3D performs worse interactive segmentation performance when the vision prompt is located away from the instance centers, limiting the reliability of the UniSeg3D and should be explored in future work.
10

---

Acknowledgements
This work is supported in part by the National Natural Science Foundation of China (Grant. No. 62225603 and 623B2038).
References
[1] Dave Zhenyu Chen, Angel X Chang, and Matthias NieBner. Scanrefer: 3d object localization in rgb-d scans using natural language. In Proc. of European Conference on Computer Vision, pages 202-221, 2020.
[2] Shaoyu Chen, Jiemin Fang. Qian Zhang, Wenyu Liu, and Xinggang Wang. Hierarchical aggregation for 3d instance segmentation. Porc. of IEÉE Intl. Conf. on Computer Vision, pages 15467-15476, 2021.
[3] Shizhe Chen, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Sugar: Pre-training 3d visual repre- sentations for robotics. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, pages 18049-18060, 2024.
[4] Silin Cheng, Xiwu Chen, Xinwei He, Zhe Liu, and Xiang Bai. Pra-net: Point relation-aware network for 3d point cloud analysis. IEEE Transactions Image Processing, 30:4436-4448. 2021.
[5] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temport convnets: Minkowski convolutional neural networks. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, pages 3075-3084, 2019.
[6] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In Proc. of IEEE Inil. Conf. on Computer Vision and Pattern Recognition, pages 5828-5833 2017.
[7] Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang. Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, et al. Auggpt: Leveraging chatgpt for text data augmentation. arXiv preprint 2.XV:2302.13007 2023.
[8] Lei Han, Tian Zheng. Lan Xu, and Lu Fang. Occuseg: Occupaney-awee
instance segmentation. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, pages 2940-2949, 2020.
[91 Tong He, Chunhua Shen, and Anton Van Den Hengel. Dyco3d: Robust instance segmentation of 3d point clouds through dynamic convolution. In Proc. of IEEE Intl. Conf. on Computer Vision and Pottern Recognition, pages 354-363, 2021.
[10] Cheng-Yar Yao Hong. Yu-Ying Chou, and Tyng-Luh Liu. Attention discriminant sampling for point clouds. In Porc. of IEFE Intl. Conf. Computer Vision, pages 14429-14440, 2023.
[11] Fangzhou Hong, Lingdong Kong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. Unified 3d and 4d panoptic segmentation via dynamic shifting networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[12] Ji Hou, Angela Dai, and Matthias NieBner. 3d-sis: 3d semantic instance segmentation of rgb-d scans. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, pages 4421 4430, 2019.
[13] Jinghua Hou, Zhe Liu, Dingkang Liang, Zhikang Zou, Xiaoqing Ye, and Xiang Bai. Query-based temporal fusion with explicit motion for 3d object detection. In Proc. of Advances in Neural Information Processing Systems, volume 36, 2023.
[14] Di Huang. Sida Peng, Tong He, Honghui Yang, Xiaowei Zhou, and Wanli Ouyang. Ponder: Point cloud pre-training via neural rendering. Proc. of IEEE Inll Conf. on Computer Vision and Pattern Recognition, pages 16089-16098, 2023.
[15] Pin-Hao Huang, Han-Hung Lee. Hlwan--Tzong Chen, and Tyng- Luh Liu. Text-guided graph neural networks for referring instance segmentation. Proc. of the AAAI Conf. on Artificial Intelligence. pages 1610-1618. 2021.
[16] Zhening Huang, Xiaoyang Wu, Xi Chen, Hengshuang Zhao, Lei Zhu, and Joan Lasenby, Openins3d: Snap and lookup for 3d open-vocabulary instance segmentation. In Proc. of European Conference on Computer Vision, 2024.
[17] Muhammad 7ubair Irshad, Sergey 7akharov, Vitor Guizilini, Adrien Gaidon, 7solt Kira, and Rares Amhrus. Nerf-mae: Masked autoencoders for self-supervised 3d representation learning for neural radiance fields. In Proc. of European Conference on Computer Vision. 2024.
[18] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer rule universal image segmentation. In Proc. of IEEE Inil. Conf. on Computer Vision and Pattern Recognition, pages 2989-2998, 2023.
[19] Maximilian Jaritz, Jiayuan Gu, and Hao Su. Multi-view pointnet for 3d scene understanding. In Porc. of IEEE Intl. Conf. Computer Vision Workshops. pages 0-0, 2019.
11

---

[20] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In Proc. of IEEE Inil. Conf. on Computer Vision and Puttern Recognition, pages 4867-4876, 2020.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tote Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Porc. of IEEE Intl. Conf. on Computer Vision, pages 401 4015-4026, 2023.
[22] Maksim Kolodiazhnyi. Anna Vorontsova, Anton Konushin, and Danila Rukhovich. Top-down beats bottom-up 3d instance segmentation. In Proc. of IEEE Winter Conf. on Applications of Computer Vision, pages 3574, 2024.
[23] Maxim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, and Danila Rukhovich. Oneformer3d: One transformer for unified point cloud segmentation. Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2024.
[24] Theodora Kontogianni. Ekin Celikkan, Siyu Tang, and Konrad Schindler. Interactive object segmentation in 3d point clouds. In Proc. of the IEEE Int. Conf. on Robotics and Automation, pages 2891-2897, 2023,
[25] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jía. Stratified transformer for point cloud segmentation. In Proc. of IEEE Intl. Conf. on Computer Vision and Pottern Recognition, 2022.
[26] Xin Lai, Yuhui Yuan, Ruihang Chu, Yukang Chen, Han Hu, and Jiaya Jia. Mask-attention--fet transformer for 3d instance segmentation. In Porc. of IEEE Intl, Conf. on Computer Vision, pages 9693-3703, 2023,
[27] Loie Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. Proc. of IEEE Imtl. Conf. on Computer Vision and Puttern Recognition, pages 4558-4567, 2018.
[28] Seungjun Lee. Yuyang Zhao, and Gim Hee Lee. Segment any 3d object with language. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2024.
[29] Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. Omg-seg: Is one model good enough for all segmentation? In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2024.
[30] Dingkang Liang. Tianrui Feng, Xin Zhou, Yumeng Zhang, Zhikang Zou, and Xiang Bai. Parameter-efficient fine-tuning in spectral domain for point cloud learning. arXiv preprint arXiv:2410.08/1 2024.
[31] Dingkang Liang. Jiahao Xie, Zhikang Zou, Xiaoqing Ye, Wei Xu, and Xiang Bai. Crowdelip: Unsupervised crowd counting via vision-language model. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, pages 2893-2903, 2023.
[32] Dingkang Liang. Xin Zhou, Wei Xu. Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: simple state space model for point cloud analysis. In Pruc. of Advances in Neural Information Processing Systems, 2024.
[33] Zhihao Liang, Zhihao Li, Songeen Xu, Mingkui Tan, and Kui Jia. Instance segmentation in 3d scenes using semantic superpoint tree networks. Porc. of IEEE Inll Conf. on Computer Vision, pages 2783-2792 2021.
[34] Haojia Lin, Xiawu Zheng, Lijiang Li, Fei Chao, Shanshan Wang. Yan Wang, Yonghong Tian, and Rongrong Ji. Meta architecture for point cloud analysis. In Proc. of IEEE Intl. Conf. on Computer Vision and Puttern Recognition, pages 7682-17691, 2023.
[35] Youquan Liu, Lingdong Kong, Xiaoyang Wu, Runnan Chen, Xin Li, Liang Pan, Ziwei Liu, and Yuexin Ma. Multi-space alignments towards universal lidar segmentation. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2024.
[36] Jiahao Lu, Jiacheng Deng, Chuxin Wang, Jianfeng He, and Tianzhu Zhang. Query refinement transformer for 3d instance segmentation. In Porc. of IEEE Intl. Conf. on Computer Vision, pages
2023.
[37] Li Minghan, Li Shuai, Zhang Xindong, and Zhang Lei. Univs: Unified and universal video segmentation with prompts as queries. In Proc. of IEEF Intl. Conf. on Computer Vision and Pattern Recognition, 2024.
[38] Gaku Narita, Takashi Seno, Tomoya Ishikawa. and Yohsuke Kaji. Panopticfusion: Online volumetric semantic mapping at the level of stuff and things. In Proc. of the IEEE Int. Conf. on Intelligent Robots and Systems, pages 4205-4212, 2019.
[39] Tuan Due Ngo, Binh-Son Hua, and Khoi Nguyen. Isbnet: 3d point cloud instance segmentation network with instance-aware sampling and box-aware dynamic convolution. In Proc. of IEEE Intl. Conf. on Computer Vision and Puttern Recognition, pages 13550-13559, 2023.
[40] Phuc DA Nguyen, Tuan Duc Ngo, Chuang Gan, Evangelos Kalogerakis, Anh Tran, Cuong Pham, and Khoi Nguyen. Open3dis: Open- vocabulary 3d instance segmentation with 2d mask guidance. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2024.
12

---

[41] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi. Mare Pollefeys. Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Proc. of IEEE Inil. Conf. Computer Vision and Pattern Recognition. pages 815-824, 2023.
[42] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In Proc. of Advances in Neural Information Processing Systems, 2017.
[43] Lu Qi, Lehan Yang, Weidong Guo, Yu Xu, Bo Du, Varun Jampani, and Ming-Hsuan Yang. Unigs: Unified representation for image generation and segmentation. In Proc. of IEEE Intl. Conf. Computer Vision and Pattern Recogmition, 2024.
[44] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Proc. of Advances in Neural Information Processing Systems, pages 23192-23204, 2022.
[45] Zhipeng Qian, Yiwei Ma, Jiayi Ji, and Xiaoshuai Sun. X-refseg3d: Enhancing referring 3d instance segmentation via structured cross-modal graph neural networks. In Proc. of the AAAI Conf. on Artificial Intelligence, pages 4551 2024.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transfcrable visual models from natural language supervision. In Proc. of Inil. Conf. on Machine Learning, pages 8748-8763, 2021.
[47] David Rozenberszki, Or Litany, and Angela Dai. Language-grounded indoor 3d semantic segmentation in the wild. In Proc. of European Conference Computer Vision, pages 125-141, 2022.
[48] Jonas Schult. Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In Proc. of the IEEE Int. Conf. on Robotics and Automation, pages 8216-8223, 2023.
[49] Daniel Seichter, Benedict Stephan, Söhnke Benedikt Fischedick, Steffen Müllrr, Leonard Rabes, and Horst-Michacl Gross. Panoptiendt: Efficient and robust panoptic mapping. In Proc. of the IEEE Int. Conf. on Intelligent Robots and Systems, pages 7233 7240, 2023.
[50] Yawar Siddiqui. Lorenzo Porzi, Samuel Rota Buló, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, pages 9043-9052, 2023.
[51] Julian Straub. Thomas Whelan, Lingni Ma, Yufan Chen. Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren. Shobhit Verma, et al. The replica dataset: digital replica of indoor spaces. arXiv preprint arXiv: 057977 2019.
[52] Jiahao Sun, Chummei Qing, Junpeng Tan, and Xiangmin Xu. Superpoint transformer for 3d scene instance segmentation. In Proc. of the AAAÍ Conf. on Artificial Intelligence, volume 37, pages 2393 2201, 2023.
[53] Weiwei Sun, Daniel Rebain, Renjie Liao, Vladimir Tankovich, Soroosh Yazdani, K Kwang Moo Yi, and Andrea Tagliasacchi. Neuralbf: Neural bilateral filtering for top-down instance segmentation on point clouds. In Proc. of IEEE Winter Conf. on Applications of Computer Vision, pages 551-560, 2023.
[54] Ayca Takmaz, Elisabetta Fedele, Robert Sumner, Mare Pollefeys, Federico Tombari, and Francis En- gelmann. Openmask3d: Open-vocabulary 3d instance segmentation. In Proc. of Advances in Neural Information Processing Systems, 2023.
[55] Thang Vu, Kookhoi Kim, Tung M Luu,, Thanh Nguyen, and Chang D Yoo. Softgroup for 3d instance segmentation point clouds. In Proc. of IEFE Intl. Conf. on Computer Vision and Pattern Recognition, pages 2708 2717, 202.. 2022.
[56] Peng- Leng-Shuai Wang. Octformer: Octree-based transformers for 3d point clouds. ACM Transactions ON Graphics, 42(4):1-11, 2023.
[57] Changli Wu, Qi Chen, Jiayi Ji, Haowei Wang, Yiwci Ma, You Huang, Gen Luo, Hao Fei, Xiaoshuai Sun, and Rongrong Ji. Ra- Rulc-guided spatial awareness network for end-to-end 3d referring expression segmentation. In Proc. of Advances in Neural Information Processing Systems, 2024.
[58] Changli Wu, Yihang Liu, Jiayi Ji. Yiwei Ma. Haowei Wang, Gen Luo, Henghui Ding. Xiaoshuai Sun, and Rongrong Ji. 3d-gres: Generalized 3d referring expression segmentation. In Proc. of ACM Multimedia, 2024.
[59] Changli Wu, Yiwei Ma. Qi Chen, Haowei Wang, Gen Luo, Jiayi Ji, and Xiaoshuai Sun. 3d-stmn: Dependency-driver superpoint-ter text matching network for end-to-end 3d referring expression segmentation. In Proc. of the AAAI Conf. Artificial Intelligence, volume 38, pages 5940-594 0900-948,. 2024.
[60] Shun Chhen Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scenegraphfusson: Incremental 3d scene graph prediction from rgb-d sequences. In Proc. of IEEE Intl, Conf. on Computer Vision and Pattern Recognition, pages 7515-7525, 2021.
13

---

[61] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, pages 9621-9330, 2019.
[62] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition based pooling, In Proc. of Advances in Neural Information Processing Systems, pages 33330 33342, 2022.
[63] Yanmin Wu, Qiankun Gao, Renrui Zhang, and Jian Zhang. Language-assisted 3d scene understanding. In Proc. of the AAAI Conf. on Artificial Intelligence, 2024.
[64] Yizheng Wu, Min Shi, Shuaiyuan Du, Hao Lu, Zhiguo Cao, and Weicai Zhong. 3d instances 1d kernels. In Proc. of European Conference on Computer Vision, pages 235-252, 2022.
[65] Chenfeng Xu, Bichen Wu, Zining Wang. Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezeseyy3: Spatially-adattive convolution for efficient point-cloud segmentation. In Proc. of European Conference on Computer Vision, pages 1-19, 2020.
[66] Mingye Xu, Mutian Xu, Tong He, Wanli Ouyang, Yali Wang, Xiaoguang Han, and Yu Qiao. Mm 3dscene: 3d scene understanding by customizing masked modeling with siformative-preservee reconstruction and self-distilled consistency. In Proc. of IEEE Intl. Conf. on Computer Vision and Puttern Recognition, pages 4380-4390, 2023.
[67] Yunhan Yang. Xiaoyang Wu, Tong He, Hengshuang Zhao, and Xihui Liu. Sam3d: Segment anything in 3d scenes. Porc. of IEEE Intl. Conf. on Computer Vision Workshops. 2023.
[68] Zhiliu Yang and Chen Liu, Tupper-maps Temporal and unified panoptic perception for 3d metric-semantic mapping. In Proc. of the IEEE Int. Conf. on Intelligent Robots and Systems, pages 1094-1101, 2021.
[69] Li Yi, Wang Zhao, He Wang, Minhyuk Sung, and Leonidas J Guibas. Gspn: Generative shape proposal network for 3d instance segmentation in point cloud. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, pages 1947-3956 2019.
[70] Yuanwen Yuc, Sabarinath Mahadevan, Jonas Schult, Francis Engelmann, Bastian Leibe, Konrad Schindler, and Theodora Kontogianni. Agilc3d: Attention guided interactive multi-object 3d segmentation. In Proc. of Intl. Conf. on Learning Representations, 2024.
[71] Dingyuan Zhang, Dingkang Liang. Hongcheng Yang. Zhikang Zou, Xiaoqing Ye, Zhe Liu, and Xiang Bai. Sam3d: Zero-shot 3d object detection via segment anything model. Science China Information Sciences, 2023.
[72] Dingyuan Zhang, Dingkang Liang, Zhikang Zou, Jingyu Li, Xiaoqing Ye. Zhe Liu, Xiao Tan, and Xiang Bai. A simple vision transformer for weakly uui-supereeied 3d object detection. In Porc. of IEEE Inil. Conf. on Computer Vision, pages 8373-8330 0383, 2023.
[73] Hengshuang Zhao, Jiang, Jiaya Jia, Philip HIS Torr, and Vladlen Koltun. Point transformer. Porc. of IEEE Insl. Conf. on Computer Vision, pages 16259-16268, 2021.
[74] Weiguang Zhao, Yuyao Yan, Chaolong Yang. Jianan Ye, Xi Yang, and Kaizhu Huang. Divide and conquer: point cloud instance segmentation with point-wise wise binarization. Porc. of IEEE Inil, Conf. on Computer Vision, pages 562-571, 2023.
[75] Xin Zhou, Dingkang Liang, Wei Xu, Xingkui Zhu, Yihan Xu, Zhikang Zou, and Xiang Bai. Dynamic adapter meets prompt tuning: Parameter-efficient transfer learning for point cloud analysis. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, pages 14707-14717, 2024.
[76] Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Porc. of IEEE Intl. Conf. Computer Vision, pages 2911-2921, 2023.
14

---

Appendix
In this appendix, we provide additional content to complement the main manuscript:
Appendix A: Comparisons employing more metrics on specific tasks.
Appendix B: Inference time analysis of the proposed UniSeg3D.
Appendix C: Qualitative visualizations illustrating model effectiveness.
A Comparisons employing more metrics on specific tasks.
The experiments presented in the main manuscript primarily use overarching metrics to measure performance on each task. This section provides more comprehensive comparisons of our method on each task using detailed metrics. We train the model on ScanNer20 and assess its open-vocabulary segmentation performance on ScanNer200. Following [40], 51 classes ScanNer200 that are semantically similar to annotated classes in ScanNet20 are grouped as Base classes, while the remaining classes are divided as Novel classes. The model is then directly tested on Replica [51] to evaluate its zero-shot segmentation performance.
Table I: Comparison with existing instance seg- mentation methods on ScanNet20. UniSeg3D achieves highly competitive performance.
Table II: Comparison with previous 3D interactive segmentation methods on ScanNet20. UniSeg3D presents remarkable performance in terms of three metrics.
Method
Reference mAP2s mAPso mAP
3D 3D-SIS[12]
CVPR 19
35.7
18.7
GSPN[69]
CVPR 19
53.4
37.8
19.3
Method
Reference
AP
AP30
AP2s
PointGroup [20]
CVPR
71.3
56.7
34.8
InterObject3D [24]
ICRA 23
20.9
38.0
67.2
OccuSeg[8]
CVPR
71.9
60.7
44.2
AGILE3D [70]
ICLR
53.5
91.3
DyCo3D[9]
CVPR 21
72.9
57.6
35.4
UniSeg3D (ours)
54.5
79.4
93.2
SSTNet[33]
ICCV 21
74.0
64.3
49.4
HAIS[2]
ICCV 21
75.6
64.4
43.5
Table III: Comparison with existing 3D referring
DKNet[64]
ICCV 22
76.9
66.7
50.8
segmentation methods ScanRefer. UniSeg3D
SoftGroup(55)
CVPR
78.9
67.6
45.8
demonstrates notable performance in terms of mloU
PBNe(74]]
ICCV 23
78.9
70.5
54.3
[SBNet[39]
CVPR 23
82.5
73.1
54.5
and acc@0.25.
SPFormer[52]
AAAI 23
82.9
73.9
56.3
Method
Reference mloU acc@0.5 acc @0.25
Mask3D[44]
ICRA
83.5
73.7
55.2
MAFT[26]
ICCV 23
75.9
58.4
TGNN [15]
AAAI 21
24.9
28.2
33.2
QueryFormerr[6]
ICCV 23
83.3
74.2
56.5
X-RefSeg3D [45] AAAI 24
25.5
28.6
34.0
OneFormerDD[23]
CVPR 24
86.4
78.1
59.3
UniSeg3D (ours)
29.6
28.0
41.5
UniSeg3D (ours)
86.1
77.0
59.3
Table IV: Comparison with previous open-vocabulary segmentation methods on ScanNe200 and Replica. Our method outperforms existing approaches terms of AP.
ScanNer200
Replica
Method
Reference
AP
AP
AP APHavel
AP
APso
AP25
OpenScene [41] with [48]
CVPR 23
8.5
11.1
7.6
10.9
15.6
17.3
OpenMask3D[[4]
NeurlPS 23
12.6
14.3
11.9
13.1
18.4
24.2
SOLE[28]
CVPR 24
18.7
17.4
19.1
Open3DIS[40]
CVPR 24
19.0
25.8
16.5
18.5
24.5
UniSeg3D (ours)
19.7
24.4
18.0
19.1
24.1
29.2
15

---

B Inference time analysis of the proposed UniSeg3D.
This work proposes unified framework, achieving six tasks in one inference, which would be more efficient than running six task-specific approaches individually. We present the inference time of the proposed method for efficiency analysis. Tab. V illustrates that our method achieves effective unification across six tasks while maintaining highly competitive inference times compared to previous methods.
Table Inference time and instance segmentation performance on the ScanNet20 validation split.
Component time, ms
Total
Method
Component
Device
time, mAP
Backbone
GPU
PointGroup [20]
Grouping
GPU+CPU
218
372
34.8
ScoreNet
GPU
106
Backbone
GPU
50
HAIS [2]
Hierarchical aggregation
GPU+CPU
116
256
43.5
Intra-instance refinement
GPU
90
Backbone
GPU
48
SoftGroup [55]
Soft grouping
GPU+CPU
266
45.8
Top-down refinement
GPU
Superpoint extraction
CPU
Backbone
GPU
26
SSTNet [33]
400
49.4
Tree Network
GPU+CPU
148
ScoreNet
GPU
58
Mask3D [48]
Backbone
GPU
106
Mask module
GPU
100
221
$4.3
w/o clustering
Query refinement
GPU
Backbone
GPU
106
Mask module
GPU
100
Mask3D [48]
19851
55.2
Query refinement
GPU
15
DBSCAN clustering
CPU
19630
Superpoint extraction
CPU
168
Backbone
GPU
26
SPFormer [52]
215
56.3
Superpoint pooling
GPU
Query decoder
GPU
Superpoint extraction
CPU
Backbone
GPU
26
OncFormer3D [23]
221
59.3
Superpoint pooling
GPU
4
Query decoder
GPU
23
Superpoint extraction
CPU
168
Backbone
GPU
33
UniSeg3D (ours)
230.03
59.3
Text encoder
GPU 0.03
Mask decoder
GPU
29
C Qualitative visualizations illustrating model effectiveness.
We provide qualitative results in this section. In Fig. I, visualizations of multi-task segmentation results are presented, showcasing point clouds, ground truth, and predictions within each scene. In Fig. II, present visualizations of predictions from UniSeg3D and current SOTA methods. In Fig. III, we test our model on open-set classes not included in training data to evaluate the model's open capability. Furthermore, we even replace the class names with attribute descriptions the open vocabulary, and impressively, we observe the preliminary reasoning capabilities of our approach.
16

---

Input
GT
Prod
Input
GT
Prod
Instance Seg.
Semantic Seg.
Panaptic Sog.
Interactive
Sog.
Referring
Seg.
OV
Figure Visualization of segmentation results obtained by UniSeg3D on ScanNet20 validation split.
Inst.
Sem.
Pan.
Inter.
Ref.
OV
Input
GT
UniSeg3D (Ours)
OneForme33D
AGILE3D
X RefSeg3D
Open3DIS
Current SOTA
Figure II: Visualization of segmentation results obtained by UniSeg3D and current SOTA methods on ScanNet20 validation split.
shaie?
"Hax tleee"
Figure III: Visualization of open capabilities. Red prompts involve categories not presented in the ScanNer200 annotations, while blue prompts describe attributes of various objects, such as affordances and color.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See limitation part.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
See experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See experiments part.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [NA]
Justification:
The code will be made available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececccccccccct /ttp:///ggiieeeeeeeuuuubiiiiiiii
/CodeSubmic ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See experiments part.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See experiments part.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See experiments part.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https / http///eeerpp/eeeeee cc
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethic.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
21

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We will release the code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
22

---

Answer:
[NA]
Justification:
We use the public assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "fLp9KDQq7L", "reviewer": "Reviewer_r1wg", "summary": "For the first time, this work proposes a unified model for several point cloud segmentation tasks, including panoptic, semantic, instance, OV, interactive, and referring segmentation. This work uses the typical query-based transformer perception architecture with the proposed knowledge distillation losses for superior performance, outperforming previous methods, which shows the effectiveness of the designed model.", "questions": "Some implementations of the method are unclear.\n1. The detailed process of producing super points.\n2. The criteria for sampling the corresponding points from the clicks.\n3. For the referring segmentation, how to decide the target segmentation mask from many text tokens?\n4. For L182-186, I am confused that isn't formula-5 already satisfied by using formula-4?", "limitations": "See above.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This work addresses an important problem of 3D scene perception with a unified model and achieves great results.\n2. The proposed model is simple and effective.\n3. The overall writing is fluent and clear.", "weaknesses": "1. The main weakness of this work is that the proposed architecture is widely used for multi-modal perception tasks, limiting the model’s novelty. However, this is acceptable as long as the model’s performance is indeed great, as simple and effective models are usually similar.\n\n2. The paper claims knowledge distillation losses as one of its contributions. However, according to the ablations, adding these losses only makes marginal changes.\n\nStill, I think this work has a solid contribution to the field (if open-sourcing the codes with reproducible results) and should be accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for providing feedback and taking the time to review our work! **We promise that the training/inference codes, logs, and checkpoints will be released**.\n\n**Weaknesses:**\n\n- **To Weakness 1**: “The main weakness of this work is that the proposed architecture is widely used for multi-modal perception tasks, limiting the model’s novelty. However, this is acceptable as long as the model’s performance is indeed great, as simple and effective models are usually similar.”\n\n**Reply:** Thanks. This paper aims to establish a simple method to unify various 3D understanding tasks. Thus, we make the architecture as simple as possible, which can serve as a solid baseline for future unified research. We also appreciate that the reviewer agrees the simplicity of the proposed architecture is acceptable.\n\n- **To Weakness 2**: “The paper claims knowledge distillation losses as one of its contributions. However, according to the ablations, adding these losses only makes marginal changes.”\n\n**Reply:** Thanks. We first construct a unified baseline that achieves highly competitive performance on six different tasks. Based on this baseline, we further propose knowledge distillation losses, which not only maintain the consistent SOTA performance on referring segmentation and open-vocabulary segmentation tasks but also improve the panoptic segmentation, instance segmentation, and interactive segmentation tasks by notable 0.8 PQ, 1.3 mAP, and 2.1 AP, respectively. To fully explore its effectiveness, we choose the representative instance segmentation task for discussing the results on strict metrics (i.e., mAP$_{75}$), as shown below. We can see that the proposed knowledge distillation losses will **bring more noticeable performance gains (+2.3%) under the strict metric** compared with the baseline.\n\n| Method | mAP | mAP$_{50}$ | mAP$_{75}$ |\n| --- | --- | --- | --- |\n| Baseline | 58.0 | 76.2 | 54.5 |\n| Baseline w/ knowledge distillation losses | __59.3__(+1.3) | __76.9__(+0.7) | __56.8__(+2.3) |\n\n**Questions:**\n\nSorry for the unclear details. We give detailed explanations for each question in the following content.\n\n- **To Question 1**: “The detailed process of producing super points.”\n\n**Reply:** The superpoints are produced in an unsupervised manner, and they are formed by grouping points with locally similar geometric structures. Specifically, our procedures for producing superpoints can be divided into three steps:\n\nStep 1: generate handcrafted local geometric features of the 3D points;\nStep 2: employ the farthest point sampling (FPS) algorithm in the coordinate space of point clouds to obtain the initial superpoint centers;\nStep 3: construct an association map between the points and superpoint centers.\n\nWe will add introductions of the detailed processes, producing superpoints in the revised version.\n\n- **To Question 2**: “The criteria for sampling the corresponding points from the clicks.”\n\n**Reply:** We sample the 3D points nearest to the clicks as the corresponding points. Specifically, in practice, clicks are clicked in 2D projections of 3D scenes. Therefore, the clicks are initially represented as 2D coordinates of the corresponding 2D projections. While reverse mapping the 2D projections into original 3D scenes, the clicks are concurrently reverse mapping to concrete 3D coordinates of the 3D scenes. Afterward, we sample points nearest to these concrete 3D coordinates as the corresponding points.\n\n- **To Question 3**: “For the referring segmentation, how to decide the target segmentation mask from many text tokens?”\n\n**Reply:** Each textual expression refers to a specific object segmentation mask, and the text tokens are individually encoded from respective textual expressions. We feed the text tokens into the mask decoder and generate respective mask predictions. Therefore, the target segmentation mask, i.e., the mask predictions, are naturally corresponded to the input text tokens in a one-to-one manner. With many text tokens, we decide the target segmentation masks to the individually respective input text tokens.\n\n- **To Question 4**: “For L182-186, I am confused that isn't formula-5 already satisfied by using formula-4?”\n\n**Reply:** Sorry for the confused statement. Actually, while $\\mathbf{e}^{P} _ {i}\\mathbf{e}^{T} _ {i}<\\mathbf{e}^{P} _ {i}\\mathbf{e}^{T} _ {j}$, the formula-5 is not satisfied by using formula-4. Specifically, the formula-5 is:\n\n$\\mathcal{L} _ {rank}=\\frac{1}{B}\\sum _ {i=1}^{B}\\sum _ {j=1}^{B}\\mathrm{max\\left (0,\\mathbf{s} _ {i,j}-\\mathbf{s} _ {i,i} \\right)}$,\n\nwhere  $\\text{max}\\left ( 0,\\mathbf{s} _ {i,j}-\\mathbf{s} _ {i,i} \\right)=\n\\mathbf{s} _ {i,j}-\\mathbf{s} _ {i,i}$ \n if $\\mathbf{e}^{P} _ {i}\\mathbf{e}^{T} _ {i}<\\mathbf{e}^{P} _ {i}\\mathbf{e}^{T} _ {j}$, and  $0$ otherwise. $\\mathbf{e}^{P} _ \n {i}\\mathbf{e}^{T} _ {i}$ and $\\mathbf{e}^{P} _ {i}\\mathbf{e}^{T} _ {j}$ are input parameters of formula-4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nMyqtUoHfc", "reviewer": "Reviewer_C3ws", "summary": "This paper proposes UniSeg3D, a unified framework for six 3D segmentation tasks that achieves SOTA results on the six tasks. The authors propose to use knowledge distillation and ranking-based contrastive learning to enhance inter-task knowledge sharing and the overall performance. Extensive experiments are done to prove that UniSeg3D is powerful. Comprehensive ablation studies are performed to prove the effectiveness of the design.", "questions": "Please refer to the weaknesses.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. UniSeg3D is the first framework that unifies six tasks in 3D segmentation, and the comprehensive experimental results prove the effectiveness of the design.\n2. Interactive segmentation-guided training is insightful. Analysis of its impact on inter-task formulation is comprehensive, enlightening future directions.\n3. Well-written manuscripts with illustrative figures.", "weaknesses": "1. Since the feature for visual prompt is sampled from the superpoints, the quality of the visual prompt significantly influence the overall performance of the model.\n2. The experiments are only conducted on ScanNet-based datasets, as a unified model, the authors should provide more experiments on different datasets to validate the method as done in previous works[1][2].\n3. Unlike previous works[2][3], UniSeg3D directly learn the relation between text and 3D without any 2D supervision or guidance. It is a concern that how \"open\" is this framework in OVS task. Some visualized experiments on open-set text queries as in [2][3][4] could answer the question.\n\n\n[1] Zhu, Ziyu, et al. \"3d-vista: Pre-trained transformer for 3d vision and text alignment.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n[2] Nguyen, Phuc, et al. \"Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[3] Takmaz, Ayça, et al. \"Openmask3d: Open-vocabulary 3d instance segmentation.\" arXiv preprint arXiv:2306.13631 (2023).\n[4] Peng, Songyou, et al. \"Openscene: 3d scene understanding with open vocabularies.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback!\n- **To Weakness 1**: “Since the feature for visual prompt is sampled from the superpoints, the quality of the visual prompt significantly influence the overall performance of the model.”\n\n**Reply:** Thanks. For the interactive segmentation task, it is intuitive that using different visual prompts will report different results. To investigate its influence, we study the three variants, including random, center, and prompts provided by the SOTA method (AGILE3D [1]), and the performance is shown in the table below. We can find that under the fair setting, i.e., using the prompts from AGILE3D, the proposed UniSeg3D achieves notable performance improvement compared with the SOTA. For the other tasks, due to attention masks in cross attention layer, the visual prompt will not influence their performance. We will add these discussions in the revised version. \n\n| Method | Used sampling criteria | AP | AP$_{50}$ | AP$_{25}$ |\n| --- | --- | --- | --- | --- |\n| AGILE3D (ICLR 24) [1] | AGILE3D (ICLR 24) [1] | 53.5 | 75.6 | 91.3 |\n| UniSeg3D | AGILE3D (ICLR 24) [1] | 54.5 | 79.4 | 93.2 |\n| UniSeg3D | Center | __56.6__ | __82.1__ | __94.9__ |\n| UniSeg3D | Random | 51.3 | 75.2 | 89.6 |\n\n[1] AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation. ICLR 2024. \n\n- **To Weakness 2**: “The experiments are only conducted on ScanNet-based datasets, as a unified model, the authors should provide more experiments on different datasets to validate the method as done in previous works[1][2].”\n\n**Reply:** Good suggestion! To the best of our knowledge, ScanNet is the only publicly available dataset supporting six tasks. Thus, to explore a unified framework supporting six tasks in a single model, we mainly conduct experiments on the ScanNet-based datasets in the manuscript.\n\nUnfortunately, we note that the work[1] mainly focuses on visual grounding tasks, i.e., it is a 3D object detection task, while **this paper focuses on the various segmentation tasks**. The second work[2] explores the open-vocabulary segmentation task on the Replica dataset[3], one of our supported tasks. Hence, we follow work[2] to conduct zero-shot open-vocabulary segmentation on the Replica[3] dataset. The Table below demonstrates that the UniSeg3D surpasses the current SOTA approach Open3DIS.\n\n| Method | AP | AP$_{25}$ |\n| --- | --- | --- |\n| OpenScene (CVPR 23) | 10.9 | 17.3 |\n| OpenMask3D (NeurIPS 23) | 13.1 | 24.2 |\n| Open3DIS (CVPR 24) [2] | 18.5 | 28.2 |\n| UniSeg3D (Ours) | __19.1__ | __29.2__ |\n\nAs a supplement, we conduct experiments on the S3DIS dataset. Note that the S3DIS dataset does not provide text expressions, i.e., it can not support the referring segmentation task. Here, we evaluate the panoptic, semantic, instance, and interactive segmentation tasks on the S3DIS dataset. We can see that UniSeg3D outperforms current specialized SOTA approaches, especially for the challenging panoptic segmentation task, verifying the effectiveness of our method. We will include these new comparisons in the revised version.\n\n|  | Panoptic Segmentation | Semantic Segmentation | Instance Segmentation | Interactive Segmentation |\n| --- | :---: | :---: | :---: | :---: |\n| Method | PQ | mIoU | mAP | AP |\n| PointNeXt-XL (NeurIPS 22) | - | 70.5 | - | - |\n| PointTransformerV2 (NeurIPS 22) | - | 71.6 | - | - |\n| PBNet (ICCV 23) | - | - | 53.5 | - |\n| Mask3D (ICRA 23) | - | - | 57.8 | - |\n| OneFormer3D (CVPR 24) | 62.2 | 72.4 | __58.7__ | - |\n| UniSeg3D (Ours) | __65.7__ | __72.5__ | __58.7__ | __29.9__ |\n\n[1] 3d-vista: Pre-trained transformer for 3d vision and text alignment. ICCV 23.\n\n[2] Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance. CVPR 24.\n\n[3] The Replica dataset: A digital replica of indoor spaces. arXiv 19.\n\n- **To Weakness 3**: “Unlike previous works[2][3], UniSeg3D directly learn the relation between text and 3D without any 2D supervision or guidance. It is a concern that how \"open\" is this framework in OVS task. Some visualized experiments on open-set text queries as in [2][3][4] could answer the question.”\n\n**Reply:** Thanks for admiring the distinctiveness of our work! The proposed UniSeg3D directly learns relations between text and 3D, mitigating the dependencies on 2D supervision. Previous works[2][3][4] mainly employ two experiment settings to evaluate how “open” they are, i.e., using random open-set text prompts on the ScanNet200 dataset and the cross-dataset evaluation. Following your suggestion, we visualize the open-vocabulary segmentation results under these two experiment settings. **Please see Figure-R 1 and Figure-R 2 in the attached PDF for details**. As we can see, the UniSeg3D presents the applicability from the following aspects:\n\n(1) support open-set classes which are not included in the training data (illustrated in Figure-R 1);\n\n(2) support attribute descriptions such as affordance and color (illustrated in Figure-R 1);\n\n(3) support cross-dataset point cloud data (illustrated in Figure-R 2).\n\nSince the above three applications, we think the UniSeg3D presents a desirable performance in how “open.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xtneGnT3Uj", "reviewer": "Reviewer_PEfy", "summary": "This work proposes UniSeg3D, a framework to unify 3D point cloud segmentation tasks. Compared to previous work that unifies 3 tasks, UniSeg3D additionally incorporates interactive segmentation, text-referring segmentation, and open-vocabulary segmentation. In total, six tasks are unified in a single Transformer decoder architecture, and techniques such as knowledge distillation, contrastive learning, and two-stage fine-tuning are applied to boost performance. Consequently, UniSeg3D achieves comparable or superior performance against existing state-of-the-art baselines.", "questions": "- Equation 3: The current notation is for cross products while it looks like dot products are intended.\n- Table 1: where does the Table 1 come from?\n- Notations: $e^P$ and $e^T$ in Equation 4 are not the same as the $e$'s in Figure 2 and there are no corresponding notations in Figure 3. Consider using a different notation or adding corresponding legends in Figures 2 and 3 for better clarity.\n- Section 3.3: the motivation of inter-task association is not clearly stated, how exactly can the tasks benefit each other? For example, how can referring segmentation benefit interactive segmentation?", "limitations": "The authors have discussed their limitations in Section 5. Given the paper is titled A Unified Framework for 3D Scene Understanding, the reviewer would like to point out that point cloud segmentation is one aspect of 3D scene understanding, so it would also be worth discussing how such a unified segmentation framework can potentially help other 3D understanding tasks.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Novel architecture for incorporating six segmentation tasks under one single model, which appears to be effective and flexible for extending to additional tasks.\n2. Solid experiments. The authors conducted experiments on 3 datasets and various ablation studies to demonstrate the effectiveness of the proposed method.", "weaknesses": "1. Marginal performance gain. While referring segmentation and interactive segmentation tasks enjoy noticeable improvements in UniSeg3D, the performance of generic segmentation tasks seems to be on par or worse than training alone (see Table 1). This led to questions about the motivation for the proposed unification. Further justifications, such as the standard deviation of the performances, could make the results more convincing.\n2. Additionally,  in Table 6, unifying the additional 3 tasks turns out to hurt the performance of generic segmentation tasks. How do the authors justify their motivation?\n3. Some minor issues in writing and notations. See the Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for providing feedback and taking the time to review our work!\n\n- **To Weakness 1 & 2**: “Marginal performance gain...” / “Additionally, in Table 6, unifying…”\n\n**Reply:** Good question! Unifying the tasks into a single model could save computation consumption and benefit real-world applications, which has become a trend in the community [1][2]. In this paper, we aim to achieve six 3D segmentation tasks within a single model in one inference. We first propose a simple baseline unifying such six tasks and find that it can not achieve ideal performance, and we argue that the reasons are from two aspects: challenges of creating unified representations and difficulties of multi-task joint optimization for these tasks. We would like to clarify that the goal of Table 6 is to prove multi-task unification a challenging topic instead of our final results, e.g., introducing new tasks might hurt the performance of existing tasks. To relieve these impacts, we propose UniSeg3D, which uses queries to unify the representations and employs knowledge distillation with ranking-based contrastive learning to jointly optimize associated tasks. Extensive experiments demonstrate that our method **not only supports six tasks but also surpasses currently specialized SOTA approaches on all tasks**. Besides, following your suggestion, we provide the standard deviation of UniSeg3D by training 3 times. Referring to the Table below, we report slight standard deviations on six tasks.\n\n| Times | Panoptic Segmentation (PS) | Semantic Segmentation (SS) | Instance Segmentation (IS) | Interactive Segmentation (Inter. Seg.) | Referring Segmentation (Ref. Seg.) | Open-vocabulary Segmentation (OVS) |\n| --- | --- | --- | --- | --- | --- | --- |\n| 1 | 71.3 | __76.9__ | __59.3__ | 54.5 | __29.6__ | 19.7 |\n| 2 | __71.4__ | 76.6 | 59.2 | 54.3 | 29.4 | __19.8__ |\n| 3 | 71.2 | 76.8 | 59.2 | __54.6__ | 29.5 | 19.7 |\n| Overall | 71.30±0.10 | 76.77±0.15 | 59.23±0.06 | 54.47±0.15 | 29.50±0.10 | 19.73±0.06 |\n\n[1] UniGS: Unified Representation for Image Generation and Segmentation. CVPR 24.\n\n[2] UniVS: Unified and Universal Video Segmentation with Prompts as Queries. CVPR 24.\n\n- **To Weakness 3**: “Some minor issues…”\n\n**Reply:** Thank you for pointing out these minor issues. We will carefully revise them in the next version.\n\n- **To Question 1**: “Equation 3: The current notation…”\n\n**Reply:** Sorry for this unclear statement. Actually, they are indeed cross products. We will add descriptions in the revised version. \n\n- **To Question 2**: “Table 1: where does…”\n\n**Reply:** Thanks! To make full use of knowledge contained in superior 3D scene understanding tasks, we propose knowledge distillation to transfer knowledge from superior tasks to other tasks, which improves multi-task performance without extra inference costs. The key to knowledge distillation is to utilize the task-predicting segmentation masks of the best quality to guide the other tasks, i.e., using a teacher to guide the students. Considering that visual prompts can provide location priors indicting precise object positions, naturally making the interactive task perform best in mask prediction. To prove this, we provide quantitative results (mIoU) in Table 1 of the manuscript. Essentially, the main difference between instance and interactive segmentation is w/o or w/ visual prompt. Here, we further compare the instance segmentation with interactive segmentation. Referring to the Table below, the interactive segmentation significantly outperforms instance segmentation by 7.9% mIoU, indicating the superior quality of predicted masks provided by the interactive segmentation task. We will add these discussions to make Table 1 more clear in the revised version.\n\n| Task | mIoU |\n| --- | --- |\n| Instance segmentation (w/o visual prompt) | 68.1 |\n| Interactive segmentation (w/ visual prompt) | __76.0__(+7.9) |\n\n- **To Question 3**: “Notations: $𝑒^{𝑃}$ and $𝑒^{𝑇}$ in Equation 4…”\n\n**Reply:** Good suggestion! Following your suggestion, we now add the corresponding legends in Figure 3 for better clarity, **which can be found in Figure-R 3 of the attached PDF**.\n\n- **To Question 4**: “Section 3.3: the motivation…”\n\n**Reply:** Sorry for this unclear statement. In this paper, we propose the knowledge distillation loss to implement inter-task association, i.e., transfer the 3D scene understanding knowledge from the superior task (a good teacher) to the rest of the tasks. Therefore, considering that the visual prompts provide strong object location priors (see the reply to question 2), we employ interactive segmentation as the teacher. We use this teacher to provide auxiliary guidance, which improves multi-task performance without extra inference costs. At the same time, we would like to clarify that we mainly employ the interactive segmentation as the teacher, instead of the referring segmentation, to benefit other tasks (including the referring segmentation). The experimental results prove that under such inter-task association, we achieve considerable improvements compared with the baseline.\n\n| Method | PS | SS | IS | Inter. Seg. | Ref. Seg. | OVS |\n| --- | --- | --- | --- | --- | --- | --- |\n| Baseline | 70.4 | __76.2__ | 58.0 | 54.5 | 29.1 | __19.7__ |\n| Baseline w/ knowledge distillation loss | __71.2__ (+0.8) | __76.2__(+0.0) | __59.3__ (+1.3) | __56.6__ (+2.1) | __29.2__ (+0.1) | 19.6 (-0.1) |\n\n- **To Limitation**:\n\n**Reply:** Titling 3D segmentation research as 3D scene understanding has been a common practice in this community[1][2]. We will explore the relationships between 3D segmentation and other 3D scene understanding tasks in the future. Besides, UniSeg3D can provide pixel-level semantic information, which might serve as potential reliable input for other 3D understanding tasks.\n\n[1] OpenScene: 3D Scene Understanding With Open Vocabularies. CVPR 23.\n\n[2] Exploring Data-Efficient 3D Scene Understanding With Contrastive Scene Contexts. CVPR 21.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gNGMyrCSWC", "reviewer": "Reviewer_3AFf", "summary": "The paper proposes a unified 3D segmentation frame work for six 3D segmentation tasks. It enhance the performance through building the inter-task connections. The model could achieve the state-of-the-art performance in individual tasks even comparing to the models specialized for individual tasks.", "questions": "1. As is mentioned in the weaknesses, how do you compare models/design choice when the evaluation metrics for different tasks are not consistent? For example in table 4(a), how would you compare line 2 and line 4? Which one is better?\n2. The design choice of Mask Decoder Layer is weird. Why do you choose to do a cross-attention first then followed by a self-attention? Also, is there duplicate information when doing the cross-attention between q and f_i, since q_u in q is obtained from f_i? Why not do the self-attention for q_u and cross-attention with q_p and q_t?\n3. Table 6 shows that unifying the tasks are hurting the performance, then what is the point of unifying all the tasks together? Adding new tasks (OVS, Referring, Interactive) is hurting PS, SS, IS. Would these new tasks help PS, SS, IS?\n4. As is mentioned in the weaknesses, table 1 could not support the claim that interactive segmentation is superior in mask prediction. Also, is there any intuition why this specific segmentation task is superior?", "limitations": "The paper addresses the limitation and claims there is no societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed model unifies 6 different 3D segmentation task in one framework, with no modules specialized for specific task. The authors build explicit inter-task relations to further improve the performance of individual tasks. Detailed experiments and comparison are reported.", "weaknesses": "1. The writing needs improvement. Some notation and writing flow are confusing. For example, K_v in line 155 is not introduced. The multiplication in equation (3) is not clear (is it a matrix multiplication?). Positive sample and negative samples are not introduced in section 3.2 (line 206). mask_pos in equation (6) is not introduced.\n2. No explanation on how to compare models/design choice when the evaluation metrics for different tasks are not consistent. For example, if model A is better in interactive segmentation while model B is better in referring segmentation, how to compare?\n3. Without the extra fine-tunning trick, the proposed model has a worse performance over previous SOTA OneFormer3D [16] on SS and IS tasks (table 2). According to Table I and IV in appendix, the performance of the proposed method is questionable.\n4. The authors empirically find the interactive segmentation is the best task for mask predictions (line 57-58) but no analysis on the reason. Only one value is reported in table 1, which could not validate interactive segmentation has superior performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for providing feedback and taking the time to review our work!\n\n- **To Weakness 1**: “The writing needs improvement...”\n\n**Reply:** Thank you for carefully reading this paper. We apologize for any confusion regarding notations. Specifically, $K_{v}$ refers to the length of the class name vocabulary; the Multiplication in equation (3) is indeed a matrix multiplication; Positive and negative samples denote samples successfully matched or miss-matched with labels, respectively; and $\\mathbf{mask}_{pos}$ in equation (6) represents the mask predictions from the positive samples. We will carefully revise these notations in the revised version.\n\n- **To Weakness 2 & Question 1**: “…models/design choice…different tasks are not consistent…model A is better…model B…” / “…compare models/design choice…table 4(a)…compare line 2 and line 4…”\n\n**Reply:** UniSeg3D is the first model unifying six 3D segmentation tasks. OneFormer3D is the only existing unified model for 3D segmentation, covering just three tasks. To compare comprehensively and demonstrate our model's superiority, we first evaluate UniSeg3D on OneFormer3D's three tasks, selecting the best checkpoint. This checkpoint is used to test interactive, referring, and open-vocabulary tasks. Remarkably, without specific checkpoint selection, UniSeg3D outperforms current specialized SOTA methods. Additionally, different complete convergence checkpoints show minimal differences in performance, making both models viable. In practical applications, checkpoints can be chosen based on preferred tasks while maintaining good performance across other tasks.\n\nFor table 4(a), lines 2 and 4 both represent our methods. We empirically find that, with the fine-tuning trick, the setting of line 4 performs better compared to line 2  (see below table). Thus, we employ the setting of line 4 for our method. Note that the fine-tuning trick improves performance without extra inference costs.\n\n|  | Panoptic Segmentation  (PS) | Semantic Segmentation (SS) | Instance Segmentation (IS) | Interactive Segmentation (Inter. Seg.) | Referring Segmentation (Ref. Seg.) | Open-vocabulary Segmentation (OVS) |\n| --- | --- | --- | --- | --- | --- | --- |\n| Tab.4(a) line 2 w/ fine-tuning | __71.3__ | 76.4 | 59.1 | 54.2 | __29.6__ | __19.7__ |\n| Tab.4(a) line 4 w/ fine-tuning | __71.3__(+0.0) | __76.9__(+0.5) | __59.3__(+0.2) | __54.5__(+0.3) | __29.6__(+0.0) | __19.7__(+0.0) |\n- **To Weakness 3**: “Without the extra fine-tunning trick… Table I and IV in appendix…questionable.”\n\n**Reply:**  Thank you! **Without the trick, UniSeg3D already achieves highly competitive performance against OneFormer3D**, supporting three additional tasks. It consistently outperforms specialized SOTA methods in interactive, referring, and open-vocabulary segmentation before fine-tuning. We argue that achieving full convergence while optimizing six tasks simultaneously is challenging. Hence, we introduce a simple trick requiring only a few epochs of fine-tuning, allowing UniSeg3D to surpass all current specialized SOTA methods across six tasks. Importantly, this fine-tuning trick does not add any additional inference cost, making it a practical solution.\n\n|  | PS | SS | IS | Inter. Seg. | Ref. Seg. | OVS |\n| --- | --- | --- | --- | --- | --- | --- |\n| AGILE3D (ICLR 24) | - | - | - | 53.5 | - | - |\n| X-RefSeg3D (AAAI 24) | - | - | - | - | 25.5 | - |\n| Open3DIS (CVPR 24) | - | - | - | - | - | 19.0 |\n| OneFormer3D (CVPR 24) | 71.2 | __76.6__ | __59.3__ | - | - | - |\n| UniSeg3D w/o fine tuning trick | __71.3__(+0.1) | 76.3(-0.3) | 59.1(-0.2) | __54.1__(+0.6) | __29.5__(+4.0) | __19.6__(+0.6) |\n\nBesides, considering Table I and IV in the appendix, we would like to clarify two points: **1)** For the instance segmentation task, mAP is a more comprehensive metric than mAP$ _ {50}$  and mAP$ _ {25}$. UniSeg3D achieves the same mAP as OneFormer3D but shows notable improvement under stricter metrics (e.g., mAP$ _ {85}$, mAP$ _ {90}$), indicating more precise mask generation; **2)** For the referring segmentation task, mIoU is more comprehensive, and UniSeg3D outperforms the SOTA method X-RefSeg3D by 4.1 mIoU, proving UniSeg3D's superior performance in this task.\n\n| Method | mAP$_{85}$ | mAP$_{90}$ |\n| --- | --- | --- |\n| OneFormer3D (CVPR 24) | 41.0 | 29.3 |\n| UniSeg3D (Ours) | __42.9__(+1.9) | __30.9__(+1.6) |\n\n- **To Weakness 4 & Question 4**: “…interactive segmentation is the best task…no analysis….” / “…table 1 could not support the claim that interactive segmentation is superior in mask prediction…”\n\n**Reply:** Good suggestion! Visual prompts are crucial for interactive segmentation, providing precise object location priors that guide focus on foreground targets and reduce background noise. The interactive segmentation can be considered as an extension of instance segmentation with the addition of strong object location priors. Consequently, it excels in mask prediction, achieving 76.0 mIoU, 7.9 higher than the 68.1 mIoU of instance segmentation. We will add discussions and this comparison to Table 1 in the next version for clarity. \n\n|  | mIoU |\n| --- | --- |\n| IS (w/o visual prompt) | 68.1 |\n| Inter. Seg. (w/ visual prompt) | __76.0__(+7.9) |\n\n**For questions 2 & 3, please see the reply presented in the “comment” part.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely thank the reviewers for their thoughtful comments and feedback. We are encouraged that the reviewers appreciate the simple, novel architecture and insightful module of UniSeg3D, solid experiments of the proposed method, well-written of the paper. We provide detailed responses to each reviewer, respectively, and promise we will incorporate all feedback in the revised version.

Best regard,

Paper 1782 Authors

Meta-review: This paper presents a multi-purpose framework for 3D scene segmentation, allowing panoptic, semantic, instance, and other types of segmentation tasks by a single model. The reviewers acknowledge that unifying all these tasks is novel, and note the simplicity of the model, the clarity of the writing, and the strong results. The paper underwent improvements during the rebuttal phase, adding experiments on S3DIS and adding more baselines. Reviewer 3AFf provided constructive criticism on the writing, which the authors are encouraged to make use of. Given these positive reviews, the AC’s decision is to accept. Congratulations to the authors on the strong submission! One minor suggestion (by the AC) is to make the title more unique, considering (as mentioned in the paper) this is not the only “unified framework for 3D scene understanding".
Decision: Accept (poster)