Unified 3D Segmenter As Prototypical Classifiers
Authors: Zheyun Qin
Cheng Han
Qifan Wang
Xiushan Nie
Yilong Yin
Xiankai Lu
Venue: NeurIPS 2023 poster (2023)
Keywords: Point Cloud Segmentation
Prototypical Classifier
Unified Framework
TL;DR: The paper introduce a unified prototype-based model that treats point cloud semantic, instance, and panoptic segmentation tasks as a classification problem.

Abstract: The task of point cloud segmentation, comprising semantic, instance, and panoptic segmentation, has been mainly tackled by designing task-specific network architectures, which often lack the flexibility to generalize across tasks, thus resulting in a fragmented research landscape. In this paper, we introduce ProtoSEG, a prototype-based model that unifies semantic, instance, and panoptic segmentation tasks. Our approach treats these three homogeneous tasks as a classification problem with different levels of granularity. By leveraging a Transformer architecture, we extract point embeddings to optimize prototype-class distances and dynamically learn class prototypes to accommodate the end tasks. Our prototypical design enjoys simplicity and transparency, powerful representational learning, and ad-hoc explainability. Empirical results demonstrate that ProtoSEG outperforms concurrent well-known specialized architectures on 3D point cloud benchmarks, achieving 72.3%, 76.4% and 74.2% mIoU for semantic segmentation on S3DIS, ScanNet V2 and SemanticKITTI, 66.8% mCov and 51.2% mAP for instance segmentation on S3DIS and ScanNet V2, 62.4% PQ for panoptic segmentation on SemanticKITTI, validating the strength of our concept and the effectiveness of our algorithm.  The code and models are available at https://github.com/zyqin19/PROTOSEG.

Unified 3D Segmenter As Prototypical Classifiers
Zheyun Qin Cheng Han?; Qifan Wang', Nie Xiushan', Yilong Yin Xiankai Lult Shandong University, "Rochester Institute of Technology, MMeta AI, "Shandong Jianzhu University
Abstract
The task of point cloud segmentation, comprising semantic, instance, and panoptic segmentation, has been mainly tackled by designing task-specific network archi- tectures, which often lack the flexibility to generalize across tasks, thus resulting in a fragmented research landscape. In this paper, we introduce PROTOSEG, a prototype-base baaoe model that unifies semantic, instance, and panoptic segmentation tasks. Our approach treats these three homogeneous tasks as a classification prob- lem with different levels of granularity. By leveraging a Transformer architecture, we extract point embeddings to optimize prototype-class distances and dynamically learn class prototypes accommodate the end tasks. Our prototypical design enjoys simplicity and transparency, powerful representational learning, and ad-hoc explainability. Empirical results demonstrate that PROTOSEG outperforms concur- rent well-known specialized architectures on 3D point cloud benchmarks, achieving 72.3%, 76.4% and 74.2% mIoU for semantic segmentation on S3DIS, ScanNet V2 and SemanticKITTI, 66.8% mCov and 51 51.2% mAP for instance segmentation on S3DIS and SeanNet V2, 62.4% PQ for panoptic segmentation on SemanticKITTT, validating the strength of our concept and the effectiveness of our algorithm. The code and models are available https: //gi thub. Conp/yyqn19//POOO0
1 Introduction
Point cloud segmentation entails partitioning a collection of 3D data points into meaningful categories to interpret complex 3D scenes. Three classical subdivision segmentation tasks are involved: semantic segmentation, which assigns a class label to each point based on its class; instance segmentation, which distinguishes between individual instances of the same class; and panoptic segmentation, which combines semantic and instance segmentation to address amorphous or uncountable regions. These tasks use distinct technical approaches, which have advanced each individual task but lack flexibility to generalize to other tasks. This methodological convention leads to fragmented research efforts.
For the purpose of advancing the field of point cloud segmentation in synergistic manner, shift from task-specific network architectures towards a more universal framework is imperative. natural to question why the existing methods cannot achieve such unification, and address this issue in $ 2 by examining the relationships and distinctions between current approaches. Many these approaches are customized for individual tasks, resulting in data representation, feature extraction, and prediction head inconsistencies that hinder integration into a unified framework [1]. Furthermore, task-specific architectures tend to overfit their respective tasks, reducing their generalizability when applied to other segmentation tasks, thus hindering the practical unification of segmentation tasks.
In light of the above discussion, research question becomes more fundamental: what epistemo- logical framework can effectively embrace this shift? response, we revisit the prototype paradigm and point clouds' natural properties (irregular and sparse), and propose that point cloud segmentation naturally aligns with a classification problem. This foundational perspective shapes our research direction and motivates the creation of our prototype-anchored classification method, allowing us to unify various segmentation tasks within a single framework seamlessly. In $ 3, we introduce
Equual contribution. CCoresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Semantic Segmentation
Instance Segmentation
Panoptic Segmentation
87.7
48.7
91.5
64.7
92.0
62.4
SOTA Specialized Segmentation Architectures:
Universal Segmentation Architectures:
StratifiedFormer
DKN
Panoptic-PHNet
ProtoSEG (ours)
JSNet
PROTOSEG unifies three 3D segmentation tasks (i.e., semantic, instance, and panoptic) from the prototypical view, and greatly suppresses existing specialized and unified models.
Figure 1:
PROTOSEG, prototype-anchonee classification method that unifies semantic, instance, and panoptic segmentation. To achieve this, we employ a Transformer [2] backbone to extract point embeddings and represent classes prototypes. This approach allows for task-specific outputs based various granularity annotations (i.e., semantic, instance, and scene), without architectural modification. By utilizing dynamic association and updating mechanism, PROTOSEG captures class-wise character- istics and intra-class variance. Ultimately, PROTOSEG creates well-structured embedded spaces with inter-class separation and intra-class compactness by optimizing point-prototype classs distances.
Concretely, our innovations focus on three aspects. ! Simplicity and transparency: PROTOSEG has an intuitive working mechanism [3] as the statistical meaning of class sub-centroids makes it elegant and easy to understand. This leads to the automatic discovery underlying data structures, as the latent distribution of each class is automatically mined and fully captured as set of representative local means (see $3.2). This contrasts with non-prototype methods [4] that learn a single weight (or query) vector per class, which may struggle accommodate rich intra-class variations [5, 6]. @ Representation learning: PROTOSEG achieves point-wise classification through comparisons between data samples and class sub-centroid in the embedding space. This distance- based nature enables PROTOSEG to blend unsupervised sub-pattern mining class-wise clustering (see Eqs. 13-14) with supervised representation learning learnable prototype classification (see Eq. 15) in synergistic manner. Significant local patterns are automatically identified to facilitate classification decisions, while the supervisory signal from classification directly optimizes the representation, turn boosting meaningful clustering and discriminative features learning. C Ad-hoc explainability: PROTOSEG is transparent classifier, imbued with a built-in case-based reasoning process, establishing IF Then rules [7]. This is achieved by allowing the class sub-centroids to serve as representative samples from the training set, thus providing underttdaaaa explanations for each prediction (see Fig. 3). Such an ad-hoe explainability adheres to the internal decision- makiing process, thereby differentiating PROTOSEG from many existing methods [8] that fail elucidate precisely how model works.
Incorporating these innovations, we present an elegant, general, and flexible framework for point cloud segmentation that achieves remarkable results (see Fig. 1) widely recognized 3D point cloud semantic, instance, and panoptic segmentation tasks. Specifically, our method achieves notable performance gains in terms of 91.5% - 92.2% OAcc. 75.2% 76.4% mIoU and 75.4% 76.3% mloU for semantic segmentation on S3DIS [9] Area 5 and ScanNet V2 [10] test/val. sets, 64.7% - 66.8% mCov, 50.6% + 51.2% mAp and 67.6% - 68.4% mAPso for instance segmentation on S3DIS [9] Area and ScanNet V2 [10] test/val. sets, and 61.5% + 62.4% PQ for panoptic segmentation on SemanticKITTI [11] test set when compared to main competitors. Accompanied by a comprehensive series of ablation studies in $5.4, our extensive experiments evaluations confirm the strength of our concept and the effectiveness of our prototype-anchored anchored classification algorithm.
2 Existing Task-Specialized Segmentation Models as Non-Prototype Learners
In this section, we provide overview of existing methods for point cloud segmentation that are tailored specific tasks, such as semantic, instance, or panoptic segmentation. We will discuss the key aspects of these methods, including data representation, feature extraction, and projection head.

---

A key challenge is the inherent customization of segmentation methods for specific tasks. From broader perspective, it's evident that semantic segmentation methods might not directly apply to instance segmentation tasks and vice versa. For instance, the semantic segmentation method PTv2 [2] lacks modules for instance-specifiy localization and segmentation. This makes inadequate for capturing the detailed spatial relationships required for precise instance boundary delineation and separation. contrast, the instance segmentation method SoftGroup [12] uses grouping mechanisms to link points belonging to the same instance. While such mechanism is essential for instance segmentation, could introduce unwarranted complexity and overhead in semantic segmentation scenarios, where distinguishing individual instances isn't the primary goal.
On finer scale, we observe inconsistencies across data representation, feature extraction, and prediction heads (Eqs. 1,2,3) in these methods. To illustrate, let's first explore the differences between the data representations that make up the initial steps of the segmentation. For given 3D) point cloud scene M comprised of a set of points, data can be represented in multiple ways, including projection- based [13], voxel-based [14, 15] and point-based [16, 17, 18, 19, 2] representations. Mathematically, the input data representation through function p(-) can be expressed as:
X = P(M)
(1)
After the data representation is obtained, encoder is employed extract point cloud embeddings:
= = (X),
(2)
where incorporates techniques such as object detection [20, 21], clustering [22, 23] or graph learning [24, 25] to enhance the embedding representation based on the backbone network.
Finally, each point embedding e, E is fed into the projection head for segmentation [26]. Let V == [u] x E RCxD be a set of class-specific vectors, where C' is the number of class, D is vector dimension, vr RDD represents the linear weight or query vector in MLP-based [18. 2, 19, 2] or Query-based based models [27, 28] for the th class. The probability that a point example æ; E X with embedding e; R.D is assigned to class e can be expressed with inner product as follows:
exp((v") ee)
p(c...) Exp((x((00"
(3)
These above inconsistencies data representation, feature extraction, and projection head further complicate the unification of segmentation tasks, as the learned features may not be directly trans- ferable across different tasks. In addition, the segmentation methods employed may not be optimal for specific tasks, limiting their applicability and performance in unified framework. Despite these methods necessitating different processing techniques, thereby complicating the unification of the tasks, they can collectively be categorized as Non-Prototype Learners.
Universal Segmentation Models as Prototype Classifiers
3.1 Problem Formulation
To address the challenges, we propose unified framework that interprets different segmentation tasks as unique granularity classification problems. This approach crafts a prototype classifier, eliminating extra computational load and task-specifie architectures. Specifically, let M be 3D point cloud scene containing a set of points (æ:). The objective of point cloud segmentation is to assign each point æ; to the class assignment
PointSeg(M)) : =z ++ (A.(z.) Ac(a))) E (0.1), = (1,.,,,,....
(4)
where Az;)) indicates whether the point x; E . belongs (1) or does not belong (0) to class c. Distinct from Eqs. 1-3 in $2, we seamlessly integrate 3D point cloud segmentation under Eq.4, adopting a perspective of prototype-anchored classification, considering various levels of granularity.
3.2 Prototypical Classifiers for Segmentation
We propose a unified prototypical classifier for specific segmentation tasks. To achieve this, we leverage a set of prototypes P P= = C.K E ER where pf IR denotes the center of the th sub-cluster (property) of training point samples belonging to class For given point sample x,,
3

---

Pk
Fneader
Trans Trans fonmerr e.
(Eq, 10) Prototype Association
Ad-hoc
semantic-level
Prototype Updane (Fq. 12)
instance-level
Scheme
Embedding Space
Prototypical Classification
scene-level
3D Point Cloud Scene
Eres 13)
Classification resalls
Figure 2: Overall pipeline of PROTOSEG. The classification of points is predicted by evaluating the minimum distance between the embedding (extracted by encoder d) and prototype P via our Prototype Association and Update mechanism (see Eq. 10, Eq. 12 in $3.2). Subsequently, supervision scheme optimizes ø by minimizing the peint-proootype-llas distance (see Eqs. 13-15 in $3.3). Ad- hoc explains the decision- decision-making process of our classification model, detailed in $5.4.
we make predictions by comparing point embedding ER,, extracted using simple Transformer encoder Φ, with prototype pl and assigning the corresponding prototype's 's class as the response:
exp( de,.c)
p(c|,,))
with de..e arg min (e,, (e) K
(5)
Ee exp( -ooddde,,) dej.et)
where the negative cosine distance measure (.,.) is defined as (e..p)) = -
To get the informative prototype that can represent the properties of the class, we first associate the point embeddings to the prototypes belonging to the same class (i.e., Prototype Association) and then dynamically update prototypes according to the assignments (i.e., Prototype Update).
Prototype Association. Given embeddings point samples E {e)) E DxN. in a training batch and prototypes P== { [k= D for class c, we aim to maximize the similarities J ((M) between the associated point sample embeddings and corresponding class prototypes:
max M
J(M) = Tr((M)) (P*) E",
(6)
K,N, where Tr(.) denotes the matrix trace, M 1 kii= E {0, 1jx.,, is point-to-prototye permutation matrix that denotes the association between point and prototype, m {0, 1} denotes the one-hot assignment vector that assigns the point sample æ; to the prototype & of the class
To accommodate the representation all classes for M we impose two constraints aimed at avoiding a trivial solution in which all point samples are assigned to single prototype [29]:
No 1k,
(M*) T IN., (M*)
where 1 1N. denotes the tensor of all ones of K or N₂ dimensions. The former Unique assignment Constraint ensures that each point embedding is assigned one and only one prototype across all classes. The latter Equipartition Constraint enforces that each prototype is selected an equal number of times within the batch on average. The implementation of these constraints is of utmost importance as it significantly enhances the representative capability of the prototypes for each class.
For more intuitive, Eq. 6 can be rewritten estimating the optimal s..:
M
max Tr((M)) (PP(M))((PPTTTT TP))
(8)
M
Following [30], we smooth Eq. 8 with an entropic regularization term h(M²):
S max TM((M)TTTPP)) = s.t. M M K
(9)
M
where ((M) = Et. m Jogmm, reduces the randomness of association and avoids most point samples associating one single prototype. We control the association smoothness using a small value K to ensure that each point embedding is assigned to only one prototype.

---

With the entropic regularization term in Eq. 9, the problem can be solved using the Sinkhorn- iteration [30] on the renormalization vectors u E RRK and vf E IRN". After a few iterations, the vectors u" and v" converge, and then the permutation matrix M can be directly calculated as:
(P
E²
M (tlag(")) exp
diag(v"),
(10)
where diag(-) denotes a diagonal matrix formed by the input vector. The hyper-parameter x: balances the convergence speed and stability of Eq. 10 in addition smoothing the association (Eq. 9). A smaller x can lead to slower convergence but more accurate results, while a larger K does the opposite. We just use K 0.05 following [30] for our experiments. not extensively fine-tuned. The resulting matrix M satisfies both the unique assignment and equipartition constraints, effectively mapping each point sample 2; in the training batch to its corresponding prototype P for each class c.
Prototype Update. In previous solutions [31, 5], the prototypes are typically computed by determin- ing the centers of the corresponding embedded point samples and subsequently updating them based on the online clustering results. However, this strategy may not effectively mitigate the widespread class imbalance issue, resulting in sub-optimal prototypes specifically for rare classes.
To overcome this limitation, we introduce prototype calibration strategy with a learnable calibration factor, which estimates the representative level of each class prototype showing the importance of tailed class prototype embeddings. The calibration factor is determined by rectified sigmoid function that adjusts the distance between the prototypes P {pf K of class and the point No embedding set {ef] assigned p by prototype association current training batch:
w² 1
Σ No Ex
K 1 exp((ee)) pi)
w E 1].
(11)
NoK
Thus, the formula for the modified prototype updating method with prototype calibration is as follows: p + upl + wn(1 =,
(12)
where μ E [0, 1] is momentum coefficient, and E E RR is the mean vector of [
3.3
Supervision Scheme
Based on prototype assignment and update, we get the learned prototypes where { has class representation ability and is typical for their corresponding class c, and P represents htth property within the class c. order to shape well-structured embedded spaces, we designed a comprehensive supervision scheme based on metric learning to enhance inter-class prototypes separation (see Leps in Eq. 13), intra-class prnnt-protttype compactness (see Lyrc in Eq. 14), and inter-class point-class distances.
separation (see Lecs in Eq. 15) through optimizing chint-prototype-coe Prototype-Prototype Distances Optimization. order to differentiate between different prototypes within the same class (i.e., inter-class prototypes separation), we utilize wrototype-based metric learning strategy to enhance the relationship between points and their assigned prototypes within specific class. By utilizing the point-to-proton each point embedding ef to be similar to its assigned ('positive') prototype pk; and dissimilar to
permutation matrix M this strategy encourages
other irrelevant ("negative") prototypes P {P Pk E C.K. and pt, # *P
exp((ef) /(./T)
Lrps = - log
exp((ef) P /T) Enp pijee- eP- exp((e)) (0p((e)))P1)1) P2,/T)*
(13)
where 7 controls the concentration level of the distributions over classes. Point-Prototype Distance Optimization. To reduce intra-cluster variation, directly minimize the distance between each embedded point eê and its assigned prototype P (Le. intra-class point- prototype compactness), resulting in tighter and more coherent class representations:
Laee = (1 - (et) P
(14)
Point-Class Distances Optimization. In addition to the constraints between point-prototype, for more accurate classification prediction, we aim to correctly associate points with their respective classes (i.e., inter-class point-class separation). Specifically, push the embedded points closer to
5

---

their corresponding class (i.e., o²) and enlarge the distance from other irrelevant classes (i.e., c' * cc). Given the groundtruth class G of each point embedding e,, the cross-entropy loss is used for LpCs as:
exp( exp(-der.e)
Lecs - log exp(-dea,:)) Lelfte, exp(-de,,c) with de,c . = arg (e:
(15)
where the point-class distance de E [-1, 1] is the distance to the closest prototype of class c. Total Losses. Our prototypical classifier can effectively optimize the point embedding space extracted through o by minimizing the combined loss over all the training point samples:
LTowa
Lecs + a<Cpps + 3LPCC.
(16)
where 3 are hyperparameters that control the trade-off between inter-class separation. intra-class compactness, and correct classification. By minimizing Toall our model segments the point cloud effectively while producing compact and discriminative class representations in the embedding space.
3.4 Implementation Details
We use PointTransformer V2 [2] the encoder e and replace the original MI.--based semantic prediction head with our prototypical classifier. Instance segmentation and panorama segmentation tasks can be viewed as more fine- aeneggrined prototype classification problems. For these tasks, we additionally learn fine-grained prototypes pe.. E ER p E IR T xk and a fine-grained permutation matrix M 1] 1 M {0, 1) where o E O, T, O are the number of instance prototypes and instances per class. The association and update of prototypes follow $3.2. Instance- level supervision using a cross- coos-pyntropy loss is incorporated into our supervision scheme ($3.3).
Reproducibility. Training and testing are conducted on eight NVIDIA A100 GPUs. More details and parameter settings can be found in the Appendix.
4 Important Knows and Knowledge Gap
Specialized point cloud segmentation. Point cloud segmentation comprises three separate tasks, i.e. semantic, instance and panoptic segmentation, each of which focuses on different semantic aspects. Historically, researchers have devised specialized models and optimization objectives for each task.
Semantic segmentation is to achieve comprehension of high-level semantic concepts through the grouping of points into discrete semantic units. The advent of point convolution [16] has facilitated the development of sophisticated models, encompassing context aggregation [27, 19], graph convolution integration [24, 17], and contrastive learning [32, 33]. Recently, Transformer architectures [18, 2], which has been successful in vision [34, 35, 36), have obtained considerable research interest.
Instance segmentation involves assigning foreground points to individual object instances, similar to video tasks [37], which can be achieved through three main approaches: @ top-down models [21, 12] that initially detect object bounding boxes and generate an instance mask for each box, @ bottom- up models [38, 39] that learn distinctive point embeddings on instance boundaries, energy levels, geometric structures, and point-center offsets, and group them into instances, and & single-shot models [22] that directly predict instance masks through set of learnable object queries.
Panoptic segmentation seeks comprehensive scene understanding, considering semantic relations between background points and instance memberships of foreground points. Traditional solutions
involve decomposing the problem into manageable tasks such as box-based segmentation [11, 20], thing-stuff merging [40], instance clustering [23, 41, 25). More recent DETR- approaches [28] shifted towards an end-to-end scheme using Transformer [42] and achieved compelling performance.
Prototypical classifier. Prototype-basee based classification, an exemplar-driven approach that compares observations with representative examples, has gained particular attention among various machine learning algorithms, which include classical statistics-bases methods, Support Vector Machines, and Multilayer Perceptrons. The nearest neighbors rule, the earliest prototype learning method [43], has led to the development of many well-Known, con-paraettiic classifiers [44], such as generalized Learning Vector Quantization (LVQ) [45], and Neighborhood Component Analysis [46]. Metric learning [47, 48] is also naturally related to prototype learning. Recent efforts have attempted to integrate deep learning into prototype learning, demonstrating its potential in few-shot [49], self- supervised learning [50], weakly supervised learning [51, 52], as well as supervised learning [53, 54]

---

and interpretable networks [55]. Building on these successes, we aim to advance this research by developing a universal 3D segmentation framework based on prototypical classification. Our method, PROTOSEG, incorporates prototype-anchore classification design and simple Transformer archi- tecture to better capture the nature of prototypical learning and effectively handle the heterogeneity across various segmentation tasks using the same architecture.
Universal 3D Segmentation. Universal segmentation strives to adopt unified architecture capable of addressing diverse segmentation tasks. One of the early attempts at unification through multi-task learning is MT-PNeet [56]. which uses two branches to predict semantic class labels and instance embedding labels. respectively. More recent studies, such as those presented in [57]. employ clustering schemes to integrate semantic and instance information, but their heuristic post- post-procssiing technique (e.g., mean shift) can bring serious computational burden. In comparison to these pioneering efforts, PROTOSEG distinguishes itself by being: @ more transparent leveraging straightforward, case- based reasoning for 3D tasks; @ more flexible handling multiple segmentation tasks simultaneously; and @ more powerful leading specialized methods by notable margins.
Multi-Task Image Segmentation Multi-Task Image Segmentation aims to develop a cohesive archi- tecture to address various segmentation challenges. K-Net [58] pioneered this approach, leveraging dynamic kernel learning for mask generation. In recent developments, several architectures inspired by DETR [59] have formulated different tasks within a mask classification paradigm. For instance, [61] negate the need for box predictions by employing conditional convolutions, bridging the gap between box-dependent and box- -ex-indeenndent methods for the first time. MaskFormer [62] and its subsequent versions [63, 64] introduce streamlined and effective inference strategy to merge outputs into task-specific prediction format using a collection of binary masks. In addition, methods based on dense prediction [65], optimal transport [66]. graph modeling [67], and clustering [68, 69, 70J also have achieved great success.
Max-DeepLan [60] and kMaX-DeepLa
Unlike the above methods, our approach perceives these three closely-related tasks as a prototype- anchored classification challenge with distinct granularity levels. We define prototypes
class cub-centroids derived from the feature embeddings of training samples. Then, a test sample is directly classified based on its proximity to the nearest centroids. Such an approach, rooted in case- based reasoning, introduces a distinct element of ad-hoc interpretability to our method (see Fig. 3 in $5.4).
5 Empirical Evidence
PROTOSEG is the first framework to support three core point cloud segmentation tasks with a single unified architecture. To demonstrate its broad applicability and wide benefit, conduct extensive experiments on semantic ($5.1). I). instance ($5.2), panoptic ($5.3) segmentation, and carry out ablation studies ($5.4) related our framework design. More qualitative results are available in Appendix.
5.1 Experiment Semantic Segmentation
Dataset. S3DIS [9], a large-scale indoor point cloud dataset, encompasses point clouds from 271 rooms across 6 areas. Each room represents a medium-sized point cloud, annotating every point with semantic label from one of the 13 classes. ScanNet V2 [10] provides over 1,500 indoor scenes and around 2.5 million annotated RGB-D images with approximately 90% surface coverage. The benchmark is evaluated on 20 semantic classes, which include 18 different object classes. SemanticKITTI [11] is introduced based on the well-known KITTI Vision [71] benchmark illustrating complex outdoor traffic scenarios. It contains 22 data sequences, 43,552 frames of outdoor scenes, of which 23,201 frames with panoptic labels are used for training and validation, and the remaining 20.351 frames without labels are used for testing. There are annotated point-wise labels in 20 classes for segmentation tasks, 8 of which are defined as thing classes.
Metric. To evaluate our semantic segmentation performance, we apply the mean Intersection-over- Union (mloU 1), the overall accuracy (OAce 1) taking all points into consideration and the average class accuracy (mAce 1) of all semantic classes upon the whole dataset.
Performance Comparison. Table 1 and Table show the results of our PROTOSEG model compared with previous methods on S3DIS [9] and ScanNet v2 [10], respectively. Our model outperforms prior methods across all evaluation metrics. Notably, PROTOŠEG significantly out- performs PTv2 [2] by 0.9 % and 1.2% % mIoU on the ScanNet v2 [10] validation and test set, and
7

---

Table I: Comparisons semantic segmentation performance on S3DIS [9] Area 5 (see $5.1).
Table 2: Comparisons semantic segmentation with mloU ScanNet [10] (see $5.
Method
OAce
mAce
mloU
Method
Test
Val.
JSNet [57] [AAAF22]]
87.7
61.4
54.5
SegGCN [72] (CVPR
58.9
SegGCN [72] [CVPR
88.2
70.4
63.6
RandLA-Net [79] [CVPR220]
64.5
SCF-Net [73] [CVPR221]
88.4
71.6
82.7
PointASNL [27] [CVPR220]
66.6
63.5
PAConv [74] [CVPR221]
66.6
RPNet [80] [ICCV22]]
68.2
RepSurf-U [75] [CVPR222]
90.2
76.0
68.9
FusionNet [81] (RCCV120)
68.8
CBL [33] (CVPR222
90.6
75.2
69.4
JSENet [82] [BCCV120]
69.9
PTv1 [18] [[CCV22]]
90.8
76.5
70.4
RepSurf-U [75] [CVPR223
70.2
FastPT [76] (CVPR(23
77.9
70.3
PTv1 [18] [CCC
70.6
PointMixer [77] [BCCV/22]
77.4
71.4
PointNeXt [19] [NeurPP522]
71.2
71.5
PTv2 [2] [Ne)
91.1
77.9
71.6
StratifiedFoomer [78] [CVPR/22)
73.7
74.3
StratifiedFomer [78] (CVPR)22)
91.5
78.1
72.0
PTv2 [2] [NeurPS:22]
75.2
75.4
Ours (Area
92.2
78.6
72.3
Ours
76.4
76.3
surpasses other methods in all metries on the S3DIS [9] dataset. These results further support the robustness and effectiveness of our method for large-scale point cloud semantic segmentation tasks. Notably, our method yields a more substantial performance enhancement on ScanNet V2 than on S3DIS compared to the main competitor P2T22 [2]. This discrepancy can be attributed to PROTOSEG's statistical characteristics, which can mine richer intra-class variations on the larger scale and more categories ScanNet V2. To evaluate the performance more comprehen- sively, we report the results of semantic segmentation on SemanticKITTI validation set. PROTOSEG achieves mlou of 74.2%, confirming the performance benefits of our model.
Table 3: Comparisons seman-
tic segmentation performance on Se- manticKITTI val set (see 15.1).
Method
mloU
Cylinde33D [V3R
65.9
2DPASS [84]]50CV
69.3
(AF)' -33eet (85](CVPR'2)
74.2
Ours
74.2
5.2 Experiment Instance Segmentation
Dataset. S3DIS [9] and ScanNet V2 [10] are utilized for instance segmentation experiments.
Metric. We use mean coverage (mCov t), mean weighed coverage (mWCov t), mean preci- sion (mPrec 1), and mean recall (mRec t) the evaluation metrics for S3DIS [9]. Additionally, ScanNet V2 [10] evaluated using mean average precisions (mAPs f) under different loU thresholds.
Performance Comparison. As is de- picted in Tables 4-5, our method signifi- cantly outperforms other methods on the S3DIS [9] dataset. The main criteria, mCov and m mCCo, take into account both the completeness and accuracy of instance segmentation results. Specifi- cally, terms of the main criteria, our method yields mCov of 66.8%. mW- Cov of 68.4%. a mPrec of 69.7% % and mRec of 66.3% in Area 5, surpassing most methods. Furthermore, the supe- rior performance
Table 4: Comparisons of instance segmentation performance on S3DIS [9] Area 5 (see $5.2 for more details).
Method
mCov
mWCov
mPrre
mRec
PointGroup [86] (CVPR:20)
61.9
62.1
MaskGroup [39] [CME'22]
62.9
64.7
SSTNet [22] picev1a]]
42.7
59.3
65.5
64.2
DyCo3D [87] [CVPR'21]
63.5
64.6
64.3
64.2
HAIS [38] [CCCv221]
64.3
66.0
71.1
65.0
DKNet [88] [BCCV122]
64.7
Ours
68.4
Table Comparisons of instance segmentation performance on ScanNet v2 [10] (see $5.2 for more details).
Text
Val.
Module
ScanNet V2 [10] dataset in both the val. and test sets also highlights the effectivvneess of our ap- proach for instance segmentation. The significant improvements across various metrics and benchmarks demonstrate the efficacy and robustness of PROTO- SEG, even in challenging scenarios.
mAP
mAP
mAP
mAP
DyCo3D [87] [CVPR
39.5
64.1
35.4
57.6
PointGroup [86] (CVPR
40.7
63.6
34.8
56.7
MaskGroup [39] ICME
43.4
66.4
42.0
63.3
HATS [38] [CCV/21]
45.7
69.9
43.5
64.1
SoftGroup [12] (CVPR/22)
50.4
76.1
46.0
67.6
SSTNet [22) [ICCV121]
50.6
69.8
49.4
64.3
Ours
51.2
78.1
47.8
68.4
5.3 Experiment Panoptic Segmentation
Dataset. SemanticKITTI [11] is utilized for panoptic segmentation experiments.

---

Table 6: Comparisons of panoptic segmentation performance on SemanticKITTI [11] test (see $5.3).
Method
PQ
mlou
PQ
RQ
SQ
PQ
RQ
SQ
PQ"
RQ
SQ"t
LPSAD [90] [IROS
38.0
50.9
47.0
48.2
76.5
25.6
31.8
76.8
47.1
60.1
76.2
Panoster [91] [RAL/1]]
52.7
59.9
59.9
64.1
80.7
49.4
58.5
83.3
55.1
78.8
Panoptic-PolarNer [23] [CVPR'21]
54.1
60.7
81.4
60.6
87.2
54.8
68.1
77.2
DS-Net [41] [CVPR21
55.9
62.5
55.1
62.8
87.2
56.5
78.7
EfficientLPPS [40] [TR2
57.4
63.2
60.5
87.8
60.5
79.5
GP-S3Net [25] (ICCV22]
60.0
70.8
69.0
72.1
82.0
65.0
74.5
86.6
56.4
70.4
78.7
SCAN [28] [AAA122]]
61.5
67.7
67.5
72.1
84.5
61.4
69.3
88.1
61.5
81.8
Panoptic-PHNet [15) [CVPR/22]
61.5
66.0
67.9
72.1
84.8
63.8
70.4
90.7
59.9
73.3
80.5
Ours
62.4
67.5
68.5
74.4
83.4
65.6
72.6
89.1
61.8
75.7
79.2
chair
ARCE
EE
And
8
ELIF(EE
LE
And
table
[E
>
[E.
0.7%
0.0NEN
ELIRLE
E
[E
And
84.1%
[A
Our prediction: chair
window
92.3%
GroundTruth: chair
THEN IN class chair
prototype instances
rule created for "chuir"
decision-making mode
Figure 3: Interpretable prototype (i.e., class sub-centroids) with Then rules (left) and decision- making process based distance to prototypes (right). colors represent prototype instance for three S3DIS [9] classes, [:-] stands for distance between point embedding E' and instances.
Metric. We use the panoptic quality (PQ) [89] as our main metric evaluate the performance of panoptic segmentation. PQ can be seen as the multiplication of segmentation quality (SQ) and recognition quality (RQ). These three metries can be extended to things and stuff classes, denoted as So
PQ PQSt, RQTH RQSt. SQ SQ respectively. PQ' replaces PQ with IoU for stuff' classes. Performance Comparison. Table 6 presents comparison of panoptic segmentation performance on the SemanticKITTI [11] test. Our proposed method demonstrates significant improvement over the berf-perf baseline method, with 62.4% increase in PQ. It noteworthy that our unified framework demonstrates superior performance in both things and stuff classes, evidenced by outperforming the second-best method, Panoptic-PHNet [15], in out of metrics.
Overall, PROTOSEG demonstrates excellent performance across various segmentation tasks (see 15.133.3) both indoor and outdoor datasets. Notably, does not require inherent modifications to the architecture or the training regime, which highlights its generality and flexibility.
5.4 Diagnostic Experiment
This section conducts ablation studies for PROTOSEG's semantic segmentation on S3DIS [9] Area5.
Study of Ad-hoc Explainability. In $2 and 55.1-5.3, we have explained the simple yet transparent mechanism of PROTOSEG and verified its robustness and effectiveness representation learning. We now highlight PROTOSEG's interpretability in Fig. 3, which depicts prototypes that represent specific objects in the training set. Fig. (left) displays representative instances, serving as interpretable prototypes (class sub-centroids) for three S3DIS [9] classes. These objects, diverse in appearance, viewpoints, illuminations, and scales, characterize their respective classes, providing a human- interpretable overview. Leveraging the simplicity of the Nearest Centroids mechanism, we formulate intuitive IF.. - rules, providing an intuitive glimpse into PROTOSEG's deccision-makigg process. Furthermore, can elucidate PROTOSEG's predictions by visualizing the computed distance (similarity) between the query and prototypes. As illustrated in Fig. 3 (right), the model accurately classifies an observation, perceiving it more closely resembling particular chair exemplar (see the top right of Fig. 2). Overall, this visualization clearly shows that the classification decision procedure relies the notion of distance beyond the selection of the encoder é.
Key Components. We first investigate our Prototypical Classifier ($3.2) and supervision scheme ($3.3) S3DIS [9] and ScanNet V2 [10] datasets. As shown in Table 7a, introducing prototypical classifiers in conjunction with Lecs improves the performance to 71.74% and 75.84%. Adding either Lpes or LPPC to the prototypical classifiers and LPcs promotes performance gains, indicating the

---

Table 7: A set of ablative studies (see $5.4) on S3DIS [9] Area 5. (a) Key Components
(b) Prototype Number K
Prototypical Classifiers
PPCs (Eq. 15)
Lass
Lyve (Eq. 14)
S3DIS
ScanNetv2 (mloU)
# Prototype
mloU (%)
(Eq. 13)
(mloU)
K =
71.48
71.53
75.36
K 5
71.84
71.74
75.84
K 10
72.34
72.05
76.07
72.06
76.12
K 20
72.18
72.34
76.32
K 50
72.05
(c) Distance Measure
(d) Momentum Coefficient
(e) Class Balance
Sinkhorn Iterations S
mloU
Coefficien
mloU (%)
Strategy
mloU (%)
Iterations
mloU
Distance
(%)
0.9
71.62
w/o
71.49
(Eq. 10)
(%)
Standard
72.10
0.99
71.88
w
72.34
S
71.29
Huberized
72.17
0.999
72.34
Mp
72.14
$ =
72.34
Cosine
72.34
0,9999
72.04
M,
71.83
$ =
71.43
value of explicitly learning point-prototype relations. The improvements observed by introducing prototypical classifiers and supervision scheme (mloU: 71.53% 72.34% S3DIS [9], 75.36% 76.32% on ScanNet V2 [10]) confirm the effectiveness of our proposed framework.
Prototype Number Per Class. We then analyze the impact of the number of prototypes per class on segmentation performance, as reported Table 7b. With a single prototype (K 1), each class is represented by the mean embedding of its point samples, yielding the mloU score of 71.48%. The mloU score peaks at 72.34% when only results in slight decrease in performance, with mloU scores of respectively. These results suggest an optimal balance between accuracy and computational cost at K == 10 prototypes per class, affirming rationale for leveraging multiple prototypes to encapsulate intra-class variations and enhance segmentation performance. It would be interesting to find ways to ascertain this number automatically. In exploration, encountered clustering techniques like [92] that might help, but complex algorithms led to time challenges.
Further increment of prototypes K 20, 72.18% 72.05%.
50
Distance Measure. Next, we examine the impact of different distance metrics on our Supervision Scheme (see Eqs. 13-15 in $3.3). By default, we use cosine distance (denoted as Cosine in Table 7c). We also consider two alternative metrics, i.e.. standard Euclidean distance (Standard) and Huber- like function [93] (Huberized). Table 7c demonstrates that the Cosine distance surpasses the un- normalized Euclidean metrics, affirming its superiority in measuring point-prototye similarity.
Momentum Coefficient. Table 7d quantifies the effect of the momentum coefficient (μ Eq. 12). Our model achieves the best performance when the momentum coefficient is set to 0.999. As the momentum coefficient decreases, the performance gradually declines. These results suggest that higher value (i.e., slower updating) generally leads to better results in the common case.
Class Balance Strategy. Next, we proceed to validate the effectiveness of our strategy in mitigating the issue category imbalance. As Table 7e shown, the baseline (w/o) fixes w = in Eq. 12, Mt [31] and M,, [94] utilize parameter and non-parametric augmented memory banks, respectively. The reported results provide evidence for the efficacy of our prototype calibration strategy.
Sinkhorn- Knopp Iteration. Finally, we verify the effect of iterations on the model. Table 7f shows that the optimal segmentation results can be achieved using a smaller number (S = 3) of iterations. This further verifies the efficiency of our Prototype Association (see $3.2).
Conclusion
We present PROTOSEG, novel, prototype-anchore classification framework for 3D point cloud segmentation that unifies semantic, instance, and panoptic segmentation tasks. By considering these tasks under the umbrella of classification problems but various granularity, PROTOSEG offers an elegant, general, and flexible solution that eschews task-specific network architectures. Compared to -ask-specific models, PROTOSEG offers several advantages: The prototype- based classification mechanism is simple and transparent. ii) Our supervision scheme, which leverages prototype-basee metric learning, exhibits robust representational learning ability. iii) PROTOSEG offers ad-hoc explainability by anchoring class exemplars to real observations. Our comprehensive experiments demonstrate the superior performance of PROTOSEG in terms of efficacy and enhanced interpretability. Overall, we believe that our work may motivate further research in this field.
10

---

Acknowledgement. This work was supported in part by the National Natural Science Foundation of China (No. 62176139, 62106128, 62176141), the Major basic research project of Shandong Natural Science Foundation (No. 7R20217D15), the Natural Science Foundation of Shandong Province (No. (R2201QFF01)) the Young Elite Scientists Sponsorship Program by CAST (No. 2021QNRC001), the Open project of Key Laboratory of Artificial Intelligence, Ministry of Education, the Shandong Provincial Natural Science Foundation for Distinguished Young Scholars (ZR2021JQ26), the Taishan Scholar Project of Shandong Province (Isqn202103088).
References
[1] Ben Fei, Weidong Yang. Wen-Ming Chen. Zhijun Li, Yikang Li, Tao Ma, Xing Hu, and Lipeng Ma. Comprehensive review of deep learning-based 3d point cloud completion processing and analysis. IEEE 2022.
Transactions on Intelligent Transportation Systems, 23(12):22862-223
[2] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-base based pooling. NeurIPS, 2022.
[3] Kai Zhang, James T Kwok, and Bahram Parvin. Prototype vector machine for large scale semi-supervised learning. In ICML, 2009.
Wenguan Wang. Jianhing Shen, Xiankai Lu, Steven CH Hoi, and Haibin Ling. Paying attention video object pattern understanding. IEEE transactions on pattern analysis and machine intelligence, 43(7):2413-2428, 2020.
[5] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Lue Van Gool. Rethinking semantic segmentation:
A prototype view. In CVPR, 2022.
[6] Ziruo Sun, Xiushan Nic, Xiaopeng Bi, Shaohua Wang, and Yilong Yin. Detail enhancement based vehicle re-identification with orientation-guiien re-ranking. Pattern Recognition, 137:109304, 2023.
[7] Plamen Angelov and Eduardo Soares. Towards explainable deep neural networks (xdan). Neural Networks, 130:185-1944 2020.
[8] Hanxiao Tan. Fractual projection forest: Fast and explainable point cloud classifier. In WACV, 2023.
[9] Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, loannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In CVPR, 2016.
[10] Angela Dai, Angel Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
[11] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In ICCV, 2019.
[12] Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang D Yoo. Softgroup for 3d instance segmentation point clouds. In CVPR, 2022.
[13] Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. Rangenet++: Fast and accurate lidar semantic segmentation. In IROS, 2019.
[14] Li Yi, Boqing Gong, and Thomas Funkhouser. Complete & label: A domain adaptation approach to semantic segmentation of lidar point clouds. In CVPR, 2021.
[15] Jinke Li, Xiao He, Yang Wen, Yuan Gao, Xiaoqiang Cheng, and Dan Zhang. Panoptic-phnet: Towards real-time and high-precision lidar panoptic segmentation via clustering pseudo heatmap. In CVPR, 2022.
[16] Charles R Qi. Hao Su. Kaichun Mo. and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d
classification and segmentation. In CVPR, 2017.
[17] Hengshuang Zhao, Li Jiang, Chi-Wic Wing Fu, and Jiaya Jia. Pointweb: Enhancing local neighborhood features for point cloud processing. In CVPR, 2019.
[18] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021.
[19] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. NeurIPS, 2022.
[20] Rohit Mohan and Abhinav Valada. Efficientps: Efficient panoptic segmentation. International Journal of Computer Vision, 129(5):151--157) 2021.
[21] Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias NieBner. 3d-mpa: Multi Multi-proposal aggregation for 3d semantic instance segmentation. In CVPR, 2020.
11

---

[22] Zhihao Liang, Zhihao Li. Songcen Xu. Mingkui Tan, and Kui Jia. Instance segmentation in 3d scenes using semantic superpoint tree networks. In ICCV, 2021.
[23] Zixiang Zhou, Yang Zhang, and Hassan Foroosh. Panoptic-polarnet: Proposal-free lidar point cloud panoptic segmentation. In CVPR, 2021.
[24] Lei Wang. Yuchun Huang, Yaolin Hou, Shenman Zhang, and Jie Shan. Graph attention convolution for point cloud semantic segmentation. In CVPR, 2019.
Graph-based [25] Ryan Razani, Ran Cheng, Enxu Li, Ehsan Taghavi, Yuan Ren, and Liu Bingbing. Gp-s3net: Graph based panoptic sparse semantic segmentation network. In ICCV, 2021.
[26] Yang Yang. De-Chuan Zhan, Ying Fan, Yuan Jiang. and Zhi-Hur Zhou. Deep learning for fixed model reuse. In AAAI, 2017.
[27] Xu Yan, Chaoda Zheng. Zhen Li. Sheng Wang, and Shuguang Cui. Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling. In CVPR, 2020,
[28] Shuangjie Xu, Rui Wan, Maosheng Ye, Xiaoyi Zou, and Tongyi Cao. Sparse cross-scale attention network for efficient lidar panoptic segmentation. In AAAI, 2022.
[29] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
[30] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurIPS, 2013.
[31] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[32] Mengtian Li, Yuan Xie, Yunhang Shen, Bo Ke, Ruizhi Qiao, Bo Ren, Shaohui Lin, and Lizhuang Ma. Hybrider: Weakly-supervin 3d point cloud semantic segmentation via hybrid contrastive regularization. In CVPR, 2022.
[33] Liyao Tang. Yibing Zhan, Zhe Chen, Baosheng Yu, and Dacheng Tao. Contrastive boundary learning for point cloud segmentation. In CVPR, 2022.
[34] Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. E 2vpt: An effective and efficient approach for visual prompt tuning. In ICCV, 2023.
[35] Qifan Wang, Jingang Wang, Xiaojun Quan, Fuli Feng, Zenglin Xu, Shaoliang Nie, Sinong Wang. Madian Khabsa, Hamed Firooz, and Dongfang Liu. Mustie: Multimodal structural transformer for web information extraction. In ACI. 2023.
[36] Wenguan Wang, James Liang, and Dongfang Liu. Learning equivariant segmentation with instance-unique querying. In NeurlPS, 2022.
[37] Zheyun Qin. Xiankai Lu. Xiushan Nie. Dongfang Liu. Yilong Yin, and Wenguan Wang. Coarse-to- fine video instance segmentation with factorized conditional appearance flows. IEEE/CAA Journal of Automatica Sinica, 10(5):1192-12000) 2023.
[38] Shaoyu Chen, Jiemin Fang. Qian Zhang, Wenyu Liu, and Xinggang Wang. Hicrarchical aggregation for 3d instance segmentation. In /CCV, 2021.
[39] Min Zhong, Xinghao Chen, Xiaokang Chen, Gang Zeng, and Yunhe Wang. Maskgroup: Hierarchical point grouping and masking for 3d instance segmentation. In ICME, 2022.
[40] Kshitij Sirohi, Rohit Mohan, Daniel Büscher, Wolfram Burgard, and Abhinav Valada. Efficientlps: Efficient lidar panoptic segmentation. IEEE Transactions on Robotics, (8(3):1894-1144 2021.
[41]
Fangzhou Hong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. Lidar-based panoptic segmentation via dynamic shifting network. In CVPR, 2021.
[42] Yawen Lu, Qifan Wang, Siqi Ma, Tong Geng, Yingjie Victor Chen, Huaijin Chen, and Dongfang Liu. Transflow: Transformer as flow learner. In CVPR, 2023.
[43] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1):21-27, 1967.
[44] Salvador Garcia, Joaquin Derrac, Jose Cano, and Francisco Herrera. Protolype selection for nearest neighbor classification: Taxonomy and empirical study. IEEE Transactions on pattern analysis and machine intelligence, 34(3):417443 -33,, 2012.
[45] Atsushi Sato and Keiji Yamada. Generalized learning vector quantization. NewrIPS, 1995.
[46] Jacob Goldherger, Geoffrey Hinton, Sam Roweis, and Russ R Salakhutdinov. Neighhourhood components analysis. NeurlPS, 2004.
12

---

[47] Yang Shi, Xiushan Nie. Xingbo Liu, Lu Yang, and Yilong Yin. Zero-shot hashing via asymmetric ratio similarity matrix. IEEE Trunsactions on Knowledge and Data Engineering, 35(5):426-54377 2022.
[48] Xiushan Nie. Xingbo Liu, Jie Guo, Letian Wang, and Yilong Yin. Supervised discrete multiple-length hashing for image retrieval. IEEE Transactions on Big Data, 91):312-327, 2022.
[49] Na Zhao, Tat- Chua, and Gim Hec Lee. Fow-shot 3d point cloud semantic segmentation. In CVPR, 2021.
[50] Mohamed Atham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna. and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In CVPR, 2022.
[51] Yongyi Su, Xun Xu, and Kui Jia. Weakly supervised 3d point cloud segmentation via multi prototyp learning. arXiv preprint arXiv:2205.03// 03,7,, 2022.
[52] Rong Li, Anh- Anh-Quan Cao, and Raoul de Charette. Coarse3d: Class-prototypes for contrastive learning in wcakly-supervised 3d point cloud segmentation. arXiv preprint arXiv:2210.01784 2022.
[53] Lin. Yigong Wang, Yi-Fan Li, Zhuoyi Wang, Yang Gao, and Latifur Khan. Single view point cloud generation via unifted 3d prototype. In AAAI, 2021.
[54] Yangheng Zhao, Jun Wang, Xiaolong Li, Yue Hu, Ce Zhang, Yanfeng Wang, and Siheng Chen, Number- adaptive prototype learning for 3d point cloud semantic segmentation. In ECCV, 2023.
[55] Oscar Li, Hao Liu, Chanfan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In AAAI, 2018
[56] Quang- Hieu Pham. Thanh Nguyen. Binh-Son Hua, Gemma Roig, and Sai-Kit Yeung. Jsis3d: Joint semantic-instance segmentation of 3d point clouds with multi-task pointwise networks and multi-value conditional random fields. In CVPR, 2019.
[57] Lin Zhao and Wenbing Tao. Jsnet: Joint instance and semantic segmentation of 3d point clouds. In AAAI, 2020.
[58] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. NeurIPS, 2021.
[59] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[60] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In CVPR, 2021.
[61] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun 7hu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means mask transformer. In ECCV, 2022.
[62] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurlPS, 2021.
[63] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. CVPR, 2022.
[64] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In CVPR, 2023.
[65] Dongfang Liu, James Liang, Tony Geng, Alexander Loui, and Tianfei Zhou. Tripartite feature enhanced pyramid network for dense prediction. JEEE TIP, 2023.
[66] Yang Yang. Zhao- Yang Fu, De-Chuan Zhan, Zhi-Bin Liu, and Yuan Jiang. Semi-supervised multi-modal multi-instance multi-label deep network with optimal transport. IEFF Transactions on Knowledge and Data Engineering, (3(2),996-79, 2019.
[67] Xiankai Lu, Wenguan Wang. Jianbing Shen, David Crandall, and Lue Van Gool. Segmenting ohjects from relational visual data. IEEE Trans. Pattern Anal. Mach. Intell., 44(1444111))))) 44877)78885778 2021.
[68] Yang Yang, De-Chuan Zhan, Yi-Feng Wu, Zhi-Bin Liu, Hui Xiong, and Yuan Jiang. cemi-supervised multi-modal clustering and classification with incomplete modalities. JEEE Transactions on Knowledge and Data Engineering, 33(2):682-695, 2019.
[69] James Liang. Tianfei Zhou, Dongfang Liu, and Wenguan Wang. Clustseg: Clustering for universal segmentation. In ICML, 2023.
[70] Wenguan Wang, Cheng Han, Tianfei Zhou, and Dongfang Liu. Visual recognition with deep nearest centroids. In ICLR, 2023.
13

---

[71] Andreas Geiger, Philip Leaz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[72] Huan Lei, Naveed Akhtar, and Ajmal Mian. Seggen: Efficient 3d point cloud segmentation with fuzzy spherical kernel. In CVPR, 2020.
[73] Siqi Fan, Qiulei Dong, Fenghua Zhu, Yisheng Lv, Peijun Yc, and Yuc Wang. Scf-net: Learning spatial contextual features for large-scale point cloud segmentation. In CVPR, 2021.
[74] Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. In CVPR, 2021.
[75] Haoxi Ran, Jun Liu, and Chengjie Wang. Surface representation for point clouds. In CVPR, 2022.
[76] Chunghyun Park, Yoonwoo Jeong, Minsu Cho, and Jaesik Park. Fast point transformer. In CVPR, 2022.
[77] Jacsung Choc, Chunghyun Park, Francois Rameau, Jaesik Park, and In So Kweon. Pointmixer: MIp-mixer for point cloud understanding. In ECCV, 2022.
[78] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. In CVPR, 2022.
[79] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In CVPR, 2020.
[80] Haoxi Ran, Wei Zhuo, Jun Liu, and Li Lu. Learning annr-grrop relations point clouds. In ICCV, 2021.
[81] Feihu Zhang, Jin Fang, Benjamin Wah, and Philip Torr. Deep fusionnet for point cloud semantic segmenta- tion. In ECCV, 2020.
[82] Zeyu Hu, Mingmin Zhen, Xuyang Bai, Hongho Fu, and Chiew-lan Tai. Jsenet: Joint semantic segmentation and edge detection network for 3d point clouds. In ECCV, 2020.
183] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yucxin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In CVPR, 2021.
[84] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng. Ruimao Zhang. Shuguang Cui, and Zhen Li. 2dpass: 2d priors assisted semantic segmentation lidar point clouds. In ECCV, 2022.
[85] Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. 2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In CVPR, 2021.
[86] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In CVPR, 2020.
[87] Tong He, Chunhua Shen, and Anton Van Den Hengel. Dycodd: Robust instance segmentation of 3d point clouds through dynamic convolution. In CVPR, 2021.
[88] Yizheng Wu, Min Shi, Shuaiyuan Du, Hao Lu, Zhiguo Cao, and Weicai Zhong. 3d instances 1d kemels. In ECCV, 2022.
[89]
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019.
[90] Andres Milioto, Jens Behley, Chris McCool, and Cyrill Stachniss. Lidar panoptic segmentation for autonomous driving. In IROS, 2020.
[91] Stefano Gasperini, Mohammad-Al Nikouei Mahani, Alvaro Marcos-Ramiro, Nassir Navab, and Federico Tombari, Panoster: End-to-end panoptic segmentation of lidar point clouds. IEEE Robotics and Automation Letters, 6(2):3216-322 3223,, 2021.
[92] Meitar Ronen, Shahaf F. Finder, and Oren Freifeld. Deepdpm: Deep clustering with an unknown number of clusters. In CVPR, 2022.
[93] Peter J Huber. Robust regression: asymptotics, conjectures and monte carlo. The annals of statistics, pages 799-821, 1973.
[94] Renrui Zhang, Liuhui Wang, Yali Wang, Peng Gao, Hongsheng Li, and Jianbo Shi. Parameter is not all you need: Starting from non- non-paramctric networks for 3d point cloud analysis. In CVPR, 2023.
14


Reviews:
[{"review_id": "d1TohYk61e", "reviewer": "Reviewer_MoHU", "summary": "The paper presents a unified 3D segmenter to unify different segmentation tasks, including semantic, instance, and panoptic segmentation, for 3D point clouds.  It proposes to define the segmentation task as prototype classification and designs the prototype association and prototype update process.", "questions": "I like the motivation of the paper and the results of the paper are inspiring. However, the overall paper is not very clear and hard to follow. Please refer to the weaknesses part for the questions.", "limitations": "Limitations are discussed in the supplement.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper aims to unify different 3D segmentation models for multiple segmentation tasks, which is a problem worth exploring. \n2. The ProtoSeg method achieves excellent performance on multiple tasks (semantic, instance, panoptic segmentation), showing better results than previous methods in multiple datasets (S3DIS, ScanNet, SemanticKITTI).", "weaknesses": "1. In Section 2, the paper introduces different models for different 3D segmentation tasks. However, the difficulty of unifying these tasks is not clearly stated. I expect to see the challenges of unifying segmentation tasks in Section 2 and the corresponding methods for tackling the challenges in Section 3. \n2. More figures are expected to illustrate the prototype association and prototype update process.\n3. I'd like to see how the prototype classifier is used to unify the different 3D segmentation tasks, especially for the instance and panoptic segmentation tasks, but it is only very simply introduced in Section 3.4 without any details.\n4. It is not clear how to decide the number of instance prototypes and how to deal with scenes with different instance numbers. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer MoHU for the valuable time and constructive feedback. We provide a point-to-point response below.\n### Weaknesses 1: Challenges and the corresponding methods of unifying segmentation tasks\n\nThank you for your insightful review. We greatly value your observation and will revise Section 2 to more prominently highlight the challenges associated with unifying diverse 3D segmentation tasks. A key challenge is the inherent customization of segmentation methods for specific tasks. From a broader perspective, it's evident that semantic segmentation methods might not directly apply to instance segmentation tasks and vice versa. For instance, the semantic segmentation method PTv2 [2] [NeurIPS’22] lacks modules for instance-specific localization and segmentation. This makes it inadequate for capturing the detailed spatial relationships required for precise instance boundary delineation and separation. In contrast, the instance segmentation method SoftGroup [47] [CVPR’22] uses grouping mechanisms to link points belonging to the same instance. While such a mechanism is essential for instance segmentation, it could introduce unwarranted complexity and overhead in semantic segmentation scenarios, where distinguishing individual instances isn't the primary goal. On a finer scale, we observe inconsistencies across data representation, feature extraction, and prediction heads (Eqs.1-3) in these methods. To illustrate, let's consider the prediction head: both MLP-based and Query-based methods rely on a single class-specific vector for each class (Eq.3). This approach often falls short in capturing the rich intra-class variance. Moreover, these class-specific vectors are learned fully parametrically without assessing their representational effectiveness.\n\nTo address the challenges, we propose a unified framework that interprets different segmentation tasks as unique granularity classification problems. This approach crafts a prototype classifier, eliminating extra computational load and task-specific architectures. This method offers versatility, providing crucial insights into segmentation model design and training strategies. In terms of model design, we unveil a pioneering prototype-anchored classification method. It distinctly defines sub-class centers within the point embedding space as prototypes, eliminating the need for class-specific vectors. During training, prototypes capture dataset features, inherently integrating biases like intra-class compactness and inter-class separation into supplementary optimization criteria. This shapes the embedding space holistically, beyond prediction accuracy optimization. Importantly, our unique model design and training approach grant the advantage of ad-hoc interpretability, enhancing the model's transparency and understandability.\n\nWe will include a more detailed description in the revised version for better clarity.\n### Weaknesses 2: More figures\nThanks. This suggestion is highly valuable. We will certainly incorporate more illustrations in the revision to explain the details of the prototype assignment and prototype update.\n### Weaknesses 3: About prototype classifier for instance and panoptic segmentation tasks\nSorry for not explaining it clearly. For these tasks, we additionally learn fine-grained prototypes $P^{c,o}\\in R^{D\\times T},P^{c} \\in R^{T\\times K}$, where $p_t^{c,o}$ denotes the center of the t-th sub-cluster of training point samples belonging to instance $o$ of class $c$. Similarity to Eq.5, we make predictions by comparing point embedding $e_i \\in R^D$, with prototype $p^{c,o}_{t}$ and assigning the corresponding prototype's class as the response:\n\n$$p(c,o|x_i)=\\frac{\\exp(-d_{e_i,c,o})}{\\sum^{C,O}\\_{c'=1,o'=1}\\exp(-d_{e_i,c',o'})}\\,\\quad with \\quad d_{e\\_i,c,o} = \\arg \\min \\{ <e_i, p\\_t^{c,o}> \\}_{t=1}^T.$$\n\nTo get the informative prototype, we learn a fine-grained permutation matrix $M$ for prototype assignment and update. Similarity to Eq.6, we aim to maximize the similarities $J(M)$:\n\n$$\\max_{M}\\mathcal{J}(M)=\\text{Tr}((M^{c,o})^{\\top}(P^{c,o})^{\\top}E^c)$$\n\nwhere $M^{c,o}$ is a permutation matrix, $m_{t,i}^{c,o} \\in M^{c,o}$ \ndenotes the one-hot assignment vector that assigns the point sample $x_i$ to the prototype $t$ of instance $o$ of class $c$. Prototype association and update are as in Section 3.2. For instance-level supervision, we include cross-entropy loss (Section 3.3).  We will add the detail of this part to the revised version.\n### Weaknesses 4: Number of instance prototypes\nSorry for this confusion. We determined the number of instance prototypes needed through ablation experiments in Table 6 (b), Section 5.4. It would be interesting to find ways to ascertain this number automatically. In our exploration, we encountered clustering techniques like [R1] that might help, but complex algorithms led to time challenges. We will elaborate on this in the appendix of our revised version.\n\nOur experiments encompassed scenes with varying numbers of instances. We established the number of instance prototypes based on the percentage of training samples for each class. S3DIS training points per class range from 0.2% to over 30%. We assigned $K=1$ to classes with 0.2-1% training samples, $K=2$ for 1-10% samples, $K=4$ for 10-20% samples, $K=6$ for 20-30% samples, and $K=10$ for classes with over 30\\% samples. This approach resulted in slightly better performance than using a fixed $K=10$ for all classes. However, in our current version, we used $K=10$ for all classes for simplicity. We will discuss the number of prototypes and future work in the revised version. \n\n| $K$ range    | S3DIS mIoU |\n|--------------|------------|\n| unique value | 72.3       |\n| $[1,10]$     | 72.4       |\n\n[R1] DeepDPM: Deep Clustering With an Unknown Number of Clusters\n\nFinally, thanks for your valuable comments, which helped position our work relative to previous efforts.\n"}, {"review_id": "uSyGFOuyNL", "reviewer": "Reviewer_VQFt", "summary": "This work aims to unify different 3D segmentation tasks, i.e., semantic, instance, and panoptic segmentation on point clouds. The authors propose unified prototypical classifiers --- ProtoSeg -- on top of a Transformer backbone, for multi-task 3D segmentation. The main idea is to establish prototype association and update so that the model would be able to output task-specific predictions based on prototypes with different levels of granularity. The authors conducted semantic and instance segmentation tasks on S3DIS and ScanNet V2 and panoptic segmentation on SemanticKITTI. The results show that ProtoSeg outperforms prior works on most evaluation metrics.", "questions": "- Q1: Is the *'IF-Then'* explanation logic applicable to outdoor point clouds? For example, can it also interpret the panoptic segmentation process on SemanticKITTI?\n\n- Q2: How to ensure the convergence of $u^c$ and $v^c$ in the Sinkhorn-Knopp iteration?\n\n- Q3: Ablation study, do you have some analysis or observations on the ScanNet V2 dataset? The scores on the S3DIS benchmark seem saturated and hard to depict influencing factors.\n\n- Minor 1: Line 33, what is the meaning of *'epistemological'* here?\n\n- Minor 2: Line 109, *'dynamic'* should be revised to 'dynamically'.\n \n", "limitations": "The authors discussed some limitations of this work in the supplementary material, which is related to model efficiency. The authors did not discuss the potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- S1: This work contributes to the area of multi-task 3D segmentation, which is definitely a valuable research direction for point cloud understanding.\n\n- S2: The proposed ProtoSeg is showing good explainability during the point cloud segmentation process.\n\n- S3: Convincing results on S3DIS and ScanNet V2 demonstrate the effectiveness and superiority.\n\n- S4: Detailed implementation examples, including pseudo-code, are provided in the supplementary material, which is helpful in facilitating future research.\n", "weaknesses": "- W1: Lacking discussions with multi-task image segmentation methods. The pursuit of unifying semantic, instance, and panoptic segmentation on images is not new for the image community. In fact, several works have explored multi-task image segmentation from a mask proposal view (or other related aspects), which is directly linked to the *'prototypical classifier'* in ProtoSeg. The authors are recommended to discuss the main differences between these methods.\n  - MaX-DeepLab [R1] proposed a task-conditioned joint training strategy with a task token to support multi-task training and inference.\n  - MaskFormer [R2] proposed a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction, for semantic- and instance-level segmentation tasks.\n  - K-Net [R3] proposed a group of learnable kernels for unified image segmentation, where each kernel is responsible for generating a mask for either a potential instance or a stuff class.\n  - Mask2Former [R4] and OneFormer [R5] extended MaskFormer [R2] for better segmentation accuracy.\n  - kMaX-DeepLab [R6] proposed kMaX decoders that transform cluster centers into mask embedding vectors, which multiply with the pixel features to generate the predicted masks.\n\n- W2: It is a bit strange that the authors conducted semantic segmentation experiments on S3DIS and ScanNet V2, but not SemanticKITTI. Although the panoptic segmentation results of ProtoSeg (Table 5) seem reasonable, the authors are recommended to supplement the semantic segmentation results as well to further consolidate the effectiveness.\n\n- Minor: The training complexity of ProtoSeg seems high, since it introduces extra losses for optimization and extra procedures (e.g. prototype update). Although it is not a major concern in this work, the authors are recommended to compare the complexity of ProtoSeg with the baseline models.\n\n#### References\n[R1] H. Wang, et al. \"Max-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers,\" CVPR, 2021.\n\n[R2] B. Cheng, et al. \"Per-Pixel Classification is Not All You Need for Semantic Segmentation,\" NeurIPS, 2021.\n\n[R3] W. Zhang, et al. \"K-Net: Towards Unified Image Segmentation,\" NeurIPS, 2021.\n\n[R4] B. Cheng, et al. \"Masked-Attention Mask Transformer for Universal Image Segmentation,\" CVPR, 2022.\n\n[R5] J. Jain, et al. \"OneFormer: One Transformer to Rule Universal Image Segmentation,\" CVPR, 2023.\n\n[R6] Q. Yu, et al. \"K-Means Mask Transformer,\" ECCV, 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks reviewer VQFt for the valuable time and constructive feedback. We provide point-to-point response below.\n### W1: Discussions with multi-task image segmentation methods\nThank you for your reminder. As you mentioned, we plan to introduce a new subsection in Section 4 to address multi-task image segmentation, which is intimately related to our work.\n\nMulti-task Image Segmentation aims to develop a cohesive architecture to address various segmentation challenges. K-Net [R3] pioneered this approach, leveraging dynamic kernel learning for mask generation. The dynamic kernel updating mechanism and binary matching strategy within K-Net underline its versatility and straightforwardness. Of note, this method stands out due to its ability to significantly reduce memory usage and improve inference speeds, making it especially suitable for real-time applications.\n\nIn recent developments, several architectures inspired by DETR have formulated different tasks within a mask classification paradigm, utilizing a transformer decoder with object queries. For instance, Max-DeepLab [R1] negates the need for box predictions by employing conditional convolutions, bridging the gap between box-dependent and box-independent methods for the first time. kMaX-DeepLab [R6], which views the object queries as cluster centers with adaptable embedding vectors, reimagines the cross-attention through the lens of k-means clustering. MaskFormer [R2] and its subsequent versions [R4] [R5] introduce a streamlined and effective inference strategy to merge outputs into a task-specific prediction format using a collection of binary masks.\n\nUnlike the above methods, our approach perceives these three closely-related tasks as a prototype-anchored classification challenge with distinct granularity levels. We define prototypes as class sub-centroids derived from the feature embeddings of training samples. Then, a test sample is directly classified based on its proximity to the nearest centroids. Such an approach, rooted in case-based reasoning, introduces a distinct element of ad-hoc interpretability to our method, as detailed in Section 5.4, Fig. 3.\n### W2: Semantic seg results for SemanticKITTI\nThanks. We agree to add SemanticKITTI experiments to support our claim better. The below results show that our method is +0.14% higher in mIoU than current works. We will include more comparisons in the revised version.\n\n| Method | mIoU | \n|---|---|\n| RPVNet [ICCV21]| 70.3 |   \n| $(AF)^2$-S3Net [CVPR21]| 70.8 |   \n| PVKD [CVPR22]| 71.2 |   \n| Ours| **72.6** |   \n### Weaknesses Minor: Complexity\nThanks. The discussion regarding the complexity of our ProtoSeg can be found in Appendix. In real-world implementations, it's minimal computation. After a few iterations, about 2.5ms to classify 10K points into $K=10$ prototypes. We've compared our method's mIoU scores, training and inference speed with the baseline model PTv2 on S3DIS, as shown in response to Mj9E's W1 due to the word limit. We find that prototype updates cause $\\sim4$% delay in training yet improve performance. Our method has faster inference due to no added overhead. These results and analysis would be updated in revised version.\n### Q1: 'IF-Then' for outdoor point clouds\nSorry for this confusion. 'IF-Then'[R1] explanation logic, please see Eqs.1-2 in Appendix. Please note that the interpretability of a classifier mainly comes from its decision-making mode, i.e., a test sample is directly classified into the class with the closest centroids instead of the training strategy or datasets. Therefore, 'IF-Then' applies to SemanticKITTI. We will update the related discussion to clarify this point in the revised version.\n\n[R1] Towards explainable deep neural networks (xdnn).\n### Q2: Convergence of $u^c$ and $v^c$ \nThanks. The Sinkhorn-Knopp iteration involves a regularization parameter $\\kappa$ (see Eq.10) that helps balance convergence speed and stability [34]. A smaller $\\kappa$ can lead to slower convergence but more accurate results, while a larger $\\kappa$ does the opposite. We just use $\\kappa=0.05$ following [34] for our experiments, not extensively fine-tuned. We will update the related discussion to clarify this point in the revised version.\n### Q3: Ablation study on ScanNetV2 \nThanks for your valuable comments. We performed ablation experiments for key components (like Table 6 (a)) on the Scannetv2 dataset to verify the validity of our Prototypical Classifier (Section 3.2) and supervision scheme (Section 3.3). The results are included in the response to saLq'W3 due to word limit. Other parameters are set experientially. To make our conclusions more convincing, we will use both S3DIS and ScanNetv2 datasets for evaluation in the revision. \n### Questions Minor 1: Meaning of 'epistemological'\nWe clarify that the 'epistemological framework' discusses the fundamental conceptual framework that guides our methodology design. Given the irregular and sparse nature of point clouds, we propose that point cloud segmentation naturally aligns with a classification problem. This foundational perspective shapes our research direction and motivates the creation of our prototype-anchored classification method, allowing us to seamlessly unify various segmentation tasks within a single framework.\n### Questions Minor 2: Some mistaking in Line 109\nOur apologies. Proofreading will be performed in the revised version.\n### Limitations: Potential negative societal impact\nThanks. On the upside, our method shows considerable promise for applications like autonomous vehicles, medical imaging. Nonetheless, it's crucial to recognize potential drawbacks. Inaccurate predictions in hands-on settings could risk safety. To mitigate any adverse impact, we recommend implementing a stringent security protocol should our method fail to operate as expected in real-world contexts. We will incorporate this discussion in our revised version.\n\nThanks for your thorough review. Let us know if you need more information."}, {"review_id": "Yx36QBgOgZ", "reviewer": "Reviewer_saLq", "summary": "This paper introduces PROTOSEG, a prototype-based model that aims to unify semantic, instance, and panoptic segmentation on point cloud. It treats these tasks as a classification problem with varying levels of granularity and achieves high performance on multiple benchmarks.", "questions": "Please refer to the **Weaknesses**.", "limitations": "Please refer to the **Weaknesses**.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The research topic addressed in this paper is the unification of three different segmentation tasks, which holds significant importance.\n2. The overall structure of the paper is well-presented and easy to follow.\n3. The formulations provided are detailed and comprehensive.\n4. The proposed method demonstrates excellent performance on both indoor and outdoor benchmarks.", "weaknesses": "1. The analysis of the parameters, flops, and time consumption of the proposed PROTOSEG is missing in this paper. The backbone network used is PTv2, and the original prediction head is replaced with the proposed prototypical classifier. Therefore, it is important to understand how this replacement affects the parameters, flops, and training/testing time, particularly considering that Sinkhorn-Knopp iteration is required to solve equation 9.\n2. How can we obtain the prototype instances in Figure 3 from the class sub-centroids, which are D-dimensional vectors?\n3. The ablation studies in Table 6 use OAcc as the metric, but mIoU is more commonly used in the Area5 S3DIS. It would be helpful if the authors explain why they chose OAcc and also provide the results of mIoU.\n4. The released code of PTv2 on S3DIS differs from the original paper and can achieve a higher mIoU of 72.0+ on Area 5. PROTOSEG achieves a comparable mIoU of 72.3 on S3DIS but notably improves to 76.4 mIoU on ScanNetV2. It would be helpful if the authors could provide some analysis or insights regarding this difference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Mj9E for the valuable time and constructive feedback. We provide point-to-point response below.\n\n### Weaknesses 1: Parameters, flops, and speed consumption\n\nWe detail the complexity analysis in the supplementary material. Although the Sinkhorn-Knopp iteration is utilized, in practice, this procedure adds only a minimal computational burden and affects speed marginally over a few iterations. We present a comparison between our proposed method and the baseline model, PTv2, in terms of the number of parameters, GFLOPs, and inference speed. The inference speed is averaged over the validation set, and both GFLOPs and inference speed measurements are obtained using an NVIDIA A100 GPU.  We will elaborate on efficiency considerations in our revised version.\n\n| Model | mIoU | \\# Parameters | GFLOPs | # Epoch | Training speed (hour/epoch) | Inference speed |\n|-------|------|---------------|--------|--------|--------|-----------------|\n| PTv2  | 71.6 | 3.5m          | 6.8    | 100    | 0.16 h           | 420 ms          |\n| Our   | 72.3 | 3.5m          | 6.8    | 100    | 0.166 h         | 404 ms          |\n\n### Weaknesses 2: Prototype instances based on the class sub-centroids\n\nSorry for the confusion. To investigate Ad-hoc Explainability, we train the model in the usual manner and calculate the class sub-centroids as prototypes. During the final stages of training, each sub-centroid is anchored to the training points closest to it, determined by the cosine similarity of their embeddings. Specifically, in the last 30 epochs, the prototypes are updated solely based on the embeddings of their corresponding anchored training points. As a result, the prototype instance depicted in Figure 3 represents the partial point cloud that is most similar to the class sub-centroids. The supplementary material provides a more comprehensive experimental setup for the ad-hoc explainability study.\n\n### Weaknesses 3: Metric of the ablation studies\n\nThanks for your valuable comments. We acknowledge that mIoU is a commonly used metric in the Area5 S3DIS benchmark. The following table presents the mIoU scores in the ablation experiments. We will use both OAcc and mIOU as indicators of ablation experiments in the revised version. \n\nAblation studies of Key Components\n\n| Prototypical | $\\mathcal{L}_{\\text{PCS}}$ | $\\mathcal{L}_{\\text{PPS}}$ | $\\mathcal{L}_{\\text{PPC}}$ | S3DIS  | ScanNetv2 |\n|--------------|----------------------------|----------------------------|----------------------------|--------|-----------|\n| Classifiers  | (Eq.15)                    | (Eq.13)                    | (Eq.14)                    | (mIoU) | (mIoU)    |\n|              | √                    |                            |                            | 71.53  | 75.36     |\n| √       | √                     |                            |                            | 71.74  | 75.84     |\n| √       |√                     | √                     |                            | 72.05  | 76.07     |\n| √      | √                     |                            | √                     | 72.06  | 76.12     |\n| √       | √                     | √                     |√                     | 72.34  | 76.32     |\n\n\nAblation studies of Prototype Number $K$\n\n| # Prototype | S3DIS mIoU (%) |\n|--------------|-----------|\n| K=1     | 71.48     |\n| K=5     | 71.84     |\n| K=10    | 72.34     |\n| K=20    | 72.18     |\n| K=50    | 72.05     |\n\nAblation studies of Distance Measure\n\n| Distance    | S3DIS mIoU (%)   |\n|-------------|------------|\n| Standard    | 72.10      |\n| Huberized   | 72.17      |\n| Cosine      | 72.34      |\n\nAblation studies of Momentum Coefficient\n\n| Coefficient $\\mu$ | S3DIS mIoU (%) |\n|-------------------|-----------|\n| 0.9         | 71.62     |\n| 0.99        | 71.88     |\n| 0.999       | 72.34     |\n| 0.9999      | 72.04     |\n\nAblation studies of Class Balance\n\n| Strategy    | S3DIS mIoU (%) |\n|-------------|-----------|\n| w/o      | 71.49     |\n| $w^c$   | 72.34     |\n| $M_{p}$ | 72.14     |\n| $M_{n}$ | 71.83     |\n\nAblation studies of Sinkhorn Iterations $S$\n\n| Iterations (Eq.10) | S3DIS mIoU (%) |\n|------------|-------|\n| $S=1$    | 71.29 |\n| $S=3$    | 72.34 |\n| $S=5$    | 71.43 |\n\n\n### Weaknesses 4: Improvement on S3DIS and ScanNetV2 datasets\n\nGood point! Thanks to PROTOSEG's intuitive mechanism, the statistical significance of the class sub-centroid allows it to automatically discern the potential distribution of each class and capture a representative set of local means. In contrast, PTv2, a non-prototype approach, learns only a single weight (or query) vector per class. PROTOSEG is better equipped to adapt to the rich in-class variations, leading to a marked performance improvement on ScanNetV2, a larger dataset with a greater number of classes.\n\nWe sincerely appreciate your thorough review and invaluable feedback, which have significantly contributed to the improvement of our work. We hope we have addressed all of your concerns. Please let us know if you require any additional information."}, {"review_id": "1H5HLd6paq", "reviewer": "Reviewer_Mj9E", "summary": "In this paper, the authors propose to use prototype paradigm to develop a generic framework for semantic segmentation, instance segmentation and panoptic segmentation. The proposed ProtoSeg use prototype association to distribute points to different prototypes and dynamic update prototypes according to the assignments. Three optimization terms are introduced to realize different level of distance regularization.", "questions": "Please refer to the weakness part. ", "limitations": "There is no explicit limitation discussion section in this paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed prototype-based method is reliable and convincing.\n2. The discussed problem of designing unified segmentation model is interesting and important.\n3. The performance of the proposed method is strong and achieves SOTA under most conditions.\n4. The ablations and discussions are thorough to analyze the designing of the model.\n5. The illustrations are clear and helpful for better comprehension.\n", "weaknesses": "1. Limitation discussion is missing.\n2. In ablation studies, why use OAcc as metric instead of mIoU? In segmentation, mIoU is more important and more readers care about mioU than OAcc. It would be better if the metric can be switched to mIoU.\n3. What is the computation efficiency of the model? Since it is a unified model, will it be much slower than task-specific models? What is the relation between efficiency and prototype number?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Mj9E for the valuable time and constructive feedback. We provide a point-to-point response below.\n### Weaknesses 1: Limitation\n\nSorry for the confusion. We discussed some limitations of this work in the supplementary material, which is related to model efficiency.\n\n### Weaknesses 2: Metric of the ablation studies\n\nThanks for your valuable comments. We understand that mIoU (mean Intersection over Union) is a widely recognized and essential metric in segmentation tasks. The following table presents the mIoU scores in the ablation experiments. We will use OAcc and mIOU as indicators of ablation experiments in the revised version.\n\nAblation studies of Key Components\n\n| Prototypical | $\\mathcal{L}_{\\text{PCS}}$ | $\\mathcal{L}_{\\text{PPS}}$ | $\\mathcal{L}_{\\text{PPC}}$ | mIoU  |\n|--------------|----------------------------|----------------------------|----------------------------|-------|\n| Classifiers  | (Eq.15)                    | (Eq.13)                    | (Eq.14)                    | (\\%)  |\n|              | √                     |                            |                            | 71.53 |\n| √       | √                     |                            |                            | 71.74 |\n| √       | √                     | √                     |                            | 72.07 |\n| √       | √                     |                            | √                     | 72.08 |\n| √       | √                     | √                     | √                     | 72.34 |\n\n\nAblation studies of Prototype Number $K$\n\n| # Prototype | mIoU (%) |\n|--------------|-----------|\n| K=1     | 71.48     |\n| K=5     | 71.84     |\n| K=10    | 72.34     |\n| K=20    | 72.18     |\n| K=50    | 72.05     |\n\nAblation studies of Distance Measure\n\n| Distance    | mIoU (%)   |\n|-------------|------------|\n| Standard    | 72.10      |\n| Huberized   | 72.17      |\n| Cosine      | 72.34      |\n\nAblation studies of Momentum Coefficient\n\n| Coefficient $\\mu$ | mIoU (%) |\n|-------------------|-----------|\n| 0.9         | 71.62     |\n| 0.99        | 71.88     |\n| 0.999       | 72.34     |\n| 0.9999      | 72.04     |\n\nAblation studies of Class Balance\n\n| Strategy    | mIoU (%) |\n|-------------|-----------|\n| w/o      | 71.49     |\n| $w^c$   | 72.34     |\n| $M_{p}$ | 72.14     |\n| $M_{n}$ | 71.83     |\n\nAblation studies of Sinkhorn Iterations $S$\n\n| Iterations (Eq.10) | mIoU (%) |\n|------------|-------|\n| $S=1$    | 71.29 |\n| $S=3$    | 72.34 |\n| $S=5$    | 71.43 |\n\n\n### Weaknesses 3: Computation efficiency\n\nThanks for your careful review. (i) Computation Efficiency: We present a comparison between our proposed method and the baseline model, PTv2, in terms of the number of parameters, GFLOPs, and inference speed. The inference speed is averaged over the validation set, and both GFLOPs and inference speed measurements are obtained using an NVIDIA A100 GPU.\n\n| Model | mIoU | \\# Parameters | GFLOPs | # Epoch | Training speed (hour/epoch) | Inference speed |\n|-------|------|---------------|--------|--------|--------|-----------------|\n| PTv2  | 71.6 | 3.5m          | 6.8    | 100    | 0.16 h           | 420 ms          |\n| Our   | 72.3 | 3.5m          | 6.8    | 100    | 0.166 h         | 404 ms          |\n\n\n(ii) Inference Speed: One limitation of our algorithm arises from the additional clustering loops (referenced in Eq.10) present in each training iteration. These loops could potentially influence computation efficiency in terms of time complexity. However, the inclusion of three recursive clusterings ensures global model convergence and introduces only a slight computational overhead. This overhead leads to approximately a 4\\% reduction in training speed compared to the baseline model PTv2. Notably, while the number of parameters and GFLOPs remain comparable, our method yields better results and a faster inference rate.\n\n(iii) The table below highlights the relationship between the number of prototypes and the efficiency of the semantic segmentation task on the S3DIS dataset. We found that the number of prototypes doesn't markedly influence the model's computational efficiency. We will elaborate on efficiency considerations in our revised version.\n\n| \\# Prototype | # Epoch | Training speed (hour/epoch) | Inference speed |\n|--------------|--------|----------------|-----------------|\n| K = 1        |  100    |0.166 h         | 400 ms          |\n| K = 5        |  100    |0.166 h         | 402 ms          |\n| K = 10       |  100    |0.166 h         | 404 ms          |\n| K = 20       |  100    |0.167 h         | 420 ms          |\n| K = 50       |  100    |0.167 h         | 430 ms          |\n\nWe appreciate your thoughtful review again and hope we have addressed your concerns. Please let us know if you'd like any further information."}]

Author rebuttal:
Thank you so much for your careful review and suggestive comments. We will revise our paper according to your comments. The major changes are as follows:

+ We will detail the challenges and methods of unified segmentation tasks, introduce a diagram of the prototyping process, and detail the use of prototype classifiers in instance segmentation and panoramic segmentation tasks, according to Reviewer MoHU's comments. In addition, we will add ablation experiments on the effect of the number of instances on the prototype, and look forward to future work.

+ We will discuss relations to multi-task methods, adds experiments with SemanticKITTI data, thoroughly examines computation efficiency and convergence in Sinkhorn-Knopp iteration, and conducts ablative experiments on Scannetv2, according to Reviewer VQFt's comments. It also ensures language accuracy and discusses societal implications.

+ We will offer more detailed discussions regarding the computation efficiency, including complexity, parameters, flops, and time consumption, according to Reviewer VQFt, saLq, and Mj9E comments.

+ Lastly, as per Reviewer saLq and Mj9E's suggestions, ablative experiment results for mIoU on S3DIS and Scannetv2 will be included.

Sincerely yours,

Authors.


Meta-review: The paper proposes a transformer based unified framework for semantic, instance, and panoptic segmentation. Experiments on benchmark datasets confirm the efficacy of the design. Reviewers are initially less clear about the challenge of such a unification, how it is different from multi-task segmentation, complexity of the proposed method, and other technical details; they also require additional experiments on additional benchmarks. The authors provide detailed responses that address these concerns. Please include these additional results in the final version of the paper. Congratulations!
Decision: Accept (poster)