SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use PointTransformer V2 [2] as the encoder  $\phi$  and replace the original MLP-based semantic prediction head with our prototypical classifier. Instance segmentation and panorama segmentation tasks can be viewed as more fine-grained prototype classification problems. For these tasks, we additionally learn fine-grained prototypes  $P^{c,o} \in \mathbb{R}^{D \times T}$ ,  $P^c \in \mathbb{R}^{T \times K}$ , and a fine-grained permutation matrix  $M^{c,o} \in \{0,1\}^{T \times N_c}$ ,  $M^c \in \{0,1\}^{K \times T}$ , where  $o \in O$ ,  $T, O$  are the number of instance prototypes and instances per class. The association and update of prototypes follow §3.2. Instance-level supervision using a cross-entropy loss is incorporated into our supervision scheme (§3.3).
Reproducibility. Training and testing are conducted on eight NVIDIA A100 GPUs. More details and parameter settings can be found in the Appendix.
4 Important Knows and Knowledge Gap
Specialized point cloud segmentation. Point cloud segmentation comprises three separate tasks, i.e., semantic, instance and panoptic segmentation, each of which focuses on different semantic aspects. Historically, researchers have devised specialized models and optimization objectives for each task.
Semantic segmentation is to achieve comprehension of high-level semantic concepts through the grouping of points into discrete semantic units. The advent of point convolution [16] has facilitated the development of sophisticated models, encompassing context aggregation [27, 19], graph convolution integration [24, 17], and contrastive learning [32, 33]. Recently, Transformer architectures [18, 2], which has been successful in vision [34, 35, 36], have obtained considerable research interest.
Instance segmentation involves assigning foreground points to individual object instances, similar to video tasks [37], which can be achieved through three main approaches: ① top-down models [21, 12] that initially detect object bounding boxes and generate an instance mask for each box, ② bottom-up models [38, 39] that learn distinctive point embeddings on instance boundaries, energy levels, geometric structures, and point-center offsets, and group them into instances, and ③ single-shot models [22] that directly predict instance masks through a set of learnable object queries.
Panoptic segmentation seeks comprehensive scene understanding, considering semantic relations between background points and instance memberships of foreground points. Traditional solutions involve decomposing the problem into manageable tasks such as box-based segmentation [11, 20], thing-stuff merging [40], instance clustering [23, 41, 25]. More recent DETR-like approaches [28] shifted towards an end-to-end scheme using Transformer [42] and achieved compelling performance.
Prototypical classifier. Prototype-based classification, an exemplar-driven approach that compares observations with representative examples, has gained particular attention among various machine learning algorithms, which include classical statistics-based methods, Support Vector Machines, and Multilayer Perceptrons. The nearest neighbors rule, the earliest prototype learning method [43], has led to the development of many well-known, non-parametric classifiers [44], such as generalized Learning Vector Quantization (LVQ) [45], and Neighborhood Component Analysis [46]. Metric learning [47, 48] is also naturally related to prototype learning. Recent efforts have attempted to integrate deep learning into prototype learning, demonstrating its potential in few-shot [49], self-supervised learning [50], weakly supervised learning [51, 52], as well as supervised learning [53, 54]
and interpretable networks [55]. Building on these successes, we aim to advance this research by developing a universal 3D segmentation framework based on prototypical classification. Our method, PROTOSEG, incorporates prototype-anchored classification design and simple Transformer architecture to better capture the nature of prototypical learning and effectively handle the heterogeneity across various segmentation tasks using the same architecture.
Universal 3D Segmentation. Universal segmentation strives to adopt a unified architecture capable of addressing diverse segmentation tasks. One of the early attempts at unification through multi-task learning is MT-PNet [56], which uses two branches to predict semantic class labels and instance embedding labels, respectively. More recent studies, such as those presented in [57], employ clustering schemes to integrate semantic and instance information, but their heuristic post-processing technique (e.g., mean shift) can bring serious computational burden. In comparison to these pioneering efforts, PROTOSEG distinguishes itself by being:  $①$  more transparent - leveraging straightforward, case-based reasoning for 3D tasks;  $②$  more flexible - handling multiple segmentation tasks simultaneously; and  $③$  more powerful - leading specialized methods by notable margins.
Multi-Task Image Segmentation Multi-Task Image Segmentation aims to develop a cohesive architecture to address various segmentation challenges. K-Net [58] pioneered this approach, leveraging dynamic kernel learning for mask generation. In recent developments, several architectures inspired by DETR [59] have formulated different tasks within a mask classification paradigm. For instance, Max-DeepLab [60] and kMaX-DeepLab [61] negate the need for box predictions by employing conditional convolutions, bridging the gap between box-dependent and box-independent methods for the first time. MaskFormer [62] and its subsequent versions [63, 64] introduce a streamlined and effective inference strategy to merge outputs into a task-specific prediction format using a collection of binary masks. In addition, methods based on dense prediction [65], optimal transport [66], graph modeling [67], and clustering [68, 69, 70] also have achieved great success.
Unlike the above methods, our approach perceives these three closely-related tasks as a prototype-anchored classification challenge with distinct granularity levels. We define prototypes as class sub-centroids derived from the feature embeddings of training samples. Then, a test sample is directly classified based on its proximity to the nearest centroids. Such an approach, rooted in case-based reasoning, introduces a distinct element of ad-hoc interpretability to our method (see Fig. 3 in §5.4).
5 Empirical Evidence
PROTOSEG is the first framework to support three core point cloud segmentation tasks with a single unified architecture. To demonstrate its broad applicability and wide benefit, we conduct extensive experiments on semantic ( $\S 5.1$ ), instance ( $\S 5.2$ ), panoptic ( $\S 5.3$ ) segmentation, and carry out ablation studies ( $\S 5.4$ ) related to our framework design. More qualitative results are available in Appendix.
5.1 Experiment on Semantic Segmentation
Dataset. S3DIS [9], a large-scale indoor point cloud dataset, encompasses point clouds from 271 rooms across 6 areas. Each room represents a medium-sized point cloud, annotating every point with a semantic label from one of the 13 classes. ScanNet V2 [10] provides over 1,500 indoor scenes and around 2.5 million annotated RGB-D images with approximately  $90\%$  surface coverage. The benchmark is evaluated on 20 semantic classes, which include 18 different object classes. SemanticKITTI [11] is introduced based on the well-known KITTI Vision [71] benchmark illustrating complex outdoor traffic scenarios. It contains 22 data sequences, 43,552 frames of outdoor scenes, of which 23,201 frames with panoptic labels are used for training and validation, and the remaining 20,351 frames without labels are used for testing. There are annotated point-wise labels in 20 classes for segmentation tasks, 8 of which are defined as thing classes.
Metric. To evaluate our semantic segmentation performance, we apply the mean Intersection-over-Union (mIoU  $\uparrow$ ), the overall accuracy (OAcc  $\uparrow$ ) taking all points into consideration and the average class accuracy (mAcc  $\uparrow$ ) of all semantic classes upon the whole dataset.
Performance Comparison. Table 1 and Table 2 show the results of our PROTOSEG model compared with previous methods on S3DIS [9] and ScanNet v2 [10], respectively. Our model outperforms prior methods across all evaluation metrics. Notably, PROTOSEG significantly outperforms PTv2 [2] by  $0.9\%$  and  $1.2\%$  mIoU on the ScanNet v2 [10] validation and test set, and
Table 1: Comparisons of semantic segmentation performance on S3DIS [9] Area 5 (see §5.1).
Method	OAcc	mAcc	mIoU
JSNet [57] [AAAI'20]	87.7	61.4	54.5
SegGCN [72] [CVPR'20]	88.2	70.4	63.6
SCF-Net [73] [CVPR'21]	88.4	71.6	82.7
PACConv [74] [CVPR'21]	-	-	66.6
RepSurf-U [75] [CVPR'22]	90.2	76.0	68.9
CBL [33] [CVPR'22]	90.6	75.2	69.4
PTv1 [18] [ICCV'21]	90.8	76.5	70.4
FastPT [76] [CVPR'22]	-	77.9	70.3
PointMixer [77] [ECCV'22]	-	77.4	71.4
PTv2 [2] [NeurIPS'22]	91.1	77.9	71.6
StratifiedFormer [78] [CVPR'22]	91.5	78.1	72.0
Ours (Area 5)	92.2	78.6	72.3
Table 2: Comparisons of semantic segmentation with mIoU on ScanNet v2 [10] (see §5.1).
Method	Test	Val.
SegGCN [72] [CVPR'20]	58.9	-
RandLA-Net [79] [CVPR'20]	64.5	-
PointASNL [27] [CVPR'20]	66.6	63.5
RPNet [80] [ICCV'21]	68.2	-
FusionNet [81] [ECCV'20]	68.8	-
JSENet [82] [ECCV'20]	69.9	-
RepSurf-U [75] [CVPR'22]	70.2	-
PTv1 [18] [ICCV'21]	-	70.6
PointNeXt [19] [NeurIPS'22]	71.2	71.5
StratifiedFormer [78] [CVPR'22]	73.7	74.3
PTv2 [2] [NeurIPS'22]	75.2	75.4
Ours	76.4	76.3
surpasses other methods in all metrics on the S3DIS [9] dataset. These results further support the robustness and effectiveness of our method for large-scale point cloud semantic segmentation tasks.
Notably, our method yields a more substantial performance enhancement on ScanNet V2 than on S3DIS compared to the main competitor PTv2 [2]. This discrepancy can be attributed to PROTOSEG's statistical characteristics, which can mine richer intra-class variations on the larger scale and more categories ScanNet V2. To evaluate the performance more comprehensively, we report the results of semantic segmentation on the SemanticKITTI validation set. PROTOSEG achieves a mIou of  $74.2\%$ , confirming the performance benefits of our model.
Table 3: Comparisons of semantic segmentation performance on SemanticKITTI val set (see §5.1).
Method	mIoU
Cylinder3D [83][CVPR'21]	65.9
2DPASS [84][ECCV'22]	69.3
(AF)2-S3Net [85][CVPR'21]	74.2
Ours	74.2
5.2 Experiment on Instance Segmentation
Dataset. S3DIS [9] and ScanNet V2 [10] are utilized for instance segmentation experiments.
Metric. We use mean coverage (mCov ↑), mean weighed coverage (mW Cov ↑), mean precision (mPrec ↑), and mean recall (mRec ↑) as the evaluation metrics for S3DIS [9]. Additionally, ScanNet V2 [10] is evaluated using mean average precisions (mAPs ↑) under different IoU thresholds.
Performance Comparison. As is depicted in Tables 4-5, our method significantly outperforms other methods on the S3DIS [9] dataset. The main criteria, mCov and mWCoV, take into account both the completeness and accuracy of instance segmentation results. Specifically, in terms of the main criteria, our method yields a mCov of  $66.8\%$ , a mW-Cov of  $68.4\%$ , a mPrec of  $69.7\%$  and a mRec of  $66.3\%$  in Area 5, surpassing most methods. Furthermore, the superior performance on ScanNet V2 [10] dataset in both the val. and test sets also highlights the effectiveness of our approach for instance segmentation. The significant improvements across various metrics and benchmarks demonstrate the efficacy and robustness of PROSEG, even in challenging scenarios.
Table 4: Comparisons of instance segmentation performance on S3DIS [9] Area 5 (see §5.2 for more details).
Method	mCov	mW Cov	mPrec	mRec
PointGroup [86] [CVPR'20]	-	-	61.9	62.1
MaskGroup [39] [ICME'22]	-	-	62.9	64.7
SSTNet [22] [ICCV'21]	42.7	59.3	65.5	64.2
DyCo3D [87] [CVPR'21]	63.5	64.6	64.3	64.2
HAIS [38] [ICCV'21]	64.3	66.0	71.1	65.0
DKNet [88] [ECCV'22]	64.7	65.6	70.8	65.3
Ours	66.8	68.4	69.7	66.3
Table 5: Comparisons of instance segmentation performance on ScanNet v2 [10] (see §5.2 for more details).
Module	Test	Val.
mAP	\( mAP_{50} \)	mAP	\( mAP_{50} \)
DyCo3D [87] [CVPR'21]	39.5	64.1	35.4	57.6
PointGroup [86] [CVPR'20]	40.7	63.6	34.8	56.7
MaskGroup [39] [ICME'22]	43.4	66.4	42.0	63.3
HAIS [38] [ICCV'21]	45.7	69.9	43.5	64.1
SoftGroup [12] [CVPR'22]	50.4	76.1	46.0	67.6
SSTNet [22] [ICCV'21]	50.6	69.8	49.4	64.3
Ours	51.2	78.1	47.8	68.4
5.3 Experiment on Panoptic Segmentation
Dataset. SemanticKITTI [11] is utilized for panoptic segmentation experiments.
Table 6: Comparisons of panoptic segmentation performance on SemanticKITTI [11] test (see §5.3).
Method	PQ	mIou	\( PQ^† \)	RQ	SQ	\( PQ^{Th} \)	\( RQ^{Th} \)	\( SQ^{Th} \)	\( PQ^{St} \)	\( RQ^{St} \)	\( SQ^{St} \)
LPSAD [90] [IROS'20]	38.0	50.9	47.0	48.2	76.5	25.6	31.8	76.8	47.1	60.1	76.2
Panoster [91] [RAL'21]	52.7	59.9	59.9	64.1	80.7	49.4	58.5	83.3	55.1	68.2	78.8
Panoptic-PolarNet [23] [CVPR'21]	54.1	59.5	60.7	65.0	81.4	53.3	60.6	87.2	54.8	68.1	77.2
DS-Net [41] [CVPR'21]	55.9	61.6	62.5	66.7	82.3	55.1	62.8	87.2	56.5	69.5	78.7
EfficientLPS [40] [TR'21]	57.4	61.4	63.2	68.7	83.0	53.1	60.5	87.8	60.5	74.6	79.5
GP-S3Net [25] [ICCV'21]	60.0	70.8	69.0	72.1	82.0	65.0	74.5	86.6	56.4	70.4	78.7
SCAN [28] [AAAI'22]	61.5	67.7	67.5	72.1	84.5	61.4	69.3	88.1	61.5	74.1	81.8
Panoptic-PHNet [15] [CVPR'22]	61.5	66.0	67.9	72.1	84.8	63.8	70.4	90.7	59.9	73.3	80.5
Ours	62.4	67.5	68.5	74.4	83.4	65.6	72.6	89.1	61.8	75.7	79.2
Figure 3: Interpretable prototype (i.e., class sub-centroids) with  $IF \cdots$ . Then rules (left) and decision-making process based on distance to prototypes (right). colors represent prototype instance for three S3DIS [9] classes,  $[\cdot, \cdot]$  stands for distance between point embedding  $E$  and instances.
Metric. We use the panoptic quality (PQ) [89] as our main metric to evaluate the performance of panoptic segmentation. PQ can be seen as the multiplication of segmentation quality (SQ) and recognition quality (RQ). These three metrics can be extended to things and stuff classes, denoted as  $\mathrm{PQ^{Th}}$ ,  $\mathrm{PQ^{St}}$ ,  $\mathrm{RQ^{Th}}$ ,  $\mathrm{RQ^{St}}$ ,  $\mathrm{SQ^{Th}}$ ,  $\mathrm{SQ^{St}}$ , respectively.  $\mathrm{PQ}^{\dagger}$  replaces PQ with IoU for stuff classes. Performance Comparison. Table 6 presents a comparison of panoptic segmentation performance on the SemanticKITTI [11] test. Our proposed method demonstrates a significant improvement over the best-performing baseline method, with a  $62.4\%$  increase in PQ. It is noteworthy that our unified framework demonstrates superior performance in both things and stuff classes, as evidenced by outperforming the second-best method, Panoptic-PhNet [15], in 7 out of 11 metrics.
Overall, PROTOSEG demonstrates excellent performance across various segmentation tasks (see §5.1-5.3) on both indoor and outdoor datasets. Notably, it does not require inherent modifications to the architecture or the training regime, which highlights its generality and flexibility.
5.4 Diagnostic Experiment
This section conducts ablation studies for PROTOSEG's semantic segmentation on S3DIS [9] Area5.
Study of Ad-hoc Explainability. In §2 and §5.1-5.3, we have explained the simple yet transparent mechanism of PROTOSEG and verified its robustness and effectiveness in representation learning. We now highlight PROTOSEG's interpretability in Fig. 3, which depicts prototypes that represent specific objects in the training set. Fig. 3 (left) displays representative instances, serving as interpretable prototypes (class sub-centroids) for three S3DIS [9] classes. These objects, diverse in appearance, viewpoints, illuminations, and scales, characterize their respective classes, providing a human-interpretable overview. Leveraging the simplicity of the Nearest Centroids mechanism, we formulate intuitive  $IF\cdots$  Then rules, providing an intuitive glimpse into PROTOSEG's decision-making process. Furthermore, we can elucidate PROTOSEG's predictions by visualizing the computed distance (similarity) between the query and prototypes. As illustrated in Fig. 3 (right), the model accurately classifies an observation, perceiving it as more closely resembling a particular chair exemplar (see the top right of Fig. 2). Overall, this visualization clearly shows that the classification decision procedure relies on the notion of distance beyond the selection of the encoder  $\phi$ .
Key Components. We first investigate our Prototypical Classifier (§3.2) and supervision scheme (§3.3) on S3DIS [9] and ScanNet V2 [10] datasets. As shown in Table 7a, introducing prototypical classifiers in conjunction with  $\mathcal{L}_{\mathrm{PCS}}$  improves the performance to  $71.74\%$  and  $75.84\%$ . Adding either  $\mathcal{L}_{\mathrm{PPS}}$  or  $\mathcal{L}_{\mathrm{PPC}}$  to the prototypical classifiers and  $\mathcal{L}_{\mathrm{PCS}}$  promotes performance gains, indicating the
Table 7: A set of ablative studies (see §5.4) on S3DIS [9] Area 5.
(a) Key Components
Prototypical
Classifiers	LPCS
(Eq. 15)	LPPS
(Eq. 13)	LPPC
(Eq. 14)	S3DIS
(mIoU)	ScanNetv2
(mIoU)
✓	71.53	75.36
✓	✓	71.74	75.84
✓	✓	✓	72.05	76.07
✓	✓	✓	72.06	76.12
✓	✓	✓	✓	72.34	76.32
(b) Prototype Number  $K$
# Prototype	mIoU (%)
K=1	71.48
K=5	71.84
K=10	72.34
K=20	72.18
K=50	72.05
(c) Distance Measure
Distance	mIoU (%)
Standard	72.10
Huberized	72.17
Cosine	72.34
(d) Momentum Coefficient
Coefficient μ	mIoU (%)
0.9	71.62
0.99	71.88
0.999	72.34
0.9999	72.04
(e) Class Balance
Strategy	mIoU (%)
w/o	71.49
w^c	72.34
M_p	72.14
M_n	71.83
(f) Sinkhorn Iterations  $S$
Iterations
(Eq. 10)	mIoU
(%)
S = 1	71.29
S = 3	72.34
S = 5	71.43
value of explicitly learning point-prototype relations. The improvements observed by introducing prototypical classifiers and supervision scheme (mIoU:  $71.53\% \rightarrow 72.34\%$  on S3DIS [9],  $75.36\% \rightarrow 76.32\%$  on ScanNet V2 [10]) confirm the effectiveness of our proposed framework.
Prototype Number Per Class. We then analyze the impact of the number of prototypes per class on segmentation performance, as reported in Table 7b. With a single prototype ( $K = 1$ ), each class is represented by the mean embedding of its point samples, yielding the mIoU score of  $71.48\%$ . The mIoU score peaks at  $72.34\%$  when  $K = 10$ . Further increment of prototypes to  $K = 20$ ,  $K = 50$  only results in a slight decrease in performance, with mIoU scores of  $72.34\% \rightarrow 72.18\% \rightarrow 72.05\%$ , respectively. These results suggest an optimal balance between accuracy and computational cost at  $K = 10$  prototypes per class, affirming our rationale for leveraging multiple prototypes to encapsulate intra-class variations and enhance segmentation performance. It would be interesting to find ways to ascertain this number automatically. In our exploration, we encountered clustering techniques like [92] that might help, but complex algorithms led to time challenges.
Distance Measure. Next, we examine the impact of different distance metrics on our Supervision Scheme (see Eqs. 13-15 in §3.3). By default, we use cosine distance (denoted as Cosine in Table 7c). We also consider two alternative metrics, i.e., standard Euclidean distance (Standard) and Huber-like function [93] (Huberized). Table 7c demonstrates that the Cosine distance surpasses the unnormalized Euclidean metrics, affirming its superiority in measuring point-prototype similarity.
Momentum Coefficient. Table 7d quantifies the effect of the momentum coefficient  $(\mu$  in Eq. 12). Our model achieves the best performance when the momentum coefficient is set to 0.999. As the momentum coefficient decreases, the performance gradually declines. These results suggest that a higher value (i.e., slower updating) generally leads to better results in the common case.
Class Balance Strategy. Next, we proceed to validate the effectiveness of our strategy in mitigating the issue of category imbalance. As Table 7e shown, the baseline  $(w / o)$  fixes  $w^{c} = 1$  in Eq. 12,  $M_{l}$  [31] and  $M_{u}$  [94] utilize parameter and non-parametric augmented memory banks, respectively. The reported results provide evidence for the efficacy of our prototype calibration strategy.
Sinkhorn-Knopp Iteration. Finally, we verify the effect of iterations on the model. Table 7f shows that the optimal segmentation results can be achieved using a smaller number ( $S = 3$ ) of iterations. This further verifies the efficiency of our Prototype Association (see §3.2).
We present PROtoSEG, a novel, prototype-anchored classification framework for 3D point cloud segmentation that unifies semantic, instance, and panoptic segmentation tasks. By considering these tasks under the umbrella of classification problems but at various granularity, PROtoSEG offers an elegant, general, and flexible solution that eschews task-specific network architectures. Compared to task-specific models, PROtoSEG offers several advantages: i) The prototype-based classification mechanism is simple and transparent. ii) Our supervision scheme, which leverages prototype-based metric learning, exhibits robust representational learning ability. iii) PROtoSEG offers ad-hoc explainability by anchoring class exemplars to real observations. Our comprehensive experiments demonstrate the superior performance of PROtoSEG in terms of efficacy and enhanced interpretability. Overall, we believe that our work may motivate further research in this field.
Acknowledgement. This work was supported in part by the National Natural Science Foundation of China (No. 62176139, 62106128, 62176141), the Major basic research project of Shandong Natural Science Foundation (No. ZR2021ZD15), the Natural Science Foundation of Shandong Province (No. ZR2021QF001), the Young Elite Scientists Sponsorship Program by CAST (No. 2021QNRC001), the Open project of Key Laboratory of Artificial Intelligence, Ministry of Education, the Shandong Provincial Natural Science Foundation for Distinguished Young Scholars (ZR2021JQ26), the Taishan Scholar Project of Shandong Province (tsqn202103088).