SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {P} (c; q _ {j}) = \frac {\mathbb {P} _ {c l s} (c ; q _ {j})}{\sum_ {l \in \mathcal {K} ^ {t}} \mathbb {P} _ {c l s} (l ; q _ {j})} (1 - \mathbb {P} (\mathbf {0}; q _ {j}))
$$
4.6 Alleviating catastrophic forgetting for incremental learning
Following the success of exemplar replay in avoiding catastrophic forgetting of the old classes during incremental learning for object detection [18, 11, 41], we adopt it for the task of incremental learning in 3D instance segmentation where we use exemplars from the classes of the previous task to fine-tune the model trained on the novel classes. In our setting, we use the same dataset for the three tasks and mask the classes of the previous task when training on the novel classes from the current task. As a result, the novel classes of the current task might be encountered again when replaying the exemplars from the previous task, as the same scenes are being used in fine-tuning.
5.1 Open-world evaluation protocol
We use our proposed splits of classes which mimic the challenges that are mostly faced in the open-world to ensure a strict performance evaluation for 3D instance segmentation models.
Evaluation metrics. We adopt three common evaluation metrics, wilderness impact (WI) [9], absolute open set error (A-OSE) [26], and the recall of the unknown classes (U-Recall) [1, 24, 11]
Table 2: State-of-the-Art comparison for 3D-OWIS model. We show a comparison of performance under the three open-world scenarios, where 3D-OWIS-PC - CT is our model 3D-OWIS without Probability Correction (PC) and Confidence Thresholding (CT). We rely on the metrics used in the open-world literature, A-OSE which quantifies the number of unknown objects misclassified as one of the known classes, WI which measures the impact of the unknown class on the precision of the model on the known classes, and the U-Recall to evaluate the model's ability to recover the unknown objects. We show that 3D-OWIS performs remarkably better than the other models under all scenarios when dealing with the known classes, and superior performance in split A and B, and slightly less performance in split C when handling the unknown objects. We also provide a closed-setting comparison between Mask3D and Oracle (Ours with access to unknown labels).
Task IDs (→)	Task 1	Task 2	Task 3
WI(↓)	A-OSE(↓)	U-Recall(↑)	mAP (↑)	WI(↓)	A-OSE(↓)	U-Recall(↑)	mAP (↑)	mAP (↑)
Current known	All	Previously known	Current known	All	Previously known	Current known	All
Split A
Oracle	0.129	227	55.94	38.75	38.60	0.03	112	45.40	38.25	20.91	29.40	29.58	17.78	26.10
Mask3D [29]	-	-	-	39.12	39.12	-	-	-	38.30	20.57	29.15	28.61	18.33	25.58
3D-OW-DETR [11]	0.547	721	22.14	35.56	35.05	0.282	253	26.24	18.18	13.62	15.76	21.56	08.38	17.67
3D-OWIS-PC - CT	1.589	707	30.72	37.50	37.00	0.000	4	04.75	11.00	17.30	14.10	21.40	08.00	17.50
Ours: 3D-OWIS	0.397	607	34.75	40.2	39.7	0.007	126	27.03	29.40	16.40	22.70	20.20	15.20	18.70
Split B
Oracle	1.126	939	70.31	24.57	24.80	0.180	441	73.16	25.50	20.30	23.40	23.40	30.40	26.00
Mask3D [29]	-	-	-	23.48	23.48	-	-	-	21.81	18.91	20.37	24.20	29.22	26.06
3D-OW-DETR [11]	3.229	1935	17.18	20.00	19.73	2.053	1389	33.31	12.36	13.86	12.93	07.27	18.96	11.62
3D-OWIS-PC - CT	3.133	1895	21.67	18.94	18.70	3.169	1081	26.63	18.00	16.40	17.20	17.30	20.10	18.30
Ours: 3D-OWIS	3.684	1780	24.79	23.60	23.30	0.755	581	24.21	18.70	17.30	17.90	18.70	24.60	20.90
Split C
Oracle	1.039	651	71.61	23.30	23.6	0.249	591	62.83	20.50	18.40	19.60	25.30	28.20	26.30
Mask3D [29]	-	-	-	20.82	21.15	-	-	-	22.67	26.67	24.13	25.41	25.21	25.35
3D-OW-DETR [11]	1.463	1517	13.00	14.81	14.59	1.330	847	16.04	08.00	17.41	12.40	08.81	15.63	11.01
3D-OWIS-PC - CT	2.901	1752	15.66	15.00	14.80	1.799	666	15.99	13.50	19.70	16.40	17.50	17.70	17.50
Ours: 3D-OWIS	0.419	1294	14.34	18.00	17.60	0.152	303	15.80	13.90	22.20	17.80	17.80	17.70	17.80
to evaluate the performance of our model on the unknown classes and to provide a fair comparison with and without contributions. For the known classes, we use mean Average Precision (mAP). WI measures the impact of the unknown classes on the precision of the model at a specific confidence level. Ideally, WI is nil, i.e., there are no unknown objects predicted as known. For our evaluation, we report WI at 0.5 confidence. It can be computed as follows:  $\mathrm{WI} = \frac{P_{\mathcal{K}}}{P_{\mathcal{K}\cup u}} -1$
We also report A-OSE, which represents the count of unknown instances misclassified as one of the known classes, and the U-Recall at 0.5 IoU, which reflects the ability of the model to recover unknown objects.
5.2 Implementation details
We adapt Mask3D [29] for the task of open-world instance segmentation. We add an extra prediction output for the unknown class. In training, we assign an ignore label to the classes of the future and previous tasks, while we keep the labels of the previous task and assign an unknown class label to the classes of the future task during evaluation. For contrastive clustering, we use the indices obtained after matching the predictions with the target using Hungarian matching to assign a label to the queries and store them in the Query Store  $\mathcal{Q}_{\text{store}}$ . The store is then averaged per class and used to periodically update the prototypes every 10 iterations for the hinge loss computation. Finally, we use 40 exemplars per class on average for incremental learning. The classes from the current task are kept during class exemplar replay since we are using the same dataset for the three tasks.
5.3 Open-world results
Table 2 provides a comprehensive performance comparison between the Oracle, our implementation of [11] as 3D-OW-DETR, 3D-OWIS, and 3D-OWIS-PC - CT when excluding the Probability Correction (PC) and Confidence Thresholding (CT) components. Across all scenarios and tasks,
Table 3: Extensive ablation of the added components. We perform the ablation by adding Probability Correction (PC) and Confidence Thresholding (CT) components to 3D-OWIS-PC-CT. We conduct the performance comparison in terms of mAP, U-Recall, WI, and A-OSE. Even though 3D-OWIS is performing well in retrieving the unknown classes without PC and CT, which is reflected by the high U-Recall, it is still performing poorly on the known classes, based on the high WI and A-OSE. This negative impact on the known classes accumulates over the tasks and results in further reduction in mAP. When adding the CT, the performance on the known classes improves considerably and remains consistent throughout the incremental learning process. Probability correction (PC) significantly improves the U-Recall in all cases. Even though the latter shows lower performance in terms of WI and A-OSE, the overall mAP slightly improves or remains higher with a large margin compared 3D-OWIS-PC-CT. This shows that adding PC and CT gives the best compromise in performance on both known and unknown classes.
Task IDs (→)	Task 1	Task 2	Task 3
w/ Finetuning	CT PC	WI(↓)	A-OSE(↓)	U-Recall(↑)	mAP(↑)	WI(↓)	A-OSE(↓)	U-Recall(↑)	mAP(↑)	mAP(↑)
Current known	All	Previously known	Current known	All	Previously known	Current known	All
Split A
×	× ×	1.589	707	30.72	37.50	37.00	0.870	321	19.42	00.00	16.74	08.40	00.00	09.30	02.80
×	✓ ×	0.237	443	30.00	40.30	39.70	0.306	129	14.96	00.00	21.00	10.50	00.00	17.45	05.20
✓	× ×	1.589	707	30.72	37.50	37.00	0.000	4	04.75	11.00	17.30	14.10	21.40	08.00	17.50
✓	✓ ×	0.237	443	30.00	40.30	39.70	0.004	102	23.62	29.22	15.80	22.30	19.70	15.70	18.50
✓	✓	0.398	607	34.75	40.2	39.70	0.007	126	27.03	29.40	16.40	22.70	No unknown labels for evaluation
Split B
×	× ×	3.133	1895	21.67	18.94	18.70	1.82	829	17.20	00.00	15.40	06.60	00.00	20.20	07.50
×	✓ ×	2.147	21.70	21.70	23.80	23.50	1.563	375	13.08	00.00	18.30	07.90	00.00	25.40	09.40
✓	× ×	3.219	1905	21.70	18.94	18.70	3.169	1081	26.63	18.00	16.40	17.20	17.30	20.10	18.30
✓	✓ ×	2.147	1397	21.70	23.80	23.50	0.466	413	20.90	18.60	16.90	17.70	18.50	24.20	20.60
✓	✓	3.684	1780	24.79	23.6	23.30	0.755	581	24.21	18.70	17.30	17.90	No unknown labels for evaluation
Split C
×	× ×	2.901	1752	15.66	15.00	14.80	6.294	857	11.05	0.00	15.70	07.50	00.00	14.60	04.70
×	✓ ×	0.227	828	11.44	18.70	18.40	1.361	365	10.16	00.00	19.50	09.40	00.00	19.10	6.20
✓	× ×	2.901	1752	15.66	15.00	14.80	1.799	666	15.99	13.50	19.70	16.40	17.50	17.70	17.50
✓	✓ ×	0.227	828	11.44	18.70	18.40	0.088	208	12.63	14.50	22.10	18.00	17.80	17.70	17.80
✓	✓	0.419	1294	14.34	18	17.60	0.152	303	15.80	13.90	22.20	17.80	No unknown labels for evaluation
3D-OWIS-PC - CT consistently exhibits inferior performance in terms of mAP. Additionally, it demonstrates considerably lower U-Recall performance in splits A and B, with slightly higher
performance in split C. Of particular note, our 3D-OWIS demonstrates remarkable proficiency in preserving knowledge of the previous classes after fine-tuning. This proficiency is attributed to better pseudo-label selection for the unknown classes. 3D-OWIS outperforms 3D-OWIS-PC - CT in most cases while minimizing the impact of the unknown classes on the known classes, as evidenced by lower WI and A-OSE scores and higher mAP.
Table 4 presents a comparison between our model, 3D-OWIS, and our implementation of two methods, GGN [32] and OLN [19]. For both models, we adapt Mask3D and train it with mask loss only for OLN. In the case of GGN, we train a Minkowski backbone to predict affinity maps and use Connected Components to generate class-agnostic proposals. These results underscore the effectiveness and potential of our approach in addressing the three proposed open-world challenges.
5.4 Incremental learning results
Our model's performance in incremental learning is evaluated based on its ability to preserve knowledge from previous classes. With the utilization of exemplar replay, the 3D-OWIS model demonstrates significant improvement on previous classes mAP. Table 2 presents the results, indicating that our model consistently outperforms the others in terms of mean Average Precision (mAP) for the previous classes in all cases.
5.5 Discussion and analysis
Ablation study. We show in Table 3 that 3D-OWIS-PC - CT model performs poorly on the known classes because of the high number of low-quality pseudo-labels generated by Auto-labeler, which is also explained by the high value of Wilderness Impact and Absolute open set error. The U-Recall drops considerably when fine-tuning the 3D-OWIS-PC - CT, while the WI and A-OSE either decrease or increase with the mAP on the unknown. On the other hand, our model limits the training only to the best pseudo-labels, which maintain good performance on the known classes in all cases, before and after fine-tuning, and also achieve results on the unknown class comparable to the 3D-OWIS-PC - CT in most of the cases. Adding the proba
Table 4: Open-world instance segmentation comparison. We provide the results of our implementation of two methods for 2D open-world instance segmentation models. We show that our model performs comparatively better than others across all metrics.
Split A
Task ID	Task 1
WI(↓)	A-OSE(↓)	U-Recall(↑)	mAP(↑)
Current known	All
3D-GGN [32]	15.68	1452	21.33	20.51	20.12
3D-OLN [19]	-	-	02.45	-	-
Ours: 3D-OWIS	0.397	607	34.75	40.2	39.7
bility correction module helps in improving the U-Recall while keeping the mAP of the known classes much above the 3D-OWIS-PC - CT. However, it results in an increase in WI and A-OSE because of the increase of false positives in the known classes.
tSNE analysis The tSNE plot shown in Fig. 6 illustrates the below-par performance of the 3D-OWIS-PC - CT in clustering the unknown classes, where most queries are still maintaining features representative of the known classes. This behavior is a result of the weak supervision of the unknown class, which shows the need for correcting the predictions, and explains the improvement in U-Recall when applying the probability correction with nil deterioration in the known classes mAP in most cases.
Qualitative analysis. Fig. 5 shows that 3D-OWIS is able to correctly identify background and unknown objects as unknown. Also note the second scene, where predictions are corrected from known to unknown without affecting the predictions of the known classes.
Figure 6: tSNE visualization of the queries for known & unknown classes
6 Limitations
Confidence Thresholding (CT) enhances the performance of the model on known classes; nonetheless, it diminishes the model's capacity to segment unknown classes, mainly due to its reliance on a smaller number of pseudo-labels during training. Additionally, the effectiveness of Probability Correction (PC) is contingent upon the inherent characteristics of the clusters within the known classes. In scenarios characterized by data imbalance, the performance of probability correction may deteriorate when applied to the undersampled classes.
In this paper, we address the challenge of 3D instance segmentation in open-world scenarios, which is a novel problem formulation. We propose an innovative approach that incorporates an unknown object identifier to detect objects not present in the training set. To facilitate evaluation and experimentation, we present three dataset splits of ScanNet200 based on different criteria for selecting unknown objects. Our experimental results demonstrate that our proposed unknown object identifier significantly improves the detection of unknown objects across various tasks and dataset splits. This work contributes to advancing the localization and segmentation of 3D objects in real-world environments and paves the way for more robust and adaptable vision systems.
Acknowledgement The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement No. 2022-06725, and by the Berzelius resource, provided by Knut and Alice Wallenberg Foundation at the National Supercomputer Center.
[33] W. Wang, M. Feiszli, H. Wang, and D. Tran. Unidentified video objects: A benchmark for dense, open-world segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10776-10785, 2021. 1
[34] W. Wang, R. Yu, Q. Huang, and U. Neumann. Sgpn: Similarity group proposal network for 3d point cloud instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2569-2578, 2018. 2
[35] S. Xie, J. Gu, D. Guo, C. R. Qi, L. Guibas, and O. Litany. Pointcontrast: Unsupervised pretraining for 3d point cloud understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 574-591. Springer, 2020. 2
[36] B. Yang, J. Wang, R. Clark, Q. Hu, S. Wang, A. Markham, and N. Trigoni. Learning object bounding boxes for 3d instance segmentation on point clouds. Advances in neural information processing systems, 32, 2019. 2
[37] L. Yi, W. Zhao, H. Wang, M. Sung, and L. J. Guibas. Gspn: Generative shape proposal network for 3d instance segmentation in point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3947-3956, 2019. 2
[38] B. Zhang and P. Wonka. Point cloud instance segmentation using probabilistic embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8883-8892, 2021. 2
[39] J. Zhang, R. Dong, and K. Ma. Clip-for3d: Learning free open-world 3d scene representations from 2d dense clip. arXiv preprint arXiv:2303.04748, 2023. 3
[40] X. Zhu, R. Zhang, B. He, Z. Zeng, S. Zhang, and P. Gao. Pointclip v2: Adapting clip for powerful 3d open-world learning. arXiv preprint arXiv:2211.11682, 2022. 3
[41] O. Zohar, K.-C. Wang, and S. Yeung. Prob: Probabilistic objectness for open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11444-11453, 2023. 3, 7
A Scalability of 3D-OWIS
We show in Table 5 that 3D-OWIS can accommodate a large number of classes without a major size increase
Table 5: Demonstrating the Scalability of 3D-OWIS with Respect to the maximum number of classes it can learn.
# of classes	200	1000	5000	10000	50000	100000
Size of 3D-OWIS	39.7M	39.8M	40.7M	41.9M	50.9M	62.2M
B Additional details on Split B
We utilize the 20 scene types present in the ScanNet200 dataset to distribute the 200 classes over the three tasks. Initially, we establish a notion of similarity between two scene types by assessing the extent of their shared classes. This similarity is quantified through the intersection over the union  $(IoU)$  metric, which measures the ratio of common classes to the total count of unique classes across both scenes. By employing this metric, we identify scene types that exhibit a substantial  $IoU$ , indicating a higher degree of similarity. The similarity matrix, depicted in Fig. 7, showcases the relationships between the 20 scene types within the ScanNet200 dataset.
Subsequently, we employed three criteria to group the classes:  $(i)$  the likelihood of encountering them first when accessing an indoor area,  $(ii)$  their affiliation with similar scene types, and  $(iii)$  the proximity in the number of known classes across tasks. By taking these factors into consideration, we arrived at the split of scenes presented in Table 6.
Figure 7: Similarity matrix between the 20 scene types in ScanNet200 dataset. We show the ratio of common classes to the total count of unique classes between two scene types.
Table 6: Frequently occurring scene when training during the three tasks in Split B. Scene types are grouped into tasks based on three criteria: (i) the likelihood of encountering the classes within the scene types when entering an indoor area, (ii) similarity of scene types containing the classes, and (iii) consistency in the overall number of classes within the scene types across all tasks. This grouping ensures a cohesive organization of scene types for effective evaluation of 3D instance segmentation models integrated with tasks such as robot navigation within indoor environments.
Split B
Task 1	Task 2	Task 3
Bedroom / Hotel	Kitchen	ComputerCluster	Mail Room	Game room	Office
Dining Room	Bathroom	Misc.	Hallway	Apartment
Lounge	Closet	Gym	Classroom	Lobby
Garage	Library	Conference Room	Stairs
C Additional details on the experimentation
Training: We train the model on the entire ScanNet200 dataset for all tasks. In Task 1, objects belonging to the classes from Task 2 and Task 3 are masked, excluding them from the learning process. Moving to Task 2, we utilize the last saved checkpoint of the model from Task 1 as a starting point and mask the objects with labels that correspond to the current known classes of Task 1 and Task 3. This allows the model to focus solely on learning and distinguishing the specific objects associated with the current task. Finally, Task 3 builds upon the progress made in Task 2. We load the
latest checkpoint of the model from Task 2 and incorporate an exemplar replay. Similar to Task 2, the objects with labels belonging to the known classes in Task 1 and Task 2 are masked during training. This step further refines the model's understanding and discrimination abilities for the specific objects relevant to the current task.
Evaluation: To conduct the evaluation during a task, we assign the "unknown" label to the known classes from all the future tasks.
D Additional qualitative results
D.1 Unknown objects identification
The qualitative results depicted in Fig. 10, 12, 13, and 11 highlight the superior performance of our contribution in retrieving unknown objects. Across the majority of scenes, our model consistently corrects the mispredicted unknown classes while preserving the accuracy of known objects, thus demonstrating its robustness and effectiveness.
D.2 Learning novel classes
Fig. 8 and Fig. 9 illustrate the sequential process of learning novel classes after identifying unknown objects from the previous task. In Fig. 8, we demonstrate the effectiveness of our method in successfully retrieving unknown classes in all tasks. Additionally, in Fig. 9, we highlight the potential of exemplar replay in retaining knowledge of the old classes after learning the novel classes in Task 2 and Task 3.
Figure 8: Illustration of the process of unknown identification and learning novel classes. We use orange circles to highlight the differences between 3D-OWIS and 3D-OWIS-PC-CT. The objects depicted in green represent the known classes, while those in blue represent the unknown objects. The gray objects correspond to the background. The qualitative results demonstrate that 3D-OWIS outperforms 3D-OWIS-PC-CT in retrieving unknown objects. Notably, 3D-OWIS correctly identifies the background objects as unknown, whereas 3D-OWIS-PC-CT misclassifies them as known objects.
Figure 9: Alleviating catastrophic forgetting during incremental learning. The capability of 3D-OWIS in retaining knowledge of the previously known classes after learning the new one is demonstrated across Task 2 and Task 3 for both scenes, where all objects of old known classes are still being predicted as known.
Figure 10: Qualitative results. The objects depicted in green represent the known classes, while the ones in blue represent the "unknown" class, and the gray objects represent the background. To emphasize the differences between 3D-OWIS and 3D-OWIS-PC-CT, we highlight them with orange circles.
3D-OWIS
Figure 11: Additional qualitative results We demonstrate the better performance of our model in accurately identifying background objects (depicted in gray) as unknown (represented by the blue color), and also correcting the predictions from known class to unknown class. This capability greatly reduces the misclassification of background objects as known objects, leading to improved overall classification accuracy.
Figure 12: Additional qualitative results
Figure 13: Additional qualitative results