3D Indoor Instance Segmentation in an Open-World
Authors: Mohamed El Amine Boudjoghra
Salwa K. Al Khatib
Jean Lahoud
Hisham Cholakkal
Rao Muhammad Anwer
Salman Khan
Fahad Khan
Venue: NeurIPS 2023 poster (2023)
Keywords: open-world
3d instance segmentation

Abstract: Existing 3D instance segmentation methods typically assume that all semantic classes to be segmented would be available during training and only seen categories are segmented at inference. We argue that such a closed-world assumption is restrictive and explore for the first time 3D indoor instance segmentation in an open-world setting, where the model is allowed to distinguish a set of known classes as well as identify an unknown object as unknown and then later incrementally learning the semantic category of the unknown when the corresponding category labels are available. To this end, we introduce an open-world 3D indoor instance segmentation method, where an auto-labeling scheme is employed to produce pseudo-labels during training and induce separation to separate known and unknown category labels. We further improve the pseudo-labels quality at inference by adjusting the unknown class probability based on the objectness score distribution. We also introduce carefully curated open-world splits leveraging realistic scenarios based on inherent object distribution, region-based indoor scene exploration and randomness aspect of open-world classes. Extensive experiments reveal the efficacy of the proposed contributions leading to promising open-world 3D instance segmentation performance. Code and splits are available at: https://github.com/aminebdj/3D-OWIS.

3D Indoor Instance Segmentation in an Open-Worke
Mohamed El Amine Boudjoghra' Salwa K. Al Khatib', Jean Lahoud', Hisham Cholakkal', Rao Muhammad Anwer! Salman Khan 1.3 Fahad Shahbaz Khan 1.4
"Mohamed Bin Zayed University of Artificial Intelligence (MBZUAD,
Aalto University, 3 Australlin National University, "Linköping University {mohamed. boudjoghra, salwa. khatib, jean. lahoud, hisham. cholakkal, rao. anwer, salman kalman.khan, fahad. Baazunimbuaa...m ac.
Abstract
Existing 3D instance segmentation methods typically assume that all semantic classes be segmented would available during training and only seen cate- gories segmented at inference. We argue that such a glosed-world assumption is restrictive and explore for the first time 3D indoor instance segmentation open- setting, where the model is allowed distinguish set of known classes as well as identify an unknown object as unknown and then later incre- mentally learning the semantic category of the unknown when the corresponding category labels are available. To this end, we introduce an open-world 3D indoor instance segmentation method, where an auto-labeling scheme is employed produce pseudo-labels during training and induce separation to separate known and unknown category labels. We further improve the pseudo-labels quality inference by adjusting the unknown class probability based on the objectness score distribution. We also introduce carefully curated open-world splits leverag- ing realistic scenarios based on inherent object distribution, region-based indoor scene exploration and randomness aspect of open-world classes. Extensive ex- periments reveal the efficacy of the proposed contributions leading to promising open-world 3D instance segmentation performance. Code and splits are available at: https://github.. cwaobbcco//mmiiim.. com/amineebiii3D-OWWS...
Introduction
3D semantic instance segmentation aims at identifying objects in given 3D scene, represented point cloud or mesh, by providing object instance-level categorization and semantic labels. The ability segment objects in the 3D domain has numerous vision applications, including robotics, augmented reality, and autonomous driving. Following the developments in the sensors that acquire depth information, a variety of datasets has been presented the literature which provides instance-level annotations. In view of the availability of large-scale 3D datasets and the advances in deep learning methods, various 3D instance segmentation methods have been proposed in recent years.
The dependence of 3D instance segmentation methods on available datasets has a major drawback: fixed set of object labels (vocabulary) is learned. However, object classes in the real world are plentiful, and many unseen/unknown classes can be present at inference. Current methods that learn fixed as background. This prevents intelligent recognition systems from identifying unknown or novel objects that not part of the background. Given the importance of identifying unknown objects, recent works have explored open-world learning setting for 2D object detection [18, 11, 28, 33). In the open-world setting, a model expected to identify unknown objects, and once new classes are labeled, the new set is desired to be incrementally learned without retraining [18]. While previous methods have been mostly suggested for open-world 2D object detection, yet be explored
only discard the unknown classes but also supervise them be labeled
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

int known objects
Unknown
objects
20-0WS
Oparair
IDOWE
Openater
Puron
rijec't
(+2
time
Figure 1: 3D instance segmentation in an ppendworld. During each iterative learning phase, the model detects unknown objects, and human operator gradually assigns labels to some them and incorporates them into the pre-existing knowledge base for further training.
in the 3D domain. The main challenge lies in understanding how objects appear in 3D in order to separate them from the background and other object categories.
3D instance segmentation in the open world, illustrated in Fig. 1, offers more flexibility, allowing the model to identify unknown objects and request annotations for these novel classes from an oracle for further training. However, this approach presents several challenges: (i) the lack of annotations for unknown classes, necessitating quality pscudo-labeling techniques; (ii) the similarities between predicted features of known and unknown classes, requiring separation techniques for improved prediction; and (iii) the need for more reliable objectness scoring method to differentiate between good and bad predicted masks for 3D point clouds.
In this work, we investigate novel problem setting, namely open- World indoor 3D Instance Segmen- tation, which aims at segmenting objects of unknown classes while incrementally adding new classes. We define wal-world protocols and splits test the ability of 3D instance segmentation methods to identify unknown objects. In the proposed setup, unknown object labels are also added incrementally to the set of known classes, akin real-world incremental learning scenarios. We propose an unknown object identifier with probability correction scheme that enables improved recognition of objects. To the best of our knowledge, we are the first explore 3D instance segmentation in an open- world setting. The key contributions of our work are:
We propose the first open- open-world 3D indoor instance segmentation method with dedicated mechanism for accurate identification of 3D unknown objects. We employ an auto-labeling scheme to generate pseudo-labels during training and induce separation in the query embed- ding space to delineate known and unknown class labels. At inference, we further improve the quality of pscudo-labels by adjusting the probability of unknown classes based on the distribution of the objectness scores.
We introduce carefully curated open-world splits, having known vs. unknown and then incremental learning over the span of 200 classes, for rigorous evaluation of open-world 3D indoor segmentation. Our proposed splits leverage different realistic scenarios such as inherent distribution ffreuuency-basee during the exploration of indoor areas (region-based), based), and the randomness aspect of ob- ject classes in the open-world. Extensive experiments reveal the merits of the proposed contributions towards bridging the performance gap between our method and oracle.
of object classes, various class types encountered
2 Related Work
3D semantic instance segmentation: The segmentation of instances in 3D scenes has been
approached from various angles. Grouping-basse or clustering-basee -based techniques use a bottom-up pipeline by learning an embedding in the latent space to help cluster the object points. [4, 13, 14, 17, methods work in a top-down fashion, first detecting 3D bounding
20, 21, 34, 38]. Proposal-basee boxes, then segmenting the object region within the box [10, 15, 22, 36, 37]. Recently, spurred by related 2D work [5, 6]. the transformer design [31] has also been applied for the purpose of segmenting 3D instances [29, 30]. Other methods present weakly-superrised alternatives to methods
that use dense annotations order to lower the cost of annotating 3D data [7, 16, 35J. While these methods aim to improve the quality of 3D instance segmentation, they are trained on known set of semantic labels. On the other hand, our proposed method aims at segmenting objects with both known and unknown class labels.

---

Training
Inference
Contrastive Clustering
Donsolutionl Backbone
Update Prototypes
Probutiliyy Coneccion
Transformer Decadar
Query Processing
Store
Intal quariso
MLP
MLP
Auto
Quey Processing
Labeler
Crnant
Goog oponted 2EC UNGIIN
Assign target to query
Preckcted
zoftmaa
Contrastive Clustering
Figure Proposed open-world 3D instance segmentation pipeline. From left right: 3D instance segmentation model, where the point cloud goes through 3D convolutional backbone. The extracted feature maps are used in the transformer decoder refine some initial queries, which then pass through two MLPs to generate label and mask predictions. The Contrastive Clustering block takes the refined queries, the prediction masks, and labels to further process the queries by assigning target or an unknown pseudo label in the Query Processing module, and then storing them in a Query Store to finally update the class prototypes, which are finally used for contrastive clustering. During inference, the queries are used to correct the probability of the predicted labels based on their reachability to the known class prototypes.
Open-world object recognition: Open-world object recognition was introduced in [2], where the Nearest Mean Classifier was extended for an open-world setting. In the direction of open-world object detection, many studies [41, 18, 11, 25] have been conducted in the past. In[18], pseudo-labels for the unknowns are generated to perform contrastive clustering during training for a better unknown- known classes separation, where an energy- based unknown class identifier was proposed detect the unknown classes, based on the energy of the logits from the known classes. For incremental learning, they adopted exemplar replay to alleviate catastrophic forgetting of old classes. In the same task as [18). [11] used wansformer-basee model and proposed another way of unknown pseudo-labels generation, by using a new method of objectness estimation, and introduced a foreground objectness branch that separates the background from the foreground. For the task of outdoor 3D point cloud semantic segmentation, [3] proposed a model that predicts old, novel, and unknown classes from three separate classification heads. The latter is trained on the labels of the known classes and pseudo-labels old classes generated by the same model to alleviate catastrophic forgetting, while the unknown class is assigned the second-highest score for better unknown class segmentation. Other methods proposed [40, 12. 39], primarily focus on enhancing the generalizability 3D models for novel classes by leveraging supervision from 2D Vision Language Models for object recognition and 3D semantic segmentation tasks. However, these approaches exhibit several limitations, including (i) The 3D model's performance becomes dependent on the 2D Vision Language model. (ii) The 3D geometric properties of unseen objects in the training data are neglected during the training process. (iii) There exists avenue for enhancing the model's performance novel classes in cases where new labels are introduced. (iv) The training process necessitates pairs of images and corresponding 3D scenes.
3 Closed-world 3D Instance Segmentation
We adopted the state-of-the- thetth----rttt art.33 3D instance segmentation model Mask3D [29] as our baseline. The latter is hybrid model that combines Convolutional Neural Networks (CNNs) with transformers to learn class-agnostic masks and labels for instance separation. The backbone of Mask3D is CNN-based and used to extract feature maps from multiple levels. Meanwhile, the decoder is transformer-baser and used to refine no N instance queries Q {gy _ E (I, RO)). using the extracted

---

feature maps. The learning scheme consists of Cross-entropy loss for learning semantic class labels and binary cross-entropy loss for learning instance masks during training.
Open-World 3D Instance Segmentation
4.1 Problem formulation
We start by formulating the problem setting of open-world 3D instance segmentation. At Task T,, there exists a set of known object categories X = {1, 2).,C O} and set of unknown object categories 24 = + I.... that may exist on inference time. The training dataset D = {X' Y'} includes samples from the classes K The input set X [Pi,.. PM) made of point clouds, where P, E R2*3 is quantized point cloud of N voxels each carrying average RGB color of the points within. The corresponding labels are Y (Yı,.., Ya) ), where Y; { k object instances. Each object instance corresponding class label 4 E K
........ Vx. encodes
[B, represents binary mask B, E {0, 1) and
In our problem setting, Mcc is 3D instance segmentation model that is trained on C' object categories, and, on test time, can recognize instances from these classes, in addition to instances from new classes not seen during training by classifying them as unknown. The detected unknown instances can be used by human user to identify set of new classes not previously trained on, which can be incrementally added to the learner that updates itself to produce Mc+a without explicitly retraining on previously seen classes. At this point Task T the known class object categories are k = K U (C 1,... C + n}. This process repeats throughout the lifespan of the instance segmentation model, continuously improving itself by incorporating new information from new classes until it reaches its maximum capacity of classes it can learn. In the rest of the paper, We assign the unknown class a label 0.
4.2 Open-world scenarios
In order simulate different realistic scenarios that might be encountered in an open-world, propose three different ways of grouping classes under three tasks. These scenarios split scenes based on the inherent distribution (frequency-based) of object classes, the various classes encountered during the exploration of various indoor areas (region-based), and the randomness aspect of object classes in the open world.
Table 1: The statistics of each split across the three tasks. The number of known classes task is reported along with the count of instances (3D objects) in the training and validation set, we also show the number of non-empty scenes used during training and validation.
Split
Split B
Split C Task
Task
Task
Task
Task
Task
Task
Task
Task
Classes count
64
68
66
73
55
70
66
66
66
Train instances
24224
3791
1612
15327
8177
6123
13483
8239
7905
Validation instances
6539
1000
428
4177
2261
1529
3776
2102
2089
Train seenes
1201
924
627
1201
1002
895
1169
1089
1159
Validation scenes
312
242
165
312
264
236
307
273
300
Point frequency
10e
109
10
104
104
10
Split A
Split B
Split C
Figure 3: Point- wise count for each class across the three tasks under the three open-world scenarios

---

Split A (Instance frequency-basede We introduce split that leverages the inherent distribution of objects, with known classes being more prevalent than unknown categories. Task T encompasses all the head classes defined in the ScanNer200 benchmark [8, 27]. while tasks T and T group the common and tail classes, respectively. This division allows us to effectively capture the varying frequency and significance of object categories within the dataset.
Split B (Region-basedl); In this split, our objective to replicate the diverse class types encountered during indoor exploration. We argue that a perfect model for robot moving indoors should segment both classes it knows and classes it hasn't seen before. Additionally, it should keep learning and getting better at segmenting new classes over time. This partition draws inspiration from the sequence of classes that a robot might encounter when navigating indoors. To achieve this, we group classes that are likely to be encountered initially when accessing an indoor space and share similarities in scenes, Initially, we assign each class to a specific scene where predominantly occurs. Subsequently, we divide the classes into three distinct groups, corresponding to the three tasks.
Split C (Random sampling of classes): This third split introduces a different challenge inspired by the randomness aspect of the open-worid, where tasks can exhibit random levels of class imbalance. To create this split, we randomly shuflled the classes and sampled without replacement, selecting 66 classes three times for each task.
4.3 Generating pseudo-labels for the unknown classes
Because of the wide range of classes in open-world setting, the auto-labeler is used as an alternative to manual labeling. The former makes use of the existing target labels from the available ground truth classes (known classes) to generate pscudo-labels for the unknown class in the process of training. In [18], the model is assumed to be class agnostic, where unknown objects are predicted as known with high confidence. As result, the authors of the paper proposed to use the predictions with top-k confidence scores that do not intersect with the ground truth as pseudo-labels for the unknown class. In our study, we show that top-k pseudo-label selection can severely harm the performance of the model on the known and unknown classes. Hence, we propose a Confidence Thresholding (CT) based selection of pseudo-labels. We show that the performance on the known and the unknown classes increases by a large margin in terms of mean Average Precision (mAP).
The auto-labeler unit, depicted in Fig. 2, is used for unknown pseudo-labels generation. It takes a set of predicted binary masks B- = (B. | i (I,..., nq)]. where ng is the number of queries, B, = 1 (M, > 0.5) is a mask from single query, and M, {ma.s E [0, 1] | j E (I,.., N)) is heat map measuring the similarity between query qj E RD and the features of N voxels extracted from the high-resolution level in the backbone.
Moreover, each query qy encodes semantic information and can generate a class prediction Pets = (PP..o (z) I c E (0, .... IK) using classification head (refer to Fig. 2). Subsequently, the objectness confidence score is assigned to predictions following Eq 1.
M₁ 1(M, > 0.5)*
8j Selaj
1 (M, > 0.5)[1
where Sela.j E IR is the max output probability from the classification head Pete (g)), and 1 is the indicator function. After scoring the predictions, the auto-labeler returns m pseudo-labels Y = (v. [B., 0] i E m)) with confidence above a threshold and has a low IoU with the known classes' target masks.
4.4 Query target assignment and contrastive clustering
Similar to [18], we utilize contrastive clustering to enhance the separation of classes within the query embedding space. To achieve this, we employ set of query prototypes denoted as Op {q: E R i E (0, 1... (CC')), where qo denotes the prototype of the class unknown. We apply a contrastive loss that encourages queries with similar classes to attracted to their respective prototypes while pushing them away from those representing negative classes, as illustrated in Fig. 2. Since the queries are used to determine the class of the objects (see Fig. inference block), the class prototypes are expected to hold general semantic knowledge of their corresponding classes.
5

---

Hungarian matching is performed in the Assign target to query module, depicted in Fig. 2, where the indices of prediction-targe target are used to assign label the queries used to generate the matched prediction. The labeled queries are then stored in query store Ostore: which represents a queue with a maximum capacity. This queue is employed to update the query prototypes 2p using an exponential moving average.
Hinge embedding loss is utilized according Eq 2. This loss ensures that queries belonging the same class denoted as qe, are pulled towards their corresponding class prototype qe, while being pushed away from other prototypes representing different classes.
|A
Cconc(ge) Σ
Elge q.)
(2)
[ll9c -9il22
i=c c
E(ge, 9:)
|max(0, (0, A llue qill2)
where A the margin of the contrastive clustering.
4.5 Reachability-based probability correction (PC)
In [23], an architecture that can deal with long-tail distribution and unknown class prediction for open- world object recognition was pro- posed, where unknown classes are as- sumed to be very different in color and texture from the known classes without prior the unknown classes. However, we show in Fig. that many unknown instances hold similar fea- tures to the known ones.
Prototype of the unknown class
Prototypes the known classes
unknown class features
A Contrastive clustering margin
In our method, we relax the strict as- sumption of high dissimilarity un- known and known classes and correct the predicted output probability fol- lowing two characteristics of a feature
Correction region
Figure 4: Illustration of the region in the query embedding space where the class probability is corrected.
from an unknown object: (1) it has to be far from the nearest known class, as features of the class unknown are expected to be pushed away from the prototypes of the known classes, after applying constructive clustering, and (2) the feature should correspond to an object that not a known class. We show that applying this approach during inference boosts the performance of the model on the unknown class considerably by compensating for the weak pseudo-labels provided by the auto-labeler.
Our probability correction scheme is the following
P(O: 9)) = Pats (0; 9;) (o:ror((0
(3)
where Fets is the probability from the classification head, and Peory is the correction probability, We base our intuition the fact that unknown classes have high objectness scores, which makes them not too far from the prototypes of the known classes. To model this behavior we choose
PP Peorr (0; q5) = P Peorr (0; (3) (o
where IP "Pcorr(e (o; 9;) is the likelihood of the query to correspond to an object that is not known (either background or true unknown). Since the query prototypes encode class-specific information we propose the following method to measure the objectness of a query given all prototypes from the known classes, where it assigns high objectness probability if it is close to only few known classes. This probability distribution defines the objectness of unknown objects around a certain boundary from the prototypes as follows.
IP Poorr (o: =1
Σ k=1
P cas (k; q3)

---

Ground Truth
PC-CT
3D-OWIS
Figure 5: Qualitative results for 3D instance segmentation results some ScanNet200 validation scenes. Points highlighted blue belong to unknown classes and those highlighted green belong to known classes. We show the performance of our model in retrieving the unknown class objects compared to 3D-OWIS PC- CT for the three scenes.
while P corr (0; o, qj) the probability of the query being an unknown object, which has high value the further it is from the nearest prototype of the known classes.
Pe Poorr (;) = (1(0) b
(3(()))) :
r(g)) = min [1 q:
Here is the sigmoid function, 7(1;) is the reachability of the query 4j qi is the prototype of the 5th class, and a, are the shift and scale of the sigmoid function that assure IP c0; qj ((;)) = 0) =
0.05 and (0; o, Qj ((;)) ) 0.95, for contrastive clustering margin A.
We finally normalize the probabilities from the classification head of the known classes follows
(c)(g) (1 g0):
P(c Σ
(l:g(:))
4.6 Alleviating catastrophic forgetting for incremental learning
Following the success of exemplar replay in avoiding catastrophic forgetting of the old classes during incremental learning for object detection [18, 11, 41], we adopt it for the task of incremental learning in 3D instance segmentation where we use exemplars from the classes of the previous task fine-tune the model trained on the novel classes, In our setting, we use the same dataset for the three tasks and mask the classes of the previous task when training on the novel classes from the current task. As result, the novel classes of the current task might be encountered again when replaying the exemplars from the previous task, as the same scenes are being used in fine-tuning.
Experiments
5.1 Open-world evaluation protocol
We use our proposed splits classes which mimic the challenges that are mostly faced in the open-world ensure strict performance evaluation for 3D instance segmentation models.
Evaluation metrics. We adopt three common evaluation metrics, wilderness impact (WI) [9]. absolute open set error (A-OSE) [26], and the recall of the unknown classes (U-Recall) [1, 24, 11]
7

---

Table 2: State-of-the-Art comparison for 3D-OWIS model. We show comparison of performance under the three open-world scenarios, where 3D-OWIS PC CT is our model 3D-OWIS without Probability Correction (PC) and Confidence Thresholding (CT). We rely on the metrics used in the open-world literature, A-OSE which quantifies the number of unknown objects misclassified as one of the known classes, WI which measures the impact of the unknown class on the precision of the model on the known classes, and the U-Recall to evaluate the model's ability recover the unknown objects. We show that 3D-OWIS performs remarkably better than the other models under all scenarios when dealing with the known classes, and superior performance in split A and B. and slightly less performance in split C when handling the unknown objects. We also provide a closed-setting comparison between Mask3D and Oracle (Ours with access to unknown labels).
Task
Task
Task
WI
A-OSE U-Recall
mAP (1)
WI
A-OSE (U)
U-Recall
mAP (f)
mAP (
(
(
(t)
Current
Previously known
Carrent known
All
Previously
Carrent
Split A
Onnlle
0.129
227
55.94
38.60
38.25
20.91
Mask3D [29]
39.12
39.12
38.30
20.57
29.15
28.61
18:33
25.58
3D-OW-DETR [1]]
0.547
721
22.14
35.56
35.05
0.282
253
26.24
18.18
13.62
15.76
21.56
08.38
17.67
3D-DWIS
1.589
707
30.72
37.50
37.00
11.00
17.30
14:10
Ours: 3D-OWIS
0.397
607
34.75
40.2
39.7
0.007
126
27.03
29.40
16.40
22.70
20.20
15.20
18.70
Split B
Oracle
1.126
939
70.31
24.57
24.80
0.180
441
73.16
25.50
20.30
23.40
23.40
30.40
26.00
Mask3D [29]
23.48
23.48
21.81
18.91
20.37
24.20
29.22
26.06
3D-OW-DETR [1]]
3,229
1935
17.18
20.00
19.73
2,053
1389
33.31
12.36
13.86
12:93
07.27
18.96
11.62
3D-OWIS PC CT
3.133
1895
21.67
18.94
18,70
3,169
1081
26.63
18.00
16.40
17.20
17.30
20.10
18.30
Ours: 3D-OWIS
3.684
1780
24.79
23.60
23.30
0.755
581
24.21
18.70
17.30
17.90
18.70
24.60
20.90
Split
Oracle
LO39
71.61
23.6
0.249
591
62,83
20.50
19.60
Mask3D (29)
20.82
21.15
22.67
26.67
24.13
25.41
25.21
25.35
3D-OW-DETR [II]
1,463
1517
13.00
14.81
14.59
1,330
847
16.04
08.00
17.41
12.40
08.81
15.63
11.01
3D-OWIS- PCC
2.901
1752
15.66
Ours: 30-OWIS
0.419
1294
14.34
18.00
17.60
0.152
303
15,80
13,90
22.20
17.80
17.80
17.70
17.80
to evaluate the performance of our model on the unknown classes and to provide fair comparison with and without contributions. For the known classes, we use mean Average Precision (mAP). WI measures the impact the unknown classes on the precision of the model at a specific confidence level. Ideally, WI is nil, i.e., there are no unknown objects predicted as known. For our evaluation, we report WI at 0.5 confidence. It can be computed as follows: WI
Pkc
PELLI
We also report A-OSE, which represents the count of unknown instances misclassified as one of the known classes, and the U-Recall at 0.5 IoU, which reflects the ability of the model recover unknown objects.
5.2 Implementation details
We adapt Mask3D [29] for the task of open-world instance segmentation. We add an extra prediction output for the unknown class. In training, we assign an ignore label to the classes of the future and previous tasks, while we keep the labels of the previous task and assign an unknown class label to the classes of the future task during evaluation. For contrastive clustering, after matching the predictions with the target using Hungarian matching to assign a label to the queries and store them in the Query Store Qstore. The store is then averaged per class and used to periodically update the prototypes every 10 iterations for the hinge loss computation. Finally, we use 40 exemplars per class on average for incremental learning. The classes from the current task are kept during class exemplar replay since we are using the same dataset for the three tasks.
the indices obtained
5.3 Open-world results
Table 2 provides a comprehensive performance comparison between the Oracle, our implementation of [11] as 3D-OW-DETR, 3D-OWIS, and 3D-OWIS- PC CT when excluding the Probability Correction (PC) and Confidence Thresholding (CT) components. Across all scenarios and tasks,

---

Table 3: Extensive ablation of the added components. We perform the ablation by adding Probability Correction (PC) and Confidence Thresholding (CT) components to 3D-OWIS -OOOOI-------- We conduct the performance comparison in terms of mAP, U-Recall, WI, and A-OSE Even though 3D-OWIS is performing well in retrieving the unknown classes without PC and CT, which reflected by the high U-Recall it is still performing poorly on the known classes, based on the high WI and A-OSE. This negative impact on the known classes accumulates over the tasks and results in further reduction in mAP. When adding the CT, the performance on the known classes improves considerably and remains consistent throughout the incremental learning process. Probability correction (PC) significantly improves the U-Recall in all cases. Even though the latter shows lower performance in terms of WI and A-OSE the overall mAP slightly improves or remains higher with large margin compared 3D-OWIS PC CT. This shows that adding PC and
PC- CT.
CT gives the best compromise in performance on both known and unknown classes
Task DDx :)
Task I
Task
Task
WT A-OSE U-Recall
mAP (t)
WT
A-OSE
U-Recall
mAP (f)
mAP (1)
Current
Previously
Current known
Previously
Carrent
w/ Finctuning
CT PC
())
(1)
(1)
known
All
(1)
(1)
(t)
known
All
known
known
All
Split A
L,599
707
00.00
0.237
443
30.00
40.30
39.70
0.306
14.96
00.00
21.00
10.50
00.00
17.45
05,20
1,589
707
30.72
37,50
37.00
0,000
4
04,75
11.00
17.30
14,10
21.40
08.00
17.50
0.237
443
30.00
0.004
15.80
18.50
No unknown labels for evaluation
0.398
607
34,75
40.2
39.70
0.007
126
27.03
29.40
16.40
22.70
Split B
3.133
1895
21.67
18.94
18.70
1.82
829
17,20
00.00
15,40
06.60
00.00
20.20
07.50
2.147
21.70
21.70
23.80
23.50
1.563
375
13.08
00.00
18.30
07.90
00.00
25,40
09.40
1905
21.70
18.94
18.70
3..69
26.63
18.00
17.20
17.30
20,10
18.30
2.147
1397
21.70
0.466
20.90
18.60
16.90
17.70
18.50
20.60
No unknown labels
3.684
1780
24.79
23.6
23.50
0.755
581
24.21
18.70
17.30
17.90
for evaluation
Split C
x
2,901
1752
15.66
15.00
14.80
66.994
857
11.03
0.00
15.70
07.50
00.00
14.60
04470
0.227
838
11.44
18.70
18.40
1.361
365
10.16
00.00
19.50
09.40
00.00
19.10
6.20
2.901
1752
15.66
14.80
1.799
19.70
17.70
17.50
0.227
828
11.44
18,70
18.40
0.088
200
12.63
14,50
22.10
18.00
17.80
17:70
17.80
No unknown labels for ovaluation
0.419
1294
14.34
17.60
0.152
13,90
22.20
17.80
3D-OWIS- - CT consistently exhibits inferior performance in terms of mAP. Additionally, it demonstrates considerably lower U-Recall performance in splits A and B, with slightly higher
performance in split C. Of particular note, our 3D-OWIS demonstrates remarkable proficiency in preserving knowledge of the previous classes after fine-tuning. This proficiency is attributed to better pseudo-label selection for the unknown classes. 3D-OWIS outperforms 3D-OWIS- PC - CT in most cases while minimizing the impact of the unknown classes on the known classes, as evidenced by lower WI and A-OSE scores and higher mAP.
Table 4 presents comparison between our model, 3D-OWIS, and our implementation of two methods, GGN [32] and OLN [19]. For both models, we adapt Mask3D and train it with mask loss only for OLN. In the case of GGN, we train Minkowski backbone to predict affinity maps and use Connected Components to generate class-agnostic proposals. These results underscore the effectiveness and potential of our approach in addressing the three proposed open-world challenges.
5.4 Incremental learning results
Our model's performance in incremental learning is evaluated based on its ability to preserve knowledge from previous classes. With the utilization of exemplar replay, the 3D-OWIS model demonstrates significant improvement on previous classes mAP. Table 2 presents the results, indicating that our model consistently outperforms the others in terms of mean Average Precision (mAP) for the previous classes in all cases.

---

5.5 Discussion and analysis
Ablation study. We show in Table 3 that 3D- OWIS-PC CT model performs poorly on the known classes because of the high num- ber of low-quality pseudo-labels generated by Auto-labeler, which also explained by the high value of Wilderness Impact and Absolute open set enoor The U-Recall drops consider- ably when fine-tuning the 3D-OWIS-PP CT, while the and A-OSE either de- crease increase with the mAP on the known. On the other hand, our model lim- its the training only to the best pseudo-labek, which maintain good performance on the known classes in all cases, before and after fine- tuning, and also achieve results on the unknown class comparable to the 3D-OWIS PEWIS-PC-- CT in most of the cases. Adding the proba- bility correction module helps in improving the U-Recall while keeping the mAP of the However, it results in an in-
Table Open-world instance segmentation com- parison. We provide the results of our implemen- tation of two methods for 2D open-world instance segmentation models. We show that our model performs comparatively better than others across all metrics.
Split A
Task
Task
WI
A-OSE
U-Recall
mAP (f)
Current
())
(1)
(1)
All
known
3D-GGN [32]
15.68
1452
21.33
20.51
20.12
3D-OLN [19]
02.45
Ours: 3D-OWIS
0.397
607
34.75
40.2
39,7
known classes much above the 3D-OWIS-PC CT. crease in WI and A-OSE because of the increase of false positives in the known classes.
Known channe
ISNE analysis The (SNE plot shown in Fig. 6 illustrates the below-par performance of the 3D-OWIS-PC: CT in clustering the unknown classes, where most queries are still maintaining features representative of the known classes. This behavior is result of the weak supervision of the unknown class, which shows the need for correcting the predictions, and explains the improvement in U-Recall when applying the probability correction with nil deteri- oration the known classes mAP in most cases.
Uniinww dosc
Qualitative analysis. Fig. 5 shows that 3D-OWIS is able correctly identify background and unknown objects unknown. Also note the second scene, where predictions corrected from known unknown without affecting the predictions of the known classes.
Figure 6: tSNE visualization of the queries for known & unknown classes
Limitations
Confidence Thresholding (CT) enhances the performance of the model known classes; nonetheless, it diminishes the model's capacity to segment unknown classes, mainly due to its reliance on a smaller number of pseudo-labels during training. Additionally, the effectiveness of Probability Correction (PC) is contingent upon the inherent characteristics of the clusters within the known classes. In scenarios characterized by data imbalance, the performance of probability correction may deteriorate when applied to the undersampled classes.
7 Conclusion
In this paper, we address the challenge of 3D instance segmentation in open-world scenarios, which is a novel problem formulation. We propose an innovative approach that incorporates an unknown object identifier to detect objects not present in the training set. To facilitate evaluation and experimentation, we present three dataset splits of ScanNer200 based on different criteria for selecting unknown objects. Our experimental results demonstrate that our proposed unknown object identifier significantly improves the detection of unknown objects across various tasks and dataset splits. This work contributes advancing the localization and segmentation of 3D objects in real- gal-world environments and paves the way for more robust and adaptable vision systems.
10

---

Acknowledgement The computational resources were provided by the National Academic Infras- tructure for Supercomputing Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement No. 2022-06725, and by the Berzelius resource, provided by Knut and Alice Wallenberg Foundation the National Supercomputer Center.
References
[1] A. Bansal, K. Sikka, G. Sharma, R. Chellappa, and A. Divakaran. Zero-shot object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 384-400, 2018.
[2] A. Bendale and T. Boult. Towards open world recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1893-1902, 2015. 3
[3] J. Cen, P. Yun, S. Zhang, J. Cai, D. Luan, M. Tang, M. Liu, and M. Yu Wang. Open-world semantic segmentation for lidar point clouds. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVIII, pages 318-334, Springer, 2022. 3
[4] S. Chen, Fang, Q. Zhang, W. Liu, and X. Wang. Hierarchical aggregation for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15467-15476, 2021. 2
[5] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290-1299, 2022. 2
[6] B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:1 14:17844-77875 2021.
[7] J. Chibane, F. Engelmann, T. Anh Tran, and G. Pons-Moll. Box2mask: Weakly supervised 3d semantic instance segmentation using bounding boxes. In Computer Vision Vision-ECCV ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI, pages 681-699. Springer, 2022. 2
[8] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießnnr. Scannet: Richly- annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5831) 2017.
[9] A. Dhamija, M. Gunther, J. Ventura, and T. Boult. The overlooked elephant of object detection: Open set. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1021-1030, 2020. 7
[10] F. Engelmann, M. Bokeloh, A. Fathi, B. Leibe, and M. Nießner. 3d-mpa: Multi- polti-proopoal the aggregation for 3d semantic instance segmentation. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 9031-9040, 2020. 2
[11] A. Gupta, S. Narayan, K. Joseph, S. Khan, F. S. Khan, and M. Shah. Ow-detr: Open-world detection transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9235-9244, 2022. 3,8,7
[12] H. Ha and S. Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. In 6th Annual Conference Robot Learning, 2022. 3
[13] L. Han, T. Zheng, L. Xu, and L. Fang. Occuseg: Occupancy-awee 3d instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2020. 2
[14] T. He, C. Shen, and A. Van Den Hengel. Dyco3d: Robust instance segmentation of 3d point clouds through dynamic convolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 354-363, 2021. 2
[15] J. Hou, A. Dai, and M. Nießner. 3d-sis: 3d semantic instance segmentation of rgb-d scans. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 4421-4430 2019. 2
11

---

[16] J. Hou, B. Graham, M. Nießner. and S. Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15587-15597, 2021.
[17] L. Jiang, H. Zhao, S. Shi, S. Liu, C.-W. Fu, and J. Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and Pattern recognition, pages 4867-488 1867-4876, 2020. 2
[18) K. Joseph, S. Khan, F. S. Khan, and V. N. Balasubramanian. Towards open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5830-5840, 2021. 1,3,5,7
[19] D. Kim, T.-Y. Lin, A. Angelova, I. Kweon, and W. Kuo. Learning open-world object proposals without learning to classify. IEEE Robotics and Automation Letters, 702)545335440, 2022. 9, 10
[20] Lahoud, B. Ghanem, M. Pollefeys, and M. R. Oswald. 3d instance segmentation via multi-tasi task metric learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9256-9266, 2019. 2
[21] Z. Liang, Z. Li, S. Xu, M. Tan, and K. Jia. Instance segmentation in 3d scenes using semantic superpoint tree networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2783-2792, 2021.
[22] S.-H. Liu, S.-Y. Yu, S.-C. Wu, H.-T. Chen, and T.-L. Liu. Learning gaussian instance segmenta- tion in point clouds. arXiv preprint 20Xv:2007.09866) 2020. 2
[23] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2537-2546, 2019. 6
[24] C. Lu, Krishna, M. Bernstein, and L. Fei-Fei. Visual relationship detection with language pri- ors. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 552-869. Springer, 2016.
[25] S. Ma. Y. Wang. J. Fan, Wei, T. H. Li, H. Liu, and F. Lv. Cat: Localization and identification cascade detection transformer for open-world object detection. arXiv preprint arXiv:2301.012 009970, 2023.
[26] D. Miller, L. Nicholson, F. Dayoub, and N. Sünderhauf. Dropout sampling for robust object detection in open-set conditions. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3243-3249. IEFE, 2018.
[27] D. Rozenberszki, O. Litany, and A. Dai. in the wild. In Computer Visi 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXIII, pages 125-141. Sprinner,,,, 2022. 5
indoor 3d semantic segmentation
(28) K. Saito, P. Hu, T. Darrell, and K. Saenko. Learning to detect every thing in an open world. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 268-284. Springer, 2022.
[29] J. Schult, F. Engelmann, A. Hermans, O. Litany, S. Tang, and B. Leibe. Mask3D: Mask Transformer for 3D Semantic Instance Segmentation. In International Conference on Robotics and Automation (ICRA), 2023. 2, 8,8
[30] J. Sun, C. Qing, J. Tan, and X. Xu. Superpoint transformer for 3d scene instance segmentation. arXiv preprini arXiv:2211. 1576666. 2022. 2
[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 2
[32] W. Wang, M. Feiszli, H. Wang, J. Malik, and D. Tran. Open-world instance segmentation: Exploiting pseudo ground truth from learned pairwise affinity. In Proceedings of the IEFF/VV Conference Computer Vision and Pattern Recognition, pages 4422-4432, 2022. 9, 10
12

---

[33] W. Wang, M. Feiszli, H. Wang, and D. Tran. Unidentified video objects: A benchmark for dense, open-world segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10776-10785, 2021.
[34] W. Wang, R. Yu, Q Huang, and U. Neumann. Sgpn: Similarity group proposal network for 3d point cloud instance segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2569-2577 2578, 2018. 2
[35] S. Xie, J. Gu, D. Guo, C. R. Qi, L. Guibas, and O. Litany. Pointcontrast: Unsupervised pre- training for 3d point cloud understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part III 16, pages 574-591. Springer, 2020. 2
[36] B. Yang, J. Wang. R. Clark, Q. Hu, S. Wang, A. Markham, and N. Trigoni. Learning object bounding boxes for 3d instance segmentation on point clouds. Advances in neural information processing systems, 32, 2019.
[37] L. Yi, W. Zhao, H. Wang, M. Sung, and L. J. Guibas. Gspn: Generative shape proposal network for 3d instance segmentation in point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3947-3956, 2019. 2
[38] B. Zhang and P. Wonka. Point cloud instance segmentation using probabilistic embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8883 8892, 2021. 2
[39] Zhang, Dong, and K. Ma. Clip-fo3d: Learning free open-world scene representations from 2d dense clip. arXiv preprint wrXiv:2303.04745 2023. 3
[40] X. Zhu, R. Zhang, B. He, Z. Zeng, S. Zhang, and P. Gao. Pointelip v2: Adapting clip for powerful 3d open-world learning. arXiv preprint arXi:22211 2022.
[41] O. Zohar, K.-C. Wang, and S. Yeung. Prob: Probabilistic objectness for open world object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11444-11453, 2023. 3,7
13

---

Appendix
A Scalability of 3D-OWIS
We show in Table that 3D-OWIS can accommodate a large number of classes without a major size increase
Table 5: Demonstrating the Scalability of 3D-OWIS with Respect to the maximum number of classes it can learn.
# of classes
200
1000
5000
10000
50000
100000
Size of 3D-OWIS
39.7M
39.8M
40.7M
41.9M
50.9M
62.2M
B Additional details on Split B
We utilize the 20 scene types present in the ScanNer200 dataset distribute the 200 classes over the three tasks. Initially, we establish a notion of similarity between two scene types by assessing the extent of their shared classes. This similarity is quantified through the intersection over the union (IoU) metric, which measures the ratio of common classes to the total count of unique classes across both scenes. By employing this metric, identify scene types that exhibit substantial ToU, indicating a higher degree of similarity. The similarity matrix, depicted in Fig. 7, showcases the relationships between the 20 scene types within the ScanNer200 dataset.
Subsequently, we employed three criteria to group the classes: (i) the likelihood of encountering them first when accessing an indoor area, (ii) their affiliation with similar scene types, and (fii) the proximity in the number known classes across tasks. By taking these factors into consideration, arrived at the split of scenes presented in Table 6.
Table 6: Frequently occurring scene when training during the three tasks in Split B. Scene types are grouped into tasks based on three criteria: (i) the likeli- hood encountering the classes within the scene types when entering an indoor area, (ti) similarity of scene types containing the classes, and (iii) consistency the overall number of classes within the scene types across all tasks. This grouping ensures a cohesive organization of scene types for effective evaluation of 3D instance segmentation models integrated with tasks such as robot navigation within indoor environments.
Figure 7: Similarity matrix between the 20 scene types in ScanNet200 dataset. We show the ratio of com- mon classes to the total count of unique classes between two scene types.
Spli B
Tack
Tack
Task 3
HHodl
Kitchen Bathroom
Computerluster
Room
Office
Hallway
Aparimtent
Lounge
Gym Librry
Clasroom
Garage
Conference Room
C Additional details on the experimentation
Training: We train the model on the entire ScanNer200 dataset for all tasks. In Task 1, objects belonging to the classes from Task 2 and Task 3 are masked, excluding them from the learning process. Moving to Task we utilize the last saved checkpoint of the model from Task as starting point and mask the objects with labels that correspond to the current known classes of Task 1 and Task This allows the model to focus solely on learning and distinguishing the specific objects associated with the current task. Finally, Task builds upon the progress made Task 2. We load the
14

---

latest checkpoint of the model from Task 2 and incorporate an exemplar replay. Similar to Task 2. the objects with labels belonging to the known classes Task and Task 2 are masked during training. This step further refines the model's understanding and discrimination abilities for the specific objects relevant the current task. Evaluation: To conduct the evaluation during task, we assign the "unknown" label to the known classes from all the future tasks.
D
Additional qualitative results
D.1
Unknown objects identification
The qualitative results depicted in Fig. 10, 12, 13, and highlight the superior performance of our contribution in retrieving unknown objects. Across the majority of scenes, model consistently corrects the mispredicted unknown classes while preserving the accuracy of known objects, thus demonstrating its robustness and effectiveness.
D.2 Learning novel classes
Fig. 8 and Fig. illustrate the sequential process learning novel classes after identifying unknown objects from the previous task. In Fig. 8, we demonstrate the effectiveness of method in successfully retrieving unknown classes in all tasks. Additionally, in Fig. 9, we highlight the potential of exemplar replay in retaining knowledge of the old classes after learning the novel classes in Task and Task 3.
Task
Task
Task
Ground BEE
3D-OWIS 3D-OWIS-PO-CT Truth
Figure 8: Illustration of the process of unknown identification and learning novel classes. We use orange circles to highlight the differences between 3D-OWIS and 3D-OWIS-PPI 111-PC-CCV CT. The objects depicted in green represent the known classes, while those in blue represent the unknown objects. The gray objects correspond to the background. The qualitative results demonstrate that 3D-OWIS outperforms 3D-OWIS DD-OWIS-PC-CCE CT in retrieving unknown objects. Notably, 3D-OWIS correctly identifies the background objects as unknown, whereas 3D-OWIS PC- CT misclassifies them as known objects.
15

---

Table 7: Proposed distribution of ScanNet200 classes across tasks for each split. We show the classes that are known when training the model during a specific task for the three splits.
Talk
Teal's
Tack
staad
Euediy hoard
tedkpack
Traket
treck cus
dhskk
rack
can of buttie
Tansius pogaceir
sudl dooe
bubiucia board
cdal Euige
plllow
chout
budras
souffdd and
Laphap
boadds
spennee
teedled md)
choee Sol
<Mject
bottle
STONCE
cack
diyar
cailing
coliig
troinng beard
CONOECT C7
Ied0 papu Jugenst
Bear drect
pupe CUECT
fraed
cruch
Eas
hell
Lif's dul
Tat
andda:
Clu day::
DUET
DOUEACT
tack
laeg
follen CONE dapeaso
lauidr baclet
shrver
<bornct hecd
walling neclea
Hight caich
raid nkdge
Eght
riglssad
box
Lntic
sigh
pillce
Ees
daroer will
candl
powcr sute
clcck
Sqperoo
Brandng hucker
The
vaffect ka
p-pe:
puper
del
drak:
sigheraad
refigeniar
dae
cover Enggrnse
Su callict
eudiao
huir dayer
denags N4
N4s
spaitor
porer
chasch diec
plad
stadeedll
handdica hat
clost rnd
43se
cap
parre
Hakt pri
rocdding
pupe
Enguace
Eudge
Cloree: Fooc
decocation
cily
be fedieces surky
heach
calme
kdge
Alam deck
P22
paper
mash ore
oxt
laaalyy Eniigget
coffee sill
Iel: Jryer
deceadion
coach
dor
tabo
rang hood
packet
light
waxhwoll
hed
dtider
beytwaud pil0
light
handyy lungpc
rack
babab
piaco
wadlov
sood foe
polur
stactas
postor
peeee plat
potiolploe plane
t4 stanal
16

---

Task 1
Task 2
Task
Ground Truth
3D-OWIS
Task
Task
Task
Ground Truth
3D-OWIS
Figure 9: Alleviating catastrophic forgetting during incremental learning. The capability of 3D-OWIS retaining knowledge of the previously known classes after learning the new one is demonstrated across Task and Task for both scenes, where all objects of old known classes still being predicted as known.
17

---

Ground Truth
3D-OWIS-PC-CT
3D-OWIS
Figure 10: Qualitative results. The objects depicted in green represent the known classes, while the ones in blue represent the "unknown" class, and the gray objects represent the background. To emphasize the differences between 3D-OWIS and 3D-OWIS PC-CTT we highlight them with orange circles.
18

---

Ground Truth
3D-OWIS-PC-CT
3D-OWIS
Figure 11: Additional qualitative results We demonstrate the better performance of our model in accurately identifying background objects (depicted in gray) as unknown (represented by the blue color), and also correcting the predictions from known class to unknown class. This capability greatly reduces the misclassification of background objects as known objects, leading to improved overall classification accuracy.
19

---

Ground Truth
3D-OWIS-PC-CT
3D-OWIS
Figure 12: Additional qualitative results
20

---

Ground Truth
3D-OWIS-PC-CT
3D-OWIS
Figure 13: Additional qualitative results
21


Reviews:
[{"review_id": "EiCl71VACi", "reviewer": "Reviewer_PXM7", "summary": "This paper addresses the challenge of 3D instance segmentation in open-world scenarios.  It starts with a formulation for this problem, including the definition and setup of known and unknown objects and different splits of categories for simulating different open-world cases. Accordingly, this work proposes a framework incorporating an unknown object identifier to detect objects not present in the training set and devises several strategies to enhance the separation of classes within the query embedding space, such as contrastive clustering and reachability-based probability correction. Experiments on the new benchmark validate the effectiveness of the proposed framework.", "questions": "None.", "limitations": "The author does not discuss the limitations and potential social impacts in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper studies a new problem setting, open-world 3D instance segmentation, with a new benchmark and framework.\n- The basic idea is easy to follow and the presentation is overall clear.\n- The illustration figures are also clear such as Fig. 2-4.\n- The new benchmark considers different open-world cases when splitting different categories.\n- The corresponding modifications tailored to this new problem are reasonable, from contrastive clustering for queries to reachability-based probability correction and alleviating catastrophic forgetting for incremental learning.\n- Experiments show the effectiveness of the proposed framework. Different evaluation metrics also show different perspectives for comparing different methods.", "weaknesses": "- There is no preliminary section for the baseline framework, Mask3D, making the introduction of corresponding modifications tailored to this new setting a little confusing. In my opinion, it would be much better to introduce the baseline framework after setting up the benchmark after Sec. 3.2 and even open a new section for the \"approach\" part.\n\n- Some of the modifications are adapted from experiences in the 2D domain, and they are not revised to fit the 3D problem or tackle some specific challenges in 3D. Although some of them such as reachability-based probability correction have their own insight, it is still unclear whether the underlying reason for these challenges is related to the different data modality. It would be better to have deeper thinking from this aspect.\n\n- The evaluation benchmark involves comprehensive metrics for the new setup, but it lacks an intuitive connection or comparison between the new problem and other conventional ones, such as 2D open-world instance segmentation/detection and 3D close-set instance segmentation. Given this is a relatively new problem, it would be better to compare it with other well-explored problems to make it more friendly and acceptable for researchers from this community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging and insightful comments. Please find our responses to specific queries below. \n\n**Preliminary section after Sec 3.2.**\nWe thank the reviewer for this suggestion. As suggested by the reviewer, we will add a new section to describe our closed-set baseline “Mask3D” in the revised version. \n\n**More insights on the underlying 3D challenges.**\nThe open-world learning problem requires detecting unknowns (unknown recall) along with accurately detecting class-specific knowns (average AP). While some of the proposed adaptations are inspired from the 2D domain, we note that a direct integration of these strategies from 2D provides sub-optimal results with respect to the trade-off between unknown recall and average AP for open-world 3D indoor instance segmentation. To gain further insights, we conduct experiments by leveraging contrastive cluster (ORE) [4] or attention-driven pseudo-labeling (OW-DETR) [3] from recent 2D detection works. These experiments and their results are presented below:\n- OW-DETR [3] has introduced an objectness head to predict the instances' objectness in an input image, which significantly improves the performance of the 2D model for both known and unknown classes. However, our experiments revealed that learning the objectness of 3D features enhances the prediction recall for unknown classes (similar to 2D models) but adversely affects the performance of the 3D model on known classes (Ours with learnable objectness as in OW-DETR: 33.35 mAP vs: **Ours: 39.70** mAP).\n- Given the enriched semantic and 3D spatial information in the refined queries of the transformer decoder in Mask3D, we pursued a contrastive clustering approach to enhance the distinctiveness between queries of different classes. This facilitates a more accurate estimation of the objectness for predicted instances. To cater to the 3D nature of the problem, we tailored the clustering technique used in ORE [4] for OWOD. Unlike OWOD in [4] which clusters intermediate features in the detector, we propose to cluster the queries that encapsulate the richer spatial and semantic information extracted from the 3D transformer decoder rather than features from the output of the 3D backbone.\n\nMoreover, we observed from the t-SNE plots of the known and unknown features (**provided in the rebuttal pdf**) that the features of the unknowns are more dispersed in 3D compared to 2D, and are thus harder to segment. \n\nTherefore, we propose to tailor our method to these observations in the 3D domain and aim to provide an optimal trade-off between unknown recall and known AP. The proposed scheme provides superior performance compared to 2D strategies as shown in the table below.\n\n**Comparison to 2D open-world instance segmentation/detection and 3D closed-set instance segmentation**.\nAs recommended by the reviewer, we provide the following results after implementing techniques proposed for 2D open-world instance segmentation/detection. We also present a comparison between the results of Mask3D and Oracle (3D-OWIS with access to the training dataset of the previously known classes and the labels of the unknown classes). We show results for Split A task1, as we note that GGN [2] and OLN [1] focus on segmenting the unknowns and do not target the incrementally learned tasks. On the other hand, our method aims at both segmenting the unknowns in the current task, as well as incrementally learning classes that are introduced at each task. Our approach performs favorably compared to [1, 2, 3].\n\n|                       |             | WI ↓    | A_OSE ↓| U-Recall ↑| mAP(known)  ↑    | mAP ↑  |\n|-------------------|-----------|:---------:|:-----------:|:--------------:|:----------------------:|:----------:|\n| Closed-set    | Oracle  | 0.129 | 227   | 55.94    | 38.75                     | 38.6  |\n|                      | Mask3D  | \\-      | \\-    | \\-    | 39.12    | 39.12                     |\n|||||||\n| Open World  | OW-DETR [3] | 0.547 | 721   | 22.14    | 35.56                     | 35.05 |\n|                      |  GGN  [2]                 | 15.68      | 1452    | 21.33    | 20.51      | 20.12                        |\n|                      |  OLN   [1]            | \\-      | \\-    | 02.45  | \\-       | \\-                        |\n|                      | **3D-OWIS (Ours)**              | 0.397   | 607   | 34.75 | 40.2     | 39.7                      |\n\n\n**On limitations**. We will add the limitations and societal impact in the revised version. Similar to our base architecture Mask3D, we assume that complete reconstructed scenes are used as input, so the learning benefits from contextual information.\n\n[1] Kim et al., Learning Open-World Object Proposals without Learning to Classify, ICRA 2022 \n\n[2] Wang et al., Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity, CVPR 2022\n\n[3] Gupta, Akshita, et al. Ow-detr: Open-world detection transformer, CVPR 2022.\n\n[4] Joseph, K. J., et al. Towards open world object detection, CVPR 2021.\n"}, {"review_id": "bM6lMc36yw", "reviewer": "Reviewer_CNBM", "summary": "The paper presents a new application of open-world object detection to the setting of 3D instance segmentation. In this setup, a 3D point-cloud segmentation model is required to label each point with instance information, whether this instance is part of training or not. The paper uses ScanNet and proposes a few ways to partition the dataset to use for evaluation in open-world. Finally, the paper adopts the framework in [16] as a baseline for this problem and proposes two tricks on top this framework: PC (probability correction) and CT (confidence thresholding). ", "questions": "NA", "limitations": "Limitation is not sufficiently discussed. For example, when repurposing ScanNet200, what are some potential limitation? This dataset isn't designed specifically for this proposed task in open-world.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The task of open-world segmentation in 3D is important yet overlooked by previous work. As outlined by the paper, prior works mainly focus on 2D setups, either image or video. The use of ScanNet for evaluation seems appropriate and the splitting mechanism also seems reasonable, especially the Region-Based split has a very nice motivation from real-world application.\n", "weaknesses": "Post rebuttal comments:\n1. The author provided more insights towards the difference between 2D and 3D. I was hoping for more attributes unique to 3D. But having some insights as least improve the quality of the work. \n2. My second concern is addressed.\n\n\nOriginal review:\nAlthough there are many things to like about the proposal of the task, this work may suffer from a few important weaknesses:\n- Missing insights specifically to 3D. The major contribution of the paper is to adopt the recent new task of open-world localization to 3D setting. Unfortunately, the paper does not focus enough on aspects unique to 3D. Most components are the same as the 2D task and insights provided by the paper do not contain enough 3D-specific information. To justify the contribution, it is important for the paper to provides both intuitions and empirical proves on why it is worth studying 3D and how it is different than simple adoption of a 2D framework. In fact, the 3D-OWIS seems to be almost identical to [16] besides swapping the backbone predictor with a 3D backbone. The claimed contributions PC and CT are also not specific to 3D or reveals anything special about 3D. \n- Lack of comprehensive evaluation/ ablations. If author would prefer to claim more of their contributions on the dataset part, it is important to benchmark the task comprehensively. A suite of recent baselines in 2D should be evaluated. For example, recent works OLN [A] and GGN [B] both provide good class-agnostic proposals, and their 3D variants should be evaluated. In particular, GGN also uses a self-teaching schema. \n\nMinor:\n- The presentation of baseline 2D-OWIS-PC-CT seems really vague when first-time introduced. Better elaborate on what this is when it is discussed the first time. \n- If author prefers to claim contributions mainly to the task/ benchmark, NeurIPS D&B is more appropriate.\n\n[A] Kim et al., Learning Open-World Object Proposals without Learning to Classify, ICRA 2022 \n[B] Wang et al., Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity, CVPR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback. Detailed answers to the reviewer's queries are provided below. \n\n**3D-specific insights.**\nAs recommended, we present here both empirical and qualitative results to provide more insights specific to our 3D setting. We empirically observe that a direct integration of different strategies from 2D provides sub-optimal results with respect to the trade-off between unknown recall and average AP for the proposed open-world 3D indoor instance segmentation setting. To this end, we conduct experiments by leveraging contrastive cluster (ORE [16]) or attention-driven pseudo-labeling (OW-DETR [C]) from recent 2D detection works. These experiments and their results are presented below:\n\nOW-DETR [C] has introduced an objectness head to predict the instances' objectness in an input image, which significantly improves the performance of the 2D model for both known and unknown classes. However, our experiments revealed that learning the objectness of 3D features enhances the prediction recall for unknown classes (similar to 2D models) but adversely affects the performance of the 3D model on known classes (Ours with learnable objectness as in OW-DETR: 33.35 mAP vs: **Ours: 39.70** mAP).\n\nGiven the enriched semantic and 3D spatial information in the refined queries of the transformer decoder in Mask3D, we pursued a contrastive clustering approach to enhance the distinctiveness between queries of different classes. This facilitates a more accurate estimation of the objectness for predicted instances. To cater to the 3D nature of the problem, we tailored the clustering technique used in ORE [16] for OWOD.  Unlike OWOD in [16] which clusters intermediate features in the detector, we propose to cluster the queries that encapsulate the richer spatial and semantic information extracted from the 3D transformer decoder rather than features from the output of the 3D backbone.\n\nMoreover, to gain more insight into the behavior of unknown classes in the 3D setting, we visualize the t-SNE plots of class features used for clustering in ORE [16] for 2D images and in our 3D-OWIS for 3D point clouds. The t-SNE comparison is presented in the **rebuttal pdf**. The t-SNE features in ORE [16] are extracted from the detector where Pascal VOC classes are known and all classes from MS-COCO are grouped together as unknown. In the case of our 3D-OWIS for 3D point clouds, we use the final refined queries in Mask3D decoder. In our 3D setting, we are showing split A task 1. The t-SNE results show more sparsity in the 3D features of the unknown classes which makes them harder to segment, compared to their 2D counterparts which are easier to cluster. To cope with this challenge in 3D point cloud unknown segmentation, we propose to use the known class query prototypes to correct the features of the unknowns in the boundaries of the clusters of the known classes.\n\nThe proposed scheme provides superior performance compared to both 2D strategies as shown in the table below.\n\n**Comparison to OLN [A] and GGN [B].**\nAs recommended, we present the comparison with OLN [A] and GGN [B] in the table below. For OLN [A], we remove the classification head from Mask3D, while we keep the mask loss during training. In the case of  GGN [B], we train an affinity predictor with Minkowski as a 3D backbone and a prediction head for the affinity.  Affinity maps are generated by the affinity predictor, and fed to the Connected Components module to generate class-agnostic mask proposals. We hypothesize that OLN failed in 3D point cloud instance segmentation because it wasn't able to learn a good objectness representation from masks only given the sparsity of point clouds. Meanwhile, GGN [B] pseudo-labels are of lower quality since the affinity is not optimal for 3D because of the empty space and the disconnected object parts (qualitative results in **rebuttal pdf** show GGN pseudo labels).\n\n|                       |             | WI ↓    | A_OSE ↓| U-Recall ↑| mAP(known)  ↑    | mAP ↑  |\n|-------------------|-----------|:---------:|:-----------:|:--------------:|:----------------------:|:----------:|\n| Closed-set    | Oracle  | 0.129 | 227   | 55.94    | 38.75                     | 38.6  |\n|                      | Mask3D  | \\-      | \\-    | \\-    | 39.12    | 39.12                     |\n|||||||\n| Open World  | OW-DETR [C] | 0.547 | 721   | 22.14    | 35.56                     | 35.05 |\n|                      |  GGN  [B]                 | 15.68      | 1452    | 21.33    | 20.51      | 20.12                        |\n|                      |  OLN   [A]            | \\-      | \\-    | 02.45  | \\-       | \\-                        |\n|                      | **3D-OWIS (Ours)**              | 0.397   | 607   | 34.75 | 40.2     | 39.7                      |\n\n**Clarification on 3D-OWIS-PC-CT.**\nWe thank the reviewer for the suggestion. 3D-OWIS-PC-CT represents our method without Probability Correction (PC) and without Confidence Threshold (CT). The final model 3D-OWIS includes all components, including PC and CT. \n\n**Discussion of limitations.**\nWe note that ScanNet200 benchmark is the largest 3D indoor instance segmentation dataset in terms of the number of classes. Due to the highly challenging nature, diversity, and number of classes, we adapt it to open-world setting. We also introduce carefully curated open-world splits leveraging realistic scenarios based on inherent object distribution, region-based indoor scene exploration, and the randomness aspect of open-world classes. As noted by the reviewer, our region-based split is well motivated from the perspective of real-world application. In future, we aim to further explore open-world setting for outdoor 3D scenes.\n\n[C] Gupta, Akshita, et al. Ow-detr: Open-world detection transformer, CVPR 2022.\n\n"}, {"review_id": "W4EFWYHGXg", "reviewer": "Reviewer_WNwG", "summary": "This paper introduces an open-world 3D indoor instance segmentation method, where an auto-labeling scheme is employed to produce pseudo-labels during training and induce separation to separate known and unknown category labels. Pseudo label quality are further improved subsequently.", "questions": "Please refer to the weakness section.", "limitations": "The author does not discuss the limitation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Open world 3d instance segmentation is an important problem and the author first addresses this problem.\n2. The method designing is reasonable and complete.\n3. The experiment is also sufficient.", "weaknesses": "1. Some designs are hard to understand. For example, for probability correction, why unknown object has to be far from the nearest known class? since we have no prior about category distributions. In experiments, you can also see that PC does not bring improvement to all metrics. This should be illustrated more clearly.\n2. Although such open world task has not been studied in 3d tasks, it has been widely studied in 2d tasks. Since the core issue of this problem is actually the same, the author should implement some 2d open world methods in 3d instance segmentation and compare with them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the suggestions to improve the clarity of the paper. \n\n**On probability correction assuming unknowns are far from knowns.**\nIn our open-world 3D instance segmentation framework, we use the auto-labeler to generate pseudo-labels for the unknowns. The contrastive clustering step then minimizes the distance between the queries and the corresponding class prototype and maximizes the distance between the other class prototypes. This includes queries corresponding to the pseudo-labels that are pulled toward the unknown class prototype. Since the class prototypes are pushed away from each other, it is expected that the unknowns are also pushed away from the known class prototypes. We thank the reviewer and will add this clarification in the revised version. \n\n**Comparison to 2D baselines.**\nWe empirically observe that a direct integration of different strategies from 2D provides sub-optimal results with respect to the trade-off between unknown recall and average AP for the proposed open-world 3D indoor instance segmentation setting. To this end, we conduct experiments by leveraging contrastive cluster (ORE [4]) or attention-driven pseudo-labeling (OW-DETR [3]) from recent 2D detection works. These experiments and their results are presented below:\n\n- OW-DETR [3] has introduced an objectness head to predict the instances' objectness in an input image, which significantly improves the performance of the 2D model for both known and unknown classes. However, our experiments revealed that learning the objectness of 3D features enhances the prediction recall for unknown classes (similar to 2D models) but adversely affects the performance of the 3D model on known classes (Ours with learnable objectness as in OW-DETR: 33.35 mAP vs: **Ours: 39.70** mAP).\n\n- Given the enriched semantic and 3D spatial information in the refined queries of the transformer decoder in Mask3D, we pursued a contrastive clustering approach to enhance the distinctiveness between queries of different classes. This facilitates a more accurate estimation of the objectness for predicted instances. To cater to the 3D nature of the problem, we tailored the clustering technique used in ORE [4] for OWOD. Unlike OWOD which clusters intermediate features in the detector, we choose to cluster the queries that encapsulate the richer spatial and semantic information extracted from the 3D transformer decoder rather than features from the output of the 3D backbone.\n\nMoreover, to gain more insight into the behavior of unknown classes in the 3D setting, we visualize the t-SNE plots of class features used for clustering in ORE [4] for 2D images and in our 3D-OWIS for 3D point clouds. The t-SNE comparison is presented in the **rebuttal pdf**. The t-SNE features in ORE [4] are extracted from the detector where Pascal VOC classes are known and all classes from MS-COCO are grouped together as unknown. In the case of our 3D-OWIS for 3D point clouds, we use the final refined queries in Mask3D decoder, which are used for predicting the masks and the class labels. In our 3D setting, we are showing split A task 1. The results in the figure show more sparsity in the 3D features of the unknown classes which makes them harder to segment compared to their 2D counterparts which are easier to cluster. To cope with this challenge in 3D point cloud unknown segmentation, we propose to use the known class query prototypes to correct the features of the unknowns in the boundaries of the clusters of the known classes.\n\nTherefore, we propose to tailor our method to these observations in the 3D domain and aim to provide an optimal trade-off between unknown recall and known AP. The proposed scheme provides superior performance compared to 2D strategies as shown in the table below. Further, we also present a comparison with two other existing works OLN [1] and GGN [2] in the table below. For OLN [1] implementation, we remove the classification head. In the case of GGN [2], we train an affinity predictor with Minkowski 3D backbone. We hypothesize that OLN failed in 3D point cloud instance segmentation because it wasn't able to learn a good objectness representation from masks only given the sparsity of point clouds. Meanwhile, GGN [2] pseudo-labels are of lower quality since the affinity is not optimal for 3D because of the empty space and the disconnected object parts (qualitative results in **rebuttal pdf** show GGN pseudo labels).\n\n|                       |             | WI ↓    | A_OSE ↓| U-Recall ↑| mAP(known)  ↑    | mAP ↑  |\n|-------------------|-----------|:---------:|:-----------:|:--------------:|:----------------------:|:----------:|\n| Closed-set    | Oracle  | 0.129 | 227   | 55.94    | 38.75                     | 38.6  |\n|                      | Mask3D  | \\-      | \\-    | \\-    | 39.12    | 39.12                     |\n|||||||\n| Open World  | OW-DETR [3] | 0.547 | 721   | 22.14    | 35.56                     | 35.05 |\n|                      |  GGN  [2]                 | 15.68      | 1452    | 21.33    | 20.51      | 20.12                        |\n|                      |  OLN   [1]            | \\-      | \\-    | 02.45  | \\-       | \\-                        |\n|                      | **3D-OWIS (Ours)**              | 0.397   | 607   | 34.75 | 40.2     | 39.7                      |\n\n\n**Discussion of limitations.**\nWe will add the limitations and societal impact in the revised version. Similar to our base architecture Mask3D, we assume that complete reconstructed scenes are used as input, so the learning benefits from contextual information.\n\n[1] Kim et al., Learning Open-World Object Proposals without Learning to Classify, ICRA 2022\n\n[2] Wang et al., Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity, CVPR 2022\n\n[3] Gupta, Akshita, et al. Ow-detr: Open-world detection transformer, CVPR 2022.\n\n[4] Joseph, K. J., et al. Towards open world object detection, CVPR 2021."}, {"review_id": "cDjR4QFwot", "reviewer": "Reviewer_L857", "summary": "This paper proposes a pipeline to do 3D open-world instance segmentation. The authors provide a problem definition and introduced three different scenarios. Moreover, to overcome the possible problems that may lead to lower performance on known classes, the authors proposes different modules like probability correction and exemplar replay. The comprehensive experiments show their proposed methods' effectiveness.", "questions": "As mentioned before I'm not an expert in this field, I'll ask more questions in later reviewing process. Currently I have one concern, since the network won't be re-trained in test and the network structure is fixed, does it mean the total class number (including the known classes and the unknown classes) will have an upper bound? And will the performance of this feature-cluster-based new class discovery become lower as the new class number increases?", "limitations": "No. The authors didn't address their limitations, and also the possible negative societal impact. I would suggest authors discuss some foundamental drawbacks of current open-world discovery settings.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "As claimed by the author, this paper is the first to investigate the 3D open-world instance segmentation task, and develop some well-motivated modules to improve the performance upon the simple baseline.", "weaknesses": "I'm not very familiar with the open-world related task and common practices both in 2D and 3D. Currently I think the writing of this paper is confusing and seems to be finished in a hurry. For example, the \"3D-OWIS-PC-CT\", I assume the \"-PC-CT\" is **without PC and CT**? And I couldn't find explanations about the \"PC\" and \"CT\".\n\nIn addition, I think authors can also provide the results of their pipeline equipped with other 3D instance segmentation as a comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging and insightful comments. Please find our responses to specific queries below. \n\n**Explanation about PC and CT.**\n3D-OWIS-PC-CT represents our method without Probability Correction (PC) and without Confidence Threshold (CT). The final model 3D-OWIS includes all components, including PC and CT. Our 3D-OWIS replaces the selection of top-k-based unknown pseudo labels selection with Confidence Threshold (CT) -based selection. This change, which prioritizes quality over quantity, contributes to generating improved masks and enhancing performance on known classes. Further, we integrated Probability Correction (PC), a technique aimed at rectifying instances that were initially misclassified as known classes. Our inspiration for PC came from analyzing t-SNE plots of the features of unknown classes, a comparison that revealed greater dispersion in our unknown class features compared to those of ORE [1] in the 2D open-world object detection (see rebuttal pdf). This observation shows the challenge of segmenting unknowns in a 3D setting as opposed to 2D.\n\n**Comparison to other 3D instance segmentation.**\nWe base our work on the recently introduced Mask3D (ICRA 2023), as it achieves state-of-the-art performance on the challenging ScanNet200 benchmark for the task of 3D indoor instance segmentation. Our approach on three carefully curated open-world splits achieves promising results compared to the recent Mask3D. As a potential future work, we aim to extend our open-world setting for outdoor 3D scenes and approaches. \n\n**Bound on the number of classes.**\nIn our experiments, we set the maximum number of classes to 200, which includes all the classes provided by ScanNet200 benchmark. Nonetheless, increasing the limit of the number of classes would only add more classifiers at the last layer of Mask3D. These classifiers are only trained once the network is presented with ground-truth objects of corresponding classes, and do not affect the overall performance of previous tasks. Therefore, increasing the limit of the number of classes would yield similar results to the ones provided in the paper, but with additional memory requirements.\n\n| \\# of classes   | 200   | 1000  | 5000  | 10000 | 50000 | 100000 |\n| --------------- | ----- | ----- | ----- | ----- | ----- | ------ |\n| Size of 3D-OWIS | 39.7M | 39.8M | 40.7M | 41.9M | 50.9M | 62.2M  |\n\n**Discussion of limitations.**\nWe will add the limitations and societal impact in the revised version. Similar to our base architecture Mask3D, we assume that complete reconstructed scenes are used as input, so the learning benefits from contextual information.\n\n[1] Joseph, K. J., et al. Towards open world object detection, CVPR 2021."}, {"review_id": "d7PTK7K3vj", "reviewer": "Reviewer_HHFW", "summary": "This work tackles the task of incremental object-discovery for 3D semantic instance segmentation. Unlike numerous concurrent works, it is not unsupervised but enables users (or oracle) to label objects that were identified as unknown in each iteration. The method is evaluated on ScanNet200, the paper proposes three new splits/tasks and measures three different metrics as defined in prior work.", "questions": "Things that are not clear and should be improved:\n\n- Looking at the videos and experiments: There are two methods 3D-OWIS-PC-CT and 3D-OWIS - are both yours? Or is one the baseline that is not yours, and the other yours? Maybe you can clearly mark yours with “Ours”.\n\n- l.106-110: “the learned that updates itself” -- what does it mean? Is it correct that after one iteration of predictions by the model, the users labels the unknown classes, and then (in the next iteration) the model is trained again based in the existing and additionally provided user annotations? If this is the case, it is confusing to say “continuously improving itself (l.109)” since that somehow suggests that the model can improve without human intervention whereas here it seems to rely on human labels? I do not question the usefulness of user input after each iteration (i think it is very desirable), I simply do not understand what is happening in the end - the description should be improved.\n\n- Figure 3 / Sec 3.2 Split B - I do not understand the split - what does it mean “exploring indoor areas” / “accessing an indoor space”? Is there a robot walking around in a simulation (I guess not)?\n\n- l.132 - what is “the auto labeler”? The term is used here for the first time without explanation. The text then continues in l.136-143 explaining the drawbacks and proposed solutions. Since I don’t know what an auto-labeler is, these two paragraphs did not provide any information to me so I ignored them for writing this review. L. 144 “the auto labeller depicted in Fig.2” - it is not depicted in Fig2 - there is only a box with the name “auto labeled” - this does not provide any useful information to understand what is happening.\n\n- Eq. 2, what is a class prototype? Do we have a class for each semantic class i.e. max of 200 in ScanNet200, or does it refer to individual object instances? The text suggests that it refers to semantic classes. If that is the case, how do you separate different instances that have the same semantic class?\n\n- l.175 - what is 3D-OWIS? Is it the same as [20]? After reading l.172-177, they seem to be the same, but I could not find the name 3D-OWIS in [20]. Is 3D-OWIS maybe the name of the method proposed in this paper? The caption of  Fig.6 it seems to become clear that 3D-OWIS is the name of the method proposed in this paper. Similarly, in line 244 what is -PC-CT? What are the contributions? What does PC CT stand for? The last line of the caption in Tab.3 indicates that PC is probability corrections dn CT is confidence threshold? [I have to play detective only to understand what is the name of your method, why do you do this to me? Imagine R2 - they will directly reject the paper :( ]\n\n- Why are the Tasks 1-2-3 introduced? To my understanding, it seems to be a redefinition of the existing head-common-tail classes. Why confuse the reader and introduce an alias for sth that is already clearly defined?\n", "limitations": "No / not applicable", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The proposed approach has significant practical significance as it enables training on a labelled corpora of data and provides the option to users to label the identified unknown objects and then refine the trained model once training labels are provided by a user.\n", "weaknesses": "The manuscript is unclear on numerous occasions to the point that it is hard to understand the method (see questions below, these should be addressed in the rebuttal and incorporated in an updated version). In general, the paper might be clear for those that worked on this project (I assume for a significant amount of time) but the writing is challenging to understand by someone that reads about the project for the first time (unfortunately this makes it really difficult to write a meaningful review).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging and insightful comments. Please find our responses to specific queries below. \n\n**Clarification on 3D-OWIS-PC-CT.**\n3DOWIS-PC-CT is an extended Mask3D with predictive capabilities to encompass unknown classes. Unlike the original closed-set baseline Mask3D, which is confined to training on a predetermined set of classes and limited to predicting solely within those classes during inference, our novel approach leverages insights from ORE [1] to achieve open-world segmentation capabilities. We meticulously adapted and integrated these insights to 3D point cloud instance segmentation, where we employ the contrastive clustering on the final refined queries in Mask3D decoder to improve the objectness score estimation, which will help in increasing the quality of the generated unknown pseudo labels. An Auto-labeling module is used in 3DOWIS-PC-CT to select the predicted masks with the highest confidence (top-k) for instances that do not correspond to the ground truth of known classes. These predicted masks are then utilized as pseudo labels to represent the unknown class.\n\n**Continuous improvement.**\nIn an open-world learning scenario, a model's progression is split into distinct tasks. In each task, the model is introduced to a specific group of labels that are known, while also being presented with numerous unfamiliar classes. In the first task, the model learns from the known examples and develops the ability to recognize unfamiliar classes within the scene. Moving on to the second task, the model builds on its knowledge from the known classes in the first task and is introduced to a fresh set of known classes. The concept of continuous improvement involves the model efficiently training on these solely new classes without causing its performance on the known classes from the first task to decline. A good model can retain the segmentation ability of old classes during training on new classes. This iterative process continues until the model reaches its maximum capacity in terms of the number of classes it can accommodate, determined by the classification head's size. \n\n**Split B motivation.**\nThe perfect model for a robot moving indoors should segment both classes it knows and classes it hasn't seen before. Additionally, it should keep learning and getting better at segmenting new classes over time. In split B, we try to emulate how object classes might be sequentially labeled based on the scene types a robot first encounters. This grouping is good to assess how well models can segment objects in scenes that robots encounter while navigating in indoor areas.\n\n**Clarification on the auto-labeler.**\nA closed-set model's prediction only identifies classes it is familiar with, so all masks it predicts get labeled with one of those known classes. To create a substitute \"pseudo-ground truth\" for the unfamiliar classes, the auto-labeler picks predicted masks with the best objectness and zero overlaps with the known class's actual mask. These picked masks are marked as unknown and help teach the model about the unknown cases.\n\n**Clarification on the class prototype.**\nA class prototype is a summary feature that captures both the spatial and semantic aspects of a class. If, for example, there are 20 known classes, the model stores 21 of these summaries: one for each known class and an extra one for the unknown classes. These prototypes are estimated by keeping an exponential moving average of the features matched with the ground truth for the known classes and the pseudo-ground truth for the unknown classes. \n\nNote that in Mask3D 100 queries are refined and each query is used to generate a mask and a class, but only some of them correspond to a real ground truth. As a result, the 100 masks and labels generated from the queries are matched with the ground truth for the knowns and pseudo-ground truth for the unknowns using Hungarian matching, then the query feature is selected to represent its matched ground truth, and then stored to perform the class prototype estimation of the classes using exponential moving average. \n\n**Clarification on the 3D-OWIS and 3D-OWIS-PC-CT.**\nAs discussed above, 3D-OWIS-PC-CT represents our refined adaptation of the initial closed-set baseline Mask3D. The goal here was to enhance its capabilities for an open-world environment. On the other hand 3D-OWIS is our final model, which emerges from two key changes:\n\n- Firstly, we replaced the selection of top-k-based unknown pseudo labels selection with Confidence Threshold (CT) -based selection. This change, which prioritizes quality over quantity, contributes to generating improved masks and enhancing performance on known classes.\n\n- Secondly, we integrated Probability Correction (PC), a technique aimed at rectifying instances that were initially misclassified as known classes. Our inspiration for PC came from analyzing t-SNE plots of the features of unknown classes, a comparison that revealed greater dispersion in our unknown class features compared to those of ORE [1] in the 2D open-world object detection (see **rebuttal pdf**). This observation shows the challenge of segmenting unknowns in a 3D setting as opposed to 2D.\n\n**Need for Tasks 1, 2, and 3.**\nA task refers to what set of classes the model has to learn. It is used to keep track of the progress of a model while learning new classes. for split A, which is based on class-frequency, it follows the head/common/tail splitting in ScanNet200. However, split B and C are distinct from this classification in the original dataset, where split B is determined based on regions and scene types and split C is completely random to simulate the randomness aspect of the open-world.\n\n[1] Joseph, K. J., et al. Towards open world object detection, CVPR 2021."}]

Author rebuttal:
We thank all the reviewers (PXM7, CNBM, WNwG, L857, HHFW) for the positive and valuable feedback, and we appreciate the comments to improve our work. **Reviewer PXM7:** "Idea is easy to follow and the presentation is overall clear. Illustration figures are clear. The corresponding modifications tailored to this new problem are reasonable". **Reviewer CNBM:**  "open-world segmentation in 3D is important yet overlooked. Region-Based split has a very nice motivation from real-world application". **Reviewer WNwG:** "The method designing is reasonable and complete. The experiment is also sufficient.". **Reviewer L857:** "develop some well-motivated modules to improve the performance". **Reviewer HHFW:** "The proposed approach has significant practical significance".  

We summarize the main points presented in our response:
- We include comparisons to other strategies tackling 2D open-world.
- We present more insights on the 3D challenges and motivations to our proposed method.
- We provide clarifications and answers to the points raised, which we will include in our revised version.

Meta-review: This work proposes a method for tackling open-world 3D indoor instance segmentation by identifying when objects belong to an "unknown" category and incrementally expanding the set of known classes.  The method uses an auto-labeling scheme to separate known and unknown category labels, and use contrastive clustering to learn prototype representations that will be used for classifying new instances.  The main contribution of this work is the introduction of this problem setup and evaluation protocol (where there are different types of splits for unknown clases) for investigating open-world instance segmentation with unknown classes in 3D.

While this setup is novel and interesting, reviewers have expressed some concerns about the writing and presentation of the work.  Nevertheless, reviewers are overall positive on the work.  The AC agrees that the work has value and can foster additional work in this important direction.

The AC recommends that the authors incorporate clarifications and feedback from the reviewers including:
1. Improvements to the writing to explain the baseline method (PXM7), clarify unclear details in the method (HHFW, L857), design decisions (WNwG), additional visualizations, etc.
2. Add experiments included for the author response (including comparison of techniques for 2D open world)
3. Add a discussion of what are the challenges / insights of this task in 3D vs 2D (PXM7, CNBM, WNwG)
4. Add limitations and societal impact discussion 
5. As this is an of growing interest, the AC also recommend the authors include discussion of how this work relates to recent work in open-world 3D scene understanding (many of which are orthogonal to the contributions of this work). A selection of such work is given below:
- Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding (Ding et al. 2023)
- Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models (Ha and Song, CRL 2022)
- CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP (Zhang et al. 2023)
- PointCLIP V2: Adapting CLIP for Powerful 3D Open-world Learning (Zhu et al. 2022)
Decision: Accept (poster)