Title: 3D Indoor Instance Segmentation in an Open-World
Venue: NeurIPS 2023
Accepted: yes
Keywords: open-world

3d instance segmentation
Abstract: Existing 3D instance segmentation methods typically assume that all semantic classes to be segmented would be available during training and only seen categories are segmented at inference. We argue that such a closed-world assumption is restrictive and explore for the first time 3D indoor instance segmentation in an open-world setting, where the model is allowed to distinguish a set of known classes as well as identify an unknown object as unknown and then later incrementally learning the semantic category of the unknown when the corresponding category labels are available. To this end, we introduce an open-world 3D indoor instance segmentation method, where an auto-labeling scheme is employed to produce pseudo-labels during training and induce separation to separate known and unknown category labels. We further improve the pseudo-labels quality at inference by adjusting the unknown class probability based on the objectness score distribution. We also introduce carefully curated open-world splits leveraging realistic scenarios based on inherent object distribution, region-based indoor scene exploration and randomness aspect of open-world classes. Extensive experiments reveal the efficacy of the proposed contributions leading to promising open-world 3D instance segmentation performance. Code and splits are available at: https://github.com/aminebdj/3D-OWIS.

3D Indoor Instance Segmentation in an Open-World

Abstract

Existing 3D instance segmentation methods typically assume that all semantic classes to be segmented would be available during training and only seen categories are segmented at inference. We argue that such a closed-world assumption is restrictive and explore for the first time 3D indoor instance segmentation in an open-world setting, where the model is allowed to distinguish a set of known classes as well as identify an unknown object as unknown and then later incrementally learning the semantic category of the unknown when the corresponding category labels are available. To this end, we introduce an open-world 3D indoor instance segmentation method, where an auto-labeling scheme is employed to produce pseudo-labels during training and induce separation to separate known and unknown category labels. We further improve the pseudo-labels quality at inference by adjusting the unknown class probability based on the objectness score distribution. We also introduce carefully curated open-world splits leveraging realistic scenarios based on inherent object distribution, region-based indoor scene exploration and randomness aspect of open-world classes. Extensive experiments reveal the efficacy of the proposed contributions leading to promising open-world 3D instance segmentation performance. Code and splits are available at: https://github.com/aminebdj/3D-OWIS.

1 Introduction

3D semantic instance segmentation aims at identifying objects in a given 3D scene, represented by a point cloud or mesh, by providing object instance-level categorization and semantic labels. The ability to segment objects in the 3D domain has numerous vision applications, including robotics, augmented reality, and autonomous driving. Following the developments in the sensors that acquire depth information, a variety of datasets has been presented in the literature which provides instance-level annotations. In view of the availability of large-scale 3D datasets and the advances in deep learning methods, various 3D instance segmentation methods have been proposed in recent years.

The dependence of 3D instance segmentation methods on available datasets has a major drawback: a fixed set of object labels (vocabulary) is learned. However, object classes in the real world are plentiful, and many unseen/unknown classes can be present at inference. Current methods that learn on a fixed set not only discard the unknown classes but also supervise them to be labeled as background. This prevents intelligent recognition systems from identifying unknown or novel objects that are not part of the background. Given the importance of identifying unknown objects, recent works have explored open-world learning setting for 2D object detection [18, 11, 28, 33]. In the open-world setting, a model is expected to identify unknown objects, and once new classes are labeled, the new set is desired to be incrementally learned without retraining [18]. While previous methods have been mostly suggested for open-world 2D object detection, it is yet to be explored

Initialknownobjects

Unknownobjects

Newknown objects

Unknown

Operator

Human

New

Unknown

Operator

Human

New

objects

classes

objects

classes

t

t+1

t+2

time

in the 3D domain. The main challenge lies in understanding how objects appear in 3D in order to separate them from the background and other object categories.

3D instance segmentation in the open world, illustrated in Fig. 1, offers more flexibility, allowing the model to identify unknown objects and request annotations for these novel classes from an oracle for further training. However, this approach presents several challenges: (i) the lack of annotations for unknown classes, necessitating quality pseudo-labeling techniques; (ii) the similarities between predicted features of known and unknown classes, requiring separation techniques for improved prediction; and (iii) the need for a more reliable objectness scoring method to differentiate between good and bad predicted masks for 3D point clouds.

In this work, we investigate a novel problem setting, namely open-World indoor 3D Instance Segmentation, which aims at segmenting objects of unknown classes while incrementally adding new classes. We define real-world protocols and splits to test the ability of 3D instance segmentation methods to identify unknown objects. In the proposed setup, unknown object labels are also added incrementally to the set of known classes, akin to real-world incremental learning scenarios. We propose an unknown object identifier with a probability correction scheme that enables improved recognition of objects. To the best of our knowledge, we are the first to explore 3D instance segmentation in an open-world setting. The key contributions of our work are:

We propose the first open-world 3D indoor instance segmentation method with a dedicated mechanism for accurate identification of 3D unknown objects. We employ an auto-labeling scheme to generate pseudo-labels during training and induce separation in the query embedding space to delineate known and unknown class labels. At inference, we further improve the quality of pseudo-labels by adjusting the probability of unknown classes based on the distribution of the objectness scores.

We introduce carefully curated open-world splits, having known vs. unknown and then incremental learning over the span of 200 classes, for a rigorous evaluation of open-world 3D indoor segmentation. Our proposed splits leverage different realistic scenarios such as inherent distribution (frequency-based) of object classes, various class types encountered during the exploration of indoor areas (region-based), and the randomness aspect of object classes in the open-world. Extensive experiments reveal the merits of the proposed contributions towards bridging the performance gap between our method and oracle.

2 Related Work

3D semantic instance segmentation: The segmentation of instances in 3D scenes has been approached from various angles. Grouping-based or clustering-based techniques use a bottom-up pipeline by learning an embedding in the latent space to help cluster the object points. [4, 13, 14, 17, 20, 21, 34, 38]. Proposal-based methods work in a top-down fashion, first detecting 3D bounding boxes, then segmenting the object region within the box [10, 15, 22, 36, 37]. Recently, spurred by related 2D work [5, 6], the transformer design [31] has also been applied for the purpose of segmenting 3D instances [29, 30]. Other methods present weakly-supervised alternatives to methods that use dense annotations in order to lower the cost of annotating 3D data [7, 16, 35]. While all these methods aim to improve the quality of 3D instance segmentation, they are trained on a known set of semantic labels. On the other hand, our proposed method aims at segmenting objects with both known and unknown class labels.

Training

Inference

Contrastive Clustering

3D Convolutional Backbone

UpdatePrototypes

Probability Correction

TransformerDecoder

Repulsion

Attraction

Query Processing

QueryStore

Initial

queries

MLP

MLP

Auto

QueryProcessing

Unknown

Labeler

Lmask1

Pseudo labels

Unknown class

Assign target

MLP+

Contrastive Clustering

Mask

Predicted

Predicted labels

toquery

softmax

Open-world object recognition: Open-world object recognition was introduced in [2], where the Nearest Mean Classifier was extended for an open-world setting. In the direction of open-world object detection, many studies [41, 18, 11, 25] have been conducted in the past. In[18], pseudo-labels for the unknowns are generated to perform contrastive clustering during training for a better unknownknown classes separation, where an energy-based unknown class identifier was proposed to detect the unknown classes, based on the energy of the logits from the known classes. For incremental learning, they adopted exemplar replay to alleviate catastrophic forgetting of old classes. In the same task as [18], [11] used a transformer-based model and proposed another way of unknown pseudo-labels generation, by using a new method of objectness estimation, and introduced a foreground objectness branch that separates the background from the foreground. For the task of outdoor 3D point cloud semantic segmentation, [3] proposed a model that predicts old, novel, and unknown classes from three separate classification heads. The latter is trained on the labels of the known classes and pseudo-labels for old classes generated by the same model to alleviate catastrophic forgetting, while the unknown class is assigned the second-highest score for a better unknown class segmentation. Other methods proposed in [40, 12, 39], primarily focus on enhancing the generalizability of 3D models for novel classes by leveraging supervision from 2D Vision Language Models for object recognition and 3D semantic segmentation tasks. However, these approaches exhibit several limitations, including (i) The 3D model's performance becomes dependent on the 2D Vision Language model. (ii) The 3D geometric properties of unseen objects in the training data are neglected during the training process. (iii) There exists no avenue for enhancing the model's performance on novel classes in cases where new labels are introduced.(iv) The training process necessitates pairs of images and corresponding 3D scenes.

3 Closed-world 3D Instance Segmentation

We adopted the state-of-the-art 3D instance segmentation model Mask3D [29] as our baseline. The latter is a hybrid model that combines Convolutional Neural Networks (CNNs) with transformers to learn class-agnostic masks and labels for instance separation. The backbone of Mask3D is CNN-based and used to extract feature maps from multiple levels. Meanwhile, the decoder is transformer-based and used to refine n Q ∈ N instance queries Q = { q j ∈ R D | j ∈ (1 , ..., n Q ) } , using the extracted

feature maps. The learning scheme consists of a Cross-entropy loss for learning semantic class labels and binary cross-entropy loss for learning instance masks during training.

4 Open-World 3D Instance Segmentation

4.1 Problem formulation

We start by formulating the problem setting of open-world 3D instance segmentation. At a Task T t , there exists a set of known object categories K t = { 1 , 2 , .., C } and a set of unknown object categories U t = { C +1 , ... } that may exist on inference time. The training dataset D t = { X t , Y t } includes samples from the classes K t . The input set X t = { P 1 , .., P M } is made of M point clouds, where P i ∈ R N × 3 is a quantized point cloud of N voxels each carrying average RGB color of the points within. The corresponding labels are Y t = { Y 1 , .., Y M } , where Y i = { y 1 , .., y k } encodes k object instances. Each object instance y i = [ B i , l i ] represents a binary mask B i ∈ { 0 , 1 } N and a corresponding class label l i ∈ K t .

In our problem setting, M C is a 3D instance segmentation model that is trained on C object categories, and, on test time, can recognize instances from these classes, in addition to instances from new classes not seen during training by classifying them as unknown . The detected unknown instances can be used by a human user to identify a set of n new classes not previously trained on, which can be incrementally added to the learner that updates itself to produce M C + n without explicitly retraining on previously seen classes. At this point in Task T t +1 , the known class object categories are K t +1 = K t ∪ { C +1 , .., C + n } . This process repeats throughout the lifespan of the instance segmentation model, continuously improving itself by incorporating new information from new classes until it reaches its maximum capacity of classes it can learn. In the rest of the paper, We assign the unknown class a label 0 .

4.2 Open-world scenarios

In order to simulate different realistic scenarios that might be encountered in an open-world, we propose three different ways of grouping classes under three tasks. These scenarios split scenes based on the inherent distribution (frequency-based) of object classes, the various classes encountered during the exploration of various indoor areas (region-based), and the randomness aspect of object classes in the open world.

10

4

10

6

Split A

Point frequency

10

4

10

6

Task1

Task3

10

4

10

6

Split A (Instance frequency-based): We introduce a split that leverages the inherent distribution of objects, with known classes being more prevalent than unknown categories. Task T 1 encompasses all the head classes as defined in the ScanNet200 benchmark [8, 27], while tasks T 2 and T 3 group the common and tail classes, respectively. This division allows us to effectively capture the varying frequency and significance of object categories within the dataset.

Split B (Region-based): In this split, our objective is to replicate the diverse class types encountered during indoor exploration.We argue that a perfect model for a robot moving indoors should segment both classes it knows and classes it hasn't seen before. Additionally, it should keep learning and getting better at segmenting new classes over time. This partition draws inspiration from the sequence of classes that a robot might encounter when navigating indoors. To achieve this, we group classes that are likely to be encountered initially when accessing an indoor space and share similarities in scenes. Initially, we assign each class to a specific scene where it predominantly occurs. Subsequently, we divide the classes into three distinct groups, corresponding to the three tasks.

Split C (Random sampling of classes): This third split introduces a different challenge inspired by the randomness aspect of the open-world, where tasks can exhibit random levels of class imbalance. To create this split, we randomly shuffled the classes and sampled without replacement, selecting 66 classes three times for each task.

4.3 Generating pseudo-labels for the unknown classes

Because of the wide range of classes in an open-world setting, the auto-labeler is used as an alternative to manual labeling. The former makes use of the existing target labels from the available ground truth classes ( known classes) to generate pseudo-labels for the unknown class in the process of training. In [18], the model is assumed to be class agnostic, where unknown objects are predicted as known with high confidence. As a result, the authors of the paper proposed to use the predictions with top-k confidence scores that do not intersect with the ground truth as pseudo-labels for the unknown class. In our study, we show that top-k pseudo-label selection can severely harm the performance of the model on the known and unknown classes. Hence, we propose a Confidence Thresholding ( CT ) based selection of pseudo-labels. We show that the performance on the known and the unknown classes increases by a large margin in terms of mean Average Precision (mAP).

The auto-labeler unit, depicted in Fig. 2, is used for unknown pseudo-labels generation. It takes a set of predicted binary masks B = { B i | i ∈ (1 , ..., n Q ) } , where n Q is the number of queries, B i = 1 ( M i > 0 . 5) is a mask from a single query, and M i = { m i,j ∈ [0 , 1] | j ∈ (1 , ..., N ) } is a heat map measuring the similarity between a query q j ∈ R D and the features of N voxels extracted from the high-resolution level in the backbone.

Moreover, each query q j encodes semantic information and can generate a class prediction P cls ( q j ) = { P cls ( c ; q j ) | c ∈ (0 , 1 , ..., |K t | ) } using a classification head (refer to Fig. 2). Subsequently, the objectness confidence score is assigned to predictions following Eq 1.

where s cls,j ∈ R is the max output probability from the classification head P cls ( q j ) , and 1 is the indicator function. After scoring the predictions, the auto-labeler returns m pseudo-labels ˜ Y = { ˜ y i = [ ˜ B i , 0 ] | i ∈ (1 , ..., m ) } with confidence above a threshold and has a low IoU with the known classes' target masks.

4.4 Query target assignment and contrastive clustering

Similar to [18], we utilize contrastive clustering to enhance the separation of classes within the query embedding space. To achieve this, we employ a set of query prototypes denoted as Q p = { q i ∈ R D | i ∈ (0 , 1 , .., |K t | ) } , where q 0 denotes the prototype of the class unknown . We apply a contrastive loss that encourages queries with similar classes to be attracted to their respective prototypes while pushing them away from those representing negative classes, as illustrated in Fig. 2. Since the queries are used to determine the class of the objects (see Fig. 2 inference block), the class prototypes are expected to hold general semantic knowledge of their corresponding classes.

Hungarian matching is performed in the Assign target to query module, depicted in Fig. 2, where the indices of prediction-target are used to assign a label to the queries used to generate the matched prediction. The labeled queries are then stored in a query store Q store , which represents a queue with a maximum capacity. This queue is employed to update the query prototypes Q p using an exponential moving average.

Hinge embedding loss is utilized according to Eq 2. This loss ensures that queries belonging to the same class denoted as q c , are pulled towards their corresponding class prototype q c , while being pushed away from other prototypes representing different classes.

̸

where ∆ is the margin of the contrastive clustering.

4.5 Reachability-based probability correction (PC)

In [23], an architecture that can deal with long-tail distribution and unknown class prediction for openworld object recognition was proposed, where unknown classes are assumed to be very different in color and texture from the known classes without prior on the unknown classes. However, we show in Fig. 6 that many unknown instances hold similar features to the known ones.

In our method, we relax the strict assumption of high dissimilarity of unknown and known classes and correct the predicted output probability following two characteristics of a feature

∆

Prototypes of the known

classes

∆

Contrastive clustering

margin

Correction region

Prototype of the

unknown class

unknown class features

X

X

X

X

X

X

X

X

X

X

from an unknown object: (1) it has to be far from the nearest known class, as features of the class unknown are expected to be pushed away from the prototypes of the known classes, after applying constructive clustering, and (2) the feature should correspond to an object that is not a known class. We show that applying this approach during inference boosts the performance of the model on the unknown class considerably by compensating for the weak pseudo-labels provided by the auto-labeler.

Our probability correction scheme is the following

where P cls is the probability from the classification head, and P corr is the correction probability. We base our intuition on the fact that unknown classes have high objectness scores, which makes them not too far from the prototypes of the known classes. To model this behavior we choose

where P corr ( o ; q j ) is the likelihood of the query to correspond to an object that is not known (either background or true unknown ). Since the query prototypes encode class-specific information we propose the following method to measure the objectness of a query given all prototypes from the known classes, where it assigns a high objectness probability if it is close to only a few known classes. This probability distribution defines the objectness of unknown objects around a certain boundary from the prototypes as follows.

Ground Truth

3D-OWIS - PC - CT

3D-OWIS

while P corr ( 0 ; o, q j ) is the probability of the query being an unknown object, which has a high value the further it is from the nearest prototype of the known classes.

Here σ is the sigmoid function, γ ( q j ) is the reachability of the query q j , q i is the prototype of the i th class, and a, b are the shift and scale of the sigmoid function that assure P corr ( 0 ; o, q j , γ ( q j ) = 0) = 0 . 05 and P corr ( 0 ; o, q j , γ ( q j ) = ∆ 2 ) = 0 . 95 , for a contrastive clustering margin ∆ .

We finally normalize the probabilities from the classification head of the known classes as follows

4.6 Alleviating catastrophic forgetting for incremental learning

Following the success of exemplar replay in avoiding catastrophic forgetting of the old classes during incremental learning for object detection [18, 11, 41], we adopt it for the task of incremental learning in 3D instance segmentation where we use exemplars from the classes of the previous task to fine-tune the model trained on the novel classes. In our setting, we use the same dataset for the three tasks and mask the classes of the previous task when training on the novel classes from the current task. As a result, the novel classes of the current task might be encountered again when replaying the exemplars from the previous task, as the same scenes are being used in fine-tuning.

5 Experiments

5.1 Open-world evaluation protocol

We use our proposed splits of classes which mimic the challenges that are mostly faced in the open-world to ensure a strict performance evaluation for 3D instance segmentation models.

Evaluation metrics. We adopt three common evaluation metrics, wilderness impact (WI) [9], absolute open set error (A-OSE) [26], and the recall of the unknown classes (U-Recall) [1, 24, 11]

to evaluate the performance of our model on the unknown classes and to provide a fair comparison with and without contributions. For the known classes, we use mean Average Precision (mAP). WI measures the impact of the unknown classes on the precision of the model at a specific confidence level. Ideally, WI is nil, i.e., there are no unknown objects predicted as known . For our evaluation, we report WI at 0.5 confidence. It can be computed as follows: WI = P K P K∪U -1 .

We also report A-OSE, which represents the count of unknown instances misclassified as one of the known classes, and the U-Recall at 0.5 IoU, which reflects the ability of the model to recover unknown objects.

5.2 Implementation details

We adapt Mask3D [29] for the task of open-world instance segmentation. We add an extra prediction output for the unknown class. In training, we assign an ignore label to the classes of the future and previous tasks, while we keep the labels of the previous task and assign an unknown class label to the classes of the future task during evaluation. For contrastive clustering, we use the indices obtained after matching the predictions with the target using Hungarian matching to assign a label to the queries and store them in the Query Store Q store . The store is then averaged per class and used to periodically update the prototypes every 10 iterations for the hinge loss computation. Finally, we use 40 exemplars per class on average for incremental learning. The classes from the current task are kept during class exemplar replay since we are using the same dataset for the three tasks.

5.3 Open-world results

Table 2 provides a comprehensive performance comparison between the Oracle, our implementation of [11] as 3D-OW-DETR, 3D-OWIS , and 3D-OWIS -PC -CT when excluding the Probability Correction ( PC ) and Confidence Thresholding ( CT ) components. Across all scenarios and tasks,

3D-OWIS -PC -CT consistently exhibits inferior performance in terms of mAP. Additionally, it demonstrates considerably lower U-Recall performance in splits A and B, with slightly higher performance in split C. Of particular note, our 3D-OWIS demonstrates remarkable proficiency in preserving knowledge of the previous classes after fine-tuning. This proficiency is attributed to better pseudo-label selection for the unknown classes. 3D-OWIS outperforms 3D-OWIS -PC -CT in most cases while minimizing the impact of the unknown classes on the known classes, as evidenced by lower WI and A-OSE scores and higher mAP.

Table 4 presents a comparison between our model, 3D-OWIS , and our implementation of two methods, GGN [32] and OLN [19]. For both models, we adapt Mask3D and train it with mask loss only for OLN. In the case of GGN, we train a Minkowski backbone to predict affinity maps and use Connected Components to generate class-agnostic proposals. These results underscore the effectiveness and potential of our approach in addressing the three proposed open-world challenges.

5.4 Incremental learning results

Our model's performance in incremental learning is evaluated based on its ability to preserve knowledge from previous classes. With the utilization of exemplar replay, the 3D-OWIS model demonstrates significant improvement on previous classes mAP. Table 2 presents the results, indicating that our model consistently outperforms the others in terms of mean Average Precision (mAP) for the previous classes in all cases.

5.5 Discussion and analysis

Ablation study. We show in Table 3 that 3DOWIS -PC -CT model performs poorly on the known classes because of the high number of low-quality pseudo-labels generated by Auto-labeler , which is also explained by the high value of Wilderness Impact and Absolute open set error . The U-Recall drops considerably when fine-tuning the 3D-OWIS -PC -CT , while the WI and A-OSE either decrease or increase with the mAP on the unknown . On the other hand, our model limits the training only to the best pseudo-labels, which maintain good performance on the known classes in all cases, before and after finetuning, and also achieve results on the unknown class comparable to the 3D-OWIS -PC -CT in most of the cases. Adding the proba-

bility correction module helps in improving the U-Recall while keeping the mAP of the known classes much above the 3D-OWIS -PC -CT . However, it results in an increase in WI and A-OSE because of the increase of false positives in the known classes.

tSNE analysis The tSNE plot shown in Fig. 6 illustrates the below-par performance of the 3D-OWIS -PC -CT in clustering the unknown classes, where most queries are still maintaining features representative of the known classes. This behavior is a result of the weak supervision of the unknown class, which shows the need for correcting the predictions, and explains the improvement in U-Recall when applying the probability correction with nil deterioration in the known classes mAP in most cases.

Qualitative analysis. Fig. 5 shows that 3D-OWIS is able to correctly identify background and unknown objects as unknown . Also note the second scene, where predictions are corrected from known to unknown without affecting the predictions of the known classes.

6 Limitations

Confidence Thresholding ( CT ) enhances the performance of the model on known classes; nonetheless, it diminishes the model's capacity to segment unknown classes, mainly due to its reliance on a smaller number of pseudo-labels during training. Additionally, the effectiveness of Probability Correction ( PC ) is contingent upon the inherent characteristics of the clusters within the known classes. In scenarios characterized by data imbalance, the performance of probability correction may deteriorate when applied to the undersampled classes.

7 Conclusion

In this paper, we address the challenge of 3D instance segmentation in open-world scenarios, which is a novel problem formulation. We propose an innovative approach that incorporates an unknown object identifier to detect objects not present in the training set. To facilitate evaluation and experimentation, we present three dataset splits of ScanNet200 based on different criteria for selecting unknown objects. Our experimental results demonstrate that our proposed unknown object identifier significantly improves the detection of unknown objects across various tasks and dataset splits. This work contributes to advancing the localization and segmentation of 3D objects in real-world environments and paves the way for more robust and adaptable vision systems.

·Known classes

·Unknown class

Acknowledgement The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement No. 2022-06725, and by the Berzelius resource, provided by Knut and Alice Wallenberg Foundation at the National Supercomputer Center.

References

A. Bansal, K. Sikka, G. Sharma, R. Chellappa, and A. Divakaran. Zero-shot object detection. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 384-400, 2018. 7

A. Bendale and T. Boult. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1893-1902, 2015. 3

J. Cen, P. Yun, S. Zhang, J. Cai, D. Luan, M. Tang, M. Liu, and M. Yu Wang. Open-world semantic segmentation for lidar point clouds. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVIII , pages 318-334. Springer, 2022. 3

S. Chen, J. Fang, Q. Zhang, W. Liu, and X. Wang. Hierarchical aggregation for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15467-15476, 2021. 2

B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1290-1299, 2022. 2

B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems , 34:17864-17875, 2021. 2

J. Chibane, F. Engelmann, T. Anh Tran, and G. Pons-Moll. Box2mask: Weakly supervised 3d semantic instance segmentation using bounding boxes. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI , pages 681-699. Springer, 2022. 2

A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richlyannotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5828-5839, 2017. 5

A. Dhamija, M. Gunther, J. Ventura, and T. Boult. The overlooked elephant of object detection: Open set. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1021-1030, 2020. 7

F. Engelmann, M. Bokeloh, A. Fathi, B. Leibe, and M. Nießner. 3d-mpa: Multi-proposal aggregation for 3d semantic instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9031-9040, 2020. 2

A. Gupta, S. Narayan, K. Joseph, S. Khan, F. S. Khan, and M. Shah. Ow-detr: Open-world detection transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9235-9244, 2022. 1, 3, 7, 8

H. Ha and S. Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. In 6th Annual Conference on Robot Learning , 2022. 3

L. Han, T. Zheng, L. Xu, and L. Fang. Occuseg: Occupancy-aware 3d instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2940-2949, 2020. 2

T. He, C. Shen, and A. Van Den Hengel. Dyco3d: Robust instance segmentation of 3d point clouds through dynamic convolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 354-363, 2021. 2

J. Hou, A. Dai, and M. Nießner. 3d-sis: 3d semantic instance segmentation of rgb-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4421-4430, 2019. 2

J. Hou, B. Graham, M. Nießner, and S. Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15587-15597, 2021. 2

L. Jiang, H. Zhao, S. Shi, S. Liu, C.-W. Fu, and J. Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and Pattern recognition , pages 4867-4876, 2020. 2

K. Joseph, S. Khan, F. S. Khan, and V. N. Balasubramanian. Towards open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5830-5840, 2021. 1, 3, 5, 7

D. Kim, T.-Y. Lin, A. Angelova, I. S. Kweon, and W. Kuo. Learning open-world object proposals without learning to classify. IEEE Robotics and Automation Letters , 7(2):5453-5460, 2022. 9, 10

J. Lahoud, B. Ghanem, M. Pollefeys, and M. R. Oswald. 3d instance segmentation via multi-task metric learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9256-9266, 2019. 2

Z. Liang, Z. Li, S. Xu, M. Tan, and K. Jia. Instance segmentation in 3d scenes using semantic superpoint tree networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2783-2792, 2021. 2

S.-H. Liu, S.-Y. Yu, S.-C. Wu, H.-T. Chen, and T.-L. Liu. Learning gaussian instance segmentation in point clouds. arXiv preprint arXiv:2007.09860 , 2020. 2

Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2537-2546, 2019. 6

C. Lu, R. Krishna, M. Bernstein, and L. Fei-Fei. Visual relationship detection with language priors. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14 , pages 852-869. Springer, 2016. 7

S. Ma, Y. Wang, J. Fan, Y. Wei, T. H. Li, H. Liu, and F. Lv. Cat: Localization and identification cascade detection transformer for open-world object detection. arXiv preprint arXiv:2301.01970 , 2023. 3

D. Miller, L. Nicholson, F. Dayoub, and N. Sünderhauf. Dropout sampling for robust object detection in open-set conditions. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 3243-3249. IEEE, 2018. 7

D. Rozenberszki, O. Litany, and A. Dai. Language-grounded indoor 3d semantic segmentation in the wild. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII , pages 125-141. Springer, 2022. 5

K. Saito, P. Hu, T. Darrell, and K. Saenko. Learning to detect every thing in an open world. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV , pages 268-284. Springer, 2022. 1

J. Schult, F. Engelmann, A. Hermans, O. Litany, S. Tang, and B. Leibe. Mask3D: Mask Transformer for 3D Semantic Instance Segmentation. In International Conference on Robotics and Automation (ICRA) , 2023. 2, 3, 8

J. Sun, C. Qing, J. Tan, and X. Xu. Superpoint transformer for 3d scene instance segmentation. arXiv preprint arXiv:2211.15766 , 2022. 2

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 2

W. Wang, M. Feiszli, H. Wang, J. Malik, and D. Tran. Open-world instance segmentation: Exploiting pseudo ground truth from learned pairwise affinity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4422-4432, 2022. 9, 10

W. Wang, M. Feiszli, H. Wang, and D. Tran. Unidentified video objects: A benchmark for dense, open-world segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10776-10785, 2021. 1

W. Wang, R. Yu, Q. Huang, and U. Neumann. Sgpn: Similarity group proposal network for 3d point cloud instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2569-2578, 2018. 2

S. Xie, J. Gu, D. Guo, C. R. Qi, L. Guibas, and O. Litany. Pointcontrast: Unsupervised pretraining for 3d point cloud understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 574-591. Springer, 2020. 2

B. Yang, J. Wang, R. Clark, Q. Hu, S. Wang, A. Markham, and N. Trigoni. Learning object bounding boxes for 3d instance segmentation on point clouds. Advances in neural information processing systems , 32, 2019. 2

L. Yi, W. Zhao, H. Wang, M. Sung, and L. J. Guibas. Gspn: Generative shape proposal network for 3d instance segmentation in point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3947-3956, 2019. 2

B. Zhang and P. Wonka. Point cloud instance segmentation using probabilistic embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8883-8892, 2021. 2

J. Zhang, R. Dong, and K. Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. arXiv preprint arXiv:2303.04748 , 2023. 3

X. Zhu, R. Zhang, B. He, Z. Zeng, S. Zhang, and P. Gao. Pointclip v2: Adapting clip for powerful 3d open-world learning. arXiv preprint arXiv:2211.11682 , 2022. 3

O. Zohar, K.-C. Wang, and S. Yeung. Prob: Probabilistic objectness for open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11444-11453, 2023. 3, 7

Appendix

A Scalability of 3D-OWIS

We show in Table 5 that 3D-OWIS can accommodate a large number of classes without a major size increase

B Additional details on Split B

We utilize the 20 scene types present in the ScanNet200 dataset to distribute the 200 classes over the three tasks. Initially, we establish a notion of similarity between two scene types by assessing the extent of their shared classes. This similarity is quantified through the intersection over the union ( IoU ) metric, which measures the ratio of common classes to the total count of unique classes across both scenes. By employing this metric, we identify scene types that exhibit a substantial IoU , indicating a higher degree of similarity. The similarity matrix, depicted in Fig. 7, showcases the relationships between the 20 scene types within the ScanNet200 dataset.

Subsequently, we employed three criteria to group the classes: ( i ) the likelihood of encountering them first when accessing an indoor area, ( ii ) their affiliation with similar scene types, and ( iii ) the proximity in the number of known classes across tasks. By taking these factors into consideration, we arrived at the split of scenes presented in Table 6.

0.8

Harage

Table 6: Frequently occurring scene when training during the three tasks in Split B. Scene types are grouped into tasks based on three criteria: ( i ) the likelihood of encountering the classes within the scene types when entering an indoor area, ( ii ) similarity of scene types containing the classes, and ( iii ) consistency in the overall number of classes within the scene types across all tasks. This grouping ensures a cohesive organization of scene types for effective evaluation of 3D instance segmentation models integrated with tasks such as robot navigation within indoor environments.

C Additional details on the experimentation

Training: We train the model on the entire ScanNet200 dataset for all tasks. In Task 1, objects belonging to the classes from Task 2 and Task 3 are masked, excluding them from the learning process. Moving to Task 2, we utilize the last saved checkpoint of the model from Task 1 as a starting point and mask the objects with labels that correspond to the current known classes of Task 1 and Task 3. This allows the model to focus solely on learning and distinguishing the specific objects associated with the current task. Finally, Task 3 builds upon the progress made in Task 2. We load the

latest checkpoint of the model from Task 2 and incorporate an exemplar replay. Similar to Task 2, the objects with labels belonging to the known classes in Task 1 and Task 2 are masked during training. This step further refines the model's understanding and discrimination abilities for the specific objects relevant to the current task.

Evaluation: To conduct the evaluation during a task, we assign the " unknown " label to the known classes from all the future tasks.

D Additional qualitative results

D.1 Unknown objects identification

The qualitative results depicted in Fig. 10, 12, 13, and 11 highlight the superior performance of our contribution in retrieving unknown objects. Across the majority of scenes, our model consistently corrects the mispredicted unknown classes while preserving the accuracy of known objects, thus demonstrating its robustness and effectiveness.

D.2 Learning novel classes

Fig. 8 and Fig. 9 illustrate the sequential process of learning novel classes after identifying unknown objects from the previous task. In Fig. 8, we demonstrate the effectiveness of our method in successfully retrieving unknown classes in all tasks. Additionally, in Fig. 9, we highlight the potential of exemplar replay in retaining knowledge of the old classes after learning the novel classes in Task 2 and Task 3.

Ground

Truth

3D-OWIS-PC-CT

3D-OWIS

Task 1

Task 2

Task 3

Ground

Truth

3D-OWIS

Task 1

Task 2

Task 3

Ground

Truth

3D-OWIS

Task 1

Task 2

Task 3

Ground Truth

3D-OWIS-PC-CT

3D-OWIS

Ground Truth

3D-OWIS-PC-CT

3D-OWIS

1

Ground Truth

3D-OWIS-PC-CT

3D-OWIS

Ground Truth

3D-OWIS-PC-CT

3D-OWIS

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the task of incremental object-discovery for 3D semantic instance segmentation. Unlike numerous concurrent works, it is not unsupervised but enables users (or oracle) to label objects that were identified as unknown in each iteration. The method is evaluated on ScanNet200, the paper proposes three new splits/tasks and measures three different metrics as defined in prior work.\n\nSTRENGTHS:\nThe proposed approach has significant practical significance as it enables training on a labelled corpora of data and provides the option to users to label the identified unknown objects and then refine the trained model once training labels are provided by a user.\n\nWEAKNESSES:\nThe manuscript is unclear on numerous occasions to the point that it is hard to understand the method (see questions below, these should be addressed in the rebuttal and incorporated in an updated version). In general, the paper might be clear for those that worked on this project (I assume for a significant amount of time) but the writing is challenging to understand by someone that reads about the project for the first time (unfortunately this makes it really difficult to write a meaningful review).\n\nQUESTIONS:\nThings that are not clear and should be improved:\n\n- Looking at the videos and experiments: There are two methods 3D-OWIS-PC-CT and 3D-OWIS - are both yours? Or is one the baseline that is not yours, and the other yours? Maybe you can clearly mark yours with “Ours”.\n\n- l.106-110: “the learned that updates itself” -- what does it mean? Is it correct that after one iteration of predictions by the model, the users labels the unknown classes, and then (in the next iteration) the model is trained again based in the existing and additionally provided user annotations? If this is the case, it is confusing to say “continuously improving itself (l.109)” since that somehow suggests that the model can improve without human intervention whereas here it seems to rely on human labels? I do not question the usefulness of user input after each iteration (i think it is very desirable), I simply do not understand what is happening in the end - the description should be improved.\n\n- Figure 3 / Sec 3.2 Split B - I do not understand the split - what does it mean “exploring indoor areas” / “accessing an indoor space”? Is there a robot walking around in a simulation (I guess not)?\n\n- l.132 - what is “the auto labeler”? The term is used here for the first time without explanation. The text then continues in l.136-143 explaining the drawbacks and proposed solutions. Since I don’t know what an auto-labeler is, these two paragraphs did not provide any information to me so I ignored them for writing this review. L. 144 “the auto labeller depicted in Fig.2” - it is not depicted in Fig2 - there is only a box with the name “auto labeled” - this does not provide any useful information to understand what is happening.\n\n- Eq. 2, what is a class prototype? Do we have a class for each semantic class i.e. max of 200 in ScanNet200, or does it refer to individual object instances? The text suggests that it refers to semantic classes. If that is the case, how do you separate different instances that have the same semantic class?\n\n- l.175 - what is 3D-OWIS? Is it the same as [20]? After reading l.172-177, they seem to be the same, but I could not find the name 3D-OWIS in [20]. Is 3D-OWIS maybe the name of the method proposed in this paper? The caption of  Fig.6 it seems to become clear that 3D-OWIS is the name of the method proposed in this paper. Similarly, in line 244 what is -PC-CT? What are the contributions? What does PC CT stand for? The last line of the caption in Tab.3 indicates that PC is probability corrections dn CT is confidence threshold? [I have to play detective only to understand what is the name of your method, why do you do this to me? Imagine R2 - they will directly reject the paper :( ]\n\n- Why are the Tasks 1-2-3 introduced? To my understanding, it seems to be a redefinition of the existing head-common-tail classes. Why confuse the reader and introduce an alias for sth that is already clearly defined?\n\nLIMITATIONS:\nNo / not applicable", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper proposes a pipeline to do 3D open-world instance segmentation. The authors provide a problem definition and introduced three different scenarios. Moreover, to overcome the possible problems that may lead to lower performance on known classes, the authors proposes different modules like probability correction and exemplar replay. The comprehensive experiments show their proposed methods' effectiveness.

STRENGTHS:
As claimed by the author, this paper is the first to investigate the 3D open-world instance segmentation task, and develop some well-motivated modules to improve the performance upon the simple baseline.

WEAKNESSES:
I'm not very familiar with the open-world related task and common practices both in 2D and 3D. Currently I think the writing of this paper is confusing and seems to be finished in a hurry. For example, the "3D-OWIS-PC-CT", I assume the "-PC-CT" is **without PC and CT**? And I couldn't find explanations about the "PC" and "CT".

In addition, I think authors can also provide the results of their pipeline equipped with other 3D instance segmentation as a comparison.

QUESTIONS:
As mentioned before I'm not an expert in this field, I'll ask more questions in later reviewing process. Currently I have one concern, since the network won't be re-trained in test and the network structure is fixed, does it mean the total class number (including the known classes and the unknown classes) will have an upper bound? And will the performance of this feature-cluster-based new class discovery become lower as the new class number increases?

LIMITATIONS:
No. The authors didn't address their limitations, and also the possible negative societal impact. I would suggest authors discuss some foundamental drawbacks of current open-world discovery settings.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04

comments: SUMMARY:
This paper introduces an open-world 3D indoor instance segmentation method, where an auto-labeling scheme is employed to produce pseudo-labels during training and induce separation to separate known and unknown category labels. Pseudo label quality are further improved subsequently.

STRENGTHS:
1. Open world 3d instance segmentation is an important problem and the author first addresses this problem.
2. The method designing is reasonable and complete.
3. The experiment is also sufficient.

WEAKNESSES:
1. Some designs are hard to understand. For example, for probability correction, why unknown object has to be far from the nearest known class? since we have no prior about category distributions. In experiments, you can also see that PC does not bring improvement to all metrics. This should be illustrated more clearly.
2. Although such open world task has not been studied in 3d tasks, it has been widely studied in 2d tasks. Since the core issue of this problem is actually the same, the author should implement some 2d open world methods in 3d instance segmentation and compare with them.

QUESTIONS:
Please refer to the weakness section.

LIMITATIONS:
The author does not discuss the limitation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new application of open-world object detection to the setting of 3D instance segmentation. In this setup, a 3D point-cloud segmentation model is required to label each point with instance information, whether this instance is part of training or not. The paper uses ScanNet and proposes a few ways to partition the dataset to use for evaluation in open-world. Finally, the paper adopts the framework in [16] as a baseline for this problem and proposes two tricks on top this framework: PC (probability correction) and CT (confidence thresholding).\n\nSTRENGTHS:\nThe task of open-world segmentation in 3D is important yet overlooked by previous work. As outlined by the paper, prior works mainly focus on 2D setups, either image or video. The use of ScanNet for evaluation seems appropriate and the splitting mechanism also seems reasonable, especially the Region-Based split has a very nice motivation from real-world application.\n\nWEAKNESSES:\nPost rebuttal comments:\n1. The author provided more insights towards the difference between 2D and 3D. I was hoping for more attributes unique to 3D. But having some insights as least improve the quality of the work. \n2. My second concern is addressed.\n\n\nOriginal review:\nAlthough there are many things to like about the proposal of the task, this work may suffer from a few important weaknesses:\n- Missing insights specifically to 3D. The major contribution of the paper is to adopt the recent new task of open-world localization to 3D setting. Unfortunately, the paper does not focus enough on aspects unique to 3D. Most components are the same as the 2D task and insights provided by the paper do not contain enough 3D-specific information. To justify the contribution, it is important for the paper to provides both intuitions and empirical proves on why it is worth studying 3D and how it is different than simple adoption of a 2D framework. In fact, the 3D-OWIS seems to be almost identical to [16] besides swapping the backbone predictor with a 3D backbone. The claimed contributions PC and CT are also not specific to 3D or reveals anything special about 3D. \n- Lack of comprehensive evaluation/ ablations. If author would prefer to claim more of their contributions on the dataset part, it is important to benchmark the task comprehensively. A suite of recent baselines in 2D should be evaluated. For example, recent works OLN [A] and GGN [B] both provide good class-agnostic proposals, and their 3D variants should be evaluated. In particular, GGN also uses a self-teaching schema. \n\nMinor:\n- The presentation of baseline 2D-OWIS-PC-CT seems really vague when first-time introduced. Better elaborate on what this is when it is discussed the first time. \n- If author prefers to claim contributions mainly to the task/ benchmark, NeurIPS D&B is more appropriate.\n\n[A] Kim et al., Learning Open-World Object Proposals without Learning to Classify, ICRA 2022 \n[B] Wang et al., Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity, CVPR 2022\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nLimitation is not sufficiently discussed. For example, when repurposing ScanNet200, what are some potential limitation? This dataset isn't designed specifically for this proposed task in open-world.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of 3D instance segmentation in open-world scenarios.  It starts with a formulation for this problem, including the definition and setup of known and unknown objects and different splits of categories for simulating different open-world cases. Accordingly, this work proposes a framework incorporating an unknown object identifier to detect objects not present in the training set and devises several strategies to enhance the separation of classes within the query embedding space, such as contrastive clustering and reachability-based probability correction. Experiments on the new benchmark validate the effectiveness of the proposed framework.\n\nSTRENGTHS:\n- This paper studies a new problem setting, open-world 3D instance segmentation, with a new benchmark and framework.\n- The basic idea is easy to follow and the presentation is overall clear.\n- The illustration figures are also clear such as Fig. 2-4.\n- The new benchmark considers different open-world cases when splitting different categories.\n- The corresponding modifications tailored to this new problem are reasonable, from contrastive clustering for queries to reachability-based probability correction and alleviating catastrophic forgetting for incremental learning.\n- Experiments show the effectiveness of the proposed framework. Different evaluation metrics also show different perspectives for comparing different methods.\n\nWEAKNESSES:\n- There is no preliminary section for the baseline framework, Mask3D, making the introduction of corresponding modifications tailored to this new setting a little confusing. In my opinion, it would be much better to introduce the baseline framework after setting up the benchmark after Sec. 3.2 and even open a new section for the \"approach\" part.\n\n- Some of the modifications are adapted from experiences in the 2D domain, and they are not revised to fit the 3D problem or tackle some specific challenges in 3D. Although some of them such as reachability-based probability correction have their own insight, it is still unclear whether the underlying reason for these challenges is related to the different data modality. It would be better to have deeper thinking from this aspect.\n\n- The evaluation benchmark involves comprehensive metrics for the new setup, but it lacks an intuitive connection or comparison between the new problem and other conventional ones, such as 2D open-world instance segmentation/detection and 3D close-set instance segmentation. Given this is a relatively new problem, it would be better to compare it with other well-explored problems to make it more friendly and acceptable for researchers from this community.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nThe author does not discuss the limitations and potential social impacts in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}