SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mohamed El Amine Boudjoghra<sup>1</sup>, Salwa K. Al Khatib<sup>1</sup>, Jean Lahoud<sup>1</sup>, Hisham Cholakkal<sup>1</sup>, Rao Muhammad Anwer<sup>1,2</sup>, Salman Khan<sup>1,3</sup>, Fahad Shahbaz Khan<sup>1,4</sup>
<sup>1</sup>Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI),
$^{2}$ Aalto University,  $^{3}$ Australian National University,  $^{4}$ Linköping University
{mohamed.boudjoghra, salwa.khatib, jean.lahoud,
hisham.cholakkal, rao.anwer, salman.khan, fahad.khan}@mbzuai.ac.ae
Existing 3D instance segmentation methods typically assume that all semantic classes to be segmented would be available during training and only seen categories are segmented at inference. We argue that such a closed-world assumption is restrictive and explore for the first time 3D indoor instance segmentation in an open-world setting, where the model is allowed to distinguish a set of known classes as well as identify an unknown object as unknown and then later incrementally learning the semantic category of the unknown when the corresponding category labels are available. To this end, we introduce an open-world 3D indoor instance segmentation method, where an auto-labeling scheme is employed to produce pseudo-labels during training and induce separation to separate known and unknown category labels. We further improve the pseudo-labels quality at inference by adjusting the unknown class probability based on the objectness score distribution. We also introduce carefully curated open-world splits leveraging realistic scenarios based on inherent object distribution, region-based indoor scene exploration and randomness aspect of open-world classes. Extensive experiments reveal the efficacy of the proposed contributions leading to promising open-world 3D instance segmentation performance. Code and splits are available at: https://github.com/aminebdj/3D-OWIS.
3D semantic instance segmentation aims at identifying objects in a given 3D scene, represented by a point cloud or mesh, by providing object instance-level categorization and semantic labels. The ability to segment objects in the 3D domain has numerous vision applications, including robotics, augmented reality, and autonomous driving. Following the developments in the sensors that acquire depth information, a variety of datasets has been presented in the literature which provides instance-level annotations. In view of the availability of large-scale 3D datasets and the advances in deep learning methods, various 3D instance segmentation methods have been proposed in recent years.
The dependence of 3D instance segmentation methods on available datasets has a major drawback: a fixed set of object labels (vocabulary) is learned. However, object classes in the real world are plentiful, and many unseen/unknown classes can be present at inference. Current methods that learn on a fixed set not only discard the unknown classes but also supervise them to be labeled as background. This prevents intelligent recognition systems from identifying unknown or novel objects that are not part of the background. Given the importance of identifying unknown objects, recent works have explored open-world learning setting for 2D object detection [18, 11, 28, 33]. In the open-world setting, a model is expected to identify unknown objects, and once new classes are labeled, the new set is desired to be incrementally learned without retraining [18]. While previous methods have been mostly suggested for open-world 2D object detection, it is yet to be explored
Figure 1: 3D instance segmentation in an open-world. During each iterative learning phase, the model detects unknown objects, and a human operator gradually assigns labels to some of them and incorporates them into the pre-existing knowledge base for further training.
in the 3D domain. The main challenge lies in understanding how objects appear in 3D in order to separate them from the background and other object categories.
3D instance segmentation in the open world, illustrated in Fig. 1, offers more flexibility, allowing the model to identify unknown objects and request annotations for these novel classes from an oracle for further training. However, this approach presents several challenges: (i) the lack of annotations for unknown classes, necessitating quality pseudo-labeling techniques; (ii) the similarities between predicted features of known and unknown classes, requiring separation techniques for improved prediction; and (iii) the need for a more reliable objectness scoring method to differentiate between good and bad predicted masks for 3D point clouds.
In this work, we investigate a novel problem setting, namely open-World indoor 3D Instance Segmentation, which aims at segmenting objects of unknown classes while incrementally adding new classes. We define real-world protocols and splits to test the ability of 3D instance segmentation methods to identify unknown objects. In the proposed setup, unknown object labels are also added incrementally to the set of known classes, akin to real-world incremental learning scenarios. We propose an unknown object identifier with a probability correction scheme that enables improved recognition of objects. To the best of our knowledge, we are the first to explore 3D instance segmentation in an open-world setting. The key contributions of our work are:
- We propose the first open-world 3D indoor instance segmentation method with a dedicated mechanism for accurate identification of 3D unknown objects. We employ an auto-labeling scheme to generate pseudo-labels during training and induce separation in the query embedding space to delineate known and unknown class labels. At inference, we further improve the quality of pseudo-labels by adjusting the probability of unknown classes based on the distribution of the objectness scores.
- We introduce carefully curated open-world splits, having known vs. unknown and then incremental learning over the span of 200 classes, for a rigorous evaluation of open-world 3D indoor segmentation. Our proposed splits leverage different realistic scenarios such as inherent distribution (frequency-based) of object classes, various class types encountered during the exploration of indoor areas (region-based), and the randomness aspect of object classes in the open-world. Extensive experiments reveal the merits of the proposed contributions towards bridging the performance gap between our method and oracle.
3D semantic instance segmentation: The segmentation of instances in 3D scenes has been approached from various angles. Grouping-based or clustering-based techniques use a bottom-up pipeline by learning an embedding in the latent space to help cluster the object points. [4, 13, 14, 17, 20, 21, 34, 38]. Proposal-based methods work in a top-down fashion, first detecting 3D bounding boxes, then segmenting the object region within the box [10, 15, 22, 36, 37]. Recently, spurred by related 2D work [5, 6], the transformer design [31] has also been applied for the purpose of segmenting 3D instances [29, 30]. Other methods present weakly-supervised alternatives to methods that use dense annotations in order to lower the cost of annotating 3D data [7, 16, 35]. While all these methods aim to improve the quality of 3D instance segmentation, they are trained on a known set of semantic labels. On the other hand, our proposed method aims at segmenting objects with both known and unknown class labels.
Figure 2: Proposed open-world 3D instance segmentation pipeline. From left to right: 3D instance segmentation model, where the point cloud goes through a 3D convolutional backbone. The extracted feature maps are used in the transformer decoder to refine some initial queries, which then pass through two MLPs to generate label and mask predictions. The Contrastive Clustering block takes the refined queries, the prediction masks, and labels to further process the queries by assigning a target or an unknown pseudo label in the Query Processing module, and then storing them in a Query Store to finally update the class prototypes, which are finally used for contrastive clustering. During inference, the queries are used to correct the probability of the predicted labels based on their reachability to the known class prototypes.
Open-world object recognition: Open-world object recognition was introduced in [2], where the Nearest Mean Classifier was extended for an open-world setting. In the direction of open-world object detection, many studies [41, 18, 11, 25] have been conducted in the past. In[18], pseudo-labels for the unknowns are generated to perform contrastive clustering during training for a better unknown-known classes separation, where an energy-based unknown class identifier was proposed to detect the unknown classes, based on the energy of the logits from the known classes. For incremental learning, they adopted exemplar replay to alleviate catastrophic forgetting of old classes. In the same task as [18], [11] used a transformer-based model and proposed another way of unknown pseudo-labels generation, by using a new method of objectness estimation, and introduced a foreground objectness branch that separates the background from the foreground. For the task of outdoor 3D point cloud semantic segmentation, [3] proposed a model that predicts old, novel, and unknown classes from three separate classification heads. The latter is trained on the labels of the known classes and pseudo-labels for old classes generated by the same model to alleviate catastrophic forgetting, while the unknown class is assigned the second-highest score for a better unknown class segmentation. Other methods proposed in [40, 12, 39], primarily focus on enhancing the generalizability of 3D models for novel classes by leveraging supervision from 2D Vision Language Models for object recognition and 3D semantic segmentation tasks. However, these approaches exhibit several limitations, including (i) The 3D model's performance becomes dependent on the 2D Vision Language model. (ii) The 3D geometric properties of unseen objects in the training data are neglected during the training process. (iii) There exists no avenue for enhancing the model's performance on novel classes in cases where new labels are introduced.(iv) The training process necessitates pairs of images and corresponding 3D scenes.
3 Closed-world 3D Instance Segmentation
We adopted the state-of-the-art 3D instance segmentation model Mask3D [29] as our baseline. The latter is a hybrid model that combines Convolutional Neural Networks (CNNs) with transformers to learn class-agnostic masks and labels for instance separation. The backbone of Mask3D is CNN-based and used to extract feature maps from multiple levels. Meanwhile, the decoder is transformer-based and used to refine  $n_{Q} \in \mathbb{N}$  instance queries  $Q = \{q_{j} \in \mathbb{R}^{D} \mid j \in (1, \dots, n_{Q})\}$ , using the extracted
feature maps. The learning scheme consists of a Cross-entropy loss for learning semantic class labels and binary cross-entropy loss for learning instance masks during training.
4 Open-World 3D Instance Segmentation
4.1 Problem formulation
We start by formulating the problem setting of open-world 3D instance segmentation. At a Task  $\mathcal{T}^t$ , there exists a set of known object categories  $\mathcal{K}^t = \{1,2,\dots,C\}$  and a set of unknown object categories  $\mathcal{U}^t = \{C + 1,\ldots \}$  that may exist on inference time. The training dataset  $\mathcal{D}^t = \{\mathbf{X}^t,\mathbf{Y}^t\}$  includes samples from the classes  $\mathcal{K}^t$ . The input set  $\mathbf{X}^t = \{\mathbf{P}_1,\dots,\mathbf{P}_M\}$  is made of  $M$  point clouds, where  $\mathbf{P}_i\in \mathbb{R}^{N\times 3}$  is a quantized point cloud of  $N$  voxels each carrying average RGB color of the points within. The corresponding labels are  $\mathbf{Y}^t = \{\mathbf{Y}_1,\dots,\mathbf{Y}_M\}$ , where  $\mathbf{Y}_i = \{\mathbf{y}_1,\dots,\mathbf{y}_k\}$  encodes  $k$  object instances. Each object instance  $\mathbf{y}_i = [\mathbf{B}_i,l_i]$  represents a binary mask  $\mathbf{B}_i\in \{0,1\} ^N$  and a corresponding class label  $l_{i}\in \mathcal{K}^{t}$ .
In our problem setting,  $\mathcal{M}_C$  is a 3D instance segmentation model that is trained on  $C$  object categories, and, on test time, can recognize instances from these classes, in addition to instances from new classes not seen during training by classifying them as unknown. The detected unknown instances can be used by a human user to identify a set of  $n$  new classes not previously trained on, which can be incrementally added to the learner that updates itself to produce  $\mathcal{M}_{C + n}$  without explicitly retraining on previously seen classes. At this point in Task  $\mathcal{T}^{t + 1}$ , the known class object categories are  $\mathcal{K}^{t + 1} = \mathcal{K}^t\cup \{C + 1,\dots,C + n\}$ . This process repeats throughout the lifespan of the instance segmentation model, continuously improving itself by incorporating new information from new classes until it reaches its maximum capacity of classes it can learn. In the rest of the paper, We assign the unknown class a label  $\mathbf{0}$ .
4.2 Open-world scenarios
In order to simulate different realistic scenarios that might be encountered in an open-world, we propose three different ways of grouping classes under three tasks. These scenarios split scenes based on the inherent distribution (frequency-based) of object classes, the various classes encountered during the exploration of various indoor areas (region-based), and the randomness aspect of object classes in the open world.
Table 1: The statistics of each split across the three tasks. The number of known classes per task is reported along with the count of instances (3D objects) in the training and validation set, we also show the number of non-empty scenes used during training and validation.
Split A	Split B	Split C
Task 1	Task 2	Task 3	Task 1	Task 2	Task 3	Task 1	Task 2	Task 3
Classes count	64	68	66	73	55	70	66	66	66
Train instances	24224	3791	1612	15327	8177	6123	13483	8239	7905
Validation instances	6539	1000	428	4177	2261	1529	3776	2102	2089
Train scenes	1201	924	627	1201	1002	895	1169	1089	1159
Validation scenes	312	242	165	312	264	236	307	273	300
Figure 3: Point-wise count for each class across the three tasks under the three open-world scenarios
Split A (Instance frequency-based): We introduce a split that leverages the inherent distribution of objects, with known classes being more prevalent than unknown categories. Task  $\mathcal{T}^1$  encompasses all the head classes as defined in the ScanNet200 benchmark [8, 27], while tasks  $\mathcal{T}^2$  and  $\mathcal{T}^3$  group the common and tail classes, respectively. This division allows us to effectively capture the varying frequency and significance of object categories within the dataset.
Split B (Region-based): In this split, our objective is to replicate the diverse class types encountered during indoor exploration. We argue that a perfect model for a robot moving indoors should segment both classes it knows and classes it hasn't seen before. Additionally, it should keep learning and getting better at segmenting new classes over time. This partition draws inspiration from the sequence of classes that a robot might encounter when navigating indoors. To achieve this, we group classes that are likely to be encountered initially when accessing an indoor space and share similarities in scenes. Initially, we assign each class to a specific scene where it predominantly occurs. Subsequently, we divide the classes into three distinct groups, corresponding to the three tasks.
Split C (Random sampling of classes): This third split introduces a different challenge inspired by the randomness aspect of the open-world, where tasks can exhibit random levels of class imbalance. To create this split, we randomly shuffled the classes and sampled without replacement, selecting 66 classes three times for each task.
4.3 Generating pseudo-labels for the unknown classes
Because of the wide range of classes in an open-world setting, the auto-labeler is used as an alternative to manual labeling. The former makes use of the existing target labels from the available ground truth classes (known classes) to generate pseudo-labels for the unknown class in the process of training. In [18], the model is assumed to be class agnostic, where unknown objects are predicted as known with high confidence. As a result, the authors of the paper proposed to use the predictions with top-k confidence scores that do not intersect with the ground truth as pseudo-labels for the unknown class. In our study, we show that top-k pseudo-label selection can severely harm the performance of the model on the known and unknown classes. Hence, we propose a Confidence Thresholding (CT) based selection of pseudo-labels. We show that the performance on the known and the unknown classes increases by a large margin in terms of mean Average Precision (mAP).
The auto-labeler unit, depicted in Fig. 2, is used for unknown pseudo-labels generation. It takes a set of predicted binary masks  $\mathbf{B} = \{\mathbf{B}_i \mid i \in (1, \dots, n_Q)\}$ , where  $n_Q$  is the number of queries,  $\mathbf{B}_i = \mathbb{1}(M_i > 0.5)$  is a mask from a single query, and  $M_i = \{m_{i,j} \in [0,1] \mid j \in (1, \dots, N)\}$  is a heat map measuring the similarity between a query  $q_j \in \mathbb{R}^D$  and the features of  $N$  voxels extracted from the high-resolution level in the backbone.
Moreover, each query  $q_{j}$  encodes semantic information and can generate a class prediction  $\mathbb{P}_{cls}(q_j) = \{\mathbb{P}_{cls}(c;q_j)\mid c\in (0,1,\dots,|\mathcal{K}^t |)\}$  using a classification head (refer to Fig. 2). Subsequently, the objectness confidence score is assigned to predictions following Eq 1.
$$
s _ {j} = s _ {c l s, j} \cdot \frac {M _ {j} \cdot \mathbb {1} (M _ {j} > 0 . 5) ^ {T}}{| \mathbb {1} (M _ {j} > 0 . 5) | _ {1}} \tag {1}
$$
where  $s_{cls,j} \in \mathbb{R}$  is the max output probability from the classification head  $\mathbb{P}_{cls}(q_j)$ , and  $\mathbb{1}$  is the indicator function. After scoring the predictions, the auto-labeler returns  $m$  pseudo-labels  $\tilde{\mathbf{Y}} = \{\tilde{\mathbf{y}}_i = [\tilde{\mathbf{B}}_i, \mathbf{0}] \mid i \in (1, \dots, m)\}$  with confidence above a threshold and has a low IoU with the known classes' target masks.
4.4 Query target assignment and contrastive clustering
Similar to [18], we utilize contrastive clustering to enhance the separation of classes within the query embedding space. To achieve this, we employ a set of query prototypes denoted as  $\mathcal{Q}_p = \{\mathbf{q}_i \in \mathbb{R}^D \mid i \in (0, 1,.., |\mathcal{K}^t|)\}$ , where  $\mathbf{q}_0$  denotes the prototype of the class unknown. We apply a contrastive loss that encourages queries with similar classes to be attracted to their respective prototypes while pushing them away from those representing negative classes, as illustrated in Fig. 2. Since the queries are used to determine the class of the objects (see Fig. 2 inference block), the class prototypes are expected to hold general semantic knowledge of their corresponding classes.
Hungarian matching is performed in the Assign target to query module, depicted in Fig. 2, where the indices of prediction-target are used to assign a label to the queries used to generate the matched prediction. The labeled queries are then stored in a query store  $\mathcal{Q}_{\text{store}}$ , which represents a queue with a maximum capacity. This queue is employed to update the query prototypes  $\mathcal{Q}_p$  using an exponential moving average.
Hinge embedding loss is utilized according to Eq 2. This loss ensures that queries belonging to the same class denoted as  $q_{c}$ , are pulled towards their corresponding class prototype  $\mathbf{q}_c$ , while being pushed away from other prototypes representing different classes.
$$
\mathcal {L} _ {\text {c o n t}} \left(q _ {c}\right) = \sum_ {i = 0} ^ {| \mathcal {K} ^ {t} |} \ell \left(q _ {c}, \mathbf {q} _ {i}\right) \tag {2}
$$
$$
\ell (q _ {c}, \mathbf {q} _ {i}) = \left\{ \begin{array}{l l} | | q _ {c} - \mathbf {q} _ {i} | | _ {2} & i = c \\ \max  (0, \Delta - | | q _ {c} - \mathbf {q} _ {i} | | _ {2}) & i \neq c \end{array} \right.
$$
where  $\Delta$  is the margin of the contrastive clustering.
4.5 Reachability-based probability correction (PC)
In [23], an architecture that can deal with long-tail distribution and unknown class prediction for open-world object recognition was proposed, where unknown classes are assumed to be very different in color and texture from the known classes without prior on the unknown classes. However, we show in Fig. 6 that many unknown instances hold similar features to the known ones.
In our method, we relax the strict assumption of high dissimilarity of unknown and known classes and correct the predicted output probability following two characteristics of a feature
Figure 4: Illustration of the region in the query embedding space where the class probability is corrected.
from an unknown object: (1) it has to be far from the nearest known class, as features of the class unknown are expected to be pushed away from the prototypes of the known classes, after applying constructive clustering, and (2) the feature should correspond to an object that is not a known class. We show that applying this approach during inference boosts the performance of the model on the unknown class considerably by compensating for the weak pseudo-labels provided by the auto-labeler.
Our probability correction scheme is the following
$$
\mathbb {P} (\mathbf {0}; q _ {j}) = \mathbb {P} _ {\text {c l s}} (\mathbf {0}; q _ {j}) \cup \mathbb {P} _ {\text {c o r r}} (\mathbf {0}; q _ {j}) \tag {3}
$$
where  $\mathbb{P}_{cls}$  is the probability from the classification head, and  $\mathbb{P}_{corr}$  is the correction probability. We base our intuition on the fact that unknown classes have high objectness scores, which makes them not too far from the prototypes of the known classes. To model this behavior we choose
$$
\mathbb {P} _ {c o r r} (\mathbf {0}; q _ {j}) = \mathbb {P} _ {c o r r} (\mathbf {0}; o, q _ {j}) \cdot \mathbb {P} _ {c o r r} (o; q _ {j})
$$
where  $\mathbb{P}_{corr}(o; q_j)$  is the likelihood of the query to correspond to an object that is not known (either background or true unknown). Since the query prototypes encode class-specific information we propose the following method to measure the objectness of a query given all prototypes from the known classes, where it assigns a high objectness probability if it is close to only a few known classes. This probability distribution defines the objectness of unknown objects around a certain boundary from the prototypes as follows.
$$
\mathbb {P} _ {c o r r} (o; q _ {j}) = 1 - \sum_ {k = 1} ^ {| \mathcal {K} ^ {t} |} \mathbb {P} _ {c l s} (k; q _ {j})
$$
3D-OWIS-PC-CT
3D-OWIS
Figure 5: Qualitative results for 3D instance segmentation results on some ScanNet200 validation scenes. Points highlighted in blue belong to unknown classes and those highlighted in green belong to known classes. We show the performance of our model in retrieving the unknown class objects compared to 3D-OWIS-PC-CT for the three scenes.
while  $\mathbb{P}_{corr}(\mathbf{0};o,q_j)$  is the probability of the query being an unknown object, which has a high value the further it is from the nearest prototype of the known classes.
$$
\mathbb {P} _ {\text {c o r r}} (\mathbf {0}; o, q _ {j}) = \sigma \left(\frac {\gamma (q _ {j}) - a}{b}\right); \quad \gamma (q _ {j}) = \min  _ {\mathbf {q} _ {i}} | | q _ {j} - \mathbf {q} _ {i} | | _ {2}
$$
Here  $\sigma$  is the sigmoid function,  $\gamma(q_j)$  is the reachability of the query  $q_j$ ,  $\mathbf{q}_i$  is the prototype of the  $i^{th}$  class, and  $a, b$  are the shift and scale of the sigmoid function that assure  $\mathbb{P}_{corr}(\mathbf{0}; o, q_j, \gamma(q_j) = 0) = 0.05$  and  $\mathbb{P}_{corr}(\mathbf{0}; o, q_j, \gamma(q_j) = \frac{\Delta}{2}) = 0.95$ , for a contrastive clustering margin  $\Delta$ .
We finally normalize the probabilities from the classification head of the known classes as follows