Title: Decompose Novel into Known: Part Concept Learning For 3D Novel Class Discovery
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3D point clouds

3D recognition

part-based representation

unsupervised class discovery
Abstract: In this work, we address 3D novel class discovery (NCD) that discovers novel classes from an unlabeled dataset by leveraging the knowledge of disjoint known classes. The key challenge of 3D NCD is that learned features by known class recognition are heavily biased and hinder generalization to novel classes. Since geometric parts are more generalizable across different classes, we propose to decompose novel into known parts, coined DNIK, to mitigate the above problems. DNIK learns a part concept bank encoding rich part geometric patterns from known classes so that novel 3D shapes can be represented as part concept compositions to facilitate cross-category generalization. Moreover, we formulate three constraints on part concepts to ensure diverse part concepts without collapsing. A part relation encoding module (PRE) is also developed to leverage part-wise spatial relations for better recognition. We construct three 3D NCD tasks for evaluation and extensive experiments show that our method achieves significantly superior results than SOTA baselines (+11.7%, +14.1%, and +16.3% improvements on average for three tasks, respectively). Code and data will be released.

Decomposing Novel into Known: Part Concept Learning For 3D Novel Class Discovery

Abstract

In this work, we address 3D novel class discovery (NCD) that discovers novel classes from an unlabeled dataset by leveraging the knowledge of disjoint known classes. The key challenge of 3D NCD is that learned features by known class recognition are heavily biased and hinder generalization to novel classes. Since geometric parts are more generalizable across different classes, we propose to d ecompose n ovel i nto k nown parts, coined DNIK , to mitigate the above problems. DNIK learns a part concept bank encoding rich part geometric patterns from known classes so that novel 3D shapes can be represented as part concept compositions to facilitate cross-category generalization. Moreover, we formulate three constraints on part concepts to ensure diverse part concepts without collapsing. A part relation encoding module (PRE) is also developed to leverage part-wise spatial relations for better recognition. We construct three 3D NCD tasks for evaluation and extensive experiments show that our method achieves significantly superior results than SOTA baselines ( +11 . 7% , +14 . 1% , and +16 . 3% improvements on average for three tasks, respectively). Code and data will be released.

1 Introduction

3D recognition is fundamental for 3D tasks and plays a critical role in many applications such as robotics and autonomous vehicles. Recently, deep learning-based 3D recognition [24, 25, 45] has achieved remarkable progress in this task. Unfortunately, most methods are limited to annotated classes in the training set (i.e., known classes), and cannot recognize novel classes like human beings, which limits the robustness of 3D recognition in the real world. Therefore, it is crucial and valuable to investigate the 3D novel class discovery task (NCD) so that 3D recognition can discover novel classes by leveraging the knowledge of known classes.

Though 2D NCD has been extensively investigated, a straightforward extension of 2D NCD methods [7, 19] to 3D NCD usually performs poorly due to the feature bias problem [32], i.e. , the learned features are heavily biased towards known classes and have difficulty in generalizing to novel classes. The main factors lie in two aspects. First, learned feature representations are usually dominated by known classes while being oblivious to novel classes. Second, 3D recognition usually focuses on shortcut features [8, 12] only capture a special global structure or local regions specific to known classes [21] as shown in Fig. 1.

In this work, we tackle the above problems based on the observation that humans can subconsciously decompose shapes into familiar part concepts, which along with their spatial relations provide effective cues for recognizing different novel shapes [13, 18]. Therefore part concepts can bridge the gaps between known shapes and those novel ones with a shared part space, thus alleviating the

PointNet++

PointNet

Table

Chair

….

Arms Legs Plane Back Position

Table

(a)

(b)

Importance

Sofa

feature bias problem in 2D NCD extensions. On the other hand, the compositions and spatial relations between parts make it easier to recognize novel shapes. For example, as shown in Fig. 1(b), a novel class, e.g., a chair, contains familiar concepts like plane, leg, and backrest observed in known classes (e.g., tables and sofas), therefore we can easily recognize the chair according to its part concepts and part spatial relations.

Along with this idea, we present a novel part concept-based framework coined DNIK ( d ecompose n ovel i nto k nown parts) for 3D NCD as shown in Fig. 2. Specifically, we first construct a learnable part concept bank that encodes rich geometric patterns from known classes. Then DNIK can project translation-invariant part-level features of an input shape into the spanning space of part concepts so that shapes from different categories can be represented as part-concept composition features sharing the common embedding space. Therefore learned features by known class recognition can be generalized to novel classes. We further leverage part-wise spatial relations of input shapes with a part relation encoding module (PRE) for discriminative part relation features. These two kinds of features are then combined for 3D shape recognition. To avoid the collapse of the part concept bank and encourage diverse part concepts across classes, three constraints on part concepts are also considered. Moreover, we construct three 3D NCD tasks for the evaluation of different scenarios and also extend our method to the generalized class discovery task (GCD) [35].

Our contributions can be summarized as follows: (i) the analysis of biased features and the necessity of part concepts in 3D NCD; (ii) a part concept-based framework leveraging both part concept features and part-wise relations for 3D NCD; (iii) the design of three necessary constraints on part concepts to facilitate effective part concept learning; (iv) various 3D NCD and GCD tasks for the evaluation of different scenarios. Extensive experiments show that our method significantly outperforms baseline methods on all 3D NCD and GCD tasks and can efficiently recognize novel shapes.

2 Related Work

Novel Class Discovery. As there is no existing work on 3D NCD, we mainly review related works on 2D NCD. Related works can be classified into two categories. The first category of methods [9, 14, 15] usually first train a classification network on the known dataset and then cluster the novel data according to the prediction or output features of the pre-trained network. However, learned features in this way are highly biased toward known classes, leading to performance bottlenecks in novel classes. By contrast, the other line of works [7, 10] explores both known and novel samples for NCD during the clustering phase. Therefore feature representation can be shared between known and novel classes and allow for better generalization to novel classes. AutoNovel [10] utilizes both known and novel samples for self-supervised representation learning and joint known class classification and NCD. Following works further bridge the gap between known and novel classes by mixing up known and novel samples [49], and introducing consistency between similar objects [48] and predictions from two different branches [44]. On the other hand, UNO [7] unifies the learning of both known and novel classes using a single cross-entropy loss based on optimal transports between pseudo-label assignments. Following this paradigm, the similarity relation of samples between known and novel classes is further explored by a spacing loss in the embedding space [16], compositional experts [41], and symmetric KL divergence [19]. Recently NCD has been extended to the more practical general

Feature

𝑍

!

Point

Feature

𝑍

Concept Activation Map

𝑆

Part Composite

Feature

𝑍

"

Feature Encoder

Position-invariant Part

Feature

𝑍

#

MLP

Position Relation

Encoder

MaxPool

LGA

Part Concept Bank

𝑃

…

𝑄

$

!

𝑄

%

𝑄

&

…

𝑍

#'

!

𝑍

#%

𝑍

#&

…

𝑍

"'

!

𝑍

"%

𝑍

"&

Position-aware Part

Feature

𝑍

(

Part Position

Feature

𝑍

)

…

𝑍

(*

𝑍

(&

…

𝑍

)*

𝑍

)%

𝑍

)&

FPS+KNN

Part Set

𝑄

…

𝑍

(%

Known

𝑋

Novel

𝑋

T

%

T

&

T

%

T

&

Known Classifier

ϕ

+

Novel Classifier

ϕ

,

category discovery task (GCD) [35], which assumes that unlabeled datasets contain both known and novel classes. [35] and [39] tackle this problem by using semi-supervised K-means and purely parameterized models respectively. In this work, we address 3D NCD and GCD by introducing part concepts and part-wise relations as the bridge between known classes and novel ones.

Part-based 3D Recognition. Shape parts are the basic building blocks for 3D shapes [22, 4] and play an important role in both human perception and various 3D recognition tasks. For example, Zhao et al. [46] represent repeatable shape parts as prototypes to improve few-shot 3D object detection. Chowdhury et al. [5] learn part information as a robust prior for incremental learning. Wei et al. [38] align part-based features to reduce the geometry shifts across different domains. Zhao et al. [47] propose a part codebook-based self-attention to leverage the generalized geometric information for better voxel-based feature extractors. Inspired by these works, we learn a part concept bank to map features of known and novel classes into a shared space.

3 Method

This work aims at 3D NCD that learns to group 3D shapes in the unlabeled set into several classes by leveraging the knowledge of known classes in a labeled dataset. Basically, 3D NCD assumes the availability of two datasets: an unlabeled dataset D u = { X u i } N u i =1 with 3D shapes sampled from novel classes C u and a labeled dataset D l = { ( X l i , y l i ) } N l i =1 , where X l i denotes a 3D shape and y l i ∈ C l is the corresponding categories label. The classes in the two datasets are disjoint, therefore we have C l ∩ C u = ∅ . However, shapes from D l and D u usually endow with some similarities. For example, both a chair and a desk have legs and plane surfaces. Following the literature [7, 10, 48], we assume the number of novel classes C u is known a prior.

3.1 Analysis on the Extension of 2D NCD Methods

As there is no existing work on 3D NCD, we instead extend a mainstream 2D NCD method UNO [7] to the 3D domain as the baseline. UNO learns a shared feature extractor f and two linear classifiers ϕ l and ϕ u with C l and C u output neurons. The classifiers ϕ l and ϕ u are optimized with a cross-entropy loss using labels of known classes and pseudo labels of novel classes generated by a clustering method ( i.e. , Sinkhorn-Knopp [6]). However, naive extensions of 2D NCD to the 3D domain work badly for two reasons. First, feature representation is usually dominated by known classes, while oblivious to novel classes. Second, shape-level features extracted from 3D networks confront serious

Unsupervised Novel Class

Novel Class

Known Class

Novel Class

Cosine Similarity

0.8

ACC(%)

0.6

0.4

0.2

50

100

150

200

250

Epoch

Shape-level

Part-level

0.8

0.5

0.2

Chair

Bench

Bathtub

Plant

shortcuts as illustrated in Fig. 1. Thus learned features are heavily biased toward the known classes and are hard to serve novel classes.

We conduct a toy experiment to verify our analysis. We chose table, sofa, stool as known classes and chair, bench, bathtub, plant as novel classes from ModelNet [40]. The first two novel classes have similar shape parts with the known classes, e.g., legs, while the last two do not. To reveal the wane and wax between known and novel classes during optimization, we investigated performance curves of UNO [7] in Fig. 3 Left. In the first 60 epochs, training 3D recognition with known class supervision can help novel class discovery. Therefore there exist some sharing feature representations between two different sets of classes and UNO can achieve better accuracy on novel classes than unsupervised clustering ( i.e. , w/o the cross-entropy loss). However, with the training proceeding, the performance in novel classes starts to decrease and can even be worse than unsupervised clustering, indicating that learned features are gradually biased to known classes and lose generalizability [32].

Fig. 3 Right shows the average cosine similarity between the shape-level or part-level concepts of known classes and the shape or part features of novel classes learned from the trained UNO method [7]. Shape features are direct outputs of the feature extractor while part features denote pooling features of overlapped parts split from shapes. We then define the shape-level concepts by averaging the shape features of each known class and part-level concepts by clustering the part features of known classes using K-means. For shape-level concepts, the cosine similarities of chairs and benches are low and comparable to those of bathtubs and plants, which indicates that shape features cannot be shared between known and novel classes. However, part-level concepts can provide higher cosine similarities and reflect similar properties between known and novel classes better.

3.2 Part Concepts for Cross-category Generalization

In this work, we approach 3D NCD with a learnable part concept bank shared across different classes. The overall framework is shown in Fig 2. The basic observation is that geometric patterns and shape parts, i.e. , part concepts, are more likely to occur in different categories of shapes as analyzed in Sec. 3.1. We first learn part concepts from shapes of known classes in order to deliver more generalizable feature representation and reduce biases when facing novel classes. Then we leverage the relative positions of different parts to further enhance the recognization of novel classes.

Learning Part-level Features. As we intend to learn a set of part concepts representing generalizable shape parts, we first encode part-level features inspired by the paradigm of PointMAE [23]. Specifically, we extract point-wise features Z ∈ R L × D for each point in the input point cloud X ∈ R L × 3 with an off-the-shelf 3D backbone, e.g., PointNet [24], where L denotes the number of 3D points and D is the dimension of point cloud features. Then we use farthest point sampling (FPS) to sample N q points as the centroid seeds and group K neighboring points as overlapped parts Q = { Q i } N q i =1 . Part set Q i = { z i 1 , · · · , z iK } ∈ R K × D includes features at its centroid z i 1 followed by point features of its K -1 nearest neighbors { z i 2 , · · · , z iK } . Intuitively, we can obtain part-level features by pooling point features within each part Q i , but part features obtained in this way are dependent on the part location and may hinder the learning of shared part concepts at different positions. To cope with this problem, we propose a local geometric aggregation (LGA) module following [42, 26] to decouple the spatial location and aggregate part-level features Z l ∈ R N q × D by

where σ is the scalar standard deviation across channels, and ψ denotes a multi-layer perception (MLP) with channels ( D × D 2 × D ) to encode the local geometry features.

Part Concept learning. After obtaining part-level features Z l , we use them to learn shared part concepts among categories. First, we construct a learnable part concept bank denoted with P = { P m } M m =1 , where part concept P m ∈ R 1 × D encodes features of different shape parts and can represent a plane, corners, legs as illustrated on the top of Fig. 2. In our implementation, part concepts are initialized with a Gaussian distribution N ( 0 , I ) . We project part-level features Z l into the spanning space of part concepts P according to their respective similarities. Then part-level features Z l are approximated by part composition features Z p ∈ R N q × D as follows:

where S ∈ R N q × M measures the similarity between part features Z l and the part concept bank P . The concept activation map S is defined as:

where W z , W p ∈ R D × D p are learnable weight matrixes for feature dimension reduction, h ( · ) is a softmax function to produce normalized probabilities along each row, and d ( · , · ) is a distance function that compares part features and part concepts. To ensure that the geometric knowledge of each part feature can be learned by a particular part concept, distance d ( · , · ) should produce significant similarity differences between part features and different part concepts. A common choice of d ( · , · ) is to use dot product similarity [38] or cos similarity [11] defined in the Euclidean space. However, as shown in Fig. 4(a), we found that the Euclidean distance functions lead to uniform activations of part features on different part concepts. Different from the Euclidean space where the distance grows linearly, the distance between part features and concepts grows exponentially in the hyperbolic space [17, 30, 28], which can produce sharp concept activation maps (see Fig. 4(a)). Therefore, we project features into the hyperbolic space using Poincaré sphere model following [17] and the distance function between Z l and P in Poincaré manifolds is given by:

Though learned part concepts can cover most shape parts in novel classes, there still exist some novel part concepts that are not contained in known classes. In this case, the concept activation map between a part and part concepts P is likely to be uniform and results in ambiguous composite features. In our method, we further concatenate the local part features Z l with the part composite features Z p to confront this problem.

Part Relation Encoder. As part composite features Z p only consider position-independent compositions of part concepts and neglect mutual position relations between different parts that usually deliver shape structures as shown in Fig. 1(b). To this end, we propose a novel Position Relation Module that adopts a shallow Transformer Encoder [45] with two self-attention layers, denoted as TransEnc, to learn mutual relations between part concepts as relation feature Z r . Since the position information has been decoupled from Z l , we regain position-aware part features Z g = { Z g i , ..., Z g Nq } ∈ R N q × D from Q using max-pooling and encode Z r ∈ R N q × D as follows:

Afterward, we concatenate the position relation feature Z r with the point composite features Z p and feed the result into an MLP with 2 D × D and a max-pooling layer to produce category-aware features Z c . We can classify the input shape X by feeding Z c to a linear layer predicting the class-wise probabilities.

3.3 Constraining Part Concepts

Though part concepts are proposed to represent different parts, they can easily collapse to the same values. The main reason is that the similarity between randomly initialized part concepts and part

Euclidean

Poincar ́ 𝑒

(a)

(b)

(c)

𝑆

!

"

𝑆

!

𝑝( "

𝑆

!

, 𝑡

"

)

𝑝( "

𝑆

!

, 𝑡

"

)

𝐿

!"

𝐿

!"

Class 2

Class 1

Before

Before

After

After

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

D

2

2

features is similar during the initial training stage. Consequently, part composite features based on the concept activation map would gradually be identical and part concepts updated with similarly back-propagated gradients from them converge to similar features. To ensure the representation ability and generalization of the part concept bank, we propose three constraints to train part concepts.

Self-distillation Loss. To guarantee that each part's geometric knowledge can be learned by specific part concepts, the concept activation distribution of each part should be sharp. To achieve this, we propose a self-distillation loss inspired by [3]. We enforce each part concept activation S of the shape parts Q to be consistent with a sharper one of the corresponding part concept activation ˆ S from the augmented shape parts ˆ Q (see Fig 4 (b)) as follows:

where p ( S i , τ s ) = exp( s i /τ s ) ∑ Nq j =1 exp( s j i /τ s ) is a sharp concept activation distribution of Q i controlled by temperature parameter τ s (set 0.1 empirically) and stop ( · ) means the gradients is not back-propagated. Loss L sd enhances the sharpness and consistency of the concept activation under different data augmentations for both the labeled and unlabeled datasets.

Supervised Contrastive Loss. During training, we found that even a few part concepts can support the recognition of known classes as shown in the red boxes in Fig. 4(c). In this case, the part concept bank lacks the incentive to learn more part concepts from known shapes, which reduces the information it can share with novel classes. Thus, for the labeled dataset D l , we employ a supervised contrastive loss L sc to increase the inter-class discrepancy of their concept activation maps. As shown in the green boxes in Fig. 4(c), through L sc , the part concept bank can learn additional parts ( i.e. , leg and pedestal) from known shapes to increase their inter-class discrepancy. Given a mini-batch of B shapes { X b } B b =1 and their augmentation { ˆ X b } B b =1 , we first sum their activation maps S B ∈ R 2 B × M × N q into shape concept activation T B ∈ R 2 B × M along the part dimension, where T i summarizes all part concepts in a shape. Then the supervised contrastive loss is defined on 2 B shapes as follows:

where ρ = {{ T + j ∈ T B : y j = y i } ∪ ˆ T i } is the set that contains the shape concept activation ˆ T i of the augmented sample and other samples belonging to the same class in the mini-batch, sim ( ., . ) calculates the cosine similarity, and temperature τ c is set 0.1 empirically.

Diversity Loss. To capture different patterns in 3D shapes, part concepts should be diverse. We can encourage this by minimizing the similarity between different part concepts with the following loss:

where δ is the cosine similarity threshold between a code and the other ones. In our experiments, we set δ = 0 . 1 .

3.4 The Training Losses

Following the literature [48, 10, 7], we construct two linear classifiers ϕ l and ϕ u . We employ a standard cross-entropy loss L ce to optimize the known class classifier ϕ l and a self-classification loss L self for training novel class classifier ϕ u . Loss L self minimizes the cross entropy between novel shapes and their augmentation with the assumption that the distribution of each novel class is uniform and is formulated following [2]. In summary, the overall training objective loss function in our model is:

where loss L sd in Eq.(6), L sc in Eq.(7) and L cd in Eq.(8) enforce basic assumptions of part concepts. Weight term λ ∗ balances different losses and is set to ( 0 . 1 , 0 . 1 , 0 . 1 ). During inference, part composite features of a test sample Z c are passed through both classifiers to obtain the corresponding logits, then the two logits are concatenated and fed into a softmax layer to obtain the class-wise probability.

4 Experiments

4.1 Evaluation Datasets for 3D NCD

We construct three NCD tasks to evaluate the performance of 3D NCD based on four different kinds of 3D datasets covering CAD shapes ( i.e. , ModelNet40 [40] and ShapeNetCore [4]), and indoor objects constructed using 3D scans and multi-view images ( i.e. , ScanObjectNN [33] and Co3D [27]).

Random Split 3D NCD focuses on evaluating the impact of a different number of known and novel classes on the 3D NCD following [7]. The task is denoted with DATASET-K-U , where K is the number of known classes and the other U classes as novel ones.

Similarity Split 3D NCD. The semantic similarity between known and novel classes can greatly affect the performance of NCD. A highly semantically similar label set can provide more useful information for discovering novel classes. Based on the transfer leakage metric in [20], we split novel classes as three sets (i.e., High, Medium, and Low ) according to the descending similarity w.r.t known classes.

Cross Domain 3D NCD assesses the performance of 3D NCD on cross-domain shape recognition and reflects the real-world scenarios where novel shapes may come from different domains. Therefore, we select known and novel classes with semantic similarity from different datasets respectively. The corresponding tasks are denoted with (known class dataset → novel class dataset).

4.2 Comparison with SOTA methods

To our best knowledge, there is no previous attempt at the 3D NCD problem. So we extend different kinds of SOTA 2D NCD methods to 3D NCD, including DTC [9], AutoNovel [10], NCL [48], UNO [7], and IIC [19]. In addition, training a novel class classifier with pseudo labels generated by K-means can achieve good results and we also compare with it, denoting as Kmeans+ . We implement the above baselines by substituting the image-based backbone with a 3D one ( i.e. , PointNet) for 3D NCD.

Results on the Random Split 3D NCD Task are shown in Tab. 1. For the known class recognition, all compared methods perform well, but the performance on the novel classes diverges a lot. When the number of known classes is large, the network can inherit more knowledge for the novel class recognition. The NCL and IIC can achieve better results on most datasets since they force the separation between known and novel classes while learning similarity relationships within the novel classes, so the learned features are more discriminative for the novel classes. Our method can gain consistent improvements on novel classes of all tasks ( +11 . 7% on average) while attaining the best know-class accuracy for most tasks ( 5 / 6 ). This demonstrates that part concepts can effectively bridge the gaps between known and novel shapes thus mitigating the feature bias. When the number of known classes is small, the performance of compared methods drops significantly. By contrast, our method decreases much less and still achieves the best results ( +12 . 5% increases on average). We

conjecture that the part concept bank can learn more local features for sharing than the other methods using global features.

Results on the Similarity Split 3D NCD Task are presented in Tab. 2. NCL, UNO, IIC, and Kmeans+ can achieve better results on novel classes with high similarities. However, the performance of these methods drops significantly ( -13 . 5% on average) with the decrease in semantic similarity. The method that achieves the best results on High -similarity novel classes cannot obtain the best on Medium and Low ones simultaneously. This indicates that the learned knowledge of these methods is biased towards known classes with highly similar geometry while lacking generalizability to other novel ones. By contrast, our method achieves large improvements for all similarities ( +14 . 3% , +11 . 8% , +17 . 5% on average) while reducing the gap between different similarity tasks. This is because part concepts can show up in different shapes, e.g., the back of a sofa and a chair in Fig. 1 and align all shapes into the part space, allowing for generalization even for low-similarity novel classes.

Results on the Cross-Domain 3D NCD Task are reported in Tab. 3. We can observe that compared methods lack robustness on cross-domain shapes and perform poorly. Our method performs well among different datasets and outperforms existing methods by a large margin ( +16 . 3% on average). We believe the good performance is owing to the fact that part concepts are robust to data domains and can be generalized to different kinds of data.

4.3 Ablation Study

In this part, we conduct all ablations on the ModelNet similarity split 3D NCD. Readers can find more experiments in the supplementary.

Model components. In Tab. 4② , adding the part concept bank without the LGA only achieves a very small improvement compared to the baseline, which indicates part concepts without positional invariance are hard to fit novel classes. In ③ and ④ , LGA can ensure better generalization over novel classes, and the part concept bank can align the part features from known and novel classes into the same space. Therefore they can collaborate to boost the performance for a large margin ( 5 . 8% and 3 . 9% ). In ⑤ , using the Poincaré metric can yield much sharper concept activations and make different part concepts more discriminative, therefore raising the performance ( +3 . 7% and +8 . 5% ). In ⑥ , self-distillation loss L sd encourages each part concept to learn specific patterns to avoid collapses and contributes to an average 6 . 25% gain in accuracy. In ⑦ and ⑧ , loss L sc and L cd force the network to

LGA

PCB

Poincaré

L

sd

L

sc

L

cd

PRE

High

Low

①

48.8

37.9

②

✓

49.9

40.3

③

✓

52.6

44.8

④

✓

✓

58.4

48.7

⑤

✓

✓

✓

62.1

57.2

⑥

✓

✓

✓

✓

67.9

63.9

⑦

✓

✓

✓

✓

✓

70.1

64.8

⑧

✓

✓

✓

✓

✓

✓

71.3

65.1

⑨

✓

✓

✓

✓

✓

✓

✓

73.2

66.4

(a) #part concept

𝑁

!

(b) #points in each part

𝐾

(c) #split part

𝑁

"

C(%)

75

ClusterACC(%)

.060

75

Cluster ACC(%)

75

ClusterACC

706560

0660

1

3

5

78

8

16

32

64

128

512

256

128

64

32

NCL

IIC

Ours

learn more diverse part concepts, which is more helpful for high-similarity novel classes ( 3 . 4% vs 1 . 2% ), since their concept activation maps are more similar ( e.g. , chair and stool). By comparing ⑤ and ⑧ , we can see the three proposed constraints ( L sd , L sc , and L cd ) are essential to the final good performance and strengthen the results by +8 . 8% and +7 . 9% . In ⑨ , the PRE can further incorporate discriminative position relation features to enhance the performance ( +1 . 6% on average).

Hyperparameters. In Fig. 5(a), we conduct experiments on different numbers of part concepts, i.e. , M . Considering part concepts should increase with known classes, we set M = N c × C l , When N c ≤ 3 , the results are poor, which indicates that few part concepts cannot adequately encode all part patterns in known classes. However, the performance also drops when N c > 6 . This may be because too many concepts are harder to be optimized.

The number of points contained in each overlapped part determines the part size, therefore affecting the generalization and discrimination abilities. The discrimination ability of parts decreases when the size is reduced and the generalization ability improves. Specifically, small parts are hard to contain meaningful local geometric patterns, while large parts may contain special structures that are biased towards known classes as shown in Fig. 5(b). Our approach achieves the best results when K = 64 .

Since the part set of a shape is split by FPS, few parts lead to large variations in the part sets of similar shapes, resulting in inconsistent part composite features. In Fig. 5(c), the result will decrease continuously when N q < 128 . Our method achieves the best performance when N q = 256 .

Visualizations of the Feature Embedding with t-SNE [34] are shown in Fig. 6. For the NCL [48] and IIC [19], the features of different novel classes are mixed up in the embedding space, while ours can push away embedding regions of different novel classes to a greater extent.

Part Concept Visualization. In order to reveal what part concepts can learn, we show concept activation maps between some part concepts and part features in Fig. 7. We can see that part concepts can activate similar semantic parts of both known and novel shapes, such as legs, arms, bottles, backrests, planes, and screens. This confirms that part concepts can effectively generalize knowledge learned from known classes to novel classes and represent novel classes as the composition of part concepts. For example, a chair can be represented as {legs, arms, a backrest, and a plane}. We notice that part concepts can be robustly discovered and activated even if geometric structures are slightly different, e.g., arms of chairs, sofas, and benches.

4.4 Extension for Generalized Class Discovery

DNIK can also be extended for generalized class discovery (GCD) [35], for which the unlabeled dataset D u contains 3D shapes from both known and novel classes. In order to adapt DNIK for this task, we concatenate the known class classifier ϕ l and the novel class classifier ϕ u into a unified

Table

Bench

Chair

Piano

Stool

Bench

Chair

Sofa

Bench

Chair

Sofa

Screen

Laptop

FlowerPot

Vase

Known

Known

Known

Known

Novel

Novel

Novel

Known

Novel

Novel

Table

Bench

Chair

Stool

Novel

Known

Leg

Arm

Bottle

Backrest

Plane

Screen

Importance

classifier. All samples from both the labeled dataset and unlabeled dataset go through the unified classifier to obtain the class-wise probability. Both L ce and L self are now used to optimize the unified classifier rather than the novel class classifier of the NCD setting.

We extend two 2D GCD methods to 3D GCD, including GCD [35], SimGCD [39],and also extend AutoNovel [10] and UNO [7] for GCD tasks (denoted as AutoNovel+ and UNO+) by concatenating logits of known classes and novel classes following SimGCD [39]. Results are shown in Tab. 5 and Tab. 6. It can be seen that all compared methods show decreased results compared to the NCD tasks in Tab. 1 and Tab. 2. This indicates that samples of known classes in the unlabeled dataset D u introduce noise into the training and confuse the classifier. GCD [35] can achieve better novel class accuracy (+5.6%) than other compared methods because it uses semi-supervised Kmeans to cluster overall samples, which can produce more reliable pseudo labels. Our method can still achieve superior performance (+12.1%) on novel classes of all tasks while attaining comparable know-class accuracy for most tasks. This indicates that the idea of leveraging part compositions for novel class discovery can perform well in both the NCD setting and the GCD setting.

5 Conclusion

In this work, we present a novel part concept-based framework DNIK for 3D NCD. DNIK leverages part concepts and part-wise relations that are widely shared by both known classes and novel ones. Therefore part concepts and part-wise relations learned from known classes can reinforce the recognition of novel shapes. Comprehensive experiments demonstrate the method overwhelms all baselines consistently and significantly on all metrics. We hope our work can inspire more research on find-grained concept learning and 3D NCD.

Limitations. Part concepts in the work are not capable of learning multi-scale geometric parts, thus restricting the representation ability and resulting in sensitive scale parameters in Fig. 5 (b).

References

Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9902-9912, 2022.

Elad Amrani, Leonid Karlinsky, and Alex Bronstein. Self-supervised classification network. arXiv preprint arXiv:2103.10994 , 2021.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 , 2015.

Townim Chowdhury, Ali Cheraghian, Sameera Ramasinghe, Sahar Ahmadi, Morteza Saberi, and Shafin Rahman. Few-shot class-incremental learning for 3d point cloud objects. In ECCV , 2022.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26, 2013.

Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9284-9292, 2021.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8401-8409, 2019.

Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(10):6767-6781, 2021.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. CoRR , abs/1911.05722, 2019.

Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems , 33: 9995-10006, 2020.

Donald D Hoffman and Whitman A Richards. Parts of recognition. Cognition , 18(1-3):65-96, 1984.

Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classification without multi-class labels. In International Conference on Learning Representations , 2019.

KJ Joseph, Sujoy Paul, Gaurav Aggarwal, Soma Biswas, Piyush Rai, Kai Han, and Vineeth N Balasubramanian. Spacing loss for discovering novel categories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3761-3766, 2022.

Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6418-6428, 2020.

Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science , 350(6266):1332-1338, 2015.

Wenbin Li, Zhichen Fan, Jing Huo, and Yang Gao. Modeling inter-class and intra-class constraints in novel class discovery. arXiv preprint arXiv:2210.03591 , 2022.

Ziyun Li, Jona Otholt, Ben Dai, Di Hu, Christoph Meinel, and Haojin Yang. A closer look at novel class discovery from the labeled set. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications , 2022.

Weiquan Liu, Minghao Liu, Shijun Zheng, and Cheng Wang. Interpreting hidden semantics in the intermediate layers of 3d point cloud classification neural network. arXiv preprint arXiv:2303.06652 , 2023.

Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 909-918, 2019.

Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II , pages 604-621. Springer, 2022.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 652-660, 2017.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10901-10911, 2021.

Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. Representation tradeoffs for hyperbolic embeddings. In International conference on machine learning , pages 4460-4469. PMLR, 2018.

Mert Bulent Sariyildiz, Yannis Kalantidis, Diane Larlus, and Karteek Alahari. Concept generalization in visual representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9629-9639, 2021.

Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. In Graph Drawing: 19th International Symposium, GD 2011, Eindhoven, The Netherlands, September 21-23, 2011, Revised Selected Papers 19 , pages 355-366. Springer, 2012.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In Advances in Neural Information Processing Systems 33, 2020, virtual , 2020.

Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1588-1597, 2019.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7492-7501, 2022.

Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9782-9792, 2021.

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog) , 38(5):1-12, 2019.

Xin Wei, Xiang Gu, and Jian Sun. Learning generalizable part-based feature representation for 3d point clouds. Advances in Neural Information Processing Systems , 35:29305-29318, 2022.

Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category discovery: A baseline study. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 16590-16600, October 2023.

Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1912-1920, 2015.

Muli Yang, Yuehua Zhu, Jiaping Yu, Aming Wu, and Cheng Deng. Divide and conquer: Compositional experts for generalized novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14268-14277, 2022.

Xincheng Yang, Mingze Jin, Weiji He, and Qian Chen. Pointcat: Cross-attention transformer for point cloud. arXiv preprint arXiv:2304.03012 , 2023.

Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9150-9161, 2023.

Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. Advances in Neural Information Processing Systems , 34: 22982-22994, 2021.

Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1625916268, 2021.

Shizhen Zhao and Xiaojuan Qi. Prototypical votenet for few-shot 3d point cloud object detection. In Advances in Neural Information Processing Systems , 2022.

Tianchen Zhao, Niansong Zhang, Xuefei Ning, He Wang, Li Yi, and Yu Wang. Codedvtr: Codebook-based sparse voxel transformer with geometric guidance. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 1425-1434. IEEE, 2022.

Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10867-10875, 2021.

Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9462-9470, 2021.

A In-depth Analysis of the Method

Feature Visualization of Novel Classes. We show the feature distribution of novel classes learned by all compared methods on the high similarity task of ModelNet in Fig. 8. DTC [9] mingles features of different novel classes together and impedes novel class recognition. The feature distributions of AutoNovel [10], NCL [48], UNO [7] and IIC [19] are less confusing but there is no clear category boundary for different novel classes. Kmeans+ can produce better intra-class consistency and can separate most novel classes. A possible reason is that pseudo labels by K-means clustering on the unlabeled training set can reflect the overall distribution to some extent. However, some shapes are incorrectly grouped into the wrong class due to the poor discrimination of learned features. By exploring the part concept compositions of different novel classes, our method can push away the embedding regions of novel classes to a greater extent compared to other methods and achieve much tighter intra-class representations.

DTC

AutoNovel

NCL

UNO

IIC

Kmean+

Ours

Train Curves. As described in Sec. 3.1, the learned features of compared methods are gradually biased to known classes during the training process and lose generalizability to novel classes. To further confirm our analysis, we demonstrate the converging curves of known class accuracy and novel class cluster accuracy during the training of seven methods on the high similarity task of ModelNet in Fig 9. As shown in Fig. 9a, the known class accuracy of all methods increases quickly in the first 30 epochs and grows slowly after 100 epochs. However, in Fig. 9b, with the increase in the known class accuracy after 50 epochs, the clustering accuracies for novel classes of UNO, IIC, and Kmeans+ start to decrease. This indicates the known class accuracy competes with the novel class clustering accuracy, and features beneficial to known class recognition instead turn out to harm the generalization to novel classes. The clustering accuracy of our method stays stable after 60 epochs. Meanwhile, the trend of the cluster accuracy is consistent with that of the known class accuracy suggesting that learned features on known classes encourage the recognition of novel ones. This good generalization is attributed to the fact that part concepts construct a shared embedding space between known and novel classes.

Impact of 3D pre-trained models. 3D data makes things more challenging as the 3D pretrained model does not generalize well. For example, SimGCD [39] uses the powerful DINO [3] model pre-trained on millions of images as the encoder and only fine-tunes a very small part of the network during training. This training strategy makes the extracted image features more generalizable, thus producing better results. In fact, previous work [29] suggests that DINO is very important to the generalization to unseen concepts. In contrast, the 3D field lacks both large-scale 3D shapes as well as high-quality pre-trained models. In order to evaluate the impact of 3D pre-trained models, we present results using two different self-supervised pre-trained backbones (OcCo [36] and CrossPoint [1]) under the GCD setting in the Tab. 7. Following SimGCD [39], we only fine-tune the last three layers of the PointNet++ backbone and the classification heads. The results demonstrate that utilizing more generalized features from pre-training improves the performance of the baseline methods. However,

(a) Known class

(b) Novel class

0.8

0.7

(%)

Cluster ACC

0.5

C

0.9

AC

0.3

DTC

NCL

IIC

一

Our

DTC

NCL

IIC

一

Our

Kmeans

UNO

AutoNovel

0.1

Kmeans

UNO

AutoNovel

0.8

0

50

100

150

200

250

50

100

150

200

250

Epoch

Epoch

there remains a substantial gap compared to our method. This indicates that existing 3D pre-trained models have not yet learned representations as powerful as those captured by DINO [3].

B Ablation Study

To further investigate our proposed model components, we conduct additional ablation experiments in Tab. 8. In ② , ③ and ④ , both Poincareé distance and L sd are proposed to encourage sharper part concept activation. Each of these two modules can be used individually to obtain performance gains (+3.7% vs +8.5%, +5.1% vs +7.4%), and they can be used together to achieve an even bigger improvement (+9.5% vs +15.2%). Without Poincareé distance and L sd , using L sc and L cd alone can only achieve minor improvements as shown in ⑤ and ⑥ . This is because that part concepts are less diverse and tend to encode the global feature of known classes that are hard to be generalized to novel classes. Since novel shapes may contain new part concepts that are not present in known classes, we concatenate the local part features Z l with the part composite features Z p . As shown in ⑧ , this design can obtain better results on low-similarity novel classes (+0.7% vs + 1.8%). We also evaluate the performance of only using the PRE module as shown in the ⑩ . The results show that using PRE alone can already achieve considerable performance gains (4.75% on average).

Ablation on different known classes number. Training on more known categories can certainly benefit novel class discovery. As shown in Tab.9, increasing the number of known classes leads to improved performance. Because with multiple categories we can learn more diverse part concepts that further will help us recognize novel objects better. In the table below, we evaluated the performance by training the method with an increasing number of known categories and the fixed number of novel categories. The results show that training the method with more categories does enhance the recognition of novel shapes.

C Different Backbones

We inspect how different network backbones impact the performance of 3D NCD. The comparisons are conducted on three popular 3D backbones including PointNet++ [25], DGCNN [37], and PointNeXt [26]. Among them, the PointNeXt achieves state-of-the-art performance for 3D classification and segmentation. We replace the backbone of compared methods with the above network architec-

tures and evaluate their performance on the ModelNet high similarity task. The results are shown in Tab. 10. the known class accuracy of the same 3D NCD method on different backbones does not vary too much (within 1% ). On the other hand, the backbone has a significant influence on the novel-class performance for all the compared methods. For example, the weaker backbone DGCNN can achieve better novel class accuracy than the other two backbones. This is because DGCNN uses graphs to represent the local features of the point cloud, thus increasing the feature generalization on novel classes. Our method can surpass all the baselines by a very large margin (at least +8 . 9% ) on the novel-class accuracy even using a weak backbone (PointNet), while attaining comparable good performance on the known-class recognition. This superiority is primarily attributed to the part concept-based features that are more generalizable to novel classes than other backbones.

D Results on Large-Scale Datasets

The large-scale 3D datasets (MVimgNet [43]) are available and we conduct an MVimgNet50-100 task as shown in the Tab.11. In this task, we selected 50 classes as known classes and another 100 classes as novel classes from the large-scale MVimgNet dataset. Despite the challenges when increasing the number of unknown classes, our method can still outperform baselines by a large margin.

E Estimate Cluster Numbers

We fixed the number of unseen classes following the baseline methods [10]. However, the number of novel classes can be estimated by following the strategy in AutoNovel [10] and GCD [35], where semi-supervised k-means and a validation set can be used to select the optimal number of unseen classes. We estimate the number of clusters using the above method on the ModelNet similarity task as shown in Tab. 12. The results show the estimation method may slightly underestimate the novel class number when similarities between novel classes and known classes are high. This is because some highly related classes are clustered together. Meanwhile, it slightly overestimates the number of novel classes when similarities decrease.

F Implementation Details

In all experiments, we adopt PointNet [24] as the backbone for point-wise feature learning and sample 1024 points from the original point clouds as inputs. We use scaling, rotation, jittering, and translation as data augmentation to generate two augmented point clouds for both labeled and unlabeled inputs respectively during training. The network is optimized with the Adam optimizer with an initial learning rate of 0 . 001 . In our experiments, we first train the network for 30 epochs with labeled dataset D l to learn the part concept bank, then we train all networks for another 220 epochs with both labeled and unlabeled datasets. We specify both the feature vector size D and the dimension of the part concepts to 256 . The overall method is implemented in PyTorch, and we conduct all experiments on a computer with an NVIDIA Tesla V100 GPU. Our method takes 1.26M parameters, 14.7G memory cost and 0.43s train time per batch measured on ModelNet. Readers are encouraged to refer to the released code for detailed implementation.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper proposes a novel part-based algorithm for 3D novel class discovery (NCD). Authors propose Decompose Novel Into Known parts (DNIK) that leverages knowledge about parts of known objects to discover novel classes.\n- Authors identify that the main problem with learning 3D features for object discovery is that the features are heavily biased towards the known classes. This work shows that this can be prevented by using the well known part-based modeling approach. \n- DNIK is trained to learn a part concept bank that can be used to compose different known and novel objects. Three different regularizations are proposed to prevent the collapse of this part bank.\n- Extensive experiments show the deficiencies of existing 2D NCD literature and the effectiveness of DNIK to overcome these issues.\n\nSTRENGTHS:\n- This paper builds on an age old, part-based models in visual recognition and shows impressive improvements over single holistic representations currently in use in the 2D category discovery methods. As shown in the experiments this has significant merits for identifying and grouping new classes. \n- The paper writing was smooth and was very easy to follow. An experienced engineer would be able to reproduce the work with the given details.\n- Authors support all the claims made in the paper with experiments on real world datasets or on toy problems. Sec 3.1 and Fig. 4.a were particularly interesting to understand what the authors were trying to convey.\n- The effectiveness of the method was shown with the impressive experimental results.\n\nWEAKNESSES:\n- While the problem tackled by the authors is relevant and important, the setup adopted by the authors is outdated. Generalized category discovery, as done in [1][2] is a more realistic setting and it is not clear why the current method is not suited for this setup or the authors advice against it? It is my strong suggestion to the authors to answer this question and compare with the relevant work (cited below) to justify this work among existing literature.\n- The toy example in Sec. 3.1 is not fair for the following reason. In L86, the setup states that the classes in known and unknown sets share some similarities, but in the example authors choose {table, sofa, stool} as known and {chair, bench, bathtub, plant} as unknown objects. In this case, bathtub and plant do not share any commonalities with the known objects. It looks like the authors intentionally exaggerate the problem to make their point. While this is acceptable, it is not clear how much of a serious issue is the \"overfitting to the known classes\" problem. \n- Author propose to use supervised contrastive loss to learn more parts from the known shapes. The motivation and explanation doesn't justify why this would be the case. In L204 authors pool the features along the last dimension which basically is a \"shape\" feature as opposed to a \"part\" feature. It is not clear how the contrastive loss helps learn more part features when the loss is being applied on the \"shape\" features.\n- Table 4 demonstrates the performance improvement for each of the proposed components but experiments to demonstrate that show that the regularizations on the part features actually operate as the authors claim is missing. What happens when diversity loss is missing? Do all the part features in the bank collapse to fewer representations? This can be quantified by cosine similarity between the part features. Similar analysis on the remaining two regularization terms is warranted.\n\n[1] Sagar Vaze, Kai Han, Andrea Vedaldi, Andrew Zisserman, Generalized Category Discovery, CVPR 2022.\n[2] Sai Saketh Rambhatla, Rama Chellappa, Abhinav Shrivastava, The pursuit of knowledge: Discovering and localizing novel categories using dual memory, ICCV 2021.\n\nQUESTIONS:\n- Can Fig. 5b, 5c be combined in to one plot? Two separate plots makes it hard to understand what values of K, Q are being used for each of these. \n- Legend for Fig. 6 is missing.\n\nLIMITATIONS:\nAuthors have addressed the limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

comments: SUMMARY:
In this work, they address 3D novel class discovery (NCD) that discovers novel classes from an unlabeled dataset by leveraging the knowledge of disjoint known classes. The key challenge of 3D NCD is that learned features by known class recognition are heavily biased and hinder generalization to novel classes. Since geometric parts are more generalizable across different classes, the authors propose to decompose novel into known parts, coined DNIK, to mitigate the above problems.

STRENGTHS:
1. The paper is well written and motivation (separate instances into repeatable parts) is pretty good. 
2. The model design is reasonable and the improvement is satisfied. 
3. The experimental analysis is sufficient.

WEAKNESSES:
1. This paper does not consider hierarchical part representation. 
2. why does Part Relation Encoder work for novel classes ? 
3. Does the improved representation works for some scene level tasks, such as novel class segmentation for point cloud ?

QUESTIONS:
see the weakness

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of novel category discovery in the 3D shape recognition domain, a framework leveraging the 3D parts and the part-wise relation is proposed which the motivation is learning the parts from the known classes could help the model capture more transferrable features or concepts for the novel categories.\nThis motivation is validated using experiments, and overall the framework shows better performance than some baselines.\n\nSTRENGTHS:\n1. The idea of decomposing a category into parts is interesting.\n2. I like the organization of this paper, starts with an analysis of the problem of previou method, and then proposed new ones based on the analysis.\n3. The paper also explored a bit on the design choices for 3D novel category discovery, which could be helpful.\n\nWEAKNESSES:\n1. This paper still considers the novel category discovery problem while a more generlized setting exists, generalized category discovery [R1, R2], I would suggest the paper to include more discussion and experiment on this generalized setting.\n2. It seems that the total number of categories in the datasets are quite small compared to 2D NCD, I am wondering if Objaverse [R3] can be used for this task?\n\n\n[R1] Generalized Category Discovery, CVPR 2022\n[R2] Parametric Classification for Generalized Category Discovery: A Baseline Study, arXiv.\n[R3] https://objaverse.allenai.org/\n\nQUESTIONS:\n1. In the v1 version of SimGCD fig 10 [R4], it is shown that the accuracy on known classes first increases and then drops while the novel class accuracy keeps improving, this contradicts the observation made in this paper, I am wondering if this is because of the setting (generalized category discovery v.s. novel category discovery), the data (2D v.s. 3D), or the number of categories(200 v.s. 7)? Consider this observation is the motivation for this paper, this question will be the biggest concern of mine.\n\n\n[R4] https://arxiv.org/pdf/2211.11727v1.pdf\n\nLIMITATIONS:\nI think the main limitation is that the number of categories is small, thus the conclusion made based on these small datasets may not be generalizable to larger scale datasets.\n\nOverall I think this paper is very clear, and could be of interest for the community, however the concerns I raises in the questions should be addressed first.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a framework, called Decompose Novel Into Known parts (DNIK), that addresses the challenge of 3D Novel Class Discovery (NCD) – identifying new classes from an unlabeled dataset using the knowledge of known classes. Current methods, heavily biased towards known classes, struggle to generalize to novel classes. By leveraging more generalizable geometric parts across different classes, DNIK mitigates this issue. It constructs a part concept bank encoding rich geometric patterns from known classes, which is used to represent novel 3D shapes as part concept compositions, thus facilitating cross-category generalization. DNIK also leverages part-wise spatial relations for improved recognition. The method has been tested through three 3D NCD tasks, consistently outperforming state-of-the-art baselines.\n\n\n---- after rebuttal ----\n\nAs the author's rebuttal resolved some of concerns, I raised my score to 5. However, I still feel the studied task is a bit simple, and also there are several spaces to improve for the current manuscript. I will not fight for its acceptance.\n\nSTRENGTHS:\nThe studied direction is important as we need to understand parts well to play with 3D objects generalizable. This paper takes a step towards open 3D object recognition via part understanding. Overall, the components used in the proposed framework are sound and reasonable. The paper is easy to follow. \n\n\nExtensive results shown in Table 1 & 2 demonstrate the strength of the proposed method. The proposed DNIK generally achieved state-of-the-art performance. Some detailed ablation studies are included in Table 4. The cross-domain task is interesting to see the transfer performance.\n\nWEAKNESSES:\nUtilizing the part are sharable across different 3D object categories are studied in the previous literature [1,2]. In those paper, they exploited \"harder\" task, such as segmentation. As the proposed framework can address novel class classification via known part concepts. Can the framework be extended to ground where is those known parts in novel object? Or other applications beyond object recognition?\n\n[1] Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories\n[2] 3D Compositional Zero-Shot Learning with DeCompositional Consensus\n\nHow the framework handle two different object categories with limited shared parts, such as airplane and chair? Will the framework train on multiple object categories benefit novel object discovery? It would be good to include some failure cases to analyze and provide readers a sense for the limitation of the proposed framework. \n\nL46~47 said the framework can help use part relation features. Can the framework be extended to discovery part relationship?\n\nQUESTIONS:\nPlease address the concerns raised above.\n\nLIMITATIONS:\nLine 319~320 analyzed one minor limitation. I feel there are potential more:\n\n1. if the part are not sharable between different categories, such as lamp -> chair, table -> faucet, can the framework still handle?\n\n2. the only shown application is recognition which limits the practical use of the proposed framework.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of 3D NCD (novel class discovery). The objective is to discover novel classes by leveraging information learned from the known classes. This paper proposes a novel framework, DNIK, for 3D novel class discovery (3D NCD) by leveraging part concepts and part-wise relations learned from known classes to reinforce the recognition of novel shapes. The framework consists of a learnable part concept bank, a local geometric aggregation module, a part relation encoding module, and three constraints to facilitate effective part concept learning.\n\nSTRENGTHS:\n(1) The proposed part concept bank and part relation encoding module can effectively bridge the gaps between known and novel shapes and mitigate feature bias.\n\n(2) The experiments show that the proposed method outperforms all baselines consistently and significantly on all metrics. \n\n(3) The paper is generally well-written and easy to follow.\n\nWEAKNESSES:\n(1) The unseen class number is assumed to be known, which makes the method less practical.\n\n(2) The effectiveness of the proposed PRE module is not well demonstrated. The performance is not shown by using only the part position feature from the PRE. Therefore, it is not clear about the individual role of PCB and PRE. It would be good to at least ablate the effectiveness that only uses PRE in Table 4.\n\n(3) The study of NCD has been extended to consider the case where the unlabelled data contains objects from seen and unseen classes [A]. It is more convincing to also show results under this more general and practical case. \n\n[A] Vaze et al, Generalized Category Discovery, CVPR 2022\n\n(4) Each part in Part Set Q has the same number of points, that is, K neighbors, which may be dataset dependent and affected by the scale of the objects, while a fixed value of K=64 is selected in the paper. This is unlikely to generalize well to other datasets and objects of different scales. It would be good to show how the method works on instances from the same categories but of different scales. More investigation on this is expected.\n\nQUESTIONS:\n(1) How to ensure the features from PRE include the position relationship of each part? The feature extraction by PRE seems like a process through a black box.\n(2) How are the Nq parts like in the initial point cloud? How different/similar are they? The initialization may also heavily affect the results. e.g., if the initial parts are too similar, they are unlikely to be well separated in the end. However, in the beginning, we have little (or no) control over this.\n\nLIMITATIONS:\nThe paper has described the potential limitation of multi-scale objects, not mentioning much about the societal impact, but I did not see any major problem here.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}