Decompose Novel into Known: Part Concept Learning For 3D Novel Class Discovery
Authors: Tingyu Weng
Jun Xiao
Haiyong Jiang
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D point clouds
3D recognition
part-based representation
unsupervised class discovery

Abstract: In this work, we address 3D novel class discovery (NCD) that discovers novel classes from an unlabeled dataset by leveraging the knowledge of disjoint known classes. The key challenge of 3D NCD is that learned features by known class recognition are heavily biased and hinder generalization to novel classes. Since geometric parts are more generalizable across different classes, we propose to decompose novel into known parts, coined DNIK, to mitigate the above problems. DNIK learns a part concept bank encoding rich part geometric patterns from known classes so that novel 3D shapes can be represented as part concept compositions to facilitate cross-category generalization. Moreover, we formulate three constraints on part concepts to ensure diverse part concepts without collapsing. A part relation encoding module (PRE) is also developed to leverage part-wise spatial relations for better recognition. We construct three 3D NCD tasks for evaluation and extensive experiments show that our method achieves significantly superior results than SOTA baselines (+11.7%, +14.1%, and +16.3% improvements on average for three tasks, respectively). Code and data will be released.

Decomposing Novel into Known: Part Concept Learning For 3D Novel Class Discovery
Tingyu Weng
Jun Xiao
Haiyong Jiang"
School of AI
University of Chinese Academy Sciences
wengtiingyu180maile ucas ac. cn, (xiaojun, haiyong. jiang)
Abstract
In this work, we address 3D novel class discovery (NCD) that discovers novel classes from an unlabeled dataset by leveraging the knowledge disjoint known classes. The key challenge of 3D NCD is that learned features by known class recognition are heavily biased and hinder generalization to novel classes Since geometric parts are more generalizable across different classes, we propose decompose novel into known parts, coined DNIK, to mitigate the above problems. DNIK learns part concept bank encoding rich part geometric patterns from known classes so that novel 3D shapes can be represented as part concept compositions to facilitate cross-category generalization. Moreover, on part concepts to ensure diverse part concepts without collapsing. part relation encoding module (PRE) is also developed to leverage part- wise spatial relations for better recognition. We construct three 3D NCD tasks for evaluation and extensive experiments show that method achieves significantly superior results than SOTA baselines (+1117%, +14.1% and +16.3% improvements on average for three tasks, respectively). Code and data will be released.
formulate three constraints
Introduction
3D recognition is fundamental for 3D tasks and plays a critical role in many applications such as roboties and autonomous vehicles. Recently, deep learning-based 3D recognition [24 25 45) has achieved remarkable progress in this task. Unfortunately. most methods are limited to annotated classes in the training set (i.e., known classes), and cannot recognize novel classes like human beings, which limits the robustness of 3D recognition in the real world. Therefore, it is crucial and valuable investigate the 3D novel class discovery task (NCD) so that 3D recognition can discover novel classes by leveraging the knowledge of known classes.
Though 2D NCD has been extensively investigated, straightforward extension of 2D NCD meth- ods 7 19 to 3D NCD usually performs poorly due to the feature bias problem [32, i.e., the learned features are heavily biased towards known classes and have difficulty in generalizing novel classes. The main factors lie two aspects. First, learned feature representations are usually dominated by known classes while being oblivious novel classes. Second, 3ID recognition usually focuses shortcut features [8] [2 only capture a special global structure or local regions specific to known classes [21] shown in Fig. [
In this work, we tackle the above problems based on the observation that humans can subconsciously decompose shapes into familiar part concepts, which along with their spatial relations provide effective cues for recognizing different novel shapes [13] [8]. Therefore part concepts can bridge the gaps between known shapes and those novel ones with shared part space, thus alleviating the
Haiyong the corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Part Concept Bank
Chair
Table
PointNet+*
Sofa
Importance
Lege Posilio Novel Class
PointNet (a)
Known Classes
(b)
Figure/ 1: (a) The Grad-CAM (31 of known shape (table). 3D recognition methods focus on shorteut features representing special global structure local region. (b) Novel shape can be represented part concept composition along with part-wise spatial relations.
feature bias problem in 2D NCD extensions. On the other hand, the compositions and spatial relations between parts make easier to recognize novel shapes. For example, as shown in Fig. [b), a novel class, e.g., a chair, contains familiar concepts like plane, leg, and backrest observed in known classes (e.g., tables and sofas), therefore we can easily recognize the chair according to its part concepts and part spatial relations.
Along with this idea, we present novel part concept-based framework coined DNIK (decompose novel into known parts) for 3D NCD shown in Fig. [2] Specifically, we first construct a learnable part concept bank that encodes rich geometric patterns from known classes. Then DNIK can project part-level features of an input shape into the spanning space of part concepts so that shapes from different categories can be represented as part-concept composition features sharing the common embedding space. Therefore learned features by known class recognition can be generalized to novel classes. We further leverage part-wise spatial relations of input shapes with part relation encoding module (PRE) for discriminative part relation features. These two kinds of features are then combined for 3D shape recognition. To avoid the collapse of the part concept bank and encourage diverse part concepts across classes, three constraints on part concepts also considered. Moreover, we construct three 3D NCD tasks for the evaluation different scenarios and also extend our method the generalized class discovery task (GCD) [35]
Our contributions can summarized follows: (i) the analysis of biased features and the necessity of part concepts in 3D NCD; (ii) a part concept-based framework leveraging both part concept features and part-wise relations for 3D NCD; (iii) the design of three necessary constraints on part concepts to facilitate effective part concept learning; (iv) various 3D NCD) and GCD tasks for the evaluation of different scenarios. Extensive experiments show that our method significantly outperforms baseline methods all 3D NCD and GCD tasks and can efficiently recognize novel shapes.
2 Related Work
Novel Class Discovery, As there is existing work on 3D NCD, mainly review related works 2D NCD. Related works can be classified into two categories. The first category of methods [9 [4] usually first train classification network on the known dataset and then cluster the novel data according to the prediction or output features of the pre-trained network. However, learned features in this way are highly biased toward known classes, leading to performance bottlenecks in novel classes. By contrast, the other line of works 7 10 explores both known and novel samples for NCD during the clustering phase. Therefore feature representation can be shared between known and novel classes and allow for better generalization to novel classes. AutoNovel (10) utilizes both known and novel samples for self-supervised representation learning and joint known class classification and NCD. Following works further bridge the gap between known and novel classes by mixing up known and novel samples 49 and introducing consistency between similar objects [48) and predictions from two different branches [44] On the other hand, UNO 7 unifies the learning of both known and novel classes using single cross-entropy loss based on optimal transports between pseudo-label assignments. Following this paradigm, the similarity relation of samples between known and novel classes is further explored by a spacing loss in the embedding space [16], compositional experts [4]] and symmetric KL divergence [19]. Recently NCD has been extended to the more practical general

---

Known Clsssifior
Concept Bank,"
Zyt
Point Cloud
Feature Encoder
Activation
Poctton-ivvriaan
Part Composite Feature 4p
Owl
Feature 2
MLP
T2
T2
Point
Part Set
Foatura
Feeture Ze
MaxPool
Zge
Position Relation
Zyz
Poeilion-sware Feaature 2
Novel Glassiferr
Point Gloud
Feature 2;
The overall architecture. A point cloud and its augmentation are fed into a backbone network to encode point-wise features Z and grouped into part set Q respectively. As illustrated in pink box, we propose local geometric aggregation (L.GA) module to learn position-invarian part feature Zi and calculate the concept activation map S use the Poincaré distance between Zi and learnable part concept bank P. Then we can construct part composite features Zp according to the Eq. 2 We also propose position relation encoder (PRE) module to extract the part position feature Z, as illustrated the orange box. Finally, Zp and Z, are concatenated and fed into an MLP for recognition.
Figure 2:
35 which assumes that unlabeled datasets contain both known and novel classes. (355 and 339 tackle this problem by using semi-supervised K-means and purely parameterized models respectively. In this work, address 3D NCD and GCD by introducing part concepts and part-wise relations as the bridge between known classes and novel ones.
category discovery task (GCD)
Part-based 3D Recognition. Shape parts are the basic building blocks for 3D shapes 22 and play an important role in both human perception and various 3D recognition tasks. For example, Zhao et al. [46] represent repeatable shape parts as prototypes to improve few-shot 3D object detection. robust prior for incremental learning. Wei al.
Chowdhury al. [5] learn part information align part-based features to reduce the geometry shifts across different domains. Zhao et al. 47 propose a part codebook-basee
self-attention leverage the generalized geometric information for better voxel-based feature extractors. Inspired by these works, we learn a part concept bank to map features of known and novel classes into a shared space.
Method
This work aims 3D NCD that learns to group 3D shapes in the unlabeled set into several classes by leveraging the knowledge of known classes in a labeled dataset. Basically, 3D NCD assumes the availability of two datasets: an unlabeled dataset D {X") J with 3D shapes sampled from novel classes C and labeled dataset D' ((X) ) where X denotes 3D shape and y E C¹ is the corresponding categories label. The classes in the two datasets are disjoint, therefore we have C² n n 0. However, shapes from D and n usually endow with some similarities. For example,
both a chair and desk have legs and plane surfaces. Following the literature 7 10 48 the number of novel classes C is known a prior.
we assume
3.1 Analysis on the Extension of 2D NCD Methods
As there is no existing work on 3D NCD, we instead extend mainstream 2D NCD method UNO ? the 3D domain as the baseline. UNO learns shared feature extractor f and two linear classifiers o' and d² with C' and " output neurons. The classifiers o' and &" are optimized with a cross- cross-entropy loss using labels of known classes and pseudo labels of novel classes generated by a clustering method (i.e., Sinkhorn- badly for two reasons. First, feature representation usually dominated by known classes, while oblivious to novel classes. Second, shape-level features extracted from 3D networks confront serious
(6). However, naive extensions of 2D NCD to the 3D domain work

---

Shape-level
Part-level
Cosine Similarity
ACC(%)
Novel Class Unsupervised Class
Known Class
0.8
0.4
150
200
Chair
Bench Novel Class
Bathtub
Plant
Epoch
Figure 3: Left: The training curves of UNO 7 based methods. Right: Cosine similarities between
features of novel classes and shape part-level concepts.
shortcuts as illustrated in Fig. [ Thus learned features are heavily biased toward the known classes and are hard serve novel classes.
We conduct a toy experiment to verify our analysis. We chose table, sofa, stool as known classes and chair, bench, bathtub, plant novel classes from ModelNet [40]. The first two novel classes have similar shape parts with the known classes, e.g., legs, while the last two do not. To reveal the wane and wax between known and novel classes during optimization, we investigated performance curves of UNO 7 in Fig. Left. In the first 60 epochs, training 3D recognition with known class supervision can help novel class discovery. Therefore there exist some sharing feature representations between two different sets of classes and UNO can achieve better accuracy on novel classes than unsupervised clustering (i.e., w/o the cross-entropy loss). However, with the training proceeding, the performance in novel classes starts to decrease and can even be worse than unsupervised clustering, indicating that learned features are gradually biased to known classes and lose generalizability [32)
Fig. Right shows the average cosine similarity between the shape-level or part-level concepts of known classes and the shape part features of novel classes learned from the trained UNO method [7]. Shape features are direct outputs of the feature extractor while part features denote pooling features of overlapped parts split from shapes. We then define the shape-level concepts by averaging the shape features of each known class and part-level concepts by clustering the part features of known classes using K-means. For shape-level concepts, the cosine similarities of chairs and benches are low and comparable to those of bathtubs and plants, which indicates that shape features cannot be shared between known and novel classes. However, part-level concepts can provide higher cosine similarities and reflect similar properties between known and novel classes better.
3.2 Part Concepts for Cross-category Generalization
In this work, we approach 3D NCD with learnable part concept bank shared across different classes. The overall framework is shown in Fig 2 parts, i.e., part concepts, are more likely occur in different categories of shapes as analyzed in Sec. 3.1 We first learn part concepts from shapes of known classes in order to deliver more generalizable feature representation and reduce biases when facing novel classes. Then we leverage the relative positions of different parts to further enhance the recognization of novel classes.
The basic observation is that geometric patterns and shape
Learning Part-level Features. As intend to learn set of part concepts representing generalizable shape parts, we first encode part-level features inspired by the paradigm of PointMAE [23]. Specifi- cally, we extract point-wise features Z R f.x for each point in the input point cloud X E R R.33 with an off-the-shelf 3D backbone, e.g., PointNet [24], where L denotes the number of 3D points and D is the dimension of point cloud features. Then use farthest point sampling (FPS) sample No points the centroid seeds and group K neighboring points overlapped parts Q (Q. Part set Q; {a1 4ik } IR x includes features at its centroid followed by point features of its K nearest neighbors (z22.... .... k Intuitively, we can obtain part-level features by pooling point features within each part Q: but part features obtained in this way are dependent on the part location and may hinder the learning shared part concepts different positions. To cope with this problem, we propose local geometric aggregation (LGA) module following
42
26 to decouple
the spatial location and aggregate levt- features Z₁ E R x by
Zr (maxpool Qi Zil )), Vi f1, f1.... .N9,
(1)
σ

---

where o is the scalar standard deviation across channels, and & denotes a multi-layer perception D (MLP) with channels (D
2 D) encode the local geometry features.
Part Concept learning. After obtaining part-level features Zi, we use them to learn shared part concepts among categories. First, we construct learnable part concept bank denoted with p Jm=i, where part concept Pm E I encodes features of different shape parts and can represent plane, corners, legs as illustrated on the top of Fig. 2 In our implementation, part concepts are initialized with Gaussian distribution (0, I). We project part-level features Zi into the spanning space of part concepts P according to their respective similarities. Then part-level features Zi are approximated by part composition features Zp E IR N, x as follows:
Zp. = - S's
where S E IR N M measures the similarity between part features Z and the part concept bank P. The concept activation map S is defined as:
S = h(d(Z,, PT )), 2i W. Zi, P.PPWW P.
where Wz, W E D are learnable weight matrixes for feature dimension reduction, /t-) is softmax function to produce normalized probabilities along each row, and -(:,-) is distance function that compares part features and part concepts. To ensure that the geometric knowledge of each part feature can be learned by a particular part concept. distance d(.,) .) should produce significant similarity differences between part features and different part concepts. A common choice of d(---) to use dot product similarity 38 or cos similarity [1] defined in the Fuclidean space. However, as shown in Fig- aa), we found that the Euclidean distance functions lead to uniform activations of part features on different part concepts. Different from the Fuclidean space where the distance grows linearly, the distance between part features and concepts grows exponentially in the hyperbolic space [17 30 28), which can produce sharp concept activation maps (see Fig. Ka)). Therefore, we project features into the hyperbolic space using Poincaré sphere model following 17 and the distance function between Zi and P in Poincaré manifolds is given by:
2 Pl2
d(Zr, arcosh (1- [1-1/2]]] PPl2) + +1).
Though learned part concepts can cover most shape parts in novel classes, there still exist some novel part concepts that are not contained in known classes. In this case, the concept activation map between a part and part concepts P is likely to be uniform and results in ambiguous composite features. In our method, we further concatenate the local part features Z₁ with the part composite features Zp to confront this problem.
Part Relation Encoder. As part composite features Zp only consider position-independdee composi- tions of part concepts and neglect mutual position relations between different parts that usually deliver shape structures as shown in Fig. bb). To this end, we propose a novel Position Relation Module that adopts shallow Transformer Encoder (45) with two self-attention layers, denoted as TransEne, to learn mutual relations between part concepts as relation feature Z,. Since the position information has been decoupled from Zt, we regain position-aware part features Z, = Ez ... ZgN, E N"
from Q using max-pooling and encode Z, E Nr> as follows:
Zg: maxpool(Qi).
(5)
Z, TransEnc (2,).
Afterward, we concatenate the position relation feature 7, with the point composite features Zp and feed the result into an MIP with 2D x D and a max-pooling layer to produce category- features Ze. We can classify the input shape X by feeding Z, to linear layer predicting the ciass-wise probabilities.
3.3 Constraining Part Concepts
Though part concepts are proposed to represent different parts, they can easily collapse to the same values. The main reason is that the similarity between randomly initialized part concepts and part
5

---

Lsd
Lyc
Euclidean
Poincaré
Class
Class 2
(b)
(c)
Figure 4: (a) The part concept activation is likely to be uniform in the Euclidean space but sharp in the Poincaré space. (b) Self-distillation loss Lad encourages part activations to align with the sharper distributions of their corresponding data augmentation. (c) Supervised contrastive loss Lee forces the part concept activations of different known classes to be separated from each other so that more part concepts can be learned.
features similar during the initial training stage. Consequently, part composite features based on the concept activation map would gradually be identical and part concepts updated with similarly back-propagated gradients from them converge to similar features. To ensure the representation ability and generalization of the part concept bank, we propose three constraints to train part concepts.
Self-distillation Loss. guarantee that each part's geometric knowledge can be learned by specific part concepts, the concept activation distribution of each part should be sharp. To achieve this, propose a self-distillation loss inspired by B]. We enforce each part concept activation the shape parts Q to be consistent with a sharper one of the corresponding part concept activation from the augmented shape parts Q (see Fig (b)) as follows:
Led = - stop(p(S,, Tx)) log S, TT) S.),
(6)
where TT) Σ exp(s(/T.) is a sharp concept activation distribution of Qi controlled by temperature parameter т, (set 0.1 empirically) and stop(-) means the gradients is not back-propagater Loss Lad enhances the sharpness and consistency of the concept activation under different data augmentations for both the labeled and unlabeled datasets.
exp(s./T.)
Supervised Contrastive Loss. During training, we found that even few part concepts can support the recognition of known classes shown in the red boxes in Fig. c In this case, the part concept bank lacks the incentive to learn more part concepts from known shapes, which reduces the information it can share with novel classes. Thus, for the labeled dataset D', we employ supervised contrastive loss Lgg to increase the inter-class discrepancy of their concept activation maps. As shown in the green boxes in Fig. cc, through Lsc. the part concept bank can learn additional parts (i.e., leg and pedestal) from known shapes to increase their inter-class discrepancy. Given mini-batch of B shapes { and their augmentation { we first sum their activation maps SB E 2 M into shape concept activation TB E 2 M along the part dimension, where T) summarizes all part concepts in shape. Then the supervised contrastive loss is defined on 2B shapes as follows:
Σ log (xp(-sim(Zi, Tf Σ exp( sim TT)
Lec = |pl
where p ([T) E y Uf is the set that contains the shape concept activation 1; of the augmented sample and other samples belonging to the same class in the mini-batch, sim(... calculates the cosine similarity, and temperature Te 0.1 empirically.
Diversity Loss. To capture different patterns in 3D shapes, part concepts should be diverse. We can encourage this by minimizing the similarity between different part concepts with the following loss:
M
Led
max P (Pn) m Pa..
sim(Pm. P-) - 8),
(8)
m=1

---

where ò is the cosine similarity threshold between code and the other ones. In our experiments, we set 0.1.
3.4 The Training Losses
Following the literature 48, 10 7 we construct two linear classifiers of and d*. We employ standard cross-entropy loss Lie to optimize the known class classifier o' and a self-classification loss Leets for training novel class classifier d". Loss L minimizes the cross entropy between novel shapes and their augmentation with the assumption that the distribution of each novel class is uniform and is formulated following (2). In summary, the overall training objective loss function in model is:
Lee + Lisely + A1 Lad + X2 Lsc + X Led:
where loss Led in 6). Lec in Eq. and Led in Eq. () enforce basic assumptions of part concepts. Weight term A, balances different losses and is set to (0.1, 0.1, 0.1). During inference, part composite features of a test sample Ze are passed through both classifiers to obtain the corresponding logits, then the two logits are concatenated and fed into softmax layer to obtain the class probability.
4 Experiments
4.1 Evaluation Datasets for 3D NCD
We construct three NCD tasks to evaluate the performance of 3D NCD based on four different kinds
of 3D datasets covering CAD shapes (i.e.. ModelNet40 [40] and ShapeNetCore [4), and indoor objects constructed using 3D scans and multi-view images (i.e., ScanObjectNN [3] and Co3D (27]).
Random Split 3D NCD focuses on evaluating the impact of a different number of known and novel classes on the 3D NCD following 77. The task is denoted with DATASET-K-U, where K is the number of known classes and the other U classes as novel ones.
Similarity Split 3D NCD. The semantic similarity between known and novel classes can greatly affect the performance of NCD. A highly semantically similar label set can provide more useful information for discovering novel classes. Based on the transfer leakage metric in [20]. we split novel classes as three sets (i.c., High, Medium, and Low) according to the descending similarity w.r.t known classes.
Cross Domain 3D NCD assesses the performance of 3D NCD on cross-domain shape recognition and reflects the real-world scenarios where novel shapes may come from different domains. Therefore, we select known and novel classes with semantic similarity from different datasets respectively. The corresponding tasks are denoted with (known class dataset - novel class dataset).
4.2 Comparison with SOTA methods
To our best knowledge, there is no previous attempt at the 3D NCD problem. So we extend different kinds of SOTA 2D NCD methods to 3D NCD, including DTC 9 AutoNovel 10, NCL [88, UNO M and IIC [19. In addition, training novel class classifier with pseudo labels generated by K-means can achieve good results and we also compare with it, denoting as Kmeans+. We implement the above baselines by substituting the image-based backbone with 3D one (i.e., PointNet) for 3D NCD.
Results on the Random Split 3D NCD Task are shown in Tab. T For the known class recognition, all compared methods perform well, but the performance on the novel classes diverges a lot. When the number of known classes is large, the network can inherit more knowledge for the novel class recognition. The NCL and IIC can achieve better results on most datasets since they force the separation between known and novel classes while learning similarity relationships within the novel classes, so the learned features are more discriminative for the novel classes. Our method can gain consistent improvements on novel classes of all tasks (+11.7% on average) while attaining the best know-class accuracy for most tasks (5/6). This demonstrates that part concepts can effectively bridge the gaps between known and novel shapes thus mitigating the feature bias. When the number of known classes is small, the performance of compared methods drops significantly. By contrast, our method decreases much less and still achieves the best results (+12.5% increases average). We
7

---

Table The quantitative results of the random split 3D NCD task. The results are averaged among five random trials. The best results are in bold, and the second- best ones are underlined.
ModelNet 30-10
ModelNet 10-30
Co3D 30-20
Co3D 20-30
ShapeNet 40-15
ShapeNet
20-35
Novel
Known
Novel
Known
Novel
Known
Novel
Known
Novel
Known
Novel
Known
DTC
79.6
31.1
92.5
31.1
77.0
40.8
80.8
31,0
89.9
AutoNovel
10
46.6
88.1
46,7
93.8
52.2
80.6
42.0
86.1
36.1
82.2
35.2
91.4
NCL 48
61.3
89.1
51.2
93.7
52.4
80.2
39.5
89.6
44.4
83.4
42.2
87.7
UNO
7
59.3
86.7
45.4
93.2
48.6
80.1
41.6
87.3
45.5
82.2
28.6
90.7
IIC
19
62.7
87.6
49.7
94.2
45.9
80.0
37.4
86.7
44.3
81.5
29.1
90.6
Kmeans+
83.0
41.6
92.2
52.1
76.6
84.8
80.9
36.6
90.9
Ours
72.0
88.9
63.0
96.0
61.0
85.0
58.5
90.1
62.6
84.7
53.2
92.4
Improvement
+9.3
40.2
+11.8
+1.3
+8.6
+4.4
+14.7
+0.5
+14.5
+1.3
+11.0
+1.0
Table 2: The accuracy results on the similarity split 3D NCD task. High, Medium, Low denote the novel classes have High, Medium, Low semantic similarity with known classes.
ModelNet
Cu3D
ShapeNet
High
Medium
Low
Known
High
Medium
Low
Known
High
Medium
Low
Known
DTC
28.3
39.6
32.5
95.7
41.3
35.2
26.7
89.2
37.6
36.6
36.6
87.5
Novel
10
39.2
42.2
44.6
94.8
57.1
54.7
34.7
87.5
39.2
38.3
39.2
87.1
NCL.
48
56.6
56.5
45.1
95.2
57.4
53.1
39.7
89.8
45.3
36.9
38.7
87.1
UNO
49.8
47.4
45.5
95.1
49.4
48.1
31.3
87.9
35.4
43.9
31.2
85.4
IIC
19
59.6
56.0
38.9
93.9
55.8
48.8
43.1
84.5
47.1
38.4
28.7
85.6
Kinaans+
58.5
86.1
Ours
73,2
68.3
66.4
95.2
69.3
65.2
56.9
93.3
64.7
54.6
57.1
89,2
Improvement
+13.6
+11.8
+20.9
-0.5
+11.9
+10.5
+13.8
+3.5
+17.6
+8.7
+17.9
+1.7
conjecture that the part concept bank can learn more local features for sharing than the other methods using global features.
Results on the Similarity Split 3D NCD Task are presented Tab. NCL, UNO, IIC, and Kmeans+ can achieve better results on novel classes with high similarities. However, the performance of these methods drops significantly (-13.5% on average) with the decrease in semantic similarity. The method that achieves the best results High-similarity novel classes cannot obtain the best Medium and Low ones simultancously. This indicates that the learned knowledge of these methods is biased towards known classes with highly similar geometry while lacking generalizability to other novel ones. By contrast, our method achieves large improvements for all similarities (+14.3% +11.8, +17.5% average) while reducing the gap between different similarity tasks. This is because part concepts can show up in different shapes, e.g., the back of a sofa and a chair in Fig. and align all shapes into the part space, allowing for generalization even for low-similarity novel classes
Results on the Pross--Dmmain 3D NCD Task are reported in Tab. We can observe that compared methods lack robustness on cross- cross-domain shapes and perform poorly. Our method performs well among different datasets and outperforms existing methods by large margin (-16.3%) on average). We believe the good performance is owing to the fact that part concepts are robust to data domains and can be generalized to different kinds of data.
4.3 Ablation Study
In this part, we conduct all ablations on the ModelNet similarity split 3D NCD. Readers can find more experiments in the supplementary.
Model components. In Tab. 4-2, adding the part concept bank without the LGA only achieves very small improvement compared the baseline, which indicates part concepts without positional invariance are hard to fit novel classes. In 3 and +, LGA can ensure better generalization over novel classes, and the part concept bank can align the part features from known and novel classes into the same space. Therefore they can collaborate to boost the performance for large margin (5.8% and 3.9%). In @) using the Poincaré metric can yield much sharper concept activations and make different part concepts more discriminative, therefore raising the performance (-3.7% and +8.5%). In 6, self-distillation loss L.ud encourages each part concept learn specific patterns to avoid collapses and contributes average 6.25% gain in accuracy. In Ø and @, loss Lee and Led force the network to

---

Table 3: The results on cross-domain task. M. Table 4: Ablation experiments. PCB means part con- and Scan. are ModelNet and ScanObjectNN. cept bank. LGA, Poincaré, and PRE are in Fig.
M.-Co3D
Mcan
CoaD-=Scan.
LGA
PCB
Poincaré
End
100
Ind
PRE
High
Low
Novel
Novel
Novel
49.9
40.3
DTC 2
26.6
29.6
25.2
8
52.6
44.8
AutoNavel
B0
26.8
36.7
41.6
58.4
48.7
NCL
37.4
62.
57.2
UNO
27.7
43.9
35.0
ICC
19
30.8
39.8
35.4
70.1
64.88
Kmeuns+
29.0
41.8
26.0
71.3
65,1
Ours
64.0 +28.8
83.4 +9.5
52.5 10.5
73.2
66.4
Cluster ACC (%k)
Chatter ACC
Cluser AOC
85
16
32
64
128
512
256
128
64
(a) #part concept N
(b) #points in each part K
(c)
#split part.v
NCL
IIC
Ours
Figure 5: Hyperparameter evaluations on (a) the number of
Figure 6: Visualization of the embed- ding spaces of novel classes with t- SNE for different methods.
part concepts, (b) the number of points K contained in each part, and (c) the number of split parts No of each shape.
learn more diverse part concepts, which is more helpful for high-similarity novel classes (3.4% vs 1. 1.2%). since their concept activation maps are more similar (e.g., chair and stool). By comparing t and ( we can see the three proposed constraints (L Lec. and Lea) are essential to the final good performance and strengthen the results by +8.8% and 7.9%.. In @, the PRE can further incorporate discriminative position relation features to enhance the performance (+1.6% on average).
Hyperparameters. In Fig. a]a), conduct experiments different numbers of part concepts, i.e. M. Considering part concepts should increase with known classes, we set M = N' x C¹, When N < 3, the results are poor, which indicates that few part concepts cannot adequately encode all part patterns in known classes. However, the performance also drops when N > 6. This may be because many concepts harder be optimized.
The number of points contained in each overlapped part determines the part size, therefore affecting the generalization and discrimination abilities. The discrimination ability of parts decreases when the size is reduced and the generalization ability improves. Specifically, small parts are hard to contain meaningful local geometric patterns, while large parts may contain special structures that are biased towards known classes as shown in Fig. b).. Our approach achieves the best results when K 64.
Since the part set of a shape is split by FPS, few parts lead to large variations in the part sets of similar shapes, resulting inconsistent part composite features. In Fig. cc,, the result will decrease continuously when No < 128. Our method achieves the best performance when No = 256.
Visualizations of the Feature Embedding with t-SNE 34 are shown in Fig. For the NCL [48] and IIC [9) the features of different novel classes are mixed up in the embedding space, while ours can push away embedding regions of different novel classes greater extent.
Part Concept Visualization. order to reveal what part concepts can learn, we show concept activation maps between some part concepts and part features in Fig. We can activate similar semantic parts of both known and novel shapes, such as legs, arms, bottles, backrests, planes, and screens. This confirms that part concepts can effectively generalize knowledge learned from known classes to novel classes and represent novel classes as the composition of part concepts. For example, chair can be represented as (legs, arms, backrest, and a plane). We notice that part concepts can be robustly discovered and activated even if geometric structures are slightly different, e.g., arms of chairs, sofas, and benches.
that part concepts
4.4 Extension for Generalized Class Discovery
DNIK can also be extended for generalized class discovery (GCD) 35 for which the unlabeled dataset 8 contains 3D shapes from both known and novel classes. order to adapt DNIK for this task, concatenate the known class classifier o' and the novel class classifier 6" into unified

---

Leg
Arm
Bottle
Table
Bondh
Char
Pero
Stool
Safa
Bench
Char
FlowerPPol
Visse
Backrest
Plane
Screen
Sola
Bench
Chair
Table
Dench
Chair
Stool
Laptop
Novi
Importance
Figure 7: Visualization of different part concepts and their corresponding activations on different shapes of known and novel classes. Red indicates higher activations and similarities.
Table 5: The quantitative results of the random split 3D GCD task.
Table 6: The accuracy results on the simi- larity split 3D GCD task.
ModelNet 10
ModelNer 10 30
Co3D 30 20
Co3D 20.30
ModeINaa
Co3D
Novel
Known
Novel
Known
Nowel
Known
Nevel
Known
High
Low
Known
High
Low
Known
AutoNovela
10
32.6
33.6
94.3
5088
52.1
89.6
UNU
41.7
36.5
53.5
52.1
41.9
OCD [
58.3
54.7
95.7
56.2
46.0
89.2
SimGCD
47.2
43.5
91.0
54.3
42.1
8244
88.9
92.1
Ouns
70.6
68.3
95.9
62.5
53.9
88.0
Improvement
+18.5
-0.1
-10.2
+2.6
+10.3
+4.3
+8.3
-0.3
Improvement
+12.3
+13.5
+.2
+6.3
+7.9
-1.6
classifier. All samples from both the labeled dataset and unlabeled dataset go through the unified classifier to obtain the class-wise probability. Both Lee and Leefy are now used to optimize the unified classifier rather than the novel class classifier of the NCD setting.
We extend two 2D GCD methods to 3D GCD, including GCD [35, SimGCD ,and also extend AutoNovel I0 and UNO 7 for GCD tasks (denoted as Aovel and UNO+) by concatenating logits of known classes and novel classes following SimGCD [39. Results are shown in Tab. and Tab. 6 It can be seen that all compared methods show decreased results compared to the NCD
tasks in Tab. D and Tab. 2 This indicates that samples of known classes the unlabeled dataset D introduce noise into the training and confuse the classifier. GCD 35 can achieve better novel class accuracy (+5.6%) than other compared methods because it uses semi-supervised Kmeans to cluster overall samples, which can produce more reliable pseudo labels. Our method can still achieve superior performance (+12 (+12.1%) 1%) on novel classes of tasks while attaining comparable know-class accuracy for most tasks. This indicates that the idea of leveraging part compositions for novel class discovery can perform well both the NCD setting and the GCD setting.
Conclusion
In this work, we present a novel part concept-based framework DNIK for 3D NCD. DNIK leverages part concepts and part-wise relations that are widely shared by both known classes and novel ones. Therefore part concepts and part-wise relations learned from known classes can reinforce the recognition of novel shapes. Comprehensive experiments demonstrate the method overwhelms all baselines consistently and significantly on all metrics. We hope our work can inspire more research on find-grained concept learning and 3D NCD.
Limitations. Part concepts the work are not capable of learning multi-scale geometric parts, thus restricting the representation ability and resulting in sensitive scale parameters in Fig. (b).
10

---

Acknowledgement
This work is supported by the National Natural Science Foundation of China (62271467, U2003109 U21A20515, 62102393, 62206263), China Postdoctoral Science Foundation (2022T1 50639,, 2021M703162), the State Key Laboratory of Robotics and Systems (HIT) SELLRS-202--FF-
and the Fundamental Research Funds for the Central Universities.
References
[1] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9902-9912, 2022.
[2] Elad Amrani, Leonid Karlinsky, and Alex Bronstein. Self-supervised classification network. arXiv preprint arXiv:2103. 10994, 2021.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self supervied vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[4] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et Shapenet: An information-rice -ich 3d model repository. arXiv preprini arXiv: 151200311,, 2015.
[5] Townim Chowdhury, Ali Cheraghian, Sameera Ramasinghe, Sahar Ahmadi, Morteza Saberi, and Shafin Rahman. Few-shot class-incremental learning for 3d point cloud objects. In ECCV, 2022.
[6] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[7] Enrico Fini, Enver Sangineto. Stéphane Lathuilière. Zhun Zhong, Moin Nabi, and Elisa Ricci.
A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9284-9292, 0292,992,222 2021.
[8] Robert Geirhos, förn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelligence, 2(1 1 >:665-673, 2020.
[9] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8401-8409, 2019.
[10] Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6677-671, 2021.
[11] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. CoRR, abs/ 19111991125222 2019.
[12] Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 33: 9995-10006, 2020.
[13] Donald D Hoffman and Whitman Richards. Parts of recognition. Cognition, 18(1-3):65-98 1984.
[14] Yen-Chan Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Conference Track Proceedings. OpenRe- view.net, 2018

---

[15] Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-clase classi- fication without multi-class labels. In International Conference on Learning Representations. 2019.
[16] KJ Joseph, Sujoy Paul, Gaurav Aggarwal, Soma Biswas, Piyush Rai, Kai Han, and Vineeth N Balasubramanian. Spacing loss for discovering novel categories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3761-3766, 2022.
[17] Valentin Khrulkov, Leyla Mirvakhabova, Exgeniya Ustinova, Ivan Oseledets, and Victor Lempit- sky. Hyperbolic image embeddings. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6418-6428 2020.
[18) Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1132-13333 2015.
[19] Wenbin Li, Zhichen Fan, Jing Huo, and Yang Gao. Modeling inter-class and intra-class constraints in novel class discovery. arXiv preprint arXiv:2210.03551, 2022.
[20] Ziyun Li, Jona Otholt, Ben Dai, Di Hu, Christoph Meinel, and Haojin Yang. A closer look at novel class discovery from the labeled set. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.
[21] Weiquan Liu, Minghao Liu, Shijun Zheng, and Cheng Wang. Interpreting hidden semantics in the intermediate layers of 3d point cloud classification neural network. arXiv preprint 2033v:2303.06652, 2023.
[22] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 909-918, 2019.
[23] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supe learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II, pages 604-621, Springer, 2022.
[24] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[25] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[26] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[27] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10901-10911, 2021.
[28] Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. Representation tradeoffs for hyperbolic embeddings. In International conference on machine learning, pages 4460 4469. PMLR, 2018
[29] Mert Bulent Sariyildiz, Yannis Kalantidis, Diane Larlus, and Karteek Alahari. Concept gen- eralization visual representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9629-9639 2021.
[30] Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. In Graph Drawing: 19th International Symposium, GD 2011, Findhoven, The Netherlands, September 21-23, 2011, Revised Selected Papers 19, pages 355-366. Springer, 2012.
12

---

[31] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[32] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances Neural Information Processing Systems 33, 2020, virtual, 2020.
[33] Mikaela Angelina Uy Quang-Hies Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1588-1597, 2019.
[34] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11), 2008.
[35] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7492-7501, 2022.
[36] Hanchen Wang. Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. Proceedings of the IEEE/CVF international conference computer vision, pages 9782-9792, 2021.
[37] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds, Acm Transactions On Graphics (tog), 1-, 2019.
[38] Xin Wei, Xiang Gu, and Jian Sun. Learning generalizable part-based feature representation for 3d point clouds. Advances in Neural Information Processing Systems, 35:29305-29316 2022.
[39] Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category discovery: A baseline study. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 16590-16600, October 2023.
[40] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[41] Muli Yang, Yuehua Zhu, Jiaping Yu, Aming Wu, and Cheng Deng. Divide and conquer: Compositional experts for generalized novel class discovery. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14268-14277, 2022,
[42] Xincheng Yang, Mingze Jin, Weiji He, and Qian Chen. Pointeat: Cross-attention transformer for point cloud. arXiv preprint arXiv:2304. 2.00022000.0012,, 2023.
[43] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 91 9150-9161, 2023.
[44] Bingehen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. Advances in Neural Information Processing Systems, 34: 22982-22994, 2021.
[45] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 16259-
16268, 2021.
[46] Shizhen Zhao and Xiaojuan Qi. Prototypical votenet for few-shot 3d point cloud object detection. In Advances in Neural Information Processing Systems, 2022.
13

---

[47] Tianchen Zhao, Niansong Zhang, Xuefei Ning, He Wang, Li Yi, and Yu Wang. Codedvtr: balebook-baseee sparse voxel transformer with geometric guidance. In JEEE/CVE Conference on Computer Vixion and Patlern Recognition, CVPR 2022, New Orleans, IA, USA, June 18-24, 2022, pages 1425-1434. IEFE, 2022.
[48] Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighbor- hood contrastive learning for novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10867-10875, 2021.
[49] Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9462-9470, 2021.
14

---

A In-depth Analysis of the Method
Feature Visualization of Novel Classes. We show the feature distribution of novel classes learned by all compared methods on the high similarity task of ModelNet in Fig. DTC [ mingles features of different novel classes together and impedes novel class recognition. The feature distributions of AutoNovel [0O, NCL 48), UNO 7 and IIC [1]] are less confusing but there is no clear category boundary for different novel classes, Kmeans+ can produce better intra-class consistency and can separate most novel classes. A possible reason is that pseudo labels by K-means clustering the unlabeled training set can reflect the overall distribution to some extent. However, some shapes incorrectly grouped into the wrong class due to the poor discrimination of learned features. By exploring the part concept compositions of different novel classes, our method can push away the embedding regions of novel classes greater extent compared other methods and achieve much tighter intra-class representations.
2
DTC
AutoNovel
NCL
Ours
UNO
IIC
Kmean+
Figure 8: Learned embeddings visualized by 2D I-SNE. We use the ModelNet high-similarity task for the experiment. Each color denotes a different novel class.
Train Curves. As described in Sec. 3.1, the learned features of compared methods are gradually biased to known classes during the training process and lose generalizability novel classes. To further confirm our analysis, we demonstrate the converging curves of known class accuracy and novel class cluster accuracy during the training of seven methods on the high similarity task of ModelNet in Fig 9 As shown in Fig. 9a the known class accuracy of all methods increases quickly the first 30 epochs and grows slowly after 100 epochs. However. in Fig. 99] with the increase the known class accuracy after 50 epochs, the clustering accuracies for novel classes of UNO, IIC, and Kmeans+ start decrease. This indicates the known class accuracy competes with the novel class clustering accuracy, and features beneficial to known class recognition instead turn out to harm the generalization to novel classes. The clustering accuracy of our method stays stable after 60 epochs. Meanwhile, the trend of the cluster accuracy is consistent with that of the known class accuracy suggesting that learned features on known classes encourage the recognition of novel ones. This good generalization is attributed to the fact that part concepts construct shared embedding space between known and novel classes.
Impact of 3D pre-trained models. 3D data makes things more challenging as the 3D pretrained model does not generalize well. For example, SimGCD [39] uses the powerful DINO 3 model pre-trained on millions of images as the encoder and only fine-tunes very small part of the network during training. This training strategy makes the extracted image features more generalizable, thus producing better results. In fact, previous work [29] suggests that DINO is very important to the generalization to unseen concepts. In contrast, the 3D field lacks both large-scale scale 3D shapes as well as high-quality pre-trained models. In order to evaluate the impact of 3D pre-trained models, we present results using two different selffsupprvised pre-trained backbones (OcCo (66 and CrossPoint (I) under the GCD setting in the Tab. 7 Following SimGCD [39, we only fine-tune the last three layers of the PointNet+ backbone and the classification heads. The results demonstrate that utilizing more generalized features from pre-traine pre-trainig improves the performance of the baseline methods. However,
15

---

0.8
0.7
Cluster ACC(%)
ACC(%)
0.9
0.3
DTC
NCL
IIC
DTC
NCL
IIc
Kmeons
UND
AutoNoval
0.1
Kmesns
UNO
AutoNooel
0.8
100
150
250
100
150
200
250
Epoch
Epoch
(a) Known class
(b) Novel class
Figure 9: (a) The accuracy curves of known classes during training. (b) The clustering accuracy curves of novel classes during training.
Table 7: Result using 3D pre-trained models.
Scratch
OcCo (36]
CrossPoint
UNO+
41.7
42.7
45.3
SimGCD
39
47.2
48.9
51.2
Our
70.6
there remains a substantial gap compared to our method, This indicates that existing 3D pre-trained models have yet learned representations as powerful those captured by DINO 3
B Ablation Study
To further investigate our proposed model components, we conduct additional ablation experiments in Tab. In @2, 3 and @, both Poincareé distance and L proposed encourage sharper part concept activation. Each of these two modules can be used individually to obtain performance gains (+3.7% vs +8.5%, +5.1% vs +7.4%), and they can be used together to achieve an even bigger improvement (+9.5% vs +15.2%). Without Poincareé distance and Lid. using Lac and Led alone can only achieve minor improvements as shown in 5 and @ This is because that part concepts are less diverse and tend to encode the global feature of known classes that are hard to be generalized novel classes. Since novel shapes may contain new part concepts that are not present in known classes, we concatenate the local part features Z with the part composite features Zp. As shown in @ this design can obtain better results on low-similarity novel classes (+0.7% vs + 1.8%). We also evaluate the performance of only using the PRE module as shown in the 0. The results show that using PRE alone can already achieve considerable performance gains (4. (4.55% on average).
Ablation on different known classes number. Training on more known categories can certainly benefit novel class discovery. As shown Tabl increasing the number of known classes leads to improved performance. Because with multiple categories we can learn more diverse part concepts that further will help recognize novel objects better. In the table below, we evaluated the performance by training the method with an increasing number of known categories and the fixed number of novel categories. The results show that training the method with more categories does enhance the recognition of novel shapes.
C Different Backbones
We inspect how different network backbones impact the performance of NCD. The comparisons are conducted on three popular 3D backbones including PointNet++ (25) DGCNN (37. and Point- NeXt (26). Among them, the PointNeXt achieves state-of-the-arr performance for 3D classification and segmentation. We replace the backbone of compared methods with the above network architec-
16

---

Table Ablation experiments. PCB means part concept bank. LGA, Poincaré, Concat and PRE local geometric aggregation, Poincaré distance, concatenate the local part features Z₁ and position relation encoder as shown in Fig. 2.
LGA
PCB
Poincaré
Lod
Lsc
Led
Concat
PRE
High
Low
58.4
48.7
62.1
57.2
63.5
56.1
V
67.9
63.9
V
60.1
51.5
/
59.8
49.3
70.6
63.3
/
/
71.3
65.1
48.8
37.8
52.4
43.7
73.2
66.4
Table 9: The results of different known class number.
ModelNet 5-10
10-10
20-10
30-10
Novel Acc 57.9
61.3
64.0
66.2
tures and evaluate their performance on the ModelNet high similarity task. The results are shown in Tab. IO the known class accuracy of the same 3D NCD method on different backbones does not vary too much (within 1%). On the other hand, the backbone has significant influence on the novel-class performance for all the compared methods. For example, the weaker backbone DGCNN can achieve better novel class accuracy than the other two backbones. This is because DGCNN uses graphs to represent the local features of the point cloud, thus increasing the feature generalization on novel classes. Our method can surpass all the baselines by very large margin (at least +8.9%) on the novel-class accuracy even using weak backbone (PointNet), while attaining comparable good performance on the known-class recognition. This superiority is primarily attributed to the part soncept-based features that are more generalizable to novel classes than other backbones.
Table 10: The accuracy results of different backbones the similarity split 3D NCD task. High, Low denote the novel classes have High, Low semantic similarity with known classes.
Xet
PointNet++
DGCNN
High
Low
Known
High
Low
Known
High
Low
Known
DTC 9
43.8
32.6
93.9
50.9
44.4
94.3
51.3
38.0
94.6
AutoNovel
10
51.6
46.5
96.0
59.0
43.4
96.7
49.0
40.9
96.7
NCL
48
59.9
46.5
95.9
51.3
37.6
97.0
64.3
44.9
95.1
UNO
7
61.6
32.9
95.8
53.0
43.2
93.0
62,9
45.6
96.7
IC
19
62.3
49.7
95.3
57.6
46.0
94.6
64.0
51.8
96,8
Kmeans+
61.2
50.9
55.2
40.7
96.2
64.0
45.5
92.3
Ours (PointNet)
73.2
66.4
73.2
66.4
95.2
73.2
66.4
95.2
Improvement
+10.9
+15.5
-1.2
+14.2
+20.4
-1.8
+8.9
+14.6
-1.6
Ours (PointNet++))
73.2
66,4
95.2
73.2
66.4
95.2
73.2
66.4
95.2
Improvement
+10.9
+15.5
-1.2
+14.2
+20.4
-1.8
+8.9
+14.6
-1.6
Ours (DGCNN)
73.2
66.4
95.2
73.2
66.4
95.2
73.2
66.4
95.2
Improvement
+10.9
+15.5
-1.2
+14.2
+20.4
-1.8
+8.9
+14.6
-1.6
17

---

D Results on Large-Scale Datasets
The large-scale 3D datasets (MVimgNet 433 are available and conduct an MVimgNet50- task as shown in the Tab In this task, we selected 50 classes as known classes and another 100 classes as novel classes from the large-scale MVimgNet dataset. Despite the challenges when increasing the number of unknown classes, our method can still outperform baselines by a large margin.
Table 11: The results on large-scale datasets.
Novel
Known
AutoNovel [TO]
12.8
47.8
NCL 48
11.3
53.9
UNO
10.7
44.3
IIC 19
12.5
58.2
Ours
43,9
64.0
Improvement
+31.1
+5.8
E Estimate Cluster Numbers
We fixed the number of unseen classes following the baseline methods [0] However, the number of novel classes can be estimated by following the strategy in AutoNovel [10 and GCD [35 where semi-supervised k-means and a validation classes. We estimate the number of clusters using the above method on the ModelNet similarity task as shown Tab. 12 The results show the estimation method may slightly underestimate the novel class number when similarities between novel classes and known classes are high. This is because some highly related classes are clustered together. Mcanwhile, it slightly overestimates the number of novel classes when similarities decrease.
can be used select the optimal number of unseen
Table 12: Estimate cluster numbers on modelnet dataset.
High
Medium
Low
Ground Truth
26
27
27
Ours
24
30
31
F
Implementation Details
In all experiments, we adopt PointNet 24 as the backbone for point-wise feature learning and sample 1024 points from the original point clouds as inputs. We use scaling, rotation, jittering, and translation as data augmentation generate two augmented point clouds for both labeled and unlabeled inputs respectively during training. The network is optimized with the Adam optimizer with an initial learning rate of 0.001. In our experiments, we first train the network for 30 epochs with labeled dataset D' to learn the part concept bank, then train all networks for another 220 epochs with both labeled and unlabeled datasets. We specify both the feature vector size D and the dimension of the part concepts to 256. The overall method is implemented in PyTorc, and we conduct all experiments on computer with an NVIDIA Tesla V100 GPU. Our method takes 1.26M parameters, 14.7G memory cost and 0.43s train time per batch measured ModelNet. Readers encouraged to refer to the released code for detailed implementation.
18


Reviews:
[{"review_id": "JiOPZiRrBI", "reviewer": "Reviewer_zZT9", "summary": "This paper addresses the problem of 3D NCD (novel class discovery). The objective is to discover novel classes by leveraging information learned from the known classes. This paper proposes a novel framework, DNIK, for 3D novel class discovery (3D NCD) by leveraging part concepts and part-wise relations learned from known classes to reinforce the recognition of novel shapes. The framework consists of a learnable part concept bank, a local geometric aggregation module, a part relation encoding module, and three constraints to facilitate effective part concept learning. ", "questions": "(1) How to ensure the features from PRE include the position relationship of each part? The feature extraction by PRE seems like a process through a black box.\n(2) How are the Nq parts like in the initial point cloud? How different/similar are they? The initialization may also heavily affect the results. e.g., if the initial parts are too similar, they are unlikely to be well separated in the end. However, in the beginning, we have little (or no) control over this.", "limitations": "The paper has described the potential limitation of multi-scale objects, not mentioning much about the societal impact, but I did not see any major problem here.", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "(1) The proposed part concept bank and part relation encoding module can effectively bridge the gaps between known and novel shapes and mitigate feature bias.\n\n(2) The experiments show that the proposed method outperforms all baselines consistently and significantly on all metrics. \n\n(3) The paper is generally well-written and easy to follow.", "weaknesses": "(1) The unseen class number is assumed to be known, which makes the method less practical.\n\n(2) The effectiveness of the proposed PRE module is not well demonstrated. The performance is not shown by using only the part position feature from the PRE. Therefore, it is not clear about the individual role of PCB and PRE. It would be good to at least ablate the effectiveness that only uses PRE in Table 4.\n\n(3) The study of NCD has been extended to consider the case where the unlabelled data contains objects from seen and unseen classes [A]. It is more convincing to also show results under this more general and practical case. \n\n[A] Vaze et al, Generalized Category Discovery, CVPR 2022\n\n(4) Each part in Part Set Q has the same number of points, that is, K neighbors, which may be dataset dependent and affected by the scale of the objects, while a fixed value of K=64 is selected in the paper. This is unlikely to generalize well to other datasets and objects of different scales. It would be good to show how the method works on instances from the same categories but of different scales. More investigation on this is expected.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and useful suggestions! We are glad you think the proposed method \"can bridge the gaps between known and novel shapes effectively\" and \"outperforms baselines consistently\". We address your thoughts point by point below.\n\n> Q1: The unseen class number is assumed to be known, which makes the method less practical.\n\nA1: Thank you. We fixed the number of unseen classes following the baseline methods. However, the number of novel classes can be estimated by following the strategy in AutoNovel and GCD, where semi-supervised k-means and a validation set can be used to select the optimal number of unseen classes. **We estimate the number of clusters using the above method on the ModelNet similarity task as shown in the table below.** The results show the estimation method may slightly underestimate the novel class number when similarities between novel classes and known classes are high. This is because some highly related classes are clustered together. Meanwhile, it slightly overestimates the number of novel classes when similarities decrease. We will incorporate discussions of this finding in the revised manuscript to enrich the analysis.\n\n|              | High | Medium | Low |\n|:-------------|:----:|:------:|:---:|\n| Ground Truth |  26  |   27   | 27  |\n| Ours         |  24  |   30   | 31  |\n\n> Q2: How to ensure the features from PRE include the position relationship of each part? The feature extraction by PRE seems like a process through a black box.  The performance is not shown by using only the part position feature from the PRE. Therefore, it is not clear about the individual role of PCB and PRE. \n\nA2: Thank you. We employ the standard Transformer encoder \\[1\\] for the PRE module (see model/PDG\\*.py Line#188 in the supplementary code for reference). Similar schemes to encode relations of local patches can be found in prior works \\[2-4\\], where the Transformer encoder can capture spatial relationships between patches effectively. In revision, we will add an illustration in the supplementary for clarity.\n\n**Following the reviewer’s suggestion, we evaluate the performance of only using the PRE module as shown in the table below.** The results show that using PRE alone can already achieve considerable performance gains (4.75% on average).\n\n|                |   High   |   Low    |\n|:---------------|:--------:|:--------:|\n| Baseline       |   48.8   |   37.8   |\n| Baseline + PRE |   52.4   |   43.7   |\n| Ours           | **73.2** | **66.4** |\n\nAlso, combining the PRE module with other modules can enhance the performance ( + 1.6% on average) as shown in Table 4-9. We will add the result to Table 4 in the revision.\n\n- \\[1\\] Vaswani et al. Attention is all you need. NeurIPS 2017.\n\n- \\[2\\] Pang et al., Masked Autoencoders for Point Cloud Self-supervised Learning. ECCV 2022\n\n- \\[3\\] Yu et al., Point-bert: Pre-training 3d point cloud transformers with masked point modeling. CVPR 2022\n\n- \\[4\\] Zhang et al., Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training. NeurIPS 2022\n\n> Q3: The study of NCD has been extended to generalized category discovery. It is more convincing to also show results under this more general and practical case.\n\nA3: We appreciate the reviewer raising this thoughtful point. Please see our response to this question in the General Response.\n\n> Q4: Each part in Part Set Q has the same number of points, which may be dataset dependent and affected by the scale. This is unlikely to generalize well to other datasets and objects of different scales. It would be good to show how the method works on instances from the same categories but of different scales.\n\nA4: We thank the reviewer for raising this important issue regarding scale invariance. To address it, we would like to highlight three aspects. \n- Firstly, our method is invariant to different initial scales, because all samples are normalized to a canonical scale in our pipeline as a preliminary step. \n- Secondly, **the results in the table below** show that our method is able to maintain comparable performance even for unseen point densities. In this experiment, we evaluated the robustness of our approach by testing the trained model on testing shapes with varying point densities using the ModelNet-C dataset. We believe the performance can be further improved once we train the method with shapes of diverse densities.\n- Thirdly, our method can even work well for more challenging cross-domain cases as shown in Tab. 3. Note that multi-view scans from Co3D and ScanObjectNN present more challenging variances, such as point noises, and occlusions. Our method still achieves consistent improvements (more than +9.5%) in these cross-domain tests.\n\n| Density(point number) | 700  | 800  | 900  | 1024 |\n|:----------------------|:----:|:----:|:----:|:----:|\n| Novel Acc             | 69.5 | 72.9 | 72.6 | 73.2 |\n\n> Q5: How are the Nq parts like in the initial point cloud?  The initialization may also heavily affect the results. e.g., if the initial parts are too similar, they are unlikely to be well separated in the end. However, in the beginning, we have little control over this.\n\nA5: Thank you for raising this important point. Our model is robust to part initialization. The key is that farthest point sampling (FPS) ensures the initial parts are as far as possible therefore sampled parts can cover different local geometric structures across the whole shape and are diverse enough. Moreover, since the sampled parts for the same shape are different across epochs during training, the model can learn to identify parts with diverse structures.  We evaluate the variances of five random experiments on the high-similarity task on the ModelNet. The variance is 0.0228, while our improvement is 13.6. Therefore, different initializations do not have a big impact on the results. We will add an illustration of the sampled parts and report variances for the results in the supplementary.\n"}, {"review_id": "Zyt6c8Zg9O", "reviewer": "Reviewer_qzzG", "summary": "This work presents a framework, called Decompose Novel Into Known parts (DNIK), that addresses the challenge of 3D Novel Class Discovery (NCD) – identifying new classes from an unlabeled dataset using the knowledge of known classes. Current methods, heavily biased towards known classes, struggle to generalize to novel classes. By leveraging more generalizable geometric parts across different classes, DNIK mitigates this issue. It constructs a part concept bank encoding rich geometric patterns from known classes, which is used to represent novel 3D shapes as part concept compositions, thus facilitating cross-category generalization. DNIK also leverages part-wise spatial relations for improved recognition. The method has been tested through three 3D NCD tasks, consistently outperforming state-of-the-art baselines.\n\n\n---- after rebuttal ----\n\nAs the author's rebuttal resolved some of concerns, I raised my score to 5. However, I still feel the studied task is a bit simple, and also there are several spaces to improve for the current manuscript. I will not fight for its acceptance. ", "questions": "Please address the concerns raised above. ", "limitations": "Line 319~320 analyzed one minor limitation. I feel there are potential more:\n\n1. if the part are not sharable between different categories, such as lamp -> chair, table -> faucet, can the framework still handle?\n\n2. the only shown application is recognition which limits the practical use of the proposed framework.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The studied direction is important as we need to understand parts well to play with 3D objects generalizable. This paper takes a step towards open 3D object recognition via part understanding. Overall, the components used in the proposed framework are sound and reasonable. The paper is easy to follow. \n\n\nExtensive results shown in Table 1 & 2 demonstrate the strength of the proposed method. The proposed DNIK generally achieved state-of-the-art performance. Some detailed ablation studies are included in Table 4. The cross-domain task is interesting to see the transfer performance. ", "weaknesses": "Utilizing the part are sharable across different 3D object categories are studied in the previous literature [1,2]. In those paper, they exploited \"harder\" task, such as segmentation. As the proposed framework can address novel class classification via known part concepts. Can the framework be extended to ground where is those known parts in novel object? Or other applications beyond object recognition?\n\n[1] Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories\n[2] 3D Compositional Zero-Shot Learning with DeCompositional Consensus\n\nHow the framework handle two different object categories with limited shared parts, such as airplane and chair? Will the framework train on multiple object categories benefit novel object discovery? It would be good to include some failure cases to analyze and provide readers a sense for the limitation of the proposed framework. \n\nL46~47 said the framework can help use part relation features. Can the framework be extended to discovery part relationship? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your attentive comments! We are glad you thought “the studied direction is important” and “the components \\[...\\] are sound and reasonable.” We address your feedback point by point below.\n\n>Q1: Previous literature exploited \"harder\" task, such as segmentation. Can the framework be extended to ground where is those known parts in novel object?\n\nA1: We appreciate you raising the possibility of extending our framework, but find the specific comparisons to supervised techniques [1,2] highly inapt and misleading. \n- First, these works fundamentally address different problems as the novel class discovery. \n- Second, despite their similarities in using shape parts, technical details are fundamentally different, and two related works [1,2] cannot be directly used for the 3D NCD task.\nFor example, the part segmentation produced by [1] has no semantic information, making it less useful for knowledge transfer. \nThe goal of [2] is compositional zero-shot learning, where the part combinations of novel classes are predefined.\n- Thirdly, we would like to clarify that our method actually faces a harder challenge compared to the cited literature [1,2]. \nThose works rely on ground truth part annotations (PartNet), while **we learn parts in a completely unsupervised way without any part labels**, which also makes our approach more widely applicable. \n\n\nWe believe the method can be extended to part discovery. \nWe provide extensive part visualization results of both seen shapes and unseen shapes in Supplementary Fig.I. \nNote that our method can roughly locate shape parts in unseen shapes learned from seen shapes without part annotation.\nTherefore our method can be extended to ground the known parts in novel shapes.\nWe encourage the reviewer to see more detail there. \nIn the revision, we will add some discussions about this comment.\n\n- \\[1\\] Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories\n\n- \\[2\\] 3D Compositional Zero-Shot Learning with DeCompositional Consensus\n\n\n>Q2: How the framework handle two different object categories with limited shared parts, such as airplane and chair? If the part are not sharable between different categories, such as lamp -> chair, table -> faucet, can the framework still handle?\n\nA2: We thank the reviewer for raising this important case. \n- Firstly, as stated in Line 165, the part concept activations are distinct for dissimilar objects like airplanes and chairs, providing discriminative information to distinguish the novel class. We also concatenate the original part features so that novel parts dissimilar to existing concepts can be properly handled. \n- Secondly, as shown in Table 2’s low similarity tasks, our method achieves substantial gains over baselines (17.5% on average) when known and novel classes have limited similarities. This validates that the generalized part concepts are also beneficial for knowledge transfer across categories with few shared parts.\n\n>Q3: Will the framework train on multiple object categories benefit novel object discovery?\n\nA3: Yes, training on more object categories can certainly benefit novel object discovery. As shown in Table 1, increasing the number of known classes leads to improved performance. Because with multiple categories we can learn more diverse part concepts that further will help us recognize novel objects better.  **In the table below, we evaluated the performance by training the method with an increasing number of known categories and the fixed number of novel categories.**  The results show that training the method with more categories does enhance the recognition of novel shapes.\n\n| ModelNet  | 5-10 | 10-10 | 20-10 | 30-10 |\n|:----------|:----:|:-----:|:-----:|:-----:|\n| Novel Acc | 57.9 | 61.3  | 64.0  | 66.2  |\n\n>Q4: It would be good to include some failure cases to analyze and provide readers a sense for the limitation of the proposed framework.\n\nA4: Thank you for emphasizing the need for failure case analysis. Our model does have some challenging cases in handling nearly identical parts, such as chairs and benches containing similar seating surfaces, legs and arms, and tables and desks sharing analogous tabletops and supporting legs. These situations test the model’s ability to discriminate. We will add those failure cases in the revision. We greatly appreciate your valuable guidance on analyzing and presenting the model’s limitations concisely.\n\n>Q5: Line.46-47 said the framework can help use part relation features. Can the framework be extended to discovery part relationship?\n\nA5: The PRE module is a point cloud transformer based on local patches, following prior works like \\[1\\-4\\]. These studies have shown that self-attention can capture spatial relationships between patches effectively. We propose PRE only to complement global shape information that part concepts alone cannot precisely encode and benefit the NCD task. Note that part relations are implicitly encoded and we don’t see how the framework can be extended to discover explicit part relationships. We agree that explicitly modeling part relationships is an exciting direction for future work.\n\n- \\[1\\] Pang et al., Masked Autoencoders for Point Cloud Self-supervised Learning. ECCV 2022\n\n- \\[2\\] Yu et al., Point-bert: Pre-training 3d point cloud transformers with masked point modeling. CVPR 2022\n\n- \\[3\\] Zhang et al., Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training. NeurIPS 2022\n\n- \\[4\\] Yang et al., PointCAT: Cross-Attention Transformer for point cloud. 2023\n\n>Q6: If the part are not sharable between different categories, such as lamp -> chair, table -> faucet, can the framework still handle?\n\nA6: Thank you for raising this case. Please see Q2 for the response. \n"}, {"review_id": "KDiIbSvDtn", "reviewer": "Reviewer_mbrM", "summary": "This paper tackles the problem of novel category discovery in the 3D shape recognition domain, a framework leveraging the 3D parts and the part-wise relation is proposed which the motivation is learning the parts from the known classes could help the model capture more transferrable features or concepts for the novel categories.\nThis motivation is validated using experiments, and overall the framework shows better performance than some baselines.", "questions": "1. In the v1 version of SimGCD fig 10 [R4], it is shown that the accuracy on known classes first increases and then drops while the novel class accuracy keeps improving, this contradicts the observation made in this paper, I am wondering if this is because of the setting (generalized category discovery v.s. novel category discovery), the data (2D v.s. 3D), or the number of categories(200 v.s. 7)? Consider this observation is the motivation for this paper, this question will be the biggest concern of mine.\n\n\n[R4] https://arxiv.org/pdf/2211.11727v1.pdf", "limitations": "I think the main limitation is that the number of categories is small, thus the conclusion made based on these small datasets may not be generalizable to larger scale datasets.\n\nOverall I think this paper is very clear, and could be of interest for the community, however the concerns I raises in the questions should be addressed first.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The idea of decomposing a category into parts is interesting.\n2. I like the organization of this paper, starts with an analysis of the problem of previou method, and then proposed new ones based on the analysis.\n3. The paper also explored a bit on the design choices for 3D novel category discovery, which could be helpful.", "weaknesses": "1. This paper still considers the novel category discovery problem while a more generlized setting exists, generalized category discovery [R1, R2], I would suggest the paper to include more discussion and experiment on this generalized setting.\n2. It seems that the total number of categories in the datasets are quite small compared to 2D NCD, I am wondering if Objaverse [R3] can be used for this task?\n\n\n[R1] Generalized Category Discovery, CVPR 2022\n[R2] Parametric Classification for Generalized Category Discovery: A Baseline Study, arXiv.\n[R3] https://objaverse.allenai.org/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful observations on the generalized category discovery (GCD) problem. Your careful analysis has given us many new perspectives to consider! We sincerely appreciate you thought \"the idea \\[...\\] is interesting\" and \"like the organization of our paper\". We address your thoughts point by point below.\n\n>Q1: This paper still considers the novel category discovery problem while a more generalized setting exists, generalized category discovery \\[R1, R2\\], I would suggest the paper to include more discussion and experiment on this generalized setting.\n\nA1: We appreciate the reviewer raising this thoughtful point. Please see our response to this question in the General Response.\n\n>Q2: It seems that the total number of categories in the datasets is quite small compared to 2D NCD, I am wondering if Objaverse \\[R3\\] can be used for this task.\n\nA2: We appreciate the reviewer’s suggestion to leverage larger-scale 3D datasets. The large-scale 3D datasets (MVimgNet and Objaverse) were not available when we wrote this paper. Due to time constraints, we only conducted **an MVimgNet50-100 task following the SimGCD as shown in the table below**. In this task, we selected 50 classes as known classes and another 100 classes as novel classes from the large-scale MVimgNet dataset. Despite the challenges when increasing the number of unknown classes, our method can still outperform baselines by a large margin. We will add more experiments in future versions.\n\n|           |      Novel      |     Known     |\n|:----------|:---------------:|:-------------:|\n| AutoNovel |      12.8       |     47.8      |\n| NCL       |      11.3       |     53.9      |\n| UNO       |      10.7       |     44.3      |\n| IIC       |      12.5       |     58.2      |\n| Ours      | **43.9** + 31.1 | **64.0** +5.8 |\n\n\n\n>Q3: In the v1 version of SimGCD fig 10 \\[R4\\], it is shown that the accuracy on known classes first increases and then drops while the novel class accuracy keeps improving, this contradicts the observation made in this paper, I am wondering if this is because of the setting (generalized category discovery v.s. novel category discovery), the data (2D v.s. 3D), or the number of categories(200 v.s. 7)? Consider this observation is the motivation for this paper, this question will be the biggest concern of mine.\n\nA3: We thank the reviewer for the insightful observation. Our analysis is as follows.\n\n- Firstly, **the different setting of GCD and NCD is not the cause** as our method basically follows the GCD setting. \n\n- Secondly, **3D data makes things more challenging** as the 3D pretrained model does not generalize well. For example, SimGCD[1] uses the powerful DINO model pre-trained on millions of images as the encoder and only fine-tunes a very small part of the network during training. This training strategy makes the extracted image features more generalizable, thus producing better results. In fact, previous work[2] suggests that DINO is very important to the generalization to unseen concepts. In contrast, the 3D field lacks both large-scale 3D shapes as well as high-quality pre-trained models. We also tried using some pre-trained models like OcCo[3] but did not achieve good results.\n\n - Thirdly, we would like to clarify that **the inconsistent accuracy trend is not caused by the number of categories in the toy experiment.** Supplementary Fig. III shows the accuracy curves of various methods on the ModelNet with more categories (40 classes), we can observe the same decreasing novel accuracy curve as the toy experiment.\n\nWe believe that **the main reason is still the feature bias problem** described in Sec.3.1. As visualized in the t-SNE of Fig. 6 and Supplementary Fig. II, the learned feature distributions of different novel classes by baseline methods are highly overlapping and confused, indicating a lack of discriminability. Therefore the accuracy of pseudo-labels generated by clustering algorithms like Sinkhorn and K-Means decreases with the training proceeding **as shown in the table below**. In Section 3 of SimGCD, the authors also had the conclusion that \"the key to previous parametric classifiers’ degraded performance is unreliable pseudo labels\". In fact, we attempted to improve the quality of pseudo labels with various methods, such as self-distillation and entropy regularization, but we did not manage to improve the accuracy of pseudo labels to good quality in the case of feature bias.\n\nWe would like to highlight that our method uses part concepts to obtain more generalizable features without the requirement of any large-scale pre-trained model like SimGCD. The results show this achieves better performance, presenting an interesting alternative direction for learning generically informative 3D shape representations. In the revision, we will add this discussion.  \n\n| Epoch    |  75  | 125  | 175  | 225  |\n|:---------|:----:|:----:|:----:|:----:|\n| Kmeans   | 58.1 | 52.6 | 49.3 | 47.2 |\n| Sinkhorn | 50.9 | 48.6 | 47.8 | 45.1 |\n\n- \\[1\\]Wen. et al., A Simple Parametric Classification Baseline for Generalized Category Discovery.  arxiv 2022\n\n- \\[2\\] Sariyildiz. et al., Concept Generalization in Visual Representation Learning. ICCV 2021\n\n- \\[3\\] Wang et al., Unsupervised Point Cloud Pre-training via Occlusion Completion. CVPR 2021.\n"}, {"review_id": "Ed2FYmqRBJ", "reviewer": "Reviewer_dey2", "summary": "In this work, they address 3D novel class discovery (NCD) that discovers novel classes from an unlabeled dataset by leveraging the knowledge of disjoint known classes. The key challenge of 3D NCD is that learned features by known class recognition are heavily biased and hinder generalization to novel classes. Since geometric parts are more generalizable across different classes, the authors propose to decompose novel into known parts, coined DNIK, to mitigate the above problems. ", "questions": "see the weakness ", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and motivation (separate instances into repeatable parts) is pretty good. \n2. The model design is reasonable and the improvement is satisfied. \n3. The experimental analysis is sufficient. ", "weaknesses": "1. This paper does not consider hierarchical part representation. \n2. why does Part Relation Encoder work for novel classes ? \n3. Does the improved representation works for some scene level tasks, such as novel class segmentation for point cloud ? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your attentive comments! We are glad you thought “the paper is well written and motivation is pretty good”. We address your feedback point by point below.\n\n>Q1: This paper does not consider hierarchical part representation.\n\nA1: We thank the reviewer for the suggestive comment. The focus of this paper is leveraging part concepts to obtain generalized features for improved novel category discovery therefore we did not consider hierarchical parts in the present. We do agree that using multi-scale parts can further improve the performance as noted in the Limitations section and suggested by recent works exploring hierarchical point cloud parts\\[1,2\\] for 3D recognition. In future work, we will explore this idea further.\n\n- \\[1\\] Wei et al., Multi-scale Geometry-aware Transformer for 3D Point Cloud Classification. 2023.\n\n- \\[2\\] Yang et al., PointCAT: Cross-Attention Transformer for point cloud. 2023\n\n>Q2: Why does Part Relation Encoder work for novel classes?\n\nA2: Thank you for raising this important question. The PRE module is a point cloud transformer based on local patches following prior works like \\[1-4\\]. These studies show that self-attention can capture spatial relationships between patches effectively. We propose PRE to complement global shape information that part concepts alone cannot precisely encode. For example, a table and desk contain similar parts like flat top and leg. But PRE may capture finer spatial relationships like a table’s peripheral leg layout versus a desk’s set-back legs. So despite similar parts, PRE’s positional encodings help distinguish subtle structural differences across object classes. As shown in Table 4-9, the PRE can further enhance the performance ( + 1.6% on average).\n\n- \\[1\\] Pang et al., Masked Autoencoders for Point Cloud Self-supervised Learning. ECCV 2022\n\n- \\[2\\] Yu et al., Point-bert: Pre-training 3d point cloud transformers with masked point modeling. CVPR 2022\n\n- \\[3\\] Zhang et al., Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training. NeurIPS 2022\n\n- \\[4\\] Yang et al., PointCAT: Cross-Attention Transformer for point cloud. 2023\n\n>Q3: Does the improved representation work for some scene-level tasks, such as novel class segmentation for point cloud?\n\nA3: Novel class segmentation is an intriguing new direction, with very few explorations\\[1\\] so far even in 2D vision. As demonstrated by our good performance on 3D novel class discovery, learned shape parts and their compositions can bridge the gap between seen shapes and unseen shapes and generalize learned features to domains with semantic shifts. For semantic segmentation, this assumption still holds, and novel semantic shapes can also be seen as compositions of shared shape parts. Therefore, we believe integrating our approach with part-based segmentation is a promising path to novel class segmentation in future work. We appreciate this thoughtful suggestion that brings insights advancing our approach to novel tasks.\n\n- \\[1\\] Zhao et al., Novel class discovery in semantic segmentation. CVPR 2022."}, {"review_id": "PE9eeD8COf", "reviewer": "Reviewer_eZuw", "summary": "- This paper proposes a novel part-based algorithm for 3D novel class discovery (NCD). Authors propose Decompose Novel Into Known parts (DNIK) that leverages knowledge about parts of known objects to discover novel classes.\n- Authors identify that the main problem with learning 3D features for object discovery is that the features are heavily biased towards the known classes. This work shows that this can be prevented by using the well known part-based modeling approach. \n- DNIK is trained to learn a part concept bank that can be used to compose different known and novel objects. Three different regularizations are proposed to prevent the collapse of this part bank.\n- Extensive experiments show the deficiencies of existing 2D NCD literature and the effectiveness of DNIK to overcome these issues.", "questions": "- Can Fig. 5b, 5c be combined in to one plot? Two separate plots makes it hard to understand what values of K, Q are being used for each of these. \n- Legend for Fig. 6 is missing. ", "limitations": "Authors have addressed the limitations adequately. ", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper builds on an age old, part-based models in visual recognition and shows impressive improvements over single holistic representations currently in use in the 2D category discovery methods. As shown in the experiments this has significant merits for identifying and grouping new classes. \n- The paper writing was smooth and was very easy to follow. An experienced engineer would be able to reproduce the work with the given details.\n- Authors support all the claims made in the paper with experiments on real world datasets or on toy problems. Sec 3.1 and Fig. 4.a were particularly interesting to understand what the authors were trying to convey.\n- The effectiveness of the method was shown with the impressive experimental results.", "weaknesses": "- While the problem tackled by the authors is relevant and important, the setup adopted by the authors is outdated. Generalized category discovery, as done in [1][2] is a more realistic setting and it is not clear why the current method is not suited for this setup or the authors advice against it? It is my strong suggestion to the authors to answer this question and compare with the relevant work (cited below) to justify this work among existing literature.\n- The toy example in Sec. 3.1 is not fair for the following reason. In L86, the setup states that the classes in known and unknown sets share some similarities, but in the example authors choose {table, sofa, stool} as known and {chair, bench, bathtub, plant} as unknown objects. In this case, bathtub and plant do not share any commonalities with the known objects. It looks like the authors intentionally exaggerate the problem to make their point. While this is acceptable, it is not clear how much of a serious issue is the \"overfitting to the known classes\" problem. \n- Author propose to use supervised contrastive loss to learn more parts from the known shapes. The motivation and explanation doesn't justify why this would be the case. In L204 authors pool the features along the last dimension which basically is a \"shape\" feature as opposed to a \"part\" feature. It is not clear how the contrastive loss helps learn more part features when the loss is being applied on the \"shape\" features.\n- Table 4 demonstrates the performance improvement for each of the proposed components but experiments to demonstrate that show that the regularizations on the part features actually operate as the authors claim is missing. What happens when diversity loss is missing? Do all the part features in the bank collapse to fewer representations? This can be quantified by cosine similarity between the part features. Similar analysis on the remaining two regularization terms is warranted.\n\n[1] Sagar Vaze, Kai Han, Andrea Vedaldi, Andrew Zisserman, Generalized Category Discovery, CVPR 2022.\n[2] Sai Saketh Rambhatla, Rama Chellappa, Abhinav Shrivastava, The pursuit of knowledge: Discovering and localizing novel categories using dual memory, ICCV 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and helpful feedback! We sincerely appreciate your positive feedback that our work \"has impressive improvements\" and \"is very easy to follow\". Your suggestions provide valuable guidance for us to improve this study. We address your thoughts point by point below.\n\n>Q1: Generalized category discovery is a more realistic setting and it's not clear why the method is not suited for this setup or the authors advice against it? It is my strong suggestion to the authors to answer this question and compare with the relevant work to justify this work among existing literature.\n\nA1: We appreciate the reviewer raising this thoughtful point. Please see our response to this question in the General Response.\n\n>Q2: The toy example in Sec. 3.1 is not fair for the following reason. In L86, the setup states that the classes in known and unknown sets share some similarities, but in the example authors choose table, sofa, stool as known and chair, bench, bathtub, plant as unknown objects. In this case, bathtub and plant do not share any commonalities with the known objects. It looks like the authors intentionally exaggerate the problem to make their point. While this is acceptable, it is not clear how much of a serious issue is the \"overfitting to the known classes\" problem.\n\nA2: We are sorry for the confusion. Our intention for the toy experiment is to create an illustrative example to help readers better understand the feature bias issue. But we would like to clarify that the feature bias issue illustrated in the toy example is widely present in real datasets, rather than a contrived corner case. As evidenced in Supplementary Fig. III on the high similarity ModelNet task, the same trend of decreasing novel accuracy can be observed, validating that the problem is widely present in real-world datasets. Furthermore, Fig. 6 and Supplementary Fig. 2 visualize baseline features learned on ModelNet, showing high confusion between different novel classes due to a lack of discriminability.\n\n>Q3: Author propose to use supervised contrastive loss to learn more parts from the known shapes. The motivation and explanation doesn’t justify why this would be the case. In L204 authors pool the features along the last dimension which basically is a \"shape\" feature as opposed to a \"part\" feature. It is not clear how the contrastive loss helps learn more part features when the loss is being applied on the \"shape\" features.\n\nA3: We apologize for the unclear motivation behind the contrastive loss for discovering more part concepts. Basically, our assumption is that **the accumulated part activation map should be more similar for objects from the same class than objects from different classes.** Therefore encouraging the inter-class discrepancy of their part concept activation maps (i.e., the contrastive loss) can activate more part concepts so that the part activation maps between different classes can be pushed apart as much as possible. In our method, we use the shape concept map *T* (i.e., the \"shape\" feature) to represent the accumulated part concept activation maps of an object.\n\nIf we do not consider the contrastive loss, only **a small subset of part concepts** are needed to distinguish different known classes based on their shape concept maps as shown in Fig. 4(c) red box. For example, the maps only need to activate concepts like \\[surface, table corner\\] and \\[surface, stool back\\] for the recognition of table and stool, while other part concepts like \\[table leg\\] and \\[stool base\\] are ignored. Without the contrastive loss, the concept bank lacks the impetus to learn additional concepts beyond those minimally needed parts to distinguish known classes.\n\nBy pushing maps of different classes apart and enforcing the similarity of shape concept maps within each known class via contrastive loss, more part concepts get discovered to make the histograms more distinct and discriminative so contrastive loss is further minimized. For instance, concepts like \\[table leg\\] and \\[stool base\\] will be learned to make the shape maps clearly separable as shown in Fig. 4(c) green box.\n\n>Q4: Table 4 demonstrates the performance improvement for each of the proposed components but experiments to demonstrate that show that the regularizations on the part features actually operate as the authors claim is missing. What happens when diversity loss is missing?  This can be quantified by cosine similarity between the part features. \n\nA4: We sincerely appreciate the valuable comments from the reviewer. Firstly, as shown in Tab.4-\\[6,7,8\\] and Supplementary Tab. I-\\[3,4,5,6\\], we detailed the effect of each loss term on the results. Secondly, following the reviewer’s suggestion, we added the following table to demonstrate the average cosine similarity between part concepts when using different losses:\n\n|                   | w/o ( $L_{sd}$, $L_{sc}$, $L_{cd}$ ) | w/ $L_{sd}$ | w/ $L_{sc}$ | w/ $L_{cd}$ |\n|:------------------|:-------:|:--------------------:|:--------------------:|:--------------------:|\n| Cosine Similarity |  0.251  |        0.022         |        0.045         |        -0.007        |\n\nIt can be seen that adding the diversity loss $L_{cd}$ leads to a significant decrease in the similarity between part concepts, indicating the decreased distinction between concepts and verifying the effectiveness of the diversity loss. Adding the remaining two regularization losses ( $L_{sd}, L_{sc}$ ) also decreases the similarities to varying degrees. These results further validate the positive effects of the various loss terms on model training. We sincerely appreciate the reviewer's suggestions and will include these additional analyses in our revised manuscript to more comprehensively demonstrate the efficacy of our method.\n\n>Q5: Can Fig. 5b, c be combined? Legend for Fig. 6 is missing.\n\nA5:  We will fix the confusion in Fig. 5 and Fig. 6 in the revision.  \n\n\n"}]

Author rebuttal:
# **General Response**

Dear reviewers and AC,

We sincerely appreciate your valuable time and efforts spent reviewing our manuscript. We are grateful that reviewers found "the proposed method outperforms all baselines consistently and significantly on all metrics" (Reviewer zZT9), "the studied direction is important" (Reviewer qzzG), "the idea \[...\] is interesting" (Reviewer mbrM), "motivation is pretty good" (Reviewer dey2), and "impressive improvements over single holistic representations" (Reviewer eZuw). We also thank reviewers for their appraisal of the paper’s organization and easy-to-understanding. We respond to each reviewer in a separate response window and address the common concern below. 

**Several reviewers would like to see more analysis about the setting of our method and that of the generalized category discovery (GCD) task. Actually, our method follows the setting of the GCD task** and is fundamentally designed to address the more challenging GCD task, where test samples include both known and novel classes. As described in line#221, during inference, test samples from both known and novel classes are passed to both classifiers $\phi^l$ and $\phi^u$ to obtain the corresponding logits, then the two logits are concatenated and fed into a softmax layer to obtain the class-wise probability along all categories. Thus **all the results of our method reported in the paper are based on the GCD setting**. The compared methods in the paper follow the NCD setting where known and novel classes have separate classification heads when computing accuracy. This setting benefits these methods by avoiding confusion between known and novel classes. Therefore results of the NCD setting are better than that of the GCD setting for compared methods as shown in the table below. We also show the results of our method using the NCD setting for reference.

We do not deliberately distinguish these two settings (NCD and GCD) in the paper because we focus on solving the key feature bias problem present in both NCD and GCD. However, we agree with the reviewers that specifically reporting GCD results can better showcase our method’s capabilities. To that end, we conducted additional GCD task experiments on the high similarity task of the ModelNet dataset and compare our method with the GCD methods, i.e., AutoNovel+, UNO+, GCD\[1\], SimGCD\[2\]. We extend AutoNovel and UNO for GCD tasks (denoted as AutoNovel+ and UNO+) by concatenating logits of known classes and unknown classes following \[2\]. **The results are shown in the table below.** We can see our method greatly overwhelms the extension of 2D GCD methods in novel class recognition and also achieves comparable results on known classes. Note that since the pre-trained DINO model used in the GCD and SimGCD are not available in 3D, these two methods are trained in an end-to-end way.

|      NCD settings            |      Novel      |     Known      |
|:------------------|:---------------:|:--------------:|
| AutoNovel         |      39.2       |      94.8      |
| NCL               |      56.6       |    **95.2**    |
| UNO               |      49.8       |      95.1      |
| IIC               |      59.6       |      95.0      |
| **Ours**| **74.8** + 15.2 |   95.0 - 0.2   |

|    GCD settings          |      Novel      |     Known      |
|:------------------|:---------------:|:--------------:|
| AutoNovel+        |      34.3       |      93.4      |
| UNO+              |      42.5       |      86.3      |
| GCD \[1\]              |      57.7       |      92.6      |
| SimGCD \[2\]           |      46.1       |      94.2      |
| **Ours** | **73.2** + 15.5 | **95.2** + 1.0 |

In the revision, we will add a separate section dedicated to GCD
experiments to thoroughly illustrate the strengths of our method under
this important and widely applicable setting. We hope our responses
address the reviewers’ concerns.

Thank you very much.

Best regards, 

Authors.

- \[1\]Vaze et al., Generalized Category Discovery, CVPR 2022

- \[2\]Wen et al., A Simple Parametric Classification Baseline for Generalized Category Discovery. arxiv 2022

Meta-review: This submission introduces a method for uncovering new categories of 3D data by utilizing knowledge of parts from familiar objects. Initially, the paper garnered varied feedback. However, after the authors provided clarifications in their rebuttal, all reviewers expressed positive sentiments regarding the approach. Consequently, the AC has decided to accept the submission. For the finalized camera-ready version, it would be beneficial to include additional experimental results.
Decision: Accept (poster)