SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Following the literature [48, 10, 7], we construct two linear classifiers  $\phi^l$  and  $\phi^u$ . We employ a standard cross-entropy loss  $L_{ce}$  to optimize the known class classifier  $\phi^l$  and a self-classification loss  $L_{self}$  for training novel class classifier  $\phi^u$ . Loss  $L_{self}$  minimizes the cross entropy between novel shapes and their augmentation with the assumption that the distribution of each novel class is uniform and is formulated following [2]. In summary, the overall training objective loss function in our model is:
$$
L = L _ {c e} + L _ {s e l f} + \lambda_ {1} \cdot L _ {s d} + \lambda_ {2} \cdot L _ {s c} + \lambda_ {3} \cdot L _ {c d}, \tag {9}
$$
where loss  $L_{sd}$  in Eq.(6),  $L_{sc}$  in Eq.(7) and  $L_{cd}$  in Eq.(8) enforce basic assumptions of part concepts. Weight term  $\lambda_*$  balances different losses and is set to (0.1, 0.1, 0.1). During inference, part composite features of a test sample  $Z_c$  are passed through both classifiers to obtain the corresponding logits, then the two logits are concatenated and fed into a softmax layer to obtain the class-wise probability.
4.1 Evaluation Datasets for 3D NCD
We construct three NCD tasks to evaluate the performance of 3D NCD based on four different kinds of 3D datasets covering CAD shapes (i.e., ModelNet40 [40] and ShapeNetCore [4]), and indoor objects constructed using 3D scans and multi-view images (i.e., ScanObjectNN [33] and Co3D [27]).
Random Split 3D NCD focuses on evaluating the impact of a different number of known and novel classes on the 3D NCD following [7]. The task is denoted with  $DATASET-K-U$ , where  $K$  is the number of known classes and the other  $U$  classes as novel ones.
Similarity Split 3D NCD. The semantic similarity between known and novel classes can greatly affect the performance of NCD. A highly semantically similar label set can provide more useful information for discovering novel classes. Based on the transfer leakage metric in [20], we split novel classes as three sets (i.e., High, Medium, and Low) according to the descending similarity w.r.t known classes.
Cross Domain 3D NCD assesses the performance of 3D NCD on cross-domain shape recognition and reflects the real-world scenarios where novel shapes may come from different domains. Therefore, we select known and novel classes with semantic similarity from different datasets respectively. The corresponding tasks are denoted with (known class dataset  $\rightarrow$  novel class dataset).
4.2 Comparison with SOTA methods
To our best knowledge, there is no previous attempt at the 3D NCD problem. So we extend different kinds of SOTA 2D NCD methods to 3D NCD, including DTC [9], AutoNovel [10], NCL [48], UNO [7], and IIC [19]. In addition, training a novel class classifier with pseudo labels generated by K-means can achieve good results and we also compare with it, denoting as  $Kmeans+$ . We implement the above baselines by substituting the image-based backbone with a 3D one (i.e., PointNet) for 3D NCD.
Results on the Random Split 3D NCD Task are shown in Tab. 1. For the known class recognition, all compared methods perform well, but the performance on the novel classes diverges a lot. When the number of known classes is large, the network can inherit more knowledge for the novel class recognition. The NCL and IIC can achieve better results on most datasets since they force the separation between known and novel classes while learning similarity relationships within the novel classes, so the learned features are more discriminative for the novel classes. Our method can gain consistent improvements on novel classes of all tasks ( $+11.7\%$  on average) while attaining the best know-class accuracy for most tasks  $(5/6)$ . This demonstrates that part concepts can effectively bridge the gaps between known and novel shapes thus mitigating the feature bias. When the number of known classes is small, the performance of compared methods drops significantly. By contrast, our method decreases much less and still achieves the best results ( $+12.5\%$  increases on average). We
Table 1: The quantitative results of the random split 3D NCD task. The results are averaged among five random trials. The best results are in bold, and the second-best ones are underlined.
ModelNet 30-10	ModelNet 10-30	Co3D 30-20	Co3D 20-30	ShapeNet 40-15	ShapeNet 20-35
Novel	Known	Novel	Known	Novel	Known	Novel	Known	Novel	Known	Novel	Known
DTC [9]	49.7	79.6	31.1	92.5	31.1	77.0	21.9	84.6	40.8	80.8	31.0	89.9
AutoNovel [10]	46.6	88.1	46.7	93.8	52.2	80.6	42.0	86.1	36.1	82.2	35.2	91.4
NCL [48]	61.3	89.1	51.2	93.7	52.4	80.2	39.5	89.6	44.4	83.4	42.2	87.7
UNO [7]	59.3	86.7	45.4	93.2	48.6	80.1	41.6	87.3	45.5	82.2	28.6	90.7
IIC [19]	62.7	87.6	49.7	94.2	45.9	80.0	37.4	86.7	44.3	81.5	29.1	90.6
Kmeans+	59.9	83.0	41.6	92.2	52.1	76.6	43.8	84.8	48.1	80.9	36.6	90.9
Ours	72.0	88.9	63.0	96.0	61.0	85.0	58.5	90.1	62.6	84.7	53.2	92.4
Improvement	+9.3	-0.2	+11.8	+1.3	+8.6	+4.4	+14.7	+0.5	+14.5	+1.3	+11.0	+1.0
Table 2: The accuracy results on the similarity split 3D NCD task. High, Medium, Low denote the novel classes have High, Medium, Low semantic similarity with known classes.
ModelNet	Co3D	ShapeNet
High	Medium	Low	Known	High	Medium	Low	Known	High	Medium	Low
DTC [9]	28.3	39.6	32.5	95.7	41.3	35.2	26.7	89.2	37.6	36.6	36.6
AutoNovel [10]	39.2	42.2	44.6	94.8	57.1	54.7	34.7	87.5	39.2	38.3	39.2
NCL [48]	56.6	56.5	45.1	95.2	57.4	53.1	39.7	89.8	45.3	36.9	38.7
UNO [7]	49.8	47.4	45.5	95.1	49.4	48.1	31.3	87.9	35.4	43.9	31.2
IIC [19]	59.6	56.0	38.9	93.9	55.8	48.8	43.1	84.5	47.1	38.4	28.7
Kmeans+	58.5	54.2	39.6	95.0	54.3	49.5	37.3	84.6	44.9	45.9	33.8
Ours	73.2	68.3	66.4	95.2	69.3	65.2	56.9	93.3	64.7	54.6	57.1
Improvement	+13.6	+11.8	+20.9	-0.5	+11.9	+10.5	+13.8	+3.5	+17.6	+8.7	+17.9
conjecture that the part concept bank can learn more local features for sharing than the other methods using global features.
Results on the Similarity Split 3D NCD Task are presented in Tab. 2. NCL, UNO, IIC, and Kmeans+ can achieve better results on novel classes with high similarities. However, the performance of these methods drops significantly  $(-13.5\%)$  on average) with the decrease in semantic similarity. The method that achieves the best results on High-similarity novel classes cannot obtain the best on Medium and Low ones simultaneously. This indicates that the learned knowledge of these methods is biased towards known classes with highly similar geometry while lacking generalizability to other novel ones. By contrast, our method achieves large improvements for all similarities  $(+14.3\%, +11.8\%, +17.5\%)$  on average) while reducing the gap between different similarity tasks. This is because part concepts can show up in different shapes, e.g., the back of a sofa and a chair in Fig. 1 and align all shapes into the part space, allowing for generalization even for low-similarity novel classes.
Results on the Cross-Domain 3D NCD Task are reported in Tab. 3. We can observe that compared methods lack robustness on cross-domain shapes and perform poorly. Our method performs well among different datasets and outperforms existing methods by a large margin  $(+16.3\%)$  on average). We believe the good performance is owing to the fact that part concepts are robust to data domains and can be generalized to different kinds of data.
In this part, we conduct all ablations on the ModelNet similarity split 3D NCD. Readers can find more experiments in the supplementary.
Model components. In Tab. 4-②, adding the part concept bank without the LGA only achieves a very small improvement compared to the baseline, which indicates part concepts without positional invariance are hard to fit novel classes. In ③ and ④, LGA can ensure better generalization over novel classes, and the part concept bank can align the part features from known and novel classes into the same space. Therefore they can collaborate to boost the performance for a large margin (5.8% and 3.9%). In ⑤, using the Poincaré metric can yield much sharper concept activations and make different part concepts more discriminative, therefore raising the performance (+3.7% and +8.5%). In ⑥, self-distillation loss  $L_{sd}$  encourages each part concept to learn specific patterns to avoid collapses and contributes to an average 6.25% gain in accuracy. In ⑦ and ⑧, loss  $L_{sc}$  and  $L_{cd}$  force the network to
Table 3: The results on cross-domain task. M. and Scan. are ModelNet and ScanObjectNN.
M.→Co3D	M.→Scan.	Co3D→Scan.
Novel	Novel	Novel
DTC [9]	26.6	29.6	25.2
AutoNovel [10]	26.8	36.7	41.6
NCL [48]	35.2	39.0	37.4
UNO [7]	27.7	43.9	35.0
IIC [19]	30.8	39.8	35.4
Kmeans+	29.0	41.8	26.0
Ours	64.0 +28.8	53.4 +9.5	52.5 +10.5
Table 4: Ablation experiments. PCB means part concept bank. LGA, Poincaré, and PRE are in Fig. 2.
LGA	PCB	Poincaré	Lsd	Lsc	Lcd	PRE	High	Low
①	48.8	37.9
②	✓	49.9	40.3
③	✓	52.6	44.8
④	✓	✓	58.4	48.7
⑤	✓	✓	✓	62.1	57.2
⑥	✓	✓	✓	✓	67.9	63.9
⑦	✓	✓	✓	✓	✓	70.1	64.8
⑧	✓	✓	✓	✓	✓	✓	71.3	65.1
⑨	✓	✓	✓	✓	✓	✓	✓	73.2	66.4
Figure 5: Hyperparameter evaluations on (a) the number of part concepts, (b) the number of points  $K$  contained in each part, and (c) the number of split parts  $N^q$  of each shape.
Figure 6: Visualization of the embedding spaces of novel classes with t-SNE for different methods.
learn more diverse part concepts, which is more helpful for high-similarity novel classes (3.4% vs 1.2%), since their concept activation maps are more similar (e.g., chair and stool). By comparing ⑤ and ⑧, we can see the three proposed constraints  $(L_{sd}, L_{sc},$  and  $L_{cd})$  are essential to the final good performance and strengthen the results by +8.8% and +7.9%. In ⑨, the PRE can further incorporate discriminative position relation features to enhance the performance (+1.6% on average).
Hyperparameters. In Fig. 5(a), we conduct experiments on different numbers of part concepts, i.e.,  $M$ . Considering part concepts should increase with known classes, we set  $M = N^{c} \times C^{l}$ , When  $N^c\leq 3$ , the results are poor, which indicates that few part concepts cannot adequately encode all part patterns in known classes. However, the performance also drops when  $N^c >6$ . This may be because too many concepts are harder to be optimized.
The number of points contained in each overlapped part determines the part size, therefore affecting the generalization and discrimination abilities. The discrimination ability of parts decreases when the size is reduced and the generalization ability improves. Specifically, small parts are hard to contain meaningful local geometric patterns, while large parts may contain special structures that are biased towards known classes as shown in Fig. 5(b). Our approach achieves the best results when  $K = 64$ .
Since the part set of a shape is split by FPS, few parts lead to large variations in the part sets of similar shapes, resulting in inconsistent part composite features. In Fig. 5(c), the result will decrease continuously when  $N^q < 128$ . Our method achieves the best performance when  $N^q = 256$ .
Visualizations of the Feature Embedding with t-SNE [34] are shown in Fig. 6. For the NCL [48] and IIC [19], the features of different novel classes are mixed up in the embedding space, while ours can push away embedding regions of different novel classes to a greater extent.
Part Concept Visualization. In order to reveal what part concepts can learn, we show concept activation maps between some part concepts and part features in Fig. 7. We can see that part concepts can activate similar semantic parts of both known and novel shapes, such as legs, arms, bottles, backrests, planes, and screens. This confirms that part concepts can effectively generalize knowledge learned from known classes to novel classes and represent novel classes as the composition of part concepts. For example, a chair can be represented as {legs, arms, a backrest, and a plane}. We notice that part concepts can be robustly discovered and activated even if geometric structures are slightly different, e.g., arms of chairs, sofas, and benches.
4.4 Extension for Generalized Class Discovery
$DNIK$  can also be extended for generalized class discovery (GCD) [35], for which the unlabeled dataset  $D^u$  contains 3D shapes from both known and novel classes. In order to adapt  $DNIK$  for this task, we concatenate the known class classifier  $\phi^l$  and the novel class classifier  $\phi^u$  into a unified
Figure 7: Visualization of different part concepts and their corresponding activations on different shapes of known and novel classes. Red indicates higher activations and similarities.
Table 5: The quantitative results of the random split 3D GCD task.
ModelNet 30-10	ModelNet 10-30	Co3D 30-20	Co3D 20-30
Novel	Known	Novel	Known	Novel	Known	Novel	Known
AutoNovel+ [10]	38.6	84.7	34.9	76.5	33.1	78.9	34.2	84.4
UNO+ [7]	42.1	82.1	39.3	87.2	45.2	73.8	43.3	85.9
GCD [19]	50.6	89.2	49.2	89.5	46.4	77.3	45.8	87.3
SimGCD [39]	44.8	89.0	41.9	87.8	39.7	78.8	33.3	85.0
Ours	69.1	88.9	68.4	92.1	56.7	83.2	54.1	87.0
Improvement	+18.5	-0.1	+19.2	+2.6	+10.3	+4.3	+8.3	-0.3
Table 6: The accuracy results on the similarity split 3D GCD task.
ModelNet	Co3D
High	Low	Known	High	Low	Known
AutoNovel+ [10]	32.6	33.6	94.3	50.8	32.1	89.6
UNO+ [7]	41.7	36.5	88.5	52.1	41.9	88.8
GCD [19]	58.3	54.7	95.7	56.2	46.0	89.2
SimGCD	47.2	43.5	94.0	54.3	42.1	82.4
Ours	70.6	68.2	95.9	62.5	53.9	88.0
Improvement	+12.3	+13.5	+0.2	+6.3	+7.9	-1.6
classifier. All samples from both the labeled dataset and unlabeled dataset go through the unified classifier to obtain the class-wise probability. Both  $L_{ce}$  and  $L_{self}$  are now used to optimize the unified classifier rather than the novel class classifier of the NCD setting.
We extend two 2D GCD methods to 3D GCD, including  $GCD$  [35], SimGCD [39], and also extend AutoNovel [10] and UNO [7] for GCD tasks (denoted as AutoNovel+ and UNO+) by concatenating logits of known classes and novel classes following SimGCD [39]. Results are shown in Tab. 5 and Tab. 6. It can be seen that all compared methods show decreased results compared to the NCD tasks in Tab. 1 and Tab. 2. This indicates that samples of known classes in the unlabeled dataset  $D^{u}$  introduce noise into the training and confuse the classifier. GCD [35] can achieve better novel class accuracy  $(+5.6\%)$  than other compared methods because it uses semi-supervised Kmeans to cluster overall samples, which can produce more reliable pseudo labels. Our method can still achieve superior performance  $(+12.1\%)$  on novel classes of all tasks while attaining comparable know-class accuracy for most tasks. This indicates that the idea of leveraging part compositions for novel class discovery can perform well in both the NCD setting and the GCD setting.
In this work, we present a novel part concept-based framework  $DNIK$  for 3D NCD. DNIK leverages part concepts and part-wise relations that are widely shared by both known classes and novel ones. Therefore part concepts and part-wise relations learned from known classes can reinforce the recognition of novel shapes. Comprehensive experiments demonstrate the method overwhelms all baselines consistently and significantly on all metrics. We hope our work can inspire more research on find-grained concept learning and 3D NCD.
Limitations. Part concepts in the work are not capable of learning multi-scale geometric parts, thus restricting the representation ability and resulting in sensitive scale parameters in Fig. 5 (b).
This work is supported by the National Natural Science Foundation of China (62271467, U2003109, U21A20515, 62102393, 62206263), China Postdoctoral Science Foundation (2022T150639, 2021M703162), the State Key Laboratory of Robotics and Systems (HIT) (SKLRS-2022-KF-11), and the Fundamental Research Funds for the Central Universities.
[1] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9902-9912, 2022.
[2] Elad Amrani, Leonid Karlinsky, and Alex Bronstein. Self-supervised classification network. arXiv preprint arXiv:2103.10994, 2021.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[4] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
[5] Townim Chowdhury, Ali Cheraghian, Sameera Ramasinghe, Sahar Ahmadi, Morteza Saberi, and Shafin Rahman. Few-shot class-incremental learning for 3d point cloud objects. In ECCV, 2022.
[6] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[7] Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9284-9292, 2021.
[8] Robert Geirhos, Jorn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673, 2020.
[9] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8401-8409, 2019.
[10] Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6767-6781, 2021.
[11] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. CoRR, abs/1911.05722, 2019.
[12] Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 33: 9995-10006, 2020.
[13] Donald D Hoffman and Whitman A Richards. Parts of recognition. Cognition, 18(1-3):65-96, 1984.
[14] Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[47] Tianchen Zhao, Niansong Zhang, Xuefei Ning, He Wang, Li Yi, and Yu Wang. Codedvtr: Codebook-based sparse voxel transformer with geometric guidance. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 1425-1434. IEEE, 2022.
[48] Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10867-10875, 2021.
[49] Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9462-9470, 2021.
A In-depth Analysis of the Method
Feature Visualization of Novel Classes. We show the feature distribution of novel classes learned by all compared methods on the high similarity task of ModelNet in Fig. 8. DTC [9] mingles features of different novel classes together and impedes novel class recognition. The feature distributions of AutoNovel [10], NCL [48], UNO [7] and IIC [19] are less confusing but there is no clear category boundary for different novel classes. Kmeans+ can produce better intra-class consistency and can separate most novel classes. A possible reason is that pseudo labels by K-means clustering on the unlabeled training set can reflect the overall distribution to some extent. However, some shapes are incorrectly grouped into the wrong class due to the poor discrimination of learned features. By exploring the part concept compositions of different novel classes, our method can push away the embedding regions of novel classes to a greater extent compared to other methods and achieve much tighter intra-class representations.
AutoNovel
Kmean+
Figure 8: Learned embeddings visualized by 2D t-SNE. We use the ModelNet high-similarity task for the experiment. Each color denotes a different novel class.
Train Curves. As described in Sec. 3.1, the learned features of compared methods are gradually biased to known classes during the training process and lose generalizability to novel classes. To further confirm our analysis, we demonstrate the converging curves of known class accuracy and novel class cluster accuracy during the training of seven methods on the high similarity task of ModelNet in Fig 9. As shown in Fig. 9a, the known class accuracy of all methods increases quickly in the first 30 epochs and grows slowly after 100 epochs. However, in Fig. 9b, with the increase in the known class accuracy after 50 epochs, the clustering accuracies for novel classes of UNO, IIC, and Kmeans+ start to decrease. This indicates the known class accuracy competes with the novel class clustering accuracy, and features beneficial to known class recognition instead turn out to harm the generalization to novel classes. The clustering accuracy of our method stays stable after 60 epochs. Meanwhile, the trend of the cluster accuracy is consistent with that of the known class accuracy suggesting that learned features on known classes encourage the recognition of novel ones. This good generalization is attributed to the fact that part concepts construct a shared embedding space between known and novel classes.
Impact of 3D pre-trained models. 3D data makes things more challenging as the 3D pretrained model does not generalize well. For example, SimGCD [39] uses the powerful DINO [3] model pre-trained on millions of images as the encoder and only fine-tunes a very small part of the network during training. This training strategy makes the extracted image features more generalizable, thus producing better results. In fact, previous work [29] suggests that DINO is very important to the generalization to unseen concepts. In contrast, the 3D field lacks both large-scale 3D shapes as well as high-quality pre-trained models. In order to evaluate the impact of 3D pre-trained models, we present results using two different self-supervised pre-trained backbones (OcCo [36] and CrossPoint [1]) under the GCD setting in the Tab. 7. Following SimGCD [39], we only fine-tune the last three layers of the PointNet++ backbone and the classification heads. The results demonstrate that utilizing more generalized features from pre-training improves the performance of the baseline methods. However,
(a) Known class
(b) Novel class
Figure 9: (a) The accuracy curves of known classes during training. (b) The clustering accuracy curves of novel classes during training.
Table 7: Result of using 3D pre-trained models.
Scratch	OcCo [36]	CrossPoint [1]
UNO+	41.7	42.7	45.3
SimGCD [39]	47.2	48.9	51.2
Our	70.6	-	-
there remains a substantial gap compared to our method. This indicates that existing 3D pre-trained models have not yet learned representations as powerful as those captured by DINO [3].
B Ablation Study