SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Decomposing Novel into Known: Part Concept Learning For 3D Novel Class Discovery
Tingyu Weng Jun Xiao Haiyong Jiang*
School of AI
wengtingyu18@mails.ucas.ac.cn,{xiaojun,haiyong.jiang}@ucas.ac.cn
In this work, we address 3D novel class discovery (NCD) that discovers novel classes from an unlabeled dataset by leveraging the knowledge of disjoint known classes. The key challenge of 3D NCD is that learned features by known class recognition are heavily biased and hinder generalization to novel classes. Since geometric parts are more generalizable across different classes, we propose to decompose novel into known parts, coined  $DNIK$ , to mitigate the above problems.  $DNIK$  learns a part concept bank encoding rich part geometric patterns from known classes so that novel 3D shapes can be represented as part concept compositions to facilitate cross-category generalization. Moreover, we formulate three constraints on part concepts to ensure diverse part concepts without collapsing. A part relation encoding module (PRE) is also developed to leverage part-wise spatial relations for better recognition. We construct three 3D NCD tasks for evaluation and extensive experiments show that our method achieves significantly superior results than SOTA baselines  $(+11.7\%, +14.1\%, \text{and} +16.3\%)$  improvements on average for three tasks, respectively). Code and data will be released.
3D recognition is fundamental for 3D tasks and plays a critical role in many applications such as robotics and autonomous vehicles. Recently, deep learning-based 3D recognition [24, 25, 45] has achieved remarkable progress in this task. Unfortunately, most methods are limited to annotated classes in the training set (i.e., known classes), and cannot recognize novel classes like human beings, which limits the robustness of 3D recognition in the real world. Therefore, it is crucial and valuable to investigate the 3D novel class discovery task (NCD) so that 3D recognition can discover novel classes by leveraging the knowledge of known classes.
Though 2D NCD has been extensively investigated, a straightforward extension of 2D NCD methods [7, 19] to 3D NCD usually performs poorly due to the feature bias problem [32], i.e., the learned features are heavily biased towards known classes and have difficulty in generalizing to novel classes. The main factors lie in two aspects. First, learned feature representations are usually dominated by known classes while being oblivious to novel classes. Second, 3D recognition usually focuses on shortcut features [8, 12] only capture a special global structure or local regions specific to known classes [21] as shown in Fig. 1.
In this work, we tackle the above problems based on the observation that humans can subconsciously decompose shapes into familiar part concepts, which along with their spatial relations provide effective cues for recognizing different novel shapes [13, 18]. Therefore part concepts can bridge the gaps between known shapes and those novel ones with a shared part space, thus alleviating the
*Haiyong is the corresponding author.
Figure 1: (a) The Grad-CAM [31] of a known shape (table). 3D recognition methods focus on shortcut features representing a special global structure or local region. (b) Novel shape can be represented as a part concept composition along with part-wise spatial relations.
feature bias problem in 2D NCD extensions. On the other hand, the compositions and spatial relations between parts make it easier to recognize novel shapes. For example, as shown in Fig. 1(b), a novel class, e.g., a chair, contains familiar concepts like plane, leg, and backrest observed in known classes (e.g., tables and sofas), therefore we can easily recognize the chair according to its part concepts and part spatial relations.
Along with this idea, we present a novel part concept-based framework coined  $DNIK$  (decompose novel into known parts) for 3D NCD as shown in Fig. 2. Specifically, we first construct a learnable part concept bank that encodes rich geometric patterns from known classes. Then  $DNIK$  can project translation-invariant part-level features of an input shape into the spanning space of part concepts so that shapes from different categories can be represented as part-concept composition features sharing the common embedding space. Therefore learned features by known class recognition can be generalized to novel classes. We further leverage part-wise spatial relations of input shapes with a part relation encoding module (PRE) for discriminative part relation features. These two kinds of features are then combined for 3D shape recognition. To avoid the collapse of the part concept bank and encourage diverse part concepts across classes, three constraints on part concepts are also considered. Moreover, we construct three 3D NCD tasks for the evaluation of different scenarios and also extend our method to the generalized class discovery task (GCD) [35].
Our contributions can be summarized as follows: (i) the analysis of biased features and the necessity of part concepts in 3D NCD; (ii) a part concept-based framework leveraging both part concept features and part-wise relations for 3D NCD; (iii) the design of three necessary constraints on part concepts to facilitate effective part concept learning; (iv) various 3D NCD and GCD tasks for the evaluation of different scenarios. Extensive experiments show that our method significantly outperforms baseline methods on all 3D NCD and GCD tasks and can efficiently recognize novel shapes.
Novel Class Discovery. As there is no existing work on 3D NCD, we mainly review related works on 2D NCD. Related works can be classified into two categories. The first category of methods [9, 14, 15] usually first train a classification network on the known dataset and then cluster the novel data according to the prediction or output features of the pre-trained network. However, learned features in this way are highly biased toward known classes, leading to performance bottlenecks in novel classes. By contrast, the other line of works [7, 10] explores both known and novel samples for NCD during the clustering phase. Therefore feature representation can be shared between known and novel classes and allow for better generalization to novel classes. AutoNovel [10] utilizes both known and novel samples for self-supervised representation learning and joint known class classification and NCD. Following works further bridge the gap between known and novel classes by mixing up known and novel samples [49], and introducing consistency between similar objects [48] and predictions from two different branches [44]. On the other hand, UNO [7] unifies the learning of both known and novel classes using a single cross-entropy loss based on optimal transports between pseudo-label assignments. Following this paradigm, the similarity relation of samples between known and novel classes is further explored by a spacing loss in the embedding space [16], compositional experts [41], and symmetric KL divergence [19]. Recently NCD has been extended to the more practical general
Figure 2: The overall architecture. A point cloud and its augmentation are fed into a backbone network to encode point-wise features  $Z$  and grouped into part set  $Q$  respectively. As illustrated in the pink box, we propose a local geometric aggregation (LGA) module to learn position-invariant part feature  $Z_{l}$  and calculate the concept activation map  $S$  use the Poincaré distance between  $Z_{l}$  and learnable part concept bank  $\mathcal{P}$ . Then we can construct part composite features  $Z_{p}$  according to the Eq. 2. We also propose a position relation encoder (PRE) module to extract the part position feature  $Z_{r}$  as illustrated in the orange box. Finally,  $Z_{p}$  and  $Z_{r}$  are concatenated and fed into an MLP for recognition.
category discovery task (GCD) [35], which assumes that unlabeled datasets contain both known and novel classes. [35] and [39] tackle this problem by using semi-supervised K-means and purely parameterized models respectively. In this work, we address 3D NCD and GCD by introducing part concepts and part-wise relations as the bridge between known classes and novel ones.
Part-based 3D Recognition. Shape parts are the basic building blocks for 3D shapes [22, 4] and play an important role in both human perception and various 3D recognition tasks. For example, Zhao et al. [46] represent repeatable shape parts as prototypes to improve few-shot 3D object detection. Chowdhury et al. [5] learn part information as a robust prior for incremental learning. Wei et al. [38] align part-based features to reduce the geometry shifts across different domains. Zhao et al. [47] propose a part codebook-based self-attention to leverage the generalized geometric information for better voxel-based feature extractors. Inspired by these works, we learn a part concept bank to map features of known and novel classes into a shared space.
This work aims at 3D NCD that learns to group 3D shapes in the unlabeled set into several classes by leveraging the knowledge of known classes in a labeled dataset. Basically, 3D NCD assumes the availability of two datasets: an unlabeled dataset  $D^{u} = \{X_{i}^{u}\}_{i = 1}^{N^{u}}$  with 3D shapes sampled from novel classes  $C^u$  and a labeled dataset  $D^{l} = \{(X_{i}^{l},y_{i}^{l})\}_{i = 1}^{N^{l}}$ , where  $X_{i}^{l}$  denotes a 3D shape and  $y_{i}^{l}\in C^{l}$  is the corresponding categories label. The classes in the two datasets are disjoint, therefore we have  $C^l\cap C^u = \emptyset$ . However, shapes from  $D^{l}$  and  $D^{u}$  usually endow with some similarities. For example, both a chair and a desk have legs and plane surfaces. Following the literature [7, 10, 48], we assume the number of novel classes  $C^u$  is known a prior.
3.1 Analysis on the Extension of 2D NCD Methods
As there is no existing work on 3D NCD, we instead extend a mainstream 2D NCD method UNO [7] to the 3D domain as the baseline. UNO learns a shared feature extractor  $f$  and two linear classifiers  $\phi^l$  and  $\phi^u$  with  $C^l$  and  $C^u$  output neurons. The classifiers  $\phi^l$  and  $\phi^u$  are optimized with a cross-entropy loss using labels of known classes and pseudo labels of novel classes generated by a clustering method (i.e., Sinkhorn-Knopp [6]). However, naive extensions of 2D NCD to the 3D domain work badly for two reasons. First, feature representation is usually dominated by known classes, while oblivious to novel classes. Second, shape-level features extracted from 3D networks confront serious
Figure 3: Left: The training curves of UNO [7] based methods. Right: Cosine similarities between features of novel classes and shape or part-level concepts.
shortcuts as illustrated in Fig. 1. Thus learned features are heavily biased toward the known classes and are hard to serve novel classes.
We conduct a toy experiment to verify our analysis. We chose table, sofa, stool as known classes and chair, bench, bathtub, plant as novel classes from ModelNet [40]. The first two novel classes have similar shape parts with the known classes, e.g., legs, while the last two do not. To reveal the wane and wax between known and novel classes during optimization, we investigated performance curves of UNO [7] in Fig. 3 Left. In the first 60 epochs, training 3D recognition with known class supervision can help novel class discovery. Therefore there exist some sharing feature representations between two different sets of classes and UNO can achieve better accuracy on novel classes than unsupervised clustering (i.e., w/o the cross-entropy loss). However, with the training proceeding, the performance in novel classes starts to decrease and can even be worse than unsupervised clustering, indicating that learned features are gradually biased to known classes and lose generalizability [32].
Fig. 3 Right shows the average cosine similarity between the shape-level or part-level concepts of known classes and the shape or part features of novel classes learned from the trained UNO method [7]. Shape features are direct outputs of the feature extractor while part features denote pooling features of overlapped parts split from shapes. We then define the shape-level concepts by averaging the shape features of each known class and part-level concepts by clustering the part features of known classes using K-means. For shape-level concepts, the cosine similarities of chairs and benches are low and comparable to those of bathtubs and plants, which indicates that shape features cannot be shared between known and novel classes. However, part-level concepts can provide higher cosine similarities and reflect similar properties between known and novel classes better.
3.2 Part Concepts for Cross-category Generalization
In this work, we approach 3D NCD with a learnable part concept bank shared across different classes. The overall framework is shown in Fig 2. The basic observation is that geometric patterns and shape parts, i.e., part concepts, are more likely to occur in different categories of shapes as analyzed in Sec. 3.1. We first learn part concepts from shapes of known classes in order to deliver more generalizable feature representation and reduce biases when facing novel classes. Then we leverage the relative positions of different parts to further enhance the recognition of novel classes.
Learning Part-level Features. As we intend to learn a set of part concepts representing generalizable shape parts, we first encode part-level features inspired by the paradigm of PointMAE [23]. Specifically, we extract point-wise features  $Z \in \mathbb{R}^{L \times D}$  for each point in the input point cloud  $X \in \mathbb{R}^{L \times 3}$  with an off-the-shelf 3D backbone, e.g., PointNet [24], where  $L$  denotes the number of 3D points and  $D$  is the dimension of point cloud features. Then we use farthest point sampling (FPS) to sample  $N^q$  points as the centroid seeds and group  $K$  neighboring points as overlapped parts  $Q = \{Q_i\}_{i=1}^{N^q}$ . Part set  $Q_i = \{z_{i1}, \dots, z_{iK}\} \in \mathbb{R}^{K \times D}$  includes features at its centroid  $z_{i1}$  followed by point features of its  $K-1$  nearest neighbors  $\{z_{i2}, \dots, z_{iK}\}$ . Intuitively, we can obtain part-level features by pooling point features within each part  $Q_i$ , but part features obtained in this way are dependent on the part location and may hinder the learning of shared part concepts at different positions. To cope with this problem, we propose a local geometric aggregation (LGA) module following [42, 26] to decouple the spatial location and aggregate part-level features  $Z_l \in \mathbb{R}^{N^q \times D}$  by
$$
Z _ {l _ {i}} = \psi \left(\max  \left(\frac {Q _ {i} - z _ {i 1}}{\sigma}\right)\right), \forall i \in \{1, \dots , N ^ {q} \}, \tag {1}
$$
where  $\sigma$  is the scalar standard deviation across channels, and  $\psi$  denotes a multi-layer perception (MLP) with channels  $(D\times \frac{D}{2}\times D)$  to encode the local geometry features.
Part Concept learning. After obtaining part-level features  $Z_{l}$ , we use them to learn shared part concepts among categories. First, we construct a learnable part concept bank denoted with  $\mathcal{P} = \{P_m\}_{m=1}^M$ , where part concept  $P_{m} \in \mathbb{R}^{1 \times D}$  encodes features of different shape parts and can represent a plane, corners, legs as illustrated on the top of Fig. 2. In our implementation, part concepts are initialized with a Gaussian distribution  $N(\mathbf{0}, \mathbf{I})$ . We project part-level features  $Z_{l}$  into the spanning space of part concepts  $P$  according to their respective similarities. Then part-level features  $Z_{l}$  are approximated by part composition features  $Z_{p} \in \mathbb{R}^{N_{q} \times D}$  as follows:
$$
Z _ {p _ {i}} = \sum_ {j} S _ {i j} P _ {j}, \tag {2}
$$
where  $S \in \mathbb{R}^{N^q \times M}$  measures the similarity between part features  $Z_{l}$  and the part concept bank  $P$ . The concept activation map  $S$  is defined as:
$$
S = h \left(d \left(Z _ {l}, P ^ {T}\right)\right), Z _ {l} \leftarrow W _ {z} \cdot Z _ {l}, P \leftarrow W _ {p} \cdot P, \tag {3}
$$
where  $W_{z}, W_{p} \in \mathbb{R}^{D \times D_{p}}$  are learnable weight matrixes for feature dimension reduction,  $h(\cdot)$  is a softmax function to produce normalized probabilities along each row, and  $d(\cdot, \cdot)$  is a distance function that compares part features and part concepts. To ensure that the geometric knowledge of each part feature can be learned by a particular part concept, distance  $d(\cdot, \cdot)$  should produce significant similarity differences between part features and different part concepts. A common choice of  $d(\cdot, \cdot)$  is to use dot product similarity [38] or cos similarity [11] defined in the Euclidean space. However, as shown in Fig. 4(a), we found that the Euclidean distance functions lead to uniform activations of part features on different part concepts. Different from the Euclidean space where the distance grows linearly, the distance between part features and concepts grows exponentially in the hyperbolic space [17, 30, 28], which can produce sharp concept activation maps (see Fig. 4(a)). Therefore, we project features into the hyperbolic space using Poincaré sphere model following [17] and the distance function between  $Z_{l}$  and  $P$  in Poincaré manifolds is given by:
$$
d \left(Z _ {l}, P\right) = \operatorname {a r c o s h} \left(\frac {2 \| Z _ {l} - P \| _ {2}}{\left(1 - \| Z _ {l} \| _ {2}\right) \left(1 - \| P \| _ {2}\right)} + 1\right). \tag {4}
$$
Though learned part concepts can cover most shape parts in novel classes, there still exist some novel part concepts that are not contained in known classes. In this case, the concept activation map between a part and part concepts  $P$  is likely to be uniform and results in ambiguous composite features. In our method, we further concatenate the local part features  $Z_{l}$  with the part composite features  $Z_{p}$  to confront this problem.
Part Relation Encoder. As part composite features  $Z_{p}$  only consider position-independent compositions of part concepts and neglect mutual position relations between different parts that usually deliver shape structures as shown in Fig. 1(b). To this end, we propose a novel Position Relation Module that adopts a shallow Transformer Encoder [45] with two self-attention layers, denoted as TransEnc, to learn mutual relations between part concepts as relation feature  $Z_{r}$ . Since the position information has been decoupled from  $Z_{l}$ , we regain position-aware part features  $Z_{g} = \{Z_{g_{i}},\dots,Z_{g_{N_{q}}}\} \in \mathbb{R}^{N^{q}\times D}$  from  $Q$  using max-pooling and encode  $Z_{r} \in \mathbb{R}^{N_{q}\times D}$  as follows:
$$
Z _ {g _ {i}} = \operatorname {m a x p o o l} \left(Q _ {i}\right),
$$
$$
Z _ {r} = \operatorname {T r a n s E n c} \left(Z _ {g}\right). \tag {5}
$$
Afterward, we concatenate the position relation feature  $Z_{r}$  with the point composite features  $Z_{p}$  and feed the result into an MLP with  $2D \times D$  and a max-pooling layer to produce category-aware features  $Z_{c}$ . We can classify the input shape  $X$  by feeding  $Z_{c}$  to a linear layer predicting the class-wise probabilities.
3.3 Constraining Part Concepts
Though part concepts are proposed to represent different parts, they can easily collapse to the same values. The main reason is that the similarity between randomly initialized part concepts and part
(b)
Figure 4: (a) The part concept activation is likely to be uniform in the Euclidean space but sharp in the Poincaré space. (b) Self-distillation loss  $L_{sd}$  encourages part activations to align with the sharper distributions of their corresponding data augmentation. (c) Supervised contrastive loss  $L_{sc}$  forces the part concept activations of different known classes to be separated from each other so that more part concepts can be learned.
features is similar during the initial training stage. Consequently, part composite features based on the concept activation map would gradually be identical and part concepts updated with similarly back-propagated gradients from them converge to similar features. To ensure the representation ability and generalization of the part concept bank, we propose three constraints to train part concepts.
Self-distillation Loss. To guarantee that each part's geometric knowledge can be learned by specific part concepts, the concept activation distribution of each part should be sharp. To achieve this, we propose a self-distillation loss inspired by [3]. We enforce each part concept activation  $S$  of the shape parts  $Q$  to be consistent with a sharper one of the corresponding part concept activation  $\hat{S}$  from the augmented shape parts  $\hat{Q}$  (see Fig 4 (b)) as follows:
$$
L _ {s d} = - \frac {1}{2} \left(\operatorname {s t o p} \left(p \left(S _ {i}, \tau_ {s}\right)\right) \log \hat {S} _ {i} + \operatorname {s t o p} \left(p \left(\hat {S} _ {i}, \tau_ {s}\right)\right) \log S _ {i}\right), \tag {6}
$$
where  $p(S_i, \tau_s) = \frac{\exp(s_i / \tau_s)}{\sum_{j=1}^{N^q} \exp(s_i^j / \tau_s)}$  is a sharp concept activation distribution of  $Q_i$  controlled by temperature parameter  $\tau_s$  (set 0.1 empirically) and  $\text{stop}(\cdot)$  means the gradients is not back-propagated. Loss  $L_{sd}$  enhances the sharpness and consistency of the concept activation under different data augmentations for both the labeled and unlabeled datasets.
Supervised Contrastive Loss. During training, we found that even a few part concepts can support the recognition of known classes as shown in the red boxes in Fig. 4(c). In this case, the part concept bank lacks the incentive to learn more part concepts from known shapes, which reduces the information it can share with novel classes. Thus, for the labeled dataset  $D^{l}$ , we employ a supervised contrastive loss  $L_{sc}$  to increase the inter-class discrepancy of their concept activation maps. As shown in the green boxes in Fig. 4(c), through  $L_{sc}$ , the part concept bank can learn additional parts (i.e., leg and pedestal) from known shapes to increase their inter-class discrepancy. Given a mini-batch of  $B$  shapes  $\{X_{b}\}_{b = 1}^{B}$  and their augmentation  $\{\hat{X}_b\}_{b = 1}^B$ , we first sum their activation maps  $S_B \in \mathbb{R}^{2B\times M\times N^q}$  into shape concept activation  $T_{B} \in \mathbb{R}^{2B\times M}$  along the part dimension, where  $T_{i}$  summarizes all part concepts in a shape. Then the supervised contrastive loss is defined on  $2B$  shapes as follows:
$$
L _ {s c} = - \frac {1}{| \rho |} \sum_ {T _ {j} ^ {+} \in \rho} \log \frac {\exp \left(- \operatorname {s i m} \left(T _ {i} , T _ {j} ^ {+}\right) / \tau_ {c}\right)}{\sum_ {b = 1} ^ {| B |} \exp \left(- \operatorname {s i m} \left(T _ {i} , T _ {b}\right) / \tau_ {c}\right)}, \tag {7}
$$
where  $\rho = \{\{T_j^+ \in T_B : y_j = y_i\} \cup \hat{T}_i\}$  is the set that contains the shape concept activation  $\hat{T}_i$  of the augmented sample and other samples belonging to the same class in the mini-batch,  $\mathrm{sim}(.,.)$  calculates the cosine similarity, and temperature  $\tau_c$  is set 0.1 empirically.
Diversity Loss. To capture different patterns in 3D shapes, part concepts should be diverse. We can encourage this by minimizing the similarity between different part concepts with the following loss:
$$
L _ {c d} = \sum_ {m = 1} ^ {M} \max  _ {P ^ {-} \in \left\{P _ {m} \right\} _ {m = 1} ^ {M} / P _ {m}} (0, \operatorname {s i m} \left(P _ {m}, P ^ {-}\right) - \delta), \tag {8}
$$
where  $\delta$  is the cosine similarity threshold between a code and the other ones. In our experiments, we set  $\delta = 0.1$ .
3.4 The Training Losses