SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: Statistics of datasets and their data splits for the generalized category discovery task. The first three datasets are coarse-grained image classification datasets, while the next four are fine-grained datasets. The Herbarium19 dataset is both fine-grained and long-tailed.
Dataset	Labelled	Unlabelled
#Images	#Categories	#Images	#Categories
CIFAR-10 [65]	12.5K	5	37.5K	10
CIFAR-100 [65]	20.0K	80	30.0K	100
ImageNet-100 [66]	31.9K	50	95.3K	100
CUB-200 [67]	1.5K	100	4.5K	200
SCars [69]	2.0K	98	6.1K	196
Aircraft [68]	3.4K	50	6.6K	100
Oxford-Pet [70]	0.9K	19	2.7K	37
Herbarium19 [71]	8.9K	341	25.4K	683
Oxford-Pet [70] is a fine-grained dataset of different species of cats and dogs. This is challenging since the amount of data is very limited in this dataset, which makes it prone to overfitting.
Herbarium_19 [71] is a botanical research dataset about different types of plants. Due to its long-tailed alongside fine-grained nature, it is a challenging dataset for discovering novel categories.
In this section, we provide our implementation details for each block separately. As mentioned in the main paper, the final loss function that we use to train the model is:
$$
\mathcal {L} _ {\text {f i n a l}} = \mathcal {L} _ {\text {a d r}} + \delta \mathcal {L} _ {\text {l e n g t h}} + \eta \mathcal {L} _ {\text {C a t}} + \zeta \mathcal {L} _ {\text {c o d e} _ {\text {c o n d}}} + \mu \mathcal {L} _ {\text {m a s k} _ {\text {c o n d}}}. \tag {13}
$$
In which the loss  $\mathcal{L}_{\mathrm{adr}}$  is:
$$
\mathcal {L} _ {\mathrm {a d r}} = \alpha \mathcal {L} _ {\mathrm {C} _ {-} \text {i n}} + \beta \mathcal {L} _ {\mathrm {C} _ {-} \text {c o d e}}. \tag {14}
$$
In this formula,  $\mathcal{L}_{\mathrm{C\_in}}$  is the loss function that [27] suggested, so we use the same hyperparameters as their defaults for this loss. Hence, we only expand on  $\mathcal{L}_{\mathrm{C\_code}}$ :
$$
\mathcal {L} _ {\mathrm {a d r}} = \alpha \mathcal {L} _ {\mathrm {C} _ {-} \text {i n}} + \beta \left(\left(1 - \lambda_ {\text {c o d e}}\right) \mathcal {L} _ {\mathrm {C} _ {-} \text {c o d e}} ^ {u} + \lambda_ {\text {c o d e}} \mathcal {L} _ {\mathrm {C} _ {-} \text {c o d e}} ^ {s}\right). \tag {15}
$$
In the scope of our experimentation, it was assumed by default that  $\alpha = 1$  and  $\lambda_{\mathrm{code}} = 0.35$ . The code generation process introduces a certain noise level, potentially leading to confusion in the model, particularly in fine-grained data. To mitigate this, we integrated a smoothing hyperparameter within our contrastive learning framework, aiming to balance the noise impact and avert excessive confidence in the generated code, for datasets such as CUB and Pet, the smoothing factor was set at 1, whereas for SCars, Aircraft, and Herb datasets, it was adjusted to 0.1. In contrast, we did not apply smoothing for generic datasets like CIFAR 10/100 and ImageNet, where label noise is less significant.
Furthermore, in dealing with fine-grained data, we opted to fine-tune the final two blocks of the DINO model. This approach differs from our strategy for generic datasets, where only the last block underwent fine-tuning. Additionally, we employed semi-supervised  $k$ -means at every epoch to derive pseudo-labels from unlabeled data. These pseudo-labels were then used in our supervised contrastive learning process as a supervisory signal. It is important to note that in supervised contrastive learning, the primary requirement is that paired samples belong to the same class, allowing us to disregard discrepancies between novel class pseudo-labels and their actual ground truth values. Furthermore, instead of cosine similarity for contrastive learning, we adopt Euclidean distance, a better approximation for the category problem. Finally, for balanced datasets, we use the balanced version of  $k$ -means for semi-supervised  $k$ -means.
Code Generator. To create this block, we use a fully connected network with GeLU activation functions [118]. Then, we apply a tanh activation function  $\tanh(ax)$  in which  $a$  is a hyperparameter showing the model's age. We expect that as the model's age increases or, in other words, in later epochs, the model will be more decisive because of sharper transitions from 0 to 1. Hence, we will have a stronger binary dichotomy for code values. Also, since contrastive learning makes the different samples as far as possible, this causes a problem for the Code Generator because the feature space will not smoothly transition from different samples of the same category, especially for fine-grained datasets. To alleviate this problem, we use a label smoothing hyperparameter in the contrastive objective to help make feature space smoother, which will require a smaller tree for encoding. Since the model should distinguish 0s for the mask from 0s of the code, we do not adjust the code generator to 0 and 1s and consider the  $-1$  and 1 values in practice.
Code Masker. The Code Masker block is a fully connected network with tanh activation functions at the end, which are adjusted to be 0 and 1s. We also consider the aging hyperparameter for the tanh activation function in the masking block. In the beginning, since codes are not learned, masking the embedding space might hamper its learning ability. To solve this, we start masker with all one's entries and gradually decrease it with epochs. Hence, the activation function that is applied to the masker would be  $\tanh(x + \frac{1}{a + 1})$ , in which  $a$  is the aging parameter. In practice, we observed that norm one is stable enough in this loss function while also truncating codes at a reasonable length. Since  $\mathcal{L}_{\mathrm{length}}$  grows exponentially with code length, it will mask most of the code. For fine-grained datasets, this could be detrimental for very similar categories. To alleviate this problem, instead of using 2 as a positional base, we decrease it with each epoch to  $2 - \frac{\mathrm{epoch}}{N_{\mathrm{epochs}}}$ . So, at the end of training, the values of all positions are the same. This allows the model to encode more levels to the tree. Since we start with the base 2, we are constructing the tree with a focus on nodes near the root at the start and to the leaves at the end of training.
Categorizer. We use a fully connected network for this block and train it with the one-hot encoding of the labeled samples. This module receives the truncated codes to predict the labeled data. This module cannot categorize labeled data if the masker truncates too much information. Hence, it creates error signals that prevent the masker from truncating too much. This part of the network is arbitrary, and we showed in ablations that we can ignore this module without supervision signals.
B.3 Further Ablations
Feature Space Visualization. Figure 4 illustrates the tSNE visualizations for different embedding extracted from our model. While our model's features form separate clusters, our label embedding, which is the raw code feature before binarization, makes these clusters distinctive. After that, binary embedding enhances this separation while condensing the cluster by making samples of clusters closer to each other, which is evident for the bird cluster shown in yellow. Because of its 0 or 1 nature, semantic similarity will affect the binary embedding more than visual similarity. Finally, our code embedding, which assigns positional values to the extracted binary embedding, shows indirectly that to have the most efficient code, our model should span the code space as much as possible, which explains the porous nature of these clusters.
B.4 Extracting the Implicit Tree from the Model
Suppose that the generated feature vector by the network for sample  $X$  is  $x_0x_1\cdots x_k$ , where  $k$  is the dimension of the code embedding or, equivalently, the depth of our implicit hierarchy tree. Using appropriate activation functions, we can assume that  $x_i$  is binary. The unsupervised contrastive loss forces the model to make the associated code to each sample unique. So if  $X'$  is not equivalent to  $X$  or one of its augmentations, its code  $x_0'x_1' \cdots x_k'$  will differ from the code assigned to  $X$ . For the supervised contrastive loss, instead of considering the code, we consider a sequence by assigning different positional values to each bit so the code  $x_0x_1 \cdots x_k$  can be considered as the binary number  $0.x_0x_1 \cdots x_k$ . Then, the supervised contrastive loss aims to minimize the difference between these assigned binary numbers. This means our model learns to use the first digits for discriminative information while pushing the specific information about each sample to the last digits. Then, our masker learns to minimize the number of discriminative digits. Our Theorem states that, finally, the embedded tree that the model learns this way is a good approximation of the optimal tree. Ultimately, our model generates a code for each sample, and we consider each code as a binary tree traverse from the root to the leaf. Hence, the codes delineate our tree's structure and binary classification that
(a) Feature Embedding
(c) Binary Embedding
Figure 4: t-SNE plot for different embeddings in our model. (a) Feature embedding. The embedding after the projection head which is used by contrastive loss to maximize the representation information. (b) Label embedding. The embedding after generating code features is used by unsupervised contrastive loss for codes. (c) Binary embedding. The embedding by converting code features to a binary sequence using tanh activation functions and binary conditions. (d) Code embedding. The final truncated code which is generated by assigning positional values to the binary sequence and truncating the produced code using the masker network.
(b) Label Embedding
(d) Code Embedding
happens at each node. Since our approach enables the model to use the initial bits for supervised contrastive learning and the last bits for unsupervised contrastive learning, we can benefit from their synergic advantages while preventing them from interfering with each other.
C Related Works
C.1 Open Set Recognition
The first sparks of the requirement for models that can handle real-world data were introduced by Scheirer et al. [20] and following works of [16, 18]. The first application of deep networks to address this problem was presented by OpenMax [17]. The main goal for open-set recognition is to distinguish known categories from each other while rejecting samples from novel categories. Hence many open-set methods rely on simulating this notion of otherness, either through large reconstruction errors [119, 120] distance from a set of prototypes[121-123] or by distinguishing the adversarially generated samples [124-127]. One of the shortcomings of open set recognition is that all new classes will be discarded.
C.2 Novel Class Discovery
To overcome open set recognition shortcomings, novel class discovery aims to benefit from the vast knowledge of the unknown realm and infer the categories. It can be traced back to [79], where they used the knowledge from labeled data to infer the unknown categories. Following this work, [80] solidified the novel class discovery as a new specific problem. The main goal of novel class discovery is to transfer the implicit category structure from the known categories to infer unknown categories [24, 26, 75, 76, 81, 82, 82-99]. Despite this, the novel class discovery has a limiting assumption that test data only consists of novel categories.
C.3 Generalized Category Discovery
For a more realistic setting, Generalized Category Discovery considers both known and old categories at the test time. This nascent problem was introduced by [27] and concurrently under the name open-world semi-supervised learning by [77]. In this scenario, while the model should not lose its grasp on old categories, it must discover novel categories in test time. This adds an extra challenge because when we adapt the novel class discovery methods to this scenario, they try to be biased to either novel or old categories and miss the other group. There has been a recent surge of interest in generalized category discovery [28-31, 78, 101-110]. In this work, instead of viewing categories as an end, we investigated the fundamental question of how to conceptualize category itself.
C.4 Binary Tree Distillation
Benefiting from the hierarchical nature of categories has been investigated previously. Xiao [111] and Frosst and Hinton [112] used a decision tree in order to make the categorization interpretable and as a series of decisions. Adaptive neural trees proposed by [113] assimilate representation learning to its edges. Ji et al. [114] use attention binary neural tree to distinguish fine-grained categories by attending to the nuances of these categories. However, these methods need an explicit tree structure. In this work, we let the network extract this implicit tree on its own. This way, our model is also suitable when an explicit tree structure does not exist.