SlowGuess/NeurIPS_2023_OCR huggingface dataset:
One of the advantages of contrastive learning is to find a representation that maximizes the mutual information with the input [64]. For input  $X^i$ , let's show the hidden representation learning with  $Z^i$  which is learned contrastively by minimizing the InfoNCE loss. Van den Oord et al. [64] showed that minimizing InfoNCE loss increases a lower bound for mutual information. Hence, contrastive learning with the InfoNCE loss can be a suitable choice for minimizing the  $\mathcal{L}_{\mathrm{adr}}$  in Equation 1. We will use this to our advantage on two different levels. Let's consider that  $Z^i$  has dimension  $d$ , and each latent variable  $z_k^i$  can take up  $n$  different values. The complexity of the feature space for this latent variable would be  $\mathcal{O}(n^d)$ , then as we show in supplemental, the number of structurally different binary trees for this feature space will be  $\mathcal{O}(4^{n^d})$ . So minimizing  $n$  and  $d$  will be the most effective way to limit the number of possible binary trees. Since our model and the amount of training data is bounded, we must minimize the possible search space while providing reasonable performance. At the same time, the input feature space  $X^i$  with  $N$  possible values and dimension  $D$  has  $\mathcal{O}(N^D)$  possible states. To cover it completely, we can not arbitrarily decrease  $d$  and  $n$ . Note that for a nearly continuous function  $N \to \infty$ , the probability of a random discrete tree fully covering this space would be near zero. To make the best of both worlds, we consider different levels of complexity of latent variables, each focusing on one of these goals.
Minimizing Contrastive Loss on the Inputs. Similar to [27], we use this unsupervised contrastive loss to maximize the mutual information between input  $X^i$  and the extracted latent embedding  $Z^i$ . Akin to [27], we also benefit from the supervised contrastive learning signal for the members of a particular category. Let's assume that the number of categories in the entire dataset is  $\mathcal{C}$ . Different members of a category can be seen as different views of that category, analogous to unsupervised contrastive loss. Hence, they combine these unsupervised contrastive loss or  $\mathcal{L}_{\mathrm{C\_in}}^u$  and its supervised counterpart,  $\mathcal{L}_{\mathrm{C\_in}}^s$  with a coefficient  $\lambda$ , which we call  $\lambda_{\mathrm{in}}$  in a following manner:
$$
\mathcal {L} _ {C _ {-} i n} = \left(1 - \lambda_ {\mathrm {i n}}\right) \mathcal {L} _ {\mathrm {C} _ {-} \mathrm {i n}} ^ {u} + \lambda_ {\mathrm {i n}} \mathcal {L} _ {\mathrm {C} _ {-} \mathrm {i n}} ^ {s} \tag {6}
$$
For covering the input space,  $X^i$ , the loss function from Equation 6 is more suited if we consider both input and latent features as approximately continuous. We have shown this loss by  $\mathcal{L}_{\mathrm{C\_in}}$  in Figure 3.
Minimizing Contrastive Loss on the Codes. In order to also facilitate finding the binary tree, we map the latent feature extracted from the previous section to a binary code with minimum length. Hence we effectively decrease  $n$  to 2 while actively minimizing  $d$ . Furthermore, we extract a code by making the value of each bit in this code correspond to its sequential position in a binary number, i.e., each digit has twice the value of the digit immediately to its right. This ensures the model treats this code as a binary tree coding with its root in the first digit. We consider unsupervised contrastive learning for raw binary digits  $\mathcal{L}_{\mathrm{C\_code}}^u$  and supervised variants of this loss for the extracted code, which we will show by and  $\mathcal{L}_{\mathrm{C\_code}}^s$ , the total contrastive loss for the binary embedding is defined as:
$$
\mathcal {L} _ {\mathrm {C} _ {-} \text {c o d e}} = \left(1 - \lambda_ {\text {c o d e}}\right) \mathcal {L} _ {\mathrm {C} _ {-} \text {c o d e}} ^ {u} + \lambda_ {\text {c o d e}} \mathcal {L} _ {\mathrm {C} _ {-} \text {c o d e}} ^ {s} \tag {7}
$$
In summary, the loss from Equation 6 finds a tree compatible with the input, while the loss from Equation 7 learns an implicit tree in compliance with categories. Then we consider  $\mathcal{L}_{\mathrm{adr}}$  as their combination:
$$
\mathcal {L} _ {\mathrm {a d r}} = \alpha \mathcal {L} _ {\mathrm {C} _ {-} \text {i n}} + \beta \mathcal {L} _ {\mathrm {C} _ {-} \text {c o d e}} \tag {8}
$$
4.2 Minimizing Code Length
As discussed in Section 4.1, we need to decrease the feature space complexity by using minimum length codes to reduce the search space for finding the implicit category binary tree. In addition, using Shannon's mutual information as an approximate substitute for algorithmic mutual information necessitates minimizing the sequence length. We must simultaneously learn category codes and their optimal length. Since each of these optimizations depends on the optimal solution of the other one, we use two different blocks to solve them at the same time.
Code Generator Block. In Figure 3, the Code Generator block uses the extracted embeddings to generate binary category codes. At this stage, we consider a fixed length for these binary codes. The output of this stage is used for unsupervised contrastive learning on the codes in Equation 7. We also use  $\mathcal{L}_{\mathrm{code\_cond}}$  to enforce the digits of the codes to be a decent approximation of binary values.
Code Masker Block. For this block, we use the  $\mathcal{L}_{\mathrm{mask\_cond}}$  to ensure the binary constraint of the outputs. In addition, to control the length of the code or, in other words, the sequence of 1s at the beginning, we use  $\mathcal{L}_{\mathrm{length}}$  in Equation 2.
Table 1: Ablation study on the effectiveness of each loss function. Accuracy score on the CUB dataset is reported. This table indicates each component's preference for novel or known categories. In the first row of the table, differences in results compared with [27] can be attributed to specific implementation details, which are elaborated in section B.2 in the supplemental.
\( \mathcal{L}_{\text{C_in}} \)	\( \mathcal{L}_{\text{C_code}} \)	\( \mathcal{L}_{\text{code_cond}} \)	\( \mathcal{L}_{\text{length}} \)	\( \mathcal{L}_{\text{mask_cond}} \)	\( \mathcal{L}_{\text{Cat}} \)	All	Known	Novel
✓	X	X	X	X	X	66.8	78.1	61.1
✓	✓	X	X	X	X	67.7	75.7	63.7
✓	✓	✓	X	X	X	68.5	77.5	64.0
✓	✓	✓	✓	X	X	68.4	76.1	64.5
✓	✓	✓	✓	✓	X	67.8	78.7	62.3
✓	X	X	X	X	✓	68.2	76.4	64.1
✓	✓	✓	✓	✓	✓	69.4	77.9	65.2
4.3 Aligning Codes using Supervision Signals
In the final block of the framework, we convert category codes from the Code Generator to a binary number based on the digit positions in the sequence. To truncate this code, we do a Hadamard multiplication for this number by the mask generated by the Code Masker. We use these truncated codes for supervised contrastive learning on the codes. Finally, we feed these codes to the Catgorizer block to predict the labels directly. We believe relying solely on contrastive supervision prevents the model from benefiting from learning discriminative features early on to speed up training.
Eight Datasets. We evaluate our model on three coarse-grained datasets CIFAR10/100 [65] and ImageNet-100 [66] and four fine-grained datasets: CUB-200 [67], Aircraft [68], SCars [69] and Oxford-Pet [70]. Finally, we use the challenging Herbarium19 [71] dataset, which is fine-grained and long-tailed. To acquire the train and test splits, we follow [27]. We subsample the training dataset in a ratio of  $50\%$  of known categories at the train and all samples of unknown categories. For all datasets except CIFAR100, we consider  $50\%$  of the categories as known categories at training time. For CIFAR100,  $80\%$  of the categories are known during training time, as in [27]. A summary of dataset statistics and their train test splits is shown in the supplemental.
Implementation Details. Following [27], we use ViT-B/16 as our backbone, which is pre-trained by DINO [72] on unlabelled ImageNet 1K [4]. Unless otherwise specified, we use 200 epochs and batch size of 128 for training. We present the complete implementation details in the supplemental. Our code is available at: https://github.com/SarahRastegar/InfoSieve.
Evaluation Metrics. We use semi-supervised  $k$ -means proposed by [27] to cluster the predicted embeddings. Then, the Hungarian algorithm [73] solves the optimal assignment of emerged clusters to their ground truth labels. We report the accuracy of the model's predictions on All, Known, and Novel categories. Accuracy on All is calculated using the whole unlabelled train set, consisting of known and unknown categories. For Known, we only consider the samples with labels known during training. Finally, for Novel, we consider samples from the unlabelled categories at train time.
5.2 Ablative studies
We investigate each model's component contribution to the overall performance of the model and the effect of each hyperparameter. Further ablations can be found in the supplemental.
Effect of Each Component. We first examine the effect of each component using the CUB dataset. A fine-grained dataset like CUB requires the model to distinguish between the semantic nuances of each category. Table 1 shows the effect of each loss component for the CUB dataset. As we can see from this table,  $\mathcal{L}_{\mathrm{C\_code}}$  and  $\mathcal{L}_{\mathrm{length}}$  have the most positive effect on novel categories while affecting known categories negatively. Utilizing  $\mathcal{L}_{\mathrm{code\_cond}}$  to enforce binary constraints on the embedding enhances performance for both novel and known categories. Conversely, applying  $\mathcal{L}_{\mathrm{mask\_cond}}$  boosts performance for known categories while detrimentally impacting novel categories. This is aligned with Theorem 1 and the definition of the category because these losses have opposite goals. As discussed, minimizing the search space helps the model find the implicit category tree faster.  $\mathcal{L}_{\mathrm{C\_code}}$  and  $\mathcal{L}_{\mathrm{length}}$  try to achieve this by mapping the information to a smaller feature space, while condition
Table 2: Hyperparameter analysis. This table indicates the effect on the accuracy score of each hyperparameter on the CUB dataset for novel or known categories.
(a) Effect of Code Constraint Coefficient
Code Constraint Coef	All	Known	Novel
ζ = 0.01	69.4	77.9	65.2
ζ = 0.1	69.1	76.1	65.6
ζ = 1	69.9	76.1	66.8
ζ = 2	69.5	75.5	66.5
(b) Effect of Mask Constraint Coefficient
Mask Constraint Coef	All	Known	Novel
μ = 0.01	69.4	77.9	65.2
μ = 0.1	67.2	74.1	63.8
μ = 1	69.3	76.0	65.9
μ = 2	70.6	79.3	66.3
(c) Effect of Code Contrastive Coefficient
Code Contrastive Coef	All	Known	Novel
β = 0.01	68.6	77.0	64.5
β = 0.1	69.9	76.3	66.7
β = 1	69.4	77.9	65.2
β = 2	68.6	77.9	64.0
(d) Effect of Code Length Coefficient
Code Cut Length Coef	All	Known	Novel
δ = 0.01	69.4	77.0	65.6
δ = 0.1	69.4	77.9	65.2
δ = 1	68.8	75.8	65.2
δ = 2	69.8	76.9	66.2
losses solve this by pruning and discarding unnecessary information. Finally,  $\mathcal{L}_{\mathrm{Cat}}$  on its own has a destructive effect on known categories. One reason for this is the small size of the CUB dataset, which makes the model overfit on labeled data. However, when all losses are combined, their synergic effect helps perform well for both known and novel categories.
Our model has a few hyperparameters: code binary constraint  $(\zeta)$ , mask binary constraint  $(\mu)$ , code contrastive  $(\beta)$ , code length  $(\delta)$ , and code mapping  $(\eta)$ . We examine the effect of each hyperparameter on the model's performance on the CUB dataset. Our default values for the hyperparameters are: code constraint coeff  $\zeta = 0.01$ , mask constraint coeff  $\mu = 0.01$ , code contrastive coeff  $\beta = 1$ , code mapping coeff  $\eta = 0.01$  and code length coeff  $\delta = 0.1$ .
Code Binary Constraint. This hyperparameter is introduced to satisfy the binary requirement of the code. Since we use tanh to create the binary vector, the coefficient only determines how fast the method satisfies the conditions. When codes 0 and 1 are stabilized, the hyperparameter effect will be diminished. However, we noticed that more significant coefficients somewhat affect the known accuracy. The effect of this hyperparameter for the CUB dataset is shown in Table 2 (a). We can see that the method is robust to the choice of this hyperparameter.
Mask Binary Constraint. For the mask constraint hyperparameter, we start from an all-one mask in our Lagrange multiplier approach. A more significant constraint translates to a longer category code. A higher coefficient is more useful since it better imposes the binary condition for the mask. The effect of different values of this hyperparameter for the CUB datasets is shown in Table 2 (b).
Code Contrastive. This loss maintains information about the input. From Table 2 (c), we observe that minimizing the information for a fine-grained dataset like CUB will lead to a better performance.
Code Length. Table 2 (d) reports our results for different values of code length hyperparameter. Since the code length is penalized exponentially, the code length hyperparameter's effect is not comparable to the exponential growth; hence, in the end, the model is not sensitive to this hyperparameter's value.
Code Mapping. Since current evaluation metrics rely on predefined categories, our category codes must be mapped to this scenario. This loss is not an essential part of the self-coding that our model learns, but it accelerates model training. The effect of this hyperparameter is shown in Table 3.
Overall Parameter Analysis One reason the model is not very sensitive to different hyperparameters is that our model consists of three separate parts: Code masker, Code Generator, and Categorizer. The only hyperparameters that affect all of these three parts directly are  $\beta$ , the code contrastive coefficient, and  $\delta$ , the code length coefficient. Hence, these hyperparameters affect our model's performance more.
Table 3: Effect of Code Mapping. Accuracy scores on the CUB dataset for novel and known categories.
Code	CUB
All	Known	Novel
η = 0.01	69.4	77.9	65.2
η = 0.1	68.4	75.7	64.8
η = 1	68.8	75.7	65.3
η = 2	68.9	77.5	64.6
Table 4: Comparison on fine-grained image recognition datasets. Accuracy score for the first three methods is reported from [27] and for ORCA from [28]. Bold and underlined numbers, respectively, show the best and second-best accuracies. Our method has superior performance for the three experimental settings (All, Known, and Novel). This table shows that our method is especially well suited to fine-grained settings.
Method	CUB-200	FGVC-Aircraft	Stanford-Cars	Oxford-IIIT Pet	Herbarium-19
All	Known	Novel	All	Known	Novel	All	Known	Novel	All	Known	Novel	All	Known	Novel
k-means [74]	34.3	38.9	32.1	12.9	12.9	12.8	12.8	10.6	13.8	77.1	70.1	80.7	13.0	12.2	13.4
RankStats+ [75]	33.3	51.6	24.2	26.9	36.4	22.2	28.3	61.8	12.1	-	-	-	27.9	55.8	12.8
UNO+ [76]	35.1	49.0	28.1	40.3	56.4	32.2	35.5	70.5	18.6	-	-	-	28.3	53.7	14.7
ORCA [77]	36.3	43.8	32.6	31.6	32.0	31.4	31.9	42.2	26.9	-	-	-	24.6	26.5	23.7
GCD [27]	51.3	56.6	48.7	45.0	41.1	46.9	39.0	57.6	29.9	80.2	85.1	77.6	35.4	51.0	27.0
XCon [32]	52.1	54.3	51.0	47.7	44.4	49.4	40.5	58.8	31.7	86.7	91.5	84.1	-	-	-
PromptCAL [28]	62.9	64.4	62.1	52.2	52.2	52.3	50.2	70.1	40.6	-	-	-	-	-	-
DCCL [29]	63.5	60.8	64.9	-	-	-	43.1	55.7	36.2	88.1	88.2	88.0	-	-	-
SimGCD [31]	60.3	65.6	57.7	54.2	59.1	51.8	53.8	71.9	45.0	-	-	-	44.0	58.0	36.4
GPC [78]	52.0	55.5	47.5	43.3	40.7	44.8	38.2	58.9	27.4	-	-	-	-	-	-
InfoSieve	69.4	77.9	65.2	56.3	63.7	52.5	55.7	74.8	46.4	91.8	92.6	91.3	41.0	55.4	33.2
Table 5: Comparison on coarse-grained image recognition datasets. Accuracy for the first three methods from [27] and for ORCA from [28]. Bold and underlined numbers, respectively, show the best and second-best accuracies. While our method does not reach state-of-the-art for coarse-grained settings, it has a consistent performance for all three experimental settings (All, Known, Novel).
Method	CIFAR-10	CIFAR-100	ImageNet-100
All	Known	Novel	All	Known	Novel	All	Known	Novel
k-means [74]	83.6	85.7	82.5	52.0	52.2	50.8	72.7	75.5	71.3
RankStats+ [75]	46.8	19.2	60.5	58.2	77.6	19.3	37.1	61.6	24.8
UNO+ [76]	68.6	98.3	53.8	69.5	80.6	47.2	70.3	95.0	57.9
ORCA [77]	96.9	95.1	97.8	74.2	82.1	67.2	79.2	93.2	72.1
GCD [27]	91.5	97.9	88.2	73.0	76.2	66.5	74.1	89.8	66.3
XCon[32]	96.0	97.3	95.4	74.2	81.2	60.3	77.6	93.5	69.7
PromptCAL [28]	97.9	96.6	98.5	81.2	84.2	75.3	83.1	92.7	78.3
DCCL [29]	96.3	96.5	96.9	75.3	76.8	70.2	80.5	90.5	76.2
SimGCD [31]	97.1	95.1	98.1	80.1	81.2	77.8	83.0	93.1	77.9
GPC [78]	90.6	97.6	87.0	75.4	84.6	60.1	75.3	93.4	66.7
InfoSieve	94.8	97.7	93.4	78.3	82.2	70.5	80.5	93.8	73.8
5.4 Comparison with State-of-the-Art
Fine-grained Image Classification. Fine-grained image datasets are a more realistic approach to the real world. In coarse-grained datasets, the model can use other visual cues to guess about the novelty of a category; fine-grained datasets require that the model distinguish subtle category-specific details. Table 4 summarizes our model's performance on the fine-grained datasets. As we can see from this table, our model has more robust and consistent results compared to other methods for fine-grained datasets. Herbarium 19, a long-tailed dataset, raises the stakes by having different frequencies for different categories, which is detrimental to most clustering approaches because of the extremely unbalanced cluster size. As Table 4 shows, our model can distinguish different categories even from a few examples and is robust to frequency imbalance.
Coarse-grained Image Classification. Our method is well suited for datasets with more categories and fine distinctions. Nevertheless, we also evaluate our model on three coarse-grained datasets, namely CIFAR10 and CIFAR100 [65] and ImageNet-100 [66]. Table 5 compares our results against state-of-the-art generalized category discovery methods. As we can see from this table, our method performs consistently well on both known and novel datasets. For instance, while  $\mathrm{UNO + }$  shows the highest accuracy on the Known categories of CIFAR-10, this is at the expense of performance degradation on the Novel categories. The same observations can be seen on ImageNet-100. Table 5 shows that our method consistently performs competitively for both novel and known categories. Based on our theory, the smaller CIFAR10/100 and ImageNet-100 improvement is predictable. For CIFAR 10, the depth of the implicit tree is 4; hence, the number of implicit possible binary trees with this limited depth is smaller, meaning finding a good approximation for the implicit category tree can be achieved by other models. However, as the depth of this tree increases, our model can still find the aforementioned tree; hence, we see more improvement for fine-grained data.
5.5 Qualitative results
In supplemental, we explain how to extract the implicit tree that is learned by our model. For instance, Figure 2 shows this extracted tree structure on the CUB dataset. We see that our method can extract some hierarchical structure in the data. In the leaves of this tree, we have "yellow sparrows," instances which are the descendants of a more abstract "sparrow" node; or in the path '00...1', we have a "blackbirds" node, which can encompass multiple blackbird species as its children nodes.
Novel Category Discovery can be traced back to Han et al. [79], where knowledge from labeled data was used to infer the unknown categories. Following this work, Zhong et al. [80] solidified the novel class discovery as a new specific problem. The main goal of novel class discovery is to transfer the implicit category structure from the known categories to infer unknown categories [24, 26, 75, 76, 81, 82, 82-99]. Prior to the novel class discovery, the problem of encountering new classes at the test time was investigated by open-set recognition [16, 17, 20, 100]. However, the strategy of dealing with these new categories is different. In the open-set scenario, the model rejects the samples from novel categories, while novel class discovery aims to benefit from the vast knowledge of the unknown realm and infer the categories. However, the novel class discovery has a limiting assumption that test data only consists of novel categories. For a more realistic setting, Generalized Category Discovery considers both known and old categories at test time.
Generalised Category Discovery recently has been introduced by [27] and concurrently under the name Open-world semi-supervised learning by [77]. In this scenario, while the model should not lose its grasp on known categories, it must discover novel ones at test time. This adds an extra challenge because when we adapt the novel class discovery methods to this scenario, they are biased to either novel or known categories and miss the other group. There has been a recent surge of interest in generalized category discovery [28-31, 78, 101-110]. In this work, instead of viewing categories as an end, we investigated the fundamental question of how to conceptualize the category itself.
Decision Tree Distillation. The benefits of the hierarchical nature of categories have been investigated previously. Xiao [111] and Frosst and Hinton [112] used a decision tree in order to make the categorization interpretable. Adaptive neural trees proposed by [113] assimilate representation learning to its edges. Ji et al. [114] use attention binary neural tree to distinguish fine-grained categories by attending to the nuances of these categories. However, these methods need an explicit tree structure. In this work, we let the network extract this implicit tree on its own. This way, our model is also suitable when an explicit tree structure does not exist.
For a more comprehensive related work, see the supplemental.
This paper seeks to address the often neglected question of defining a category. While the concept of a category is readily accepted as a given in the exploration of novel categories and open-world settings, the inherent subjectivity of the term poses a challenge when optimizing deep networks, which are fundamentally mathematical frameworks. To circumvent this issue, we put forth a mathematical solution to extract category codes, replacing the simplistic one-hot categorical encoding. Further, we introduce a novel framework capable of uncovering unknown categories during inference and iteratively updating its environmental representation based on newly acquired knowledge. These category codes also prove beneficial in handling fine-grained categorization, where attention to nuanced differences between categories is paramount. By illuminating the limitations of the subjective notion of category and recasting them within a mathematical framework, we hope to catalyze the development of less human-dependent models.
Despite the endeavors made in this work, there are several notable limitations. Foremost is our assumption of an implicit hierarchical tree underlying categorization. Furthermore, our optimization solution may not align perfectly with the actual implicit hierarchy, as it only satisfies the necessary conditions outlined in Theorem 1. Lastly, akin to other approaches in the generalized category discovery literature, we operate under the assumption that we have access to unlabeled data from unknown categories during training. This assumption, while convenient for model development, may not always hold true in real-world applications.
This work is part of the project Real-Time Video Surveillance Search with project number 18038, which is (partly) financed by the Dutch Research Council (NWO) domain Applied and Engineering/ Sciences (TTW). Special thanks to Dr. Dennis Koelma and Dr. Efstratios Gavves for their valuable insights about conceptualizing the category to alleviate generalized category discovery.