Title: Learn to Categorize or Categorize to Learn? Self-Coding for Generalized Category Discovery
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generalized category discovery

Open world learning

Open-set recognition
Abstract: In the quest for unveiling novel categories at test time, we confront the inherent limitations of traditional supervised recognition models that are restricted by a predefined category set. While strides have been made in the realms of self-supervised and open-world learning towards test-time category discovery, a crucial yet often overlooked question persists: what exactly delineates a category? In this paper, we conceptualize a category through the lens of optimization, viewing it as an optimal solution to a well-defined problem. Harnessing this unique conceptualization, we propose a novel, efficient and self-supervised method capable of discovering previously unknown categories at test time. A salient feature of our approach is the assignment of minimum length category codes to individual data instances, which encapsulates the implicit category hierarchy prevalent in real-world datasets. This mechanism affords us enhanced control over category granularity, thereby equipping our model to handle fine-grained categories adeptly. Experimental evaluations, bolstered by state-of-the-art benchmark comparisons, testify to the efficacy of our solution in managing unknown categories at test time. Furthermore, we fortify our proposition with a theoretical foundation, providing proof of its optimality. Our code is available at: https://github.com/SarahRastegar/InfoSieve.

Learn to Categorize or Categorize to Learn? Self-Coding for Generalized Category Discovery

Abstract

In the quest for unveiling novel categories at test time, we confront the inherent limitations of traditional supervised recognition models that are restricted by a predefined category set. While strides have been made in the realms of self-supervised and open-world learning towards test-time category discovery, a crucial yet often overlooked question persists: what exactly delineates a category ? In this paper, we conceptualize a category through the lens of optimization, viewing it as an optimal solution to a well-defined problem. Harnessing this unique conceptualization, we propose a novel, efficient and self-supervised method capable of discovering previously unknown categories at test time. A salient feature of our approach is the assignment of minimum length category codes to individual data instances, which encapsulates the implicit category hierarchy prevalent in real-world datasets. This mechanism affords us enhanced control over category granularity, thereby equipping our model to handle fine-grained categories adeptly. Experimental evaluations, bolstered by state-of-the-art benchmark comparisons, testify to the efficacy of our solution in managing unknown categories at test time. Furthermore, we fortify our proposition with a theoretical foundation, providing proof of its optimality. Our code is available at: https://github.com/SarahRastegar/InfoSieve .

1 Introduction

The human brain intuitively classifies objects into distinct categories, a process so intrinsic that its complexity is often overlooked. However, translating this seemingly innate understanding of categorization into the realm of machine learning opens a veritable Pandora's box of differing interpretations for what constitutes a category? [1, 2]. Prior to training machine learning models for categorization tasks, it is indispensable to first demystify this concept of a category .

In the realm of conventional supervised learning [3-7], each category is represented by arbitrary codes, with the expectation that machines produce corresponding codes upon encountering objects from the same category. Despite its widespread use, this approach harbors several pitfalls: it suffers from label inconsistency, overlooks category hierarchies, and, as the main topic of this paper, struggles with open-world recognition.

Pitfall I: Label Inconsistency. Assessing a model's perfor- mance becomes problematic when category assignments are subject to noise [8, 9] or exhibit arbitrary variation across different datasets. For example, if a zoologist identifies the bird in Figure 1 as a flying fox fruit bat , it is not due to a misunderstanding of what constitutes a bird or a dog . Rather, it signifies a more nuanced understanding of these categories. However, conventional machine learning

Root

models may penalize such refined categorizations if they deviate from the pre-defined ground-truth labels. This work addresses this limitation by assigning category codes to individual samples. These codes not only prevent over-dependence on specific labels but also facilitate encoding similarities across distinct categories, paving the way for a more robust and nuanced categorization.

Pitfall II: Category Hierarchies. A prevalent encoding method, such as one-hot target vectors, falls short when addressing category hierarchies. While we, as humans, intuitively distinguish the categories of plane and dog as more disparate than cat and dog , our representations within the model fail to convey this nuanced difference, effectively disregarding the shared semantics between the categories of cat and dog . While some explorations into a category hierarchy for image classification have been undertaken [10-14], these studies hinge on an externally imposed hierarchy, thus limiting their adaptability and universality. This paper proposes a self-supervised approach that enables the model to impose these implicit hierarchies in the form of binary trees into their learned representation. For instance, Figure 2 shows that we can address each sample in the dataset with its path from the root of this implicit tree, hence associating a category code to it. We show theoretically that under a set of conditions, all samples in a category have a common prefix, which delineates category membership.

Pitfall III: Open World Recognition. The final problem we consider is the encounter with the open world [15-17]. When a model is exposed to a novel category, the vague definition of category makes it hard to deduce what will be an unseen new category. While open-set recognition models [18-22] can still evade this dilemma by rejecting new categories, Novel Class Discovery [23-26] or Generalized Category Discovery [27-31] can not ignore the fundamental flaw of a lack of definition for a category . This problem is heightened when categories are fine-grained [32, 33] or follow a long-tailed distribution [34-37].

In this paper, we confront these challenges by reframing the concept of a category as the solution to an optimization problem. We argue that categories serve to describe input data and that there is not a singular correct category but a sequence of descriptions that span different levels of abstraction. We demonstrate that considering categorization as a search for a sequence of category codes not only provides more flexibility when dealing with novel categories but also, by leveraging sequences, allows us to modulate the granularity of categorization, proving especially beneficial for fine-grained novel categories. Subsequently, we illustrate how to construct a framework capable of efficiently approximating this solution. Our key contributions are as follows:

Theoretical. We conceptualize a category as a solution to an optimization problem. We then demonstrate how to fine-tune this optimization framework such that its mathematical solutions align with the human-accepted notion of categories . Furthermore, under a set of well-defined constraints, we establish that our method theoretically yields an optimal solution.

Methodological. Based on the theory we developed, we propose a practical method for tackling the generalized category discovery problem, which is also robust to different category granularities.

Experimental. We empirically show that our method outperforms state-of-the-art generalized category discovery and adapted novel class discovery methods on fine-grained datasets while performing consistently well on coarse-grained datasets.

Before detailing our contributions, we first provide some background on category discovery to better contextualize our work.

2 Background

The Generalized Category Discovery problem introduced by Vaze et al. [27] tries to categorize a set of images during inference, which can be from the known categories seen during training or novel categories. Formally, we only have access to Y S or seen categories during training time, while we aim to categorize samples from novel categories or Y U during test time. For the Novel Class Discovery problem, it is assumed that Y S ∩ Y U = ∅ . However, this assumption could be unrealistic for real-world data. So Vaze et al. [27] proposed to use instead the more realistic Generalized Category Discovery assumption in which the model can encounter both seen and unseen categories during test time. In short, for the Generalized Category Discovery problem, we have Y S ⊂ Y U .

One major conundrum with both Generalized Category Discovery and Novel Category Discovery is that the definition of the category has remained undetermined. This complication can be overlooked when the granularity of categories at test time is similar to training time. However, for more realistic applications where test data may have different granularity from training data or categories may follow a long-tailed distribution, the definition of category becomes crucial. To this end, in the next section, we formulate categories as a way to abstract information in the input data.

3 An Information Theory Approach to Category Coding

To convert a subjective concept as a category to a formal definition, we must first consider why categorization happens in the first place. There are many theories regarding this phenomenon in human [38-40] and even animal brains [41-43]. One theory is categorization was a survival necessity that the human brain developed to retrieve data as fast and as accurately as possible [44]. Studies have shown that there could be a trade-off between retrieval speed and accuracy of prediction in the brain [45-47]. Meanwhile, other studies have shown that the more frequent categories can be recognized in a shorter time, with more time needed to recognize fine-grained nested subcategories [48, 49]. These studies might suggest shorter required neural pulses for higher hierarchy levels. Inspired by these studies, we propose categorization as an optimization problem with analogous goals to the human brain. We hypothesize that we can do the category assignment to encode objects hierarchically to retrieve them as accurately and quickly as possible.

Notation and Definitions. Let us first formalize our notation and definitions. We denote the input random variable with X and the category random variable with C . The category code random variable, which we define as the embedding sequence of input X i , is denoted by z i = z i 1 z i 2 · · · z i L , in which superscript i shows the i th sample, while subscript L shows the digit position in the code sequence. In addition, I ( X ; Z ) indicates the mutual information between random variables X and Z [50, 51], which measures the amount of information we can obtain for one random variable by observing the other one. Since category codes are sequences, algorithmic information theory is most suitable for addressing the problem. We denote the algorithmic mutual information for sequences x and z with I alg (x : z) , which specifies how much information about sequence x we can obtain by observing sequence z . Both Shannon and algorithmic information-theory-based estimators are useful for hierarchical clustering [52-56], suggesting we may benefit from this quality to simulate the implicit category hierarchy. A more in-depth discussion can be found in the supplemental.

3.1 Maximizing the Algorithmic Mutual Information

Let's consider data space D = { X i , C i : i ∈ { 1 , · · · N }} where X s are inputs and C s are the corresponding category labels.

Lemma 1 For each category c and for X i with C i =c , we can find a binary decision tree T c that starting from its root, reaches each X i by following the decision tree path. Based on this path, we assign code c ( X i )= c i 1 c i 2 · · · c i M to each X i to uniquely define and retrieve it from the tree. M is the length of the binary code assigned to the sample.

Proof of Lemma 2 is provided in the supplemental. Based on this lemma, we can find a forest with categories c as the roots and samples X i s as their leaves. We apply the same logic to find a super decision tree T that has all these category roots as its leaves. If we define the path code of category c in this super tree by p ( c )= p c 1 p c 2 · · · p c K where K is the length of the path to the category c ; we find the path to each X i in the supertree by concatenating its category path code with its code in the category decision tree. So for each input X i with category c we define an address code as q i 1 q i 2 · · · q i K + M in which q i j = p c j for j ≤ K and q i j = c i j -K for j > K . Meanwhile, since all X i s are the descendants

of root c in the T c tree, we know there is one encoding to address all samples, in which samples of the same category share a similar prefix. Now consider a model that provides the binary code z i = z i 1 · · · z i L for data input X i with category c , let's define a valid encoding in Definition 2.

Definition 1 A valid encoding for input space X and category space C is defined as an encoding that uniquely identifies every X i ∈ X . At the same time, for each category c ∈ C , it ensures that there is a sequence that is shared among all members of this category but no member out of the category.

As mentioned before, if the learned tree for this encoding is isomorph to the underlying tree T , we will have the necessary conditions that Theorem 1 provides.

Theorem 1 For a learned binary code z i to address input X i , uniquely, if the decision tree of this encoding is isomorph to underlying tree T , we will have the following necessary conditions:

I alg (z : x ) ≥ I alg (˜ z : x ) ∀ ˜ z, ˜ z is a valid encoding for x

I alg (z : c ) ≥ I alg (˜ z : c ) ∀ ˜ z, ˜ z is a valid encoding for x

Proof of Theorem 1 is provided in the supplemental. Optimizing for these two measures provides an encoding that satisfies the necessary conditions. However, from the halting Theorem [57], this optimization is generally not computable [58-61].

Theorem 1 Clarification . The first part of Theorem 1 states that if there is an implicit hierarchy tree, then for any category tree that is isomorph to this implicit tree, the algorithmic mutual information between each sample and its binary code generated by the tree will be maximal for the optimal tree. Hence, maximizing this mutual information is a necessary condition for finding the optimal tree. This is equivalent to finding a tree that generates the shortest-length binary code to address each sample uniquely. The second part of Theorem 1 states that for the optimal tree, the algorithmic mutual information between each sample category and its binary code will be maximum. Hence, again, maximizing this mutual information is a necessary condition for finding the optimal tree. This is equivalent to finding a tree that generates the shortest-length binary code to address each category uniquely. This means that since the tree should be a valid tree, the prefix to the unique address of every category sample c should be the shortest-length binary code while not being the prefix of any sample from other categories.

Shannon Mutual Information Approximation . We can approximate these requirements using Shannon mutual information instead if we consider a specific set of criteria. First, since Shannon entropy does not consider the relationship between separate bits or z i j s, we convert each sequence to an equivalent random variable number by considering its binary digit representation. To this end, we consider Z i = ∑ m k =1 z i k 2 k , which is a number between 0 and 1 . To replace the first item of Theorem 1 by its equivalent Shannon mutual information, we must also ensure that z has the minimum length. For the moment, let's assume we know this length by the function l ( X i )= l i . Hence instead of Z i ,

i

we consider its truncated form Z i l i = ∑ l i k =1 z k 2 k . This term, which we call the address loss function, is defined as follows:

We can approximate this optimization with a contrastive loss. However, there are two requirements that we must consider; First, we have to obtain the optimal code length l i , and second, we have to ensure z i k is binary. In the following sections, we illustrate how we can satisfy these requirements.

3.2 Category Code Length Minimization

To find the optimal code lengths l i in Equation 1, we have to minimize the total length of the latent code. We call this loss L length, which we define as L length = 1 N ∑ N i =0 l i . However, since the l i are in the subscripts of Equation 1, we can not use the conventional optimization tools to optimize this length. To circumvent this problem, we define a binary mask sequence m i = m i 1 m i 2 · · · m i L to simulate the subscript property of l i . Consider a masked version of z i = z i 1 · · · z i L , which we will denote as ˜ z i =˜ z i 1 · · · ˜ z i L , in which for 1 ≤ k ≤ L , we define ˜ z i k = z i k m i k . The goal is to minimize the number of ones in sequence m i while forcing them to be at the beginning of the sequence. One way to ensure this is to consider the sequence ¯ m i =( m i 1 2 1 )( m i 2 2 2 ) · · · ( m i L 2 L ) and minimize its L p Norm for p ≥ 1 .

This will ensure the requirements because adding one extra bit has an equivalent loss of all previous bits. In the supplemental, we provide a more rigorous explanation.

We extract the mask from the input X i , i.e., m i = Mask ( X i ) . Mask digits should also be binary, so we need to satisfy their binary constraints, which we will address next.

Satisfying Binary Constraints. Previous optimizations are constrained to two conditions, Code Constraint: ∀ z i k , z i k = 0 or z i k = 1 and Mask Constraint: ∀ m i k , m i k = 0 or m i k = 1 . We formulate each constraint in an equivalent Lagrangian function to make sure they are satisfied. For the binary code constraint we consider f code ( z i k )=( z i k )(1 -z i k )=0 , which is only zero if z i k =0 or z i k =1 . Similarly, for the binary mask constraint, we have f mask ( m i k )=( m i k )(1 -m i k )=0 . To ensure these constraints are satisfied, we optimize them with the Lagrangian function of the overall loss.

3.3 Aligning Category Codes using Supervision Signals

The second item in Theorem 1 shows the necessary condition for maximizing mutual information with a category. If we replace algorithmic mutual information with its Shannon cousin, we will have:

Subject to satisfying the binary constraints. Note that the optimal lengths might differ for optimizing information based on categories or input. However, here we consider the same length for both scenarios for simplicity.

Overall Loss. Putting all these losses and constraints together, we will reach the constrained loss:

Note that when we do not have the supervision signals, we can consider γ =0 and extract categories in an unsupervised manner. In the supplemental, we have shown how to maximize this function based on the Lagrange multiplier. If we indicate L code_cond = ∑ N i =0 ∑ L k =1 ( z i k ) 2 (1 -z i k ) 2 and L mask_cond = ∑ N i =0 ∑ L k =1 ( m i k ) 2 (1 -m i k ) 2 . The final loss will be:

Note that for satisfying binary constraints, we can adopt other approaches. For instance, we can omit the requirement for this hyperparameter by using binary neural networks and an STE (straight-through estimator) [62]. Another approach would be to benefit from Boltzmann machines [63] to have a binary code. Having defined our theoretical objective, we are now ready to make it operational.

4 InfoSieve: Self-supervised Code Extraction

In this section, using the equations from Section 3, we devise a framework to extract category codes. Note that as Theorem 1 indicates, when we train the model to extract the category codes instead of the categories themselves, we make the model learn the underlying category tree. The model must ensure that in its implicit category tree, there is a node for each category whose descendants all share the same category while there are no non-descendants of this node with the same category.

The overall framework of our model, named InfoSieve, is depicted in Figure 3. We first extract an embedding using the contrastive loss used by [27]. Then our Code Generator uses this embedding to generate binary codes, while our Code Masker learns a mask based on these embeddings to minimize the code length. Ultimately, the Categorizer uses this truncated code to discern groundtruth categories. In the next sections, we explain each component in more detail.

Input

Feature

Extractor

Code

Generator

L

u

C_code

Code

Masker

L

code_cond

L

mask_cond

L

length

L

C_in

×

L

s

C_code

L

Cat

4.1 Contrastive Learning of Code and Input

One of the advantages of contrastive learning is to find a representation that maximizes the mutual information with the input [64]. For input X i , let's show the hidden representation learning with Z i , which is learned contrastively by minimizing the InfoNCE loss. Van den Oord et al. [64] showed that minimizing InfoNCE loss increases a lower bound for mutual information. Hence, contrastive learning with the InfoNCE loss can be a suitable choice for minimizing the L adr in Equation 1. We will use this to our advantage on two different levels. Let's consider that Z i has dimension d , and each latent variable z i k can take up n different values. The complexity of the feature space for this latent variable would be O ( n d ) , then as we show in supplemental, the number of structurally different binary trees for this feature space will be O (4 n d ) . So minimizing n and d will be the most effective way to limit the number of possible binary trees. Since our model and the amount of training data is bounded, we must minimize the possible search space while providing reasonable performance. At the same time, the input feature space X i with N possible values and dimension D has O ( N D ) possible states. To cover it completely, we can not arbitrarily decrease d and n . Note that for a nearly continuous function N → ∞ , the probability of a random discrete tree fully covering this space would be near zero. To make the best of both worlds, we consider different levels of complexity of latent variables, each focusing on one of these goals.

Minimizing Contrastive Loss on the Inputs. Similar to [27], we use this unsupervised contrastive loss to maximize the mutual information between input X i and the extracted latent embedding Z i . Akin to [27], we also benefit from the supervised contrastive learning signal for the members of a particular category. Let's assume that the number of categories in the entire dataset is C . Different members of a category can be seen as different views of that category, analogous to unsupervised contrastive loss. Hence, they combine these unsupervised contrastive loss or L u C_in and its supervised counterpart, L s C_in with a coefficient λ , which we call λ in in a following manner:

For covering the input space, X i , the loss function from Equation 6 is more suited if we consider both input and latent features as approximately continuous. We have shown this loss by L C_in in Figure 3.

Minimizing Contrastive Loss on the Codes. In order to also facilitate finding the binary tree, we map the latent feature extracted from the previous section to a binary code with minimum length. Hence we effectively decrease n to 2 while actively minimizing d . Furthermore, we extract a code by making the value of each bit in this code correspond to its sequential position in a binary number, i.e., each digit has twice the value of the digit immediately to its right. This ensures the model treats this code as a binary tree coding with its root in the first digit. We consider unsupervised contrastive learning for raw binary digits L u C_code and supervised variants of this loss for the extracted code, which we will show by and L s C_code , the total contrastive loss for the binary embedding is defined as:

In summary, the loss from Equation 6 finds a tree compatible with the input, while the loss from Equation 7 learns an implicit tree in compliance with categories. Then we consider L adr as their combination:

4.2 Minimizing Code Length

As discussed in Section 4.1, we need to decrease the feature space complexity by using minimum length codes to reduce the search space for finding the implicit category binary tree. In addition, using Shannon's mutual information as an approximate substitute for algorithmic mutual information necessitates minimizing the sequence length. We must simultaneously learn category codes and their optimal length. Since each of these optimizations depends on the optimal solution of the other one, we use two different blocks to solve them at the same time.

Code Generator Block. In Figure 3, the Code Generator block uses the extracted embeddings to generate binary category codes. At this stage, we consider a fixed length for these binary codes. The output of this stage is used for unsupervised contrastive learning on the codes in Equation 7. We also use L code_cond to enforce the digits of the codes to be a decent approximation of binary values.

Code Masker Block. For this block, we use the L mask_cond to ensure the binary constraint of the outputs. In addition, to control the length of the code or, in other words, the sequence of 1 s at the beginning, we use L length in Equation 2.

4.3 Aligning Codes using Supervision Signals

In the final block of the framework, we convert category codes from the Code Generator to a binary number based on the digit positions in the sequence. To truncate this code, we do a Hadamard multiplication for this number by the mask generated by the Code Masker . We use these truncated codes for supervised contrastive learning on the codes. Finally, we feed these codes to the Catgorizer block to predict the labels directly. We believe relying solely on contrastive supervision prevents the model from benefiting from learning discriminative features early on to speed up training.

5 Experiments

5.1 Experimental Setup

Eight Datasets. We evaluate our model on three coarse-grained datasets CIFAR10/100 [65] and ImageNet-100 [66] and four fine-grained datasets: CUB-200 [67], Aircraft [68], SCars [69] and Oxford-Pet [70]. Finally, we use the challenging Herbarium19 [71] dataset, which is fine-grained and long-tailed. To acquire the train and test splits, we follow [27]. We subsample the training dataset in a ratio of 50% of known categories at the train and all samples of unknown categories. For all datasets except CIFAR100, we consider 50% of the categories as known categories at training time. For CIFAR100, 80% of the categories are known during training time, as in [27]. A summary of dataset statistics and their train test splits is shown in the supplemental.

Implementation Details. Following [27], we use ViT-B/16 as our backbone, which is pre-trained by DINO [72] on unlabelled ImageNet 1K [4]. Unless otherwise specified, we use 200 epochs and batch size of 128 for training. We present the complete implementation details in the supplemental. Our code is available at: https://github.com/SarahRastegar/InfoSieve .

Evaluation Metrics. We use semi-supervised k -means proposed by [27] to cluster the predicted embeddings. Then, the Hungarian algorithm [73] solves the optimal assignment of emerged clusters to their ground truth labels. We report the accuracy of the model's predictions on All , Known , and Novel categories. Accuracy on All is calculated using the whole unlabelled train set, consisting of known and unknown categories. For Known , we only consider the samples with labels known during training. Finally, for Novel , we consider samples from the unlabelled categories at train time.

5.2 Ablative studies

We investigate each model's component contribution to the overall performance of the model and the effect of each hyperparameter. Further ablations can be found in the supplemental.

Effect of Each Component. We first examine the effect of each component using the CUB dataset. A fine-grained dataset like CUB requires the model to distinguish between the semantic nuances of each category. Table 1 shows the effect of each loss component for the CUB dataset. As we can see from this table, L C_code and L length have the most positive effect on novel categories while affecting known categories negatively. Utilizing L code_cond to enforce binary constraints on the embedding enhances performance for both novel and known categories. Conversely, applying L mask_cond boosts performance for known categories while detrimentally impacting novel categories. This is aligned with Theorem 1 and the definition of the category because these losses have opposite goals. As discussed, minimizing the search space helps the model find the implicit category tree faster. L C_code and L length try to achieve this by mapping the information to a smaller feature space, while condition

losses solve this by pruning and discarding unnecessary information. Finally, L Cat on its own has a destructive effect on known categories. One reason for this is the small size of the CUB dataset, which makes the model overfit on labeled data. However, when all losses are combined, their synergic effect helps perform well for both known and novel categories.

5.3 Hyperparameter Analysis

Our model has a few hyperparameters: code binary constraint ( ζ ), mask binary constraint ( µ ), code contrastive ( β ), code length ( δ ), and code mapping ( η ). We examine the effect of each hyperparameter on the model's performance on the CUB dataset. Our default values for the hyperparameters are: code constraint coeff ζ =0 . 01 , mask constraint coeff µ =0 . 01 , code contrastive coeff β =1 , code mapping coeff η =0 . 01 and code length coeff δ =0 . 1 .

Code Binary Constraint. This hyperparameter is introduced to satisfy the binary requirement of the code. Since we use tanh to create the binary vector, the coefficient only determines how fast the method satisfies the conditions. When codes 0 and 1 are stabilized, the hyperparameter effect will be diminished. However, we noticed that more significant coefficients somewhat affect the known accuracy. The effect of this hyperparameter for the CUB dataset is shown in Table 2 (a). We can see that the method is robust to the choice of this hyperparameter.

Mask Binary Constraint. For the mask constraint hyperparameter, we start from an all-one mask in our Lagrange multiplier approach. A more significant constraint translates to a longer category code. A higher coefficient is more useful since it better imposes the binary condition for the mask. The effect of different values of this hyperparameter for the CUB datasets is shown in Table 2 (b).

Code Contrastive. This loss maintains information about the input. From Table 2 (c), we observe that minimizing the information for a fine-grained dataset like CUB will lead to a better performance.

Code Length. Table 2 (d) reports our results for different values of code length hyperparameter. Since the code length is penalized exponentially, the code length hyperparameter's effect is not comparable to the exponential growth; hence, in the end, the model is not sensitive to this hyperparameter's value.

Code Mapping. Since current evaluation metrics rely on predefined categories, our category codes must be mapped to this scenario. This loss is not an essential part of the selfcoding that our model learns, but it accelerates model training. The effect of this hyperparameter is shown in Table 3.

Overall Parameter Analysis One reason the model is not very sensitive to different hyperparameters is that our model consists of three separate parts: Code masker, Code Generator, and Categorizer. The only hyperparameters that affect all of these three parts directly are β , the code contrastive coefficient, and δ , the code length coefficient. Hence, these hyperparameters affect our model's performance more.

5.4 Comparison with State-of-the-Art

Fine-grained Image Classification. Fine-grained image datasets are a more realistic approach to the real world. In coarse-grained datasets, the model can use other visual cues to guess about the novelty of a category; fine-grained datasets require that the model distinguish subtle category-specific details. Table 4 summarizes our model's performance on the fine-grained datasets. As we can see from this table, our model has more robust and consistent results compared to other methods for fine-grained datasets. Herbarium 19, a long-tailed dataset, raises the stakes by having different frequencies for different categories, which is detrimental to most clustering approaches because of the extremely unbalanced cluster size. As Table 4 shows, our model can distinguish different categories even from a few examples and is robust to frequency imbalance.

Coarse-grained Image Classification. Our method is well suited for datasets with more categories and fine distinctions. Nevertheless, we also evaluate our model on three coarse-grained datasets, namely CIFAR10 and CIFAR100 [65] and ImageNet-100 [66]. Table 5 compares our results against state-of-the-art generalized category discovery methods. As we can see from this table, our method performs consistently well on both known and novel datasets. For instance, while UNO+ shows the highest accuracy on the Known categories of CIFAR-10, this is at the expense of performance degradation on the Novel categories. The same observations can be seen on ImageNet-100. Table 5 shows that our method consistently performs competitively for both novel and known categories. Based on our theory, the smaller CIFAR10/100 and ImageNet-100 improvement is predictable. For CIFAR 10, the depth of the implicit tree is 4; hence, the number of implicit possible binary trees with this limited depth is smaller, meaning finding a good approximation for the implicit category tree can be achieved by other models. However, as the depth of this tree increases, our model can still find the aforementioned tree; hence, we see more improvement for fine-grained data.

5.5 Qulitative results

In supplemental, we explain how to extract the implicit tree that is learned by our model. For instance, Figure 2 shows this extracted tree structure on the CUB dataset. We see that our method can extract some hierarchical structure in the data. In the leaves of this tree, we have 'yellow sparrows,' instances which are the descendants of a more abstract 'sparrow' node; or in the path '00...1', we have a 'blackbirds' node, which can encompass multiple blackbird species as its children nodes.

6 Related Works

Novel Category Discovery can be traced back to Han et al. [79], where knowledge from labeled data was used to infer the unknown categories. Following this work, Zhong et al. [80] solidified the novel class discovery as a new specific problem. The main goal of novel class discovery is to transfer the implicit category structure from the known categories to infer unknown categories [24, 26, 75, 76, 81, 82, 82-99]. Prior to the novel class discovery, the problem of encountering new classes at the test time was investigated by open-set recognition [16, 17, 20, 100]. However, the strategy of dealing with these new categories is different. In the open-set scenario, the model rejects the samples from novel categories, while novel class discovery aims to benefit from the vast knowledge of the unknown realm and infer the categories. However, the novel class discovery has a limiting assumption that test data only consists of novel categories. For a more realistic setting, Generalized Category Discovery considers both known and old categories at test time.

Generalised Category Discovery recently has been introduced by [27] and concurrently under the name Open-world semi-supervised learning by [77]. In this scenario, while the model should not lose its grasp on known categories, it must discover novel ones at test time. This adds an extra challenge because when we adapt the novel class discovery methods to this scenario, they are biased to either novel or known categories and miss the other group. There has been a recent surge of interest in generalized category discovery [28-31, 78, 101-110]. In this work, instead of viewing categories as an end, we investigated the fundamental question of how to conceptualize the category itself.

Decision Tree Distillation. The benefits of the hierarchical nature of categories have been investigated previously. Xiao [111] and Frosst and Hinton [112] used a decision tree in order to make the categorization interpretable. Adaptive neural trees proposed by [113] assimilate representation learning to its edges. Ji et al. [114] use attention binary neural tree to distinguish fine-grained categories by attending to the nuances of these categories. However, these methods need an explicit tree structure. In this work, we let the network extract this implicit tree on its own. This way, our model is also suitable when an explicit tree structure does not exist.

For a more comprehensive related work, see the supplemental.

7 Conclusion

This paper seeks to address the often neglected question of defining a category . While the concept of a category is readily accepted as a given in the exploration of novel categories and open-world settings, the inherent subjectivity of the term poses a challenge when optimizing deep networks, which are fundamentally mathematical frameworks. To circumvent this issue, we put forth a mathematical solution to extract category codes, replacing the simplistic one-hot categorical encoding. Further, we introduce a novel framework capable of uncovering unknown categories during inference and iteratively updating its environmental representation based on newly acquired knowledge. These category codes also prove beneficial in handling fine-grained categorization, where attention to nuanced differences between categories is paramount. By illuminating the limitations of the subjective notion of category and recasting them within a mathematical framework, we hope to catalyze the development of less human-dependent models.

8 Limitations

Despite the endeavors made in this work, there are several notable limitations. Foremost is our assumption of an implicit hierarchical tree underlying categorization. Furthermore, our optimization solution may not align perfectly with the actual implicit hierarchy, as it only satisfies the necessary conditions outlined in Theorem 1. Lastly, akin to other approaches in the generalized category discovery literature, we operate under the assumption that we have access to unlabeled data from unknown categories during training. This assumption, while convenient for model development, may not always hold true in real-world applications.

Acknowledgments and Disclosure of Funding

This work is part of the project Real-Time Video Surveillance Search with project number 18038, which is (partly) financed by the Dutch Research Council (NWO) domain Applied and Engineering/ Sciences (TTW). Special thanks to Dr. Dennis Koelma and Dr. Efstratios Gavves for their valuable insights about conceptualizing the category to alleviate generalized category discovery.

References

William Croft and D Alan Cruse. Cognitive linguistics . Cambridge University Press, 2004.

George Lakoff. Women, fire, and dangerous things: What categories reveal about the mind . University of Chicago press, 2008.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations , 2015.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4700-4708, 2017.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1-9, 2015.

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. In Proceedings of the IEEE/CVF international Conference on Computer Vision , pages 5138-5147, 2019.

Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition. In Proceedings of the IEEE International Conference on Computer Vision , pages 2740-2748, 2015.

Tianshui Chen, Wenxi Wu, Yuefang Gao, Le Dong, Xiaonan Luo, and Liang Lin. Fine-grained representation learning and recognition by exploiting hierarchical semantic embedding. In Proceedings of the ACM International Conference on Multimedia , pages 2023-2031, 2018.

Pascal Mettes, Mina Ghadimi Atigh, Martin Keller-Ressel, Jeffrey Gu, and Serena Yeung. Hyperbolic deep learning in computer vision: A survey. arXiv preprint arXiv:2305.06611 , 2023.

Mina Ghadimi Atigh, Julian Schoep, Erman Acar, Nanne Van Noord, and Pascal Mettes. Hyperbolic image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4453-4462, 2022.

Fangfei Lin, Bing Bai, Kun Bai, Yazhou Ren, Peng Zhao, and Zenglin Xu. Contrastive multi-view hyperbolic hierarchical clustering. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence , 2022.

Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out of-distribution detection: Solutions and future challenges. Transactions of Machine Learning Research , 2022.

Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1893-1902, 2015.

Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1563-1572, 2016.

Walter J Scheirer, Lalit P Jain, and Terrance E Boult. Probability models for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 36(11):2317-2324, 2014.

Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision , pages 754-763, 2017.

Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(7):1757-1772, 2012.

Kai Katsumata, Duc Minh Vo, and Hideki Nakayama. Ossgan: Open-set semi-supervised image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Jiaming Han, Yuqiang Ren, Jian Ding, Xingjia Pan, Ke Yan, and Gui-Song Xia. Expanding low-density latent regions for open-set object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9591-9600, 2022.

Colin Troisemaine, Vincent Lemaire, Stéphane Gosselin, Alexandre Reiffers-Masson, Joachim FloconCholet, and Sandrine Vaton. Novel class discovery: an introduction and key concepts. arXiv preprint arXiv:2302.12028 , 2023.

Yuyang Zhao, Zhun Zhong, Nicu Sebe, and Gim Hee Lee. Novel class discovery in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Muli Yang, Yuehua Zhu, Jiaping Yu, Aming Wu, and Cheng Deng. Divide and conquer: Compositional experts for generalized novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14268-14277, 2022.

KJ Joseph, Sujoy Paul, Gaurav Aggarwal, Soma Biswas, Piyush Rai, Kai Han, and Vineeth N Balasubramanian. Novel class discovery without forgetting. In European Conference on Computer Vision , pages 570-586. Springer, 2022.

Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Sheng Zhang, Salman Khan, Zhiqiang Shen, Muzammal Naseer, Guangyi Chen, and Fahad Khan. Promptcal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Nan Pu, Zhun Zhong, and Nicu Sebe. Dynamic conceptional contrastive learning for generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Florent Chiaroni, Jose Dolz, Ziko Imtiaz Masud, Amar Mitiche, and Ismail Ben Ayed. Parametric information maximization for generalized category discovery. arXiv preprint arXiv:2212.00334 , 2022.

Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category discovery: A baseline study. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16590-16600, 2023.

Yixin Fei, Zhongkai Zhao, Siwei Yang, and Bingchen Zhao. Xcon: Learning with experts for fine-grained category discovery. In British Machine Vision Conference , 2022.

Wenbin An, Feng Tian, Ping Chen, Siliang Tang, Qinghua Zheng, and QianYing Wang. Fine-grained category discovery under coarse-grained supervision with hierarchical weighted self-contrastive learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , 2022.

Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Yingjun Du, Jiayi Shen, Xiantong Zhen, and Cees GM Snoek. Superdisco: Super-class discovery improves visual recognition for the long-tail. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Yingxiao Du and Jianxin Wu. No one left behind: Improving the worst categories in long-tailed learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Sumyeong Ahn, Jongwoo Ko, and Se-Young Yun. Cuda: Curriculum of data augmentation for long-tailed recognition. In Proceedings of the International Conference on Learning Representations , 2023.

Haley A Vlach. How we categorize objects is related to how we remember them: The shape bias as a memory bias. Journal of Experimental Child Psychology , 152:12-30, 2016.

Thomas J Palmeri and Isabel Gauthier. Visual object understanding. Nature Reviews Neuroscience , 5(4): 291-303, 2004.

Alec Scharff, John Palmer, and Cathleen M Moore. Evidence of fixed capacity in visual object categorization. Psychonomic bulletin & review , 18:713-721, 2011.

Sandra Reinert, Mark Hübener, Tobias Bonhoeffer, and Pieter M Goltstein. Mouse prefrontal cortex represents learned rules for categorization. Nature , 593(7859):411-417, 2021.

Pieter M Goltstein, Sandra Reinert, Tobias Bonhoeffer, and Mark Hübener. Mouse visual cortex areas represent perceptual and semantic features of learned visual categories. Nature Neuroscience , 24(10): 1441-1451, 2021.

Nakul Yadav, Chelsea Noble, James E Niemeyer, Andrea Terceros, Jonathan Victor, Conor Liston, and Priyamvada Rajasethupathy. Prefrontal feature representations drive memory recall. Nature , 608(7921): 153-160, 2022.

Eun Jin Sim and Markus Kiefer. Category-related brain activity to natural categories is associated with the retrieval of visual features: Evidence from repetition effects during visual and functional judgments. Cognitive Brain Research , 24(2):260-273, 2005.

Glyn W Humphreys and Emer ME Forde. Hierarchies, similarity, and interactivity in object recognition:'category-specific' neuropsychological deficits. Behavioral and Brain Sciences , 24(3): 453-476, 2001.

Sander Van de Cruys, Kris Evers, Ruth Van der Hallen, Lien Van Eylen, Bart Boets, Lee De-Wit, and Johan Wagemans. Precise minds in uncertain worlds: predictive coding in autism. Psychological Review , 121(4):649, 2014.

Hermann Haken. Brain dynamics: synchronization and activity patterns in pulse-coupled neural nets with delays and noise . Springer Science & Business Media, 2006.

Edward E Smith, Edward J Shoben, and Lance J Rips. Structure and process in semantic memory: A featural model for semantic decisions. Psychological Review , 81(3):214, 1974.

James J DiCarlo, Davide Zoccolan, and Nicole C Rust. How does the brain solve visual object recognition? Neuron , 73(3):415-434, 2012.

Thomas M Cover. Elements of information theory . John Wiley & Sons, 1999.

Claude E Shannon. A mathematical theory of communication. The Bell system technical journal , 27(3): 379-423, 1948.

Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057 , 2000.

Mehdi Aghagolzadeh, Hamid Soltanian-Zadeh, B Araabi, and Ali Aghagolzadeh. A hierarchical clustering based on mutual information maximization. In IEEE International Conference on Image Processing , volume 1, pages I-277, 2007.

Ming Li, Jonathan H Badger, Xin Chen, Sam Kwong, Paul Kearney, and Haoyong Zhang. An informationbased sequence distance and its application to whole mitochondrial genome phylogeny. Bioinformatics , 17(2):149-154, 2001.

Ming Li, Xin Chen, Xin Li, Bin Ma, and Paul MB Vitányi. The similarity metric. IEEE Transactions on Information Theory , 50(12):3250-3264, 2004.

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E , 69(6):066138, 2004.

Alan Mathison Turing et al. On computable numbers, with an application to the entscheidungsproblem. J. of Math , 58(345-363):5, 1936.

Paul MB Vitányi. How incomputable is kolmogorov complexity? Entropy , 22(4):408, 2020.

Ray J Solomonoff. A formal theory of inductive inference. part i. Information and Control , 7(1):1-22, 1964.

Ray J Solomonoff. A formal theory of inductive inference. part ii. Information and control , 7(2):224-254, 1964.

Andrei N Kolmogorov. Three approaches to the quantitative definition of information. Problems of Information Transmission , 1(1):1-7, 1965.

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013.

Geoffrey E Hinton. A practical guide to training restricted boltzmann machines. In Neural Networks: Tricks of the Trade: Second Edition , pages 599-619. Springer, 2012.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops , pages 554-561, 2013.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3498-3505. IEEE, 2012.

Kiat Chuan Tan, Yulong Liu, Barbara Ambrose, Melissa Tulig, and Serge Belongie. The herbarium challenge 2019 dataset. arXiv preprint arXiv:1906.05372 , 2019.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9650-9660, 2021.

MBWright. Speeding up the Hungarian algorithm. Computers & Operations Research , 17(1):95-96, 1990.

David Arthur and Sergei Vassilvitskii. K-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms , pages 1027-1035, 2007.

Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. In Proceedings of the International Conference on Learning Representations , 2020.

Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9284-9292, 2021.

Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In Proceedings of the International Conference on Learning Representations , 2022.

Bingchen Zhao, Xin Wen, and Kai Han. Learning semi-supervised gaussian mixture models for generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.

Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8401-8409, 2019.

Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10867-10875, 2021.

Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning , pages 478-487. PMLR, 2016.

Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. Advances in Neural Information Processing Systems , 34:22982-22994, 2021.

Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9462-9470, 2021.

Subhankar Roy, Mingxuan Liu, Zhun Zhong, Nicu Sebe, and Elisa Ricci. Class-incremental novel class discovery. In European Conference on Computer Vision , pages 317-333. Springer, 2022.

Mamshad Nayeem Rizve, Navid Kardan, and Mubarak Shah. Towards realistic semi-supervised learning. In European Conference on Computer Vision , pages 437-455. Springer, 2022.

Wenbin Li, Zhichen Fan, Jing Huo, and Yang Gao. Modeling inter-class and intra-class constraints in novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3449-3458, 2023.

Muli Yang, Liancheng Wang, Cheng Deng, and Hanwang Zhang. Bootstrap your own prior: Towards distribution-agnostic novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3459-3468, 2023.

Luigi Riz, Cristiano Saltori, Elisa Ricci, and Fabio Poiesi. Novel class discovery for 3d point cloud semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9393-9402, 2023.

Peiyan Gu, Chuyu Zhang, Ruijie Xu, and Xuming He. Class-relation knowledge distillation for novel class discovery. In Proceedings of the International Conference on Learning Representations , 2023.

Yiyou Sun, Zhenmei Shi, Yingyu Liang, and Yixuan Li. When and how does known class help discover unknown ones? provable understanding through spectral analysis. In International Conference on Machine Learning . PMLR, 2023.

Zhang Chuyu, Xu Ruijie, and He Xuming. Novel class discovery for long-tailed recognition. arXiv preprint arXiv:2308.02989 , 2023.

Colin Troisemaine, Joachim Flocon-Cholet, Stéphane Gosselin, Alexandre Reiffers-Masson, Sandrine Vaton, and Vincent Lemaire. An interactive interface for novel class discovery in tabular data. arXiv preprint arXiv:2306.12919 , 2023.

Ziyun Li, Jona Otholt, Ben Dai, Di Hu, Christoph Meinel, and Haojin Yang. Supervised knowledge may hurt novel class discovery performance. arXiv preprint arXiv:2306.03648 , 2023.

Jiaming Liu, Yangqiming Wang, Tongze Zhang, Yulu Fan, Qinli Yang, and Junming Shao. Open-world semi-supervised novel class discovery. arXiv preprint arXiv:2305.13095 , 2023.

Haoang Chi, Feng Liu, Bo Han, Wenjing Yang, Long Lan, Tongliang Liu, Gang Niu, Mingyuan Zhou, and Masashi Sugiyama. Meta discovery: Learning to discover novel classes given very limited data. In Proceedings of the International Conference on Learning Representations , 2022.

Xinwei Zhang, Jianwen Jiang, Yutong Feng, Zhi-Fan Wu, Xibin Zhao, Hai Wan, Mingqian Tang, Rong Jin, and Yue Gao. Grow and merge: A unified framework for continuous categories discovery. In Advances in Neural Information Processing Systems , 2022.

Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Self-labeling framework for novel category discovery over domains. In Proceedings of the AAAI Conference on Artificial Intelligence , 2022.

KJ Joseph, Sujoy Paul, Gaurav Aggarwal, Soma Biswas, Piyush Rai, Kai Han, and Vineeth N Balasubramanian. Spacing loss for discovering novel categories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3761-3766, 2022.

Ziyun Li, Jona Otholt, Ben Dai, Christoph Meinel, Haojin Yang, et al. A closer look at novel class discovery from the labeled set. arXiv preprint arXiv:2209.09120 , 2022.

Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In Proceedings of the International Conference on Learning Representations , 2022.

Shaozhe Hao, Kai Han, and Kwan-Yee K Wong. Cipr: An efficient framework with cross-instance positive relations for generalized category discovery. arXiv preprint arXiv:2304.06928 , 2023.

Ruoyi Du, Dongliang Chang, Kongming Liang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. On-the-fly category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11691-11700, 2023.

Jianhong Bai, Zuozhu Liu, Hualiang Wang, Ruizhe Chen, Lianrui Mu, Xiaomeng Li, Joey Tianyi Zhou, Yang Feng, Jian Wu, and Haoji Hu. Towards distribution-agnostic generalized category discovery. In Advances in Neural Information Processing Systems , 2023.

Sagar Vaze, Andrea Vedaldi, and Andrew Zisserman. Improving category discovery when no representation rules them all. In Advances in Neural Information Processing Systems , 2023.

Bingchen Zhao and Oisin Mac Aodha. Incremental generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.

Florent Chiaroni, Jose Dolz, Ziko Imtiaz Masud, Amar Mitiche, and Ismail Ben Ayed. Parametric information maximization for generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1729-1739, 2023.

Yanan Wu, Zhixiang Chi, Yang Wang, and Songhe Feng. Metagcd: Learning to continually learn in generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1655-1665, 2023.

Hyungmin Kim, Sungho Suh, Daehwan Kim, Daun Jeong, Hansang Cho, and Junmo Kim. Proxy anchor-based unsupervised learning for continuous generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16688-16697, 2023.

Wenbin An, Feng Tian, Qinghua Zheng, Wei Ding, QianYing Wang, and Ping Chen. Generalized category discovery with decoupled prototypical network. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, 2023.

Sagar Vaze, Andrea Vedaldi, and Andrew Zisserman. No representation rules them all in category discovery. Advances in Neural Information Processing Systems 37 , 2023.

Han Xiao. Ndt: neual decision tree towards fully functioned neural graph. arXiv preprint arXiv:1712.05934 , 2017.

Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.09784 , 2017.

Ryutaro Tanno, Kai Arulkumaran, Daniel Alexander, Antonio Criminisi, and Aditya Nori. Adaptive neural trees. In International Conference on Machine Learning , pages 6166-6175. PMLR, 2019.

Ruyi Ji, Longyin Wen, Libo Zhang, Dawei Du, Yanjun Wu, Chen Zhao, Xianglong Liu, and Feiyue Huang. Attention convolutional binary neural tree for fine-grained visual categorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10468-10477, 2020.

Steven N Evans and Daniel Lanoue. Recovering a tree from the lengths of subtrees spanned by a randomly chosen sequence of leaves. Advances in Applied Mathematics , 96:39-75, 2018.

Peter D Grünwald, Paul MB Vitányi, et al. Algorithmic information theory. Handbook of the Philosophy of Information , pages 281-320, 2008.

Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems , 31, 2018.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Poojan Oza and Vishal M Patel. C2ae: Class conditioned auto-encoder for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2307-2316, 2019.

Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto Iida, and Takeshi Naemura. Classification-reconstruction learning for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4016-4025, 2019.

Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. Adversarial reciprocal points learning for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(11): 8065-8081, 2021.

Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In European Conference on Computer Vision , pages 507-522. Springer, 2020.

Yu Shu, Yemin Shi, Yaowei Wang, Tiejun Huang, and Yonghong Tian. P-odn: Prototype-based open deep network for open set recognition. Scientific reports , 10(1):7146, 2020.

ZongYuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. arXiv preprint arXiv:1707.07418 , 2017.

Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 813-822, 2021.

Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In European Conference on Computer Vision , pages 613-628, 2018.

Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Counterfactual zero-shot and open-set visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15404-15414, 2021.

A Theory

A.1 Notation and Definitions

Let us first formalize our notation and definition for the rest of the section. Some definitions might overlap with the notations in the main paper. However, we repeat them here for ease of access.

Probabilistic Notations. We denote the input random variable with X and the category random variable with C . The category code random variable, which we define as the embedding sequence of input X i , is denoted by z i = z i 1 z i 2 · · · z i L , in which superscript i shows the i th sample, while subscript L shows the digit position in the code sequence.

Coding Notations. Let C be a countable set, we use C ∗ to show all possible finite sequences using the members of this set. For instance: { 0 , 1 } ∗ = { ϵ, 0 , 1 , 00 , 01 , 10 , 11 , · · · } in which ϵ is empty word. The length of each sequence z , which we show with l (z) , equals the number of digits present in that sequence. For instance, for the sequence l (01010) = 5 .

Shannon Information Theory Notations. We denote the Shannon entropy or entropy of the random variable X with H ( X ) . It measures the randomness of values of X when we only have knowledge about its distribution P . It also measures the minimum number of bits required on average to transmit or encode the values drawn from this probability distribution [50, 51]. The conditional entropy of a random variable X given random variable Z is shown by H ( X | Z ) , which states the amount of randomness we expect to see from X after observing Z . In addition, I ( X ; Z ) indicates the mutual information between random variables X and Z [50, 51], which measures the amount of information we can obtain for one random variable by observing the other one. Note that contrary to H ( X | Z ) , mutual information is symmetric .

Algorithmic Information Theory Notations. Similar to Shannon's information theory, Kolmogorov Complexity or Algorithmic Information Theory [59-61] measures the shortest length to describe an object. Their difference is that Shannon's information considers that the objects can be described by the characteristic of the source that produces them, but Kolmogorov Complexity considers that the description of each object in isolation can be used to describe it with minimum length. For example, a binary string consisting of one thousand zeros might be assigned a code based on the underlying distribution it has been drawn from. However, Kolmogorov Complexity shows that we can encode this particular observation by transforming a description such as "print 0 for 1000 times" . The analogon to entropy is called complexity K (x) , which specifies the minimum length of a sequence that can specify output for a particular system. We denote the algorithmic mutual information for sequences x and z with I alg (x : z) , which specifies how much information about sequence x we can obtain by observing sequence z .

A.2 Maximizing the Algorithmic Mutual Information

Let's consider data space D = { X i , C i : i ∈ { 1 , · · · N }} where X s are inputs and C s are the corresponding category labels.

Lemma 2 For each category c and for X i with C i =c , we can find a binary decision tree T c that starting from its root, reaches each X i by following the decision tree path. Based on this path, we assign code c ( X i )= c i 1 c i 2 · · · c i M to each X i to uniquely define and retrieve it from the tree.

Proof of Lemma 2. Since the number of examples in the dataset is finite, we can enumerate samples of category c with any arbitrary coding. We then can replace these enumerations with their binary equivalent codes. We start from a root, and every time we encounter 1 in digits of these codes, we add a right child node, and for 0 , we add a left child node. We then continue from the child node until we reach the code's end. Since the number of samples with category c is limited, this process should terminate. On the other hand, since the binary codes for different samples are different, these paths are unique, and by the time we traverse a path from the root to a leaf node, we can identify the unique sample corresponding to that node. □

As mentioned in the main paper, using this Lemma, we can find at least one supertree T for the whole data space that addresses all samples in which samples of the same category share a similar prefix. We can define a model that provides the binary code z i = z i 1 · · · z i L for data input X i with category c based on the path it takes in these eligible trees. We define these path encoding functions valid encoding as defined in Definition 2:

Definition 2 A valid encoding for input space X and category space C is defined as an encoding that uniquely identifies every X i ∈ X . At the same time, for each category c ∈ C , it ensures that there is a sequence that is shared among all members of this category but no member out of the category.

Since there is no condition on how to create these trees and their subtrees, many candidate trees can address the whole data space while preserving a similar prefix for the members of each category.

However, based on our inspirations for how the brain does categorization, we assume the ground truth underlying tree T has a minimum average length path from the root to each node. In other words, each sample x has the shortest description code z to describe that data point while maintaining its validity. If we use a model to learn this encoding, the optimal model tree should be isomorph to the underlying tree T ,

Lemma 3 For a learned binary code z i to address input X i , uniquely, if the decision tree of this encoding is optimal, it is isomorph to the underlying tree T .

Proof of Lemma 3. Since the underlying tree has the minimum Kolmogorov complexity for each sample, we can extract the optimal lengths of each sample by traversing the tree. Evans and Lanoue [115] showed that a tree can be recovered from the sequence of lengths of the paths from the root to leaves to the level of isomorphism. Based on our assumption about the underlying tree T , the optimal tree can not have a shorter length for any sample codes than the underlying tree. On the other hand, having longer codes contradicts its optimality. Hence the optimal tree should have similar path lengths to the underlying ground truth tree. Therefore, it is isomorphic to the underlying tree. □

Since the optimal tree with the valid encoding ˜ z is isomorph to the underlying tree, we will have the necessary conditions that Theorem 1 provides.

Theorem 1 For a learned binary code z i to address input x i , uniquely, if the decision tree of this encoding is isomorph to underlying tree T , we will have the following necessary conditions:

I alg (z : x ) ≥ I alg (˜ z : x ) ∀ ˜ z, ˜ z is a valid encoding for x

I alg (z : c ) ≥ I alg (˜ z : c ) ∀ ˜ z, ˜ z is a valid encoding for x

Proof of Theorem 1.

Part one : From the way T has been constructed, we know that K ( x | T ) ≤ K ( x |T ) in which T is an arbitrary tree. From the complexity and mutual information properties, we also have I alg (z : x ) = K (z) -K ( x | z) [116]. Since ˜ z and z have isomorph tree structures, then K (˜ z ) = K (z) , hence: I alg (z : x ) ≥ I alg (˜ z : x ) . □

Part two : In any tree that is a valid encoding, all samples of a category should be the descendants of that node. Thus, the path length to corresponding nodes should be similar in both trees. Otherwise, the length of the path to all samples of this category will not be optimal. We can use the same logic and deduce that the subtree with the category nodes as its leaves would be isomorph for both embeddings. Let's denote the path from the root to category nodes with z c and from the category node to its corresponding samples with z x . If we assume these two paths can be considered independent, we will have K ( x ) = K (z c z x ) = K (z c ) + K (z x ) , which indicates that minimizing K ( x ) in the tree implies that K ( c ) also should be minimized. By applying the same logic as part one, we can deduce that I alg (z : c ) ≥ I alg (˜ z : c ) . □

A.2.1 Shannon Mutual Information Approximation

Optimization in Theorem 1 is generally not computable [58-61]. However, We can approximate these requirements using Shannon mutual information instead. Let's consider two functions f and g , such that both are { 0 , 1 } ∗ → R . For these functions, f + < g means that there exists a constant κ , such that f ≤ g + c , when both f + < g and g + < f hold, then f + = g [116].

Theorem 2 [116] Let P be a computable probability distribution on { 0 , 1 } ∗ ×{ 0 , 1 } ∗ . Then:

This theorem states that the expected value of algorithmic mutual information is close to its probabilistic counterpart. This means that if we maximize the Shannon information, we also approximately maximize the algorithmic information and vice versa.

Since Shannon entropy does not consider the inner regularity of the symbols it codes, to make each sequence meaningful from a probabilistic perspective, we convert each sequence to an equivalent random variable number by considering its binary digit representation. To this end, we consider

Z i = ∑ m k =1 z i k 2 k , which is a number between 0 and 1 . Note that we can recover the sequence from the value of this random variable. Since the differences in the first bits affect the number more, for different error thresholds, Shannon's information will focus on the initial bits more. In dealing with real-world data, the first bits of encoding of a category sequence are more valuable than later ones due to the hierarchical nature of categories. Furthermore, with this tweak, we equip Shannon's model with a knowledge of different positions of digits in a sequence. To replace the first item of Theorem 1 by its equivalent Shannon mutual information, we must also ensure that z has the minimum length. For the moment, let's assume we know this length by the function l ( X i )= l i . Instead of Z i , we can consider its truncated form Z i l i = ∑ l i k =1 z i k 2 k . This term, which we call the address loss function, is defined as follows:

We can approximate this optimization with a reconstruction or contrastive loss.

A.2.2 Approximation with Reconstruction Loss

Let's approximate the maximization of the mutual information by minimizing the L MSE of the reconstruction from the code z . Suppose that D ( X ) is the decoder function, and it is a Lipschitz continuous function, which is a valid assumption for most deep networks with conventional activation functions [117]. We can find an upper bound for L MSE using Lemma 3.

Lemma 3 Suppose that D ( X ) is a Lipschitz continuous function with Lipschitz constant κ , then we will have the following upper bound for L MSE :

Proof of Lemma 3 . Let's consider the L MSE loss for the reconstruction ˆ X i from the code Z i . We denote reconstruction from the truncated category code Z i l i with ˆ X i l i .

If we expand this loss, we will have the following:

Let's assume the optimal model can reconstruct X i using the entire code length Z i , i.e. X i = D ( ∑ m k =0 z i k 2 k ) . Now let's replace this in the equation:

Given that D ( X ) is a Lipschitz continuous function with the Lipschitz constant κ , then we will have the following:

Lemma 3 indicates that to minimize the upper bound on L MSE , we should aim for codes with maximum length, which can also be seen intuitively. The more length of latent code we preserve, the more accurate the reconstruction would be. This is in direct contrast with the length minimization of the algorithmic mutual information. So, the tradeoff between these two objectives defines the optimal final length of the category codes.

A.2.3 Approximation with Contrastive Loss

One of the advantages of contrastive learning is to find a representation that maximizes the mutual information with the input [64]. More precisely, if for input X i , we show the hidden representation learning Z i , that is learned contrastively by minimizing the InfoNCE loss, [64] showed that the following lower bound on mutual information exists:

Here, L N is the InfoNCE loss, and N indicates the sample size consisting of one positive and N -1 negative samples. Equation 11 shows that contrastive learning with the InfoNCE loss can be a suitable choice for minimizing the L adr in Equation 10. We will use this to our advantage on two different levels. Let's consider that Z i has dimension d , and each latent variable z i k can take up n different values. The complexity of the feature space for this latent variable would be O ( n d ) , then the number of structurally different binary trees for this feature space would be O ( C n d ) , in which C i is the i th Catalan number, which asymptotically grows as O (4 i ) . Hence the number of possible binary taxonomies for the categories will be O (4 n d ) . So minimizing n and, to a lesser degree, d , will be the most effective way to limit the number of possible binary trees. Since our model and the amount of training data is bounded, we must minimize the possible search space while still providing reasonable performance. On the other hand, the input feature space X i with N possible values and dimension D has O ( N D ) possible states, and to cover it completely, we can not arbitrarily decrease d and n . Note that for a nearly continuous function N →∞ , the probability of a random discrete tree to fully covering this space would be near zero.

A.3 Category Code Length Minimization

In the main paper, we indicate the code length loss L length , which we define as L length = 1 N ∑ N i =0 l i . To minimize this loss, we define a binary mask sequence m i = m i 1 m i 2 · · · m i L to simulate the subscript property of l i . We discussed minimizing the L p Norm for the weighted version of the mask, which we denote with ¯ m i =( m i 1 2 1 )( m i 2 2 2 ) · · · ( m i L 2 L ) . This will ensure the requirements because adding one extra bit has an equivalent loss of all previous bits.

Lemma 4 Consider the weighted mask ¯ m=( m 1 2 1 )( m 2 2 2 ) · · · ( m L 2 L ) where m j s are 0 or 1 . Consider the norm ∥ ¯ m ∥ p where p ≥ 1 , the rightmost 1 digit contributes more to the norm than the entire left sequence.

Proof of Lemma 4 . Let's consider the loss function for mask ¯ m=( m 1 2 1 )( m 2 2 2 ) · · · ( m L 2 L ) and let's denote the rightmost 1 index, with k , for simplicity we consider the ∥ ¯ m ∥ p p :

given that m j = 0 , ∀ j > k and m k = 1 , we will have:

now let's compare the two subparts of the right-hand side with each other:

Hence L Length tries to minimize the position of the rightmost 1 , simulating the cutting length subscript.

A.3.1 Satisfying Binary Constraints.

In the main paper, we stated that we have two conditions, Code Constraint: ∀ z i k , z i k = 0 or z i k = 1 and Mask Constraint ∀ m i k , m i k = 0 or m i k = 1 . We formulate each constraint in an equivalent Lagrangian function to make sure they are satisfied. For the binary code constraint we consider f code ( z i k )=( z i k )(1 -z i k )=0 , which is only zero if z i k =0 or z i k =1 . Similarly, for the binary mask constraint, we have f mask ( m i k )=( m i k )(1 -m i k )=0 . To ensure these constraints are satisfied, we optimize them with the Lagrangian function of the overall loss. Consider the Lagrangian function for L total ,

This lagrangian function ensures that constraints are satisfied for η → + ∞ and µ → + ∞ . Note that our method uses a tanh activation function which has been mapped between 0 and 1 , to produce m k and z k , so the conditions are always greater or equal to zero. For an unbounded output, we can consider the squared version of constraint functions to ensure that constraints will be satisfied. This shows how we reach the final unconstrained loss function in the paper.

B Experiments

B.1 Dataset Details

To acquire the train and test splits, we follow [27]. We subsample the training dataset in a ratio of 50% of known categories at the train and all samples of unknown categories. For all datasets except CIFAR100, we consider 50% of categories as known categories at training time. For CIFAR100 as in [27] 80% of the categories are known during training time. A summary of dataset statistics and their train test splits is shown in Table 6.

CIFAR10/100 [65] are coarse-grained datasets consisting of general categories such as car, ship, airplane, truck, horse, deer, cat, dog, frog and bird .

ImageNet-100 is a subset of 100 categories from the coarse-grained ImageNet [66] dataset.

CUB or the Caltech-UCSD Birds-200-2011 (CUB-200-2011) [67] is one of the most used datasets for fine-grained image recognition. It contains different bird species, which should be distinguished by relying on subtle details.

FGVC-Aircraft or Fine-Grained Visual Classification of Aircraft [68] dataset is another fine-grained dataset, which, instead of animals, relies on airplanes. This might be challenging for image recognition models since, in this dataset, structure changes with design.

SCars or Stanford Cars [69] is a fine-grained dataset of different brands of cars. This is challenging since the same brand of cars can look different from different angles or with different colors.

Oxford-Pet [70] is a fine-grained dataset of different species of cats and dogs. This is challenging since the amount of data is very limited in this dataset, which makes it prone to overfitting.

Herbarium_19 [71] is a botanical research dataset about different types of plants. Due to its long-tailed alongside fine-grained nature, it is a challenging dataset for discovering novel categories.

B.2 Implementation details

In this section, we provide our implementation details for each block separately. As mentioned in the main paper, the final loss function that we use to train the model is:

In which the loss L adr is:

In this formula, L C_in is the loss function that [27] suggested, so we use the same hyperparameters as their defaults for this loss. Hence, we only expand on L C_code:

In the scope of our experimentation, it was assumed by default that α =1 and λ code =0 . 35 . The code generation process introduces a certain noise level, potentially leading to confusion in the model, particularly in fine-grained data. To mitigate this, we integrated a smoothing hyperparameter within our contrastive learning framework, aiming to balance the noise impact and avert excessive confidence in the generated code, for datasets such as CUB and Pet, the smoothing factor was set at 1 , whereas for SCars, Aircraft, and Herb datasets, it was adjusted to 0 . 1 . In contrast, we did not apply smoothing for generic datasets like CIFAR 10/100 and ImageNet, where label noise is less significant.

Furthermore, in dealing with fine-grained data, we opted to fine-tune the final two blocks of the DINO model. This approach differs from our strategy for generic datasets, where only the last block underwent fine-tuning. Additionally, we employed semi-supervised k -means at every epoch to derive pseudo-labels from unlabeled data. These pseudo-labels were then used in our supervised contrastive learning process as a supervisory signal. It is important to note that in supervised contrastive learning, the primary requirement is that paired samples belong to the same class, allowing us to disregard discrepancies between novel class pseudo-labels and their actual ground truth values. Furthermore, instead of cosine similarity for contrastive learning, we adopt Euclidean distance, a better approximation for the category problem. Finally, for balanced datasets, we use the balanced version of k -means for semi-supervised k -means.

Code Generator. To create this block, we use a fully connected network with GeLU activation functions [118]. Then, we apply a tanh activation function tanh( ax ) in which a is a hyperparameter showing the model's age. We expect that as the model's age increases or, in other words, in later epochs, the model will be more decisive because of sharper transitions from 0 to 1 . Hence, we will have a stronger binary dichotomy for code values. Also, since contrastive learning makes the different samples as far as possible, this causes a problem for the Code Generator because the feature space will not smoothly transition from different samples of the same category, especially for fine-grained datasets. To alleviate this problem, we use a label smoothing hyperparameter in the contrastive objective to help make feature space smoother, which will require a smaller tree for encoding. Since the model should distinguish 0 s for the mask from 0 s of the code, we do not adjust the code generator to 0 and 1 s and consider the -1 and 1 values in practice.

Code Masker. The Code Masker block is a fully connected network with tanh activation functions at the end, which are adjusted to be 0 and 1 s. We also consider the aging hyperparameter for the tanh activation function in the masking block. In the beginning, since codes are not learned, masking the embedding space might hamper its learning ability. To solve this, we start masker with all one's entries and gradually decrease it with epochs. Hence, the activation function that is applied to the masker would be tanh( x + 1 a +1 ) , in which a is the aging parameter. In practice, we observed that norm one is stable enough in this loss function while also truncating codes at a reasonable length. Since L length grows exponentially with code length, it will mask most of the code. For fine-grained datasets, this could be detrimental for very similar categories. To alleviate this problem, instead of using 2 as a positional base, we decrease it with each epoch to 2 -epoch N epochs . So, at the end of training, the values of all positions are the same. This allows the model to encode more levels to the tree. Since we start with the base 2 , we are constructing the tree with a focus on nodes near the root at the start and to the leaves at the end of training.

Categorizer. We use a fully connected network for this block and train it with the one-hot encoding of the labeled samples. This module receives the truncated codes to predict the labeled data. This module cannot categorize labeled data if the masker truncates too much information. Hence, it creates error signals that prevent the masker from truncating too much. This part of the network is arbitrary, and we showed in ablations that we can ignore this module without supervision signals.

B.3 Further Ablations

Feature Space Visualization. Figure 4 illustrates the tSNE visualizations for different embedding extracted from our model. While our model's features form separate clusters, our label embedding, which is the raw code feature before binarization, makes these clusters distinctive. After that, binary embedding enhances this separation while condensing the cluster by making samples of clusters closer to each other, which is evident for the bird cluster shown in yellow. Because of its 0 or 1 nature, semantic similarity will affect the binary embedding more than visual similarity. Finally, our code embedding, which assigns positional values to the extracted binary embedding, shows indirectly that to have the most efficient code, our model should span the code space as much as possible, which explains the porous nature of these clusters.

B.4 Extracting the Implicit Tree from the Model

Suppose that the generated feature vector by the network for sample X is x 0 x 1 · · · x k , where k is the dimension of the code embedding or, equivalently, the depth of our implicit hierarchy tree. Using appropriate activation functions, we can assume that x i is binary. The unsupervised contrastive loss forces the model to make the associated code to each sample unique. So if X ′ is not equivalent to X or one of its augmentations, its code x ′ 0 x ′ 1 · · · x ′ k will differ from the code assigned to X . For the supervised contrastive loss, instead of considering the code, we consider a sequence by assigning different positional values to each bit so the code x 0 x 1 · · · x k can be considered as the binary number 0 .x 0 x 1 · · · x k . Then, the supervised contrastive loss aims to minimize the difference between these assigned binary numbers. This means our model learns to use the first digits for discriminative information while pushing the specific information about each sample to the last digits. Then, our masker learns to minimize the number of discriminative digits. Our Theorem states that, finally, the embedded tree that the model learns this way is a good approximation of the optimal tree. Ultimately, our model generates a code for each sample, and we consider each code as a binary tree traverse from the root to the leaf. Hence, the codes delineate our tree's structure and binary classification that

airplane

airplane

car

car

bird

bird

cat

cat

deer

deer

6op

6op

frog

frog

horse

horse

ships

ships

truck

truck

(a) Feature Embedding

airplane

airplane

car

car

bird

bird

cat

cat

deer

deer

6op

6op

60.4

6o.

horse

horse

ships

ships

truck

truck

(c) Binary Embedding

(d) Code Embedding

happens at each node. Since our approach enables the model to use the initial bits for supervised contrastive learning and the last bits for unsupervised contrastive learning, we can benefit from their synergic advantages while preventing them from interfering with each other.

C Related Works

C.1 Open Set Recognition

The first sparks of the requirement for models that can handle real-world data were introduced by Scheirer et al. [20] and following works of [16, 18]. The first application of deep networks to address this problem was presented by OpenMax [17]. The main goal for open-set recognition is to distinguish known categories from each other while rejecting samples from novel categories. Hence many open-set methods rely on simulating this notion of otherness , either through large reconstruction errors [119, 120] distance from a set of prototypes[121-123] or by distinguishing the adversarially generated samples [124-127]. One of the shortcomings of open set recognition is that all new classes will be discarded.

C.2 Novel Class Discovery

To overcome open set recognition shortcomings, novel class discovery aims to benefit from the vast knowledge of the unknown realm and infer the categories. It can be traced back to [79], where they used the knowledge from labeled data to infer the unknown categories. Following this work, [80] solidified the novel class discovery as a new specific problem. The main goal of novel class discovery is to transfer the implicit category structure from the known categories to infer unknown categories [24, 26, 75, 76, 81, 82, 82-99]. Despite this, the novel class discovery has a limiting assumption that test data only consists of novel categories.

C.3 Generalized Category Discovery

For a more realistic setting, Generalized Category Discovery considers both known and old categories at the test time. This nascent problem was introduced by [27] and concurrently under the name open-world semi-supervised learning by [77]. In this scenario, while the model should not lose its grasp on old categories, it must discover novel categories in test time. This adds an extra challenge because when we adapt the novel class discovery methods to this scenario, they try to be biased to either novel or old categories and miss the other group. There has been a recent surge of interest in generalized category discovery [28-31, 78, 101-110]. In this work, instead of viewing categories as an end, we investigated the fundamental question of how to conceptualize category itself.

C.4 Binary Tree Distillation

Benefiting from the hierarchical nature of categories has been investigated previously. Xiao [111] and Frosst and Hinton [112] used a decision tree in order to make the categorization interpretable and as a series of decisions. Adaptive neural trees proposed by [113] assimilate representation learning to its edges. Ji et al. [114] use attention binary neural tree to distinguish fine-grained categories by attending to the nuances of these categories. However, these methods need an explicit tree structure. In this work, we let the network extract this implicit tree on its own. This way, our model is also suitable when an explicit tree structure does not exist.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel perspective on categorization, framing it as an optimization problem with the goal of finding optimal solutions. The authors propose a self-supervised approach that can identify new categories during testing. They achieve this by assigning concise category codes to individual data instances, which effectively captures the inherent hierarchical structure found in real-world datasets.\n\nSTRENGTHS:\n1. The paper conceptualizes a category as a solution to an optimization problem, which is novel.\n2. The paper is well-written, easy to follow\n3. The proposed method is novel\n\nWEAKNESSES:\n1. The method is too complicated, with 5 hyperparameters in the overall loss functions, which is hard to tune, is it possible to make it automatically?\n2. Some ablation studies are missing, I am curious about the effect on every hyperparameter in the loss function, how sensitive are they to the performance?\n3. Some recent methods of comparison are missing.\n* [1] PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery (CVPR 2023)\n* [2] Modeling Inter-Class and Intra-Class Constraints in Novel Class Discovery (CVPR 2023)\n* [3] Generalized Category Discovery with Decoupled Prototypical Network (AAAI 2023)\n* [4] Supervised Knowledge May Hurt Novel Class Discovery Performance (TMLR 2023)\n\nQUESTIONS:\n1. About the experiment setting, transductive or instructive?  the test set is included or excluded in the training process?\n2. How to compute mutual information, MINE[1]?\n\n\n[1]MINE: Mutual Information Neural Estimation\n\nLIMITATIONS:\n1. Complex Hyperparameter Tuning: The model's complexity with five hyperparameters in the loss functions makes tuning challenging. An automated tuning method might streamline this process.\n2. Missing Ablation Studies: The absence of ablation studies limits understanding of individual hyperparameters' impact on performance. This analysis would contribute to model optimization.\n3. Incomplete Comparison: The paper lacks comparison with some recent methods. Incorporating them can provide a more comprehensive evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper conceptualize a category through the lens of optimization, viewing it as an optimal solution to a well-defined problem. The author propose a novel, efficient and self-supervised method capable of discovering previously unknown categories at test time.

STRENGTHS:
1、The paper aims to tackle an important problem of how to better represent categories in deep learning models. The mathematical solution and novel framework proposed by the authors offer intriguing insights into addressing this issue;
2、The paper is well-structured, with a reasonable experimental design and demonstrates the effectiveness of the proposed method on multiple datasets.

WEAKNESSES:
1、It is worth noting that the proposed method might require more empirical studies to validate its generalizability across different scenarios and tasks. How does this model perform on the common ImageNet-100 dataset?
2、I agree that there are quite a few loss functions presented in this paper. Although the authors provide detailed explanations for each loss, it might be somewhat challenging to unify so many losses within a single system. So adapting this model for multi-stage training could be a viable approach to address the complexity if handling multiple losses.
3、The improvement on the commonly used CIFAR dataset is relative weak.

QUESTIONS:
1、In the Table. 3 and Table. 4, the performance gap between the method and sota methods to be quite significant. Can you analyze the possible reasons for this?
2、I am very interested in your implict binary tree and category encoding. Could you show it using a visualization?

LIMITATIONS:
Authors adequately addressed the limitations .
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper questions a foundation problem that underpins the generalized class discovery (GCD): what defines a category. To this end, the authors proposed a self-learning-based method that solves GCD by modeling the implicit category hierarchy in the data. The proposed method allows control over category granularity and is thus good at fine-grained recognition datasets. The extensive experiments and theoretical analysis have demonstrated the effectiveness of the proposed method.

STRENGTHS:
- The writing is satisfying. The paper is easy-to-follow.
- The motivation is decent.
- The proposed method is novel and proved effective.
- Also the related works are adequately discussed.

WEAKNESSES:
- The compared methods are no longer state-of-the-art. Also, due to the explicit assumption of a hierarchical structure, the proposed method is not always the best on all datasets, especially the generic ones.

QUESTIONS:
No questions.

--- Post rebuttal ---

Thank the authors for the detailed rebuttal response, which addressed my main concerns, and thus I keep my positive rating unchanged.

LIMITATIONS:
The limitations are carefully discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of generalized category discovery (GCD) by reframing the concept of a category with an implicit category code tree, which addresses three problems of conventional supervised learning, namely category hierarchies, label inconsistency, and open-world recognition. The contributions of this paper are: (1) conceptualizing a category with a theoretically optimal solution; (2) proposing a practical GCD method based on the theory; (3) achieving the state-of-the-art GCD performance on both generic and fine-grained scenarios.\n\nSTRENGTHS:\n(1) The inspiration for modeling category hierarchies by encoding a category with a binary tree is reasonable.\n\n(2) The theoretical derivation of the optimization loss is adequate and helps readers understand the subsequent model design.\n\nWEAKNESSES:\n(1) The experimental demonstration of how binary code works is missing. For example, the authors can simply compare the codes of “dog”, “cat”, and “car” to see how the number of digits differs in the codes. It is crucial to verify the category hierarchies and label consistency, which are the core problems to solve claimed in the paper.\n\n(2) The performance improvement on generic datasets is limited. CIFAR10 performance lags behind ORCA and CIFAR100 performance gains only 0.2.\n\n(3) The paper's ablation study in Table 2 raises concern. In the first row, the model solely uses the visual features to compute the loss, i.e., the original GCD [1] approach. In Table 3 CIFAR10 section, the results provided from the original GCD paper are used. If we compare the results from both Table 2 and Table 3 by evaluating them using Semi-supervised K-means (SS-KMeans [1]), both InfoSieve and GCD approaches gain an increase in accuracy for the known class, while for the novel class, only InfoSieve gains an increase, while GCD gets the worst accuracy results. This paper shows that the GCD + generic K-Means is better than GCD + SS-KMeans, which is inconsistent with what the GCD paper claimed. An investigation should be conducted to address this inconsistency.\n\nMinor: Missing notation definitions such as \"M\" and \"K\" in the main paper.\n\nQUESTIONS:\nPlease see the weaknesses above.\n\nIn addition, should the “Z^i” in line 215 be “z^i”? Because “Z^i” is a number according to line 150?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}