SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sarah Rastegar, Hazel Doughty\*Cees G. M. Snoek University of Amsterdam
In the quest for unveiling novel categories at test time, we confront the inherent limitations of traditional supervised recognition models that are restricted by a predefined category set. While strides have been made in the realms of self-supervised and open-world learning towards test-time category discovery, a crucial yet often overlooked question persists: what exactly delineates a category? In this paper, we conceptualize a category through the lens of optimization, viewing it as an optimal solution to a well-defined problem. Harnessing this unique conceptualization, we propose a novel, efficient and self-supervised method capable of discovering previously unknown categories at test time. A salient feature of our approach is the assignment of minimum length category codes to individual data instances, which encapsulates the implicit category hierarchy prevalent in real-world datasets. This mechanism affords us enhanced control over category granularity, thereby equipping our model to handle fine-grained categories adeptly. Experimental evaluations, bolstered by state-of-the-art benchmark comparisons, testify to the efficacy of our solution in managing unknown categories at test time. Furthermore, we fortify our proposition with a theoretical foundation, providing proof of its optimality. Our code is available at: https://github.com/SarahRastegar/InfoSieve.
The human brain intuitively classifies objects into distinct categories, a process so intrinsic that its complexity is often overlooked. However, translating this seemingly innate understanding of categorization into the realm of machine learning opens a veritable Pandora's box of differing interpretations for what constitutes a category? [1, 2]. Prior to training machine learning models for categorization tasks, it is indispensable to first demystify this concept of a category.
In the realm of conventional supervised learning [3-7], each category is represented by arbitrary codes, with the expectation that machines produce corresponding codes upon encountering objects from the same category. Despite its widespread use, this approach harbors several pitfalls: it suffers from label inconsistency, overlooks category hierarchies, and, as the main topic of this paper, struggles with open-world recognition.
Figure 1: What is the correct category? Photo of a flying fox fruit bat. This image can be categorized as bat, bird, mammal, flying bat, and other categories. How should we define which answer is correct? This paper uses self-supervision to learn an implicit category code tree that reveals different levels of granularity in the data.
Pitfall I: Label Inconsistency. Assessing a model's performance becomes problematic when category assignments are subject to noise [8, 9] or exhibit arbitrary variation across different datasets. For example, if a zoologist identifies the bird in Figure 1 as a flying fox fruit bat, it is not due to a misunderstanding of what constitutes a bird or a dog. Rather, it signifies a more nuanced understanding of these categories. However, conventional machine learning
*Currently at Leiden University
Figure 2: The implicit binary tree our model finds to address samples. Each leaf in the tree indicates a specific sample, and each node indicates the set of its descendants' samples. For instance, the node associated with '11...11' is the set of all birds with red beaks, while its parent is the set of all birds with red parts in their upper body.
models may penalize such refined categorizations if they deviate from the pre-defined ground-truth labels. This work addresses this limitation by assigning category codes to individual samples. These codes not only prevent over-dependence on specific labels but also facilitate encoding similarities across distinct categories, paving the way for a more robust and nuanced categorization.
Pitfall II: Category Hierarchies. A prevalent encoding method, such as one-hot target vectors, falls short when addressing category hierarchies. While we, as humans, intuitively distinguish the categories of plane and dog as more disparate than cat and dog, our representations within the model fail to convey this nuanced difference, effectively disregarding the shared semantics between the categories of cat and dog. While some explorations into a category hierarchy for image classification have been undertaken [10-14], these studies hinge on an externally imposed hierarchy, thus limiting their adaptability and universality. This paper proposes a self-supervised approach that enables the model to impose these implicit hierarchies in the form of binary trees into their learned representation. For instance, Figure 2 shows that we can address each sample in the dataset with its path from the root of this implicit tree, hence associating a category code to it. We show theoretically that under a set of conditions, all samples in a category have a common prefix, which delineates category membership.
Pitfall III: Open World Recognition. The final problem we consider is the encounter with the open world [15-17]. When a model is exposed to a novel category, the vague definition of category makes it hard to deduce what will be an unseen new category. While open-set recognition models [18-22] can still evade this dilemma by rejecting new categories, Novel Class Discovery [23-26] or Generalized Category Discovery [27-31] can not ignore the fundamental flaw of a lack of definition for a category. This problem is heightened when categories are fine-grained [32, 33] or follow a long-tailed distribution [34-37].
In this paper, we confront these challenges by reframing the concept of a category as the solution to an optimization problem. We argue that categories serve to describe input data and that there is not a singular correct category but a sequence of descriptions that span different levels of abstraction. We demonstrate that considering categorization as a search for a sequence of category codes not only provides more flexibility when dealing with novel categories but also, by leveraging sequences, allows us to modulate the granularity of categorization, proving especially beneficial for fine-grained novel categories. Subsequently, we illustrate how to construct a framework capable of efficiently approximating this solution. Our key contributions are as follows:
- Theoretical. We conceptualize a category as a solution to an optimization problem. We then demonstrate how to fine-tune this optimization framework such that its mathematical solutions align with the human-accepted notion of categories. Furthermore, under a set of well-defined constraints, we establish that our method theoretically yields an optimal solution.
- Methodological. Based on the theory we developed, we propose a practical method for tackling the generalized category discovery problem, which is also robust to different category granularities.
- Experimental. We empirically show that our method outperforms state-of-the-art generalized category discovery and adapted novel class discovery methods on fine-grained datasets while performing consistently well on coarse-grained datasets.
Before detailing our contributions, we first provide some background on category discovery to better contextualize our work.
The Generalized Category Discovery problem introduced by Vaze et al. [27] tries to categorize a set of images during inference, which can be from the known categories seen during training or novel categories. Formally, we only have access to  $\mathcal{V}_S$  or seen categories during training time, while we aim to categorize samples from novel categories or  $\mathcal{V}_{\mathcal{U}}$  during test time. For the Novel Class Discovery problem, it is assumed that  $\mathcal{V}_S \cap \mathcal{V}_{\mathcal{U}} = \emptyset$ . However, this assumption could be unrealistic for real-world data. So Vaze et al. [27] proposed to use instead the more realistic Generalized Category Discovery assumption in which the model can encounter both seen and unseen categories during test time. In short, for the Generalized Category Discovery problem, we have  $\mathcal{V}_S \subset \mathcal{V}_{\mathcal{U}}$ .
One major conundrum with both Generalized Category Discovery and Novel Category Discovery is that the definition of the category has remained undetermined. This complication can be overlooked when the granularity of categories at test time is similar to training time. However, for more realistic applications where test data may have different granularity from training data or categories may follow a long-tailed distribution, the definition of category becomes crucial. To this end, in the next section, we formulate categories as a way to abstract information in the input data.
3 An Information Theory Approach to Category Coding
To convert a subjective concept as a category to a formal definition, we must first consider why categorization happens in the first place. There are many theories regarding this phenomenon in human [38-40] and even animal brains [41-43]. One theory is categorization was a survival necessity that the human brain developed to retrieve data as fast and as accurately as possible [44]. Studies have shown that there could be a trade-off between retrieval speed and accuracy of prediction in the brain [45-47]. Meanwhile, other studies have shown that the more frequent categories can be recognized in a shorter time, with more time needed to recognize fine-grained nested subcategories [48, 49]. These studies might suggest shorter required neural pulses for higher hierarchy levels. Inspired by these studies, we propose categorization as an optimization problem with analogous goals to the human brain. We hypothesize that we can do the category assignment to encode objects hierarchically to retrieve them as accurately and quickly as possible.
Notation and Definitions. Let us first formalize our notation and definitions. We denote the input random variable with  $X$  and the category random variable with  $C$ . The category code random variable, which we define as the embedding sequence of input  $X^i$ , is denoted by  $z^i = z_1^i z_2^i \dots z_L^i$ , in which superscript  $i$  shows the  $i$ th sample, while subscript  $L$  shows the digit position in the code sequence. In addition,  $I(X; Z)$  indicates the mutual information between random variables  $X$  and  $Z$  [50, 51], which measures the amount of information we can obtain for one random variable by observing the other one. Since category codes are sequences, algorithmic information theory is most suitable for addressing the problem. We denote the algorithmic mutual information for sequences  $x$  and  $z$  with  $I_{alg}(x : z)$ , which specifies how much information about sequence  $x$  we can obtain by observing sequence  $z$ . Both Shannon and algorithmic information-theory-based estimators are useful for hierarchical clustering [52-56], suggesting we may benefit from this quality to simulate the implicit category hierarchy. A more in-depth discussion can be found in the supplemental.
3.1 Maximizing the Algorithmic Mutual Information
Let's consider data space  $\mathcal{D} = \{X^i, C^i : i \in \{1, \dots, N\}\}$  where  $X$ s are inputs and  $C$ s are the corresponding category labels.
Lemma 1 For each category  $c$  and for  $X^i$  with  $C^i = c$ , we can find a binary decision tree  $\mathcal{T}_{\mathrm{c}}$  that starting from its root, reaches each  $X^i$  by following the decision tree path. Based on this path, we assign code  $c(X^i) = c_1^i c_2^i \dots c_M^i$  to each  $X^i$  to uniquely define and retrieve it from the tree.  $M$  is the length of the binary code assigned to the sample.
Proof of Lemma 2 is provided in the supplemental. Based on this lemma, we can find a forest with categories  $c$  as the roots and samples  $X^i$ 's as their leaves. We apply the same logic to find a super decision tree  $\mathbf{T}$  that has all these category roots as its leaves. If we define the path code of category  $c$  in this super tree by  $p(c) = p_1^c p_2^c \cdots p_K^c$  where  $K$  is the length of the path to the category  $c$ ; we find the path to each  $X^i$  in the supertree by concatenating its category path code with its code in the category decision tree. So for each input  $X^i$  with category  $c$  we define an address code as  $q_1^i q_2^i \cdots q_{K + M}^i$  in which  $q_j^i = p_j^c$  for  $j \leq K$  and  $q_j^i = c_{j - K}^i$  for  $j > K$ . Meanwhile, since all  $X^i$ 's are the descendants
of root  $c$  in the  $\mathcal{T}_{\mathrm{c}}$  tree, we know there is one encoding to address all samples, in which samples of the same category share a similar prefix. Now consider a model that provides the binary code  $z^{i} = z_{1}^{i}\dots z_{L}^{i}$  for data input  $X^i$  with category  $c$ , let's define a valid encoding in Definition 2.
Definition 1 A valid encoding for input space  $\mathcal{X}$  and category space  $\mathcal{C}$  is defined as an encoding that uniquely identifies every  $X^i\in \mathcal{X}$ . At the same time, for each category  $c\in \mathcal{C}$ , it ensures that there is a sequence that is shared among all members of this category but no member out of the category.
As mentioned before, if the learned tree for this encoding is isomorph to the underlying tree  $\mathbf{T}$ , we will have the necessary conditions that Theorem 1 provides.
Theorem 1 For a learned binary code  $z^i$  to address input  $X^i$ , uniquely, if the decision tree of this encoding is isomorph to underlying tree  $T$ , we will have the following necessary conditions:
1.  $I_{alg}(z : x) \geq I_{alg}(\tilde{z} : x) \quad \forall \tilde{z}, \tilde{z}$  is a valid encoding for  $x$
2.  $I_{alg}(\mathbb{Z}:c)\geq I_{alg}(\tilde{\mathbb{Z}}:c)\quad \forall \tilde{z},\tilde{z}$  is a valid encoding for  $x$
Proof of Theorem 1 is provided in the supplemental. Optimizing for these two measures provides an encoding that satisfies the necessary conditions. However, from the halting Theorem [57], this optimization is generally not computable [58-61].
Theorem 1 Clarification. The first part of Theorem 1 states that if there is an implicit hierarchy tree, then for any category tree that is isomorph to this implicit tree, the algorithmic mutual information between each sample and its binary code generated by the tree will be maximal for the optimal tree. Hence, maximizing this mutual information is a necessary condition for finding the optimal tree. This is equivalent to finding a tree that generates the shortest-length binary code to address each sample uniquely. The second part of Theorem 1 states that for the optimal tree, the algorithmic mutual information between each sample category and its binary code will be maximum. Hence, again, maximizing this mutual information is a necessary condition for finding the optimal tree. This is equivalent to finding a tree that generates the shortest-length binary code to address each category uniquely. This means that since the tree should be a valid tree, the prefix to the unique address of every category sample  $c$  should be the shortest-length binary code while not being the prefix of any sample from other categories.
Shannon Mutual Information Approximation. We can approximate these requirements using Shannon mutual information instead if we consider a specific set of criteria. First, since Shannon entropy does not consider the relationship between separate bits or  $z_{j}^{i}$ s, we convert each sequence to an equivalent random variable number by considering its binary digit representation. To this end, we consider  $Z^{i} = \sum_{k=1}^{m} \frac{z_{k}^{i}}{2^{k}}$ , which is a number between 0 and 1. To replace the first item of Theorem 1 by its equivalent Shannon mutual information, we must also ensure that  $z$  has the minimum length. For the moment, let's assume we know this length by the function  $l(X^{i}) = l_{i}$ . Hence instead of  $Z^{i}$ , we consider its truncated form  $Z_{l_{i}}^{i} = \sum_{k=1}^{l_{i}} \frac{z_{k}^{i}}{2^{k}}$ . This term, which we call the address loss function, is defined as follows:
$$
\mathcal {L} _ {\mathrm {a d r}} = - \frac {1}{N} \sum_ {i = 0} ^ {N} I \left(X ^ {i}; Z _ {l _ {i}} ^ {i}\right) \quad s. t. \quad Z _ {l _ {i}} ^ {i} = \sum_ {k = 1} ^ {l _ {i}} \frac {z _ {k} ^ {i}}{2 ^ {k}} \text {a n d} \forall k, z _ {k} ^ {i} \in \{0, 1 \}. \tag {1}
$$
We can approximate this optimization with a contrastive loss. However, there are two requirements that we must consider; First, we have to obtain the optimal code length  $l_{i}$ , and second, we have to ensure  $z_{k}^{i}$  is binary. In the following sections, we illustrate how we can satisfy these requirements.
3.2 Category Code Length Minimization
To find the optimal code lengths  $l_{i}$  in Equation 1, we have to minimize the total length of the latent code. We call this loss  $\mathcal{L}_{\mathrm{length}}$ , which we define as  $\mathcal{L}_{\mathrm{length}} = \frac{1}{N}\sum_{i=0}^{N}l_{i}$ . However, since the  $l_{i}$  are in the subscripts of Equation 1, we can not use the conventional optimization tools to optimize this length. To circumvent this problem, we define a binary mask sequence  $\mathrm{m}^i = m_1^i m_2^i \cdots m_L^i$  to simulate the subscript property of  $l_{i}$ . Consider a masked version of  $z^i = z_1^i \cdots z_L^i$ , which we will denote as  $\tilde{z}^i = \tilde{z}_1^i \cdots \tilde{z}_L^i$ , in which for  $1 \leq k \leq L$ , we define  $\tilde{z}_k^i = z_k^i m_k^i$ . The goal is to minimize the number of ones in sequence  $\mathrm{m}^i$  while forcing them to be at the beginning of the sequence. One way to ensure this is to consider the sequence  $\bar{\mathrm{m}}^i = (m_1^i 2^1)(m_2^i 2^2) \cdots (m_L^i 2^L)$  and minimize its  $L_p$  Norm for  $p \geq 1$ .
This will ensure the requirements because adding one extra bit has an equivalent loss of all previous bits. In the supplemental, we provide a more rigorous explanation.
$$
\mathcal {L} _ {\text {l e n g t h}} \approx \frac {1}{N} \sum_ {i = 0} ^ {N} \| \bar {\mathrm {m}} ^ {i} \| _ {p}. \tag {2}
$$
We extract the mask from the input  $X^i$ , i.e.,  $\mathrm{m}^i = \text{Mask}(X^i)$ . Mask digits should also be binary, so we need to satisfy their binary constraints, which we will address next.
Satisfying Binary Constraints. Previous optimizations are constrained to two conditions, Code Constraint:  $\forall z_k^i$ ,  $z_k^i = 0$  or  $z_k^i = 1$  and Mask Constraint:  $\forall m_k^i$ ,  $m_k^i = 0$  or  $m_k^i = 1$ . We formulate each constraint in an equivalent Lagrangian function to make sure they are satisfied. For the binary code constraint we consider  $f_{\mathrm{code}}(z_k^i) = (z_k^i)(1 - z_k^i) = 0$ , which is only zero if  $z_k^i = 0$  or  $z_k^i = 1$ . Similarly, for the binary mask constraint, we have  $f_{\mathrm{mask}}(m_k^i) = (m_k^i)(1 - m_k^i) = 0$ . To ensure these constraints are satisfied, we optimize them with the Lagrangian function of the overall loss.
3.3 Aligning Category Codes using Supervision Signals
The second item in Theorem 1 shows the necessary condition for maximizing mutual information with a category. If we replace algorithmic mutual information with its Shannon cousin, we will have:
$$
\mathcal {L} _ {\mathrm {C a t}} = - \frac {1}{N} \sum_ {i = 0} ^ {N} I \left(c ^ {i}; Z _ {l _ {i}} ^ {i}\right) \quad s. t. \quad Z _ {l _ {i}} ^ {i} = \sum_ {k = 1} ^ {l _ {i}} \frac {z _ {k} ^ {i}}{2 ^ {k}} \text {a n d} \forall k, z _ {k} ^ {i} \in \{0, 1 \}. \tag {3}
$$
Subject to satisfying the binary constraints. Note that the optimal lengths might differ for optimizing information based on categories or input. However, here we consider the same length for both scenarios for simplicity.
Overall Loss. Putting all these losses and constraints together, we will reach the constrained loss:
$$
\mathcal {L} _ {\text {c o n s t r a i n e d}} = \mathcal {L} _ {\text {a d r}} + \delta \mathcal {L} _ {\text {l e n g t h}} + \gamma \mathcal {L} _ {\text {C a t}} \quad \text {s . t .} \quad \forall k, i f _ {\text {c o d e}} \left(z _ {k} ^ {i}\right) = 0, \quad \forall k, i f _ {\text {m a x}} \left(m _ {k} ^ {i}\right) = 0. \tag {4}
$$
Note that when we do not have the supervision signals, we can consider  $\gamma = 0$  and extract categories in an unsupervised manner. In the supplemental, we have shown how to maximize this function based on the Lagrange multiplier. If we indicate  $\mathcal{L}_{\mathrm{code\_cond}} = \sum_{i=0}^{N} \sum_{k=1}^{L} (z_k^i)^2 (1 - z_k^i)^2$  and  $\mathcal{L}_{\mathrm{mask\_cond}} = \sum_{i=0}^{N} \sum_{k=1}^{L} (m_k^i)^2 (1 - m_k^i)^2$ . The final loss will be:
$$
\mathcal {L} _ {\text {f i n a l}} = \mathcal {L} _ {\text {a d r}} + \delta \mathcal {L} _ {\text {l e n g t h}} + \gamma \mathcal {L} _ {\text {C a t}} + \zeta \mathcal {L} _ {\text {c o d e} _ {\text {c o n d}}} + \mu \mathcal {L} _ {\text {m a s k} _ {\text {c o n d}}}. \tag {5}
$$
Note that for satisfying binary constraints, we can adopt other approaches. For instance, we can omit the requirement for this hyperparameter by using binary neural networks and an STE (straight-through estimator) [62]. Another approach would be to benefit from Boltzmann machines [63] to have a binary code. Having defined our theoretical objective, we are now ready to make it operational.
4 InfoSieve: Self-supervised Code Extraction
In this section, using the equations from Section 3, we devise a framework to extract category codes. Note that as Theorem 1 indicates, when we train the model to extract the category codes instead of the categories themselves, we make the model learn the underlying category tree. The model must ensure that in its implicit category tree, there is a node for each category whose descendants all share the same category while there are no non-descendants of this node with the same category.
The overall framework of our model, named InfoSieve, is depicted in Figure 3. We first extract an embedding using the contrastive loss used by [27]. Then our Code Generator uses this embedding to generate binary codes, while our Code Marker learns a mask based on these embeddings to minimize the code length. Ultimately, the Categorizer uses this truncated code to discern ground-truth categories. In the next sections, we explain each component in more detail.
Figure 3: InfoSieve framework. Feature Extractor extracts an embedding by minimizing contrastive loss  $\mathcal{L}_{\mathrm{C\_in}}$ . The Code Generator uses these input embeddings to find category codes. The Code Masker simultaneously learns masks that minimize the code length with  $\mathcal{L}_{\mathrm{length}}$ . Finally, truncated category codes are used to minimize a contrastive loss for category codes while also predicting the seen categories by minimizing  $\mathcal{L}_{\mathrm{Cat}}$ .
4.1 Contrastive Learning of Code and Input