SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[112] Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.09784, 2017.
[113] Ryutaro Tanno, Kai Arulkumaran, Daniel Alexander, Antonio Criminisi, and Aditya Nori. Adaptive neural trees. In International Conference on Machine Learning, pages 6166-6175. PMLR, 2019.
[114] Ruyi Ji, Longyin Wen, Libo Zhang, Dawei Du, Yanjun Wu, Chen Zhao, Xianglong Liu, and Feiyue Huang. Attention convolutional binary neural tree for fine-grained visual categorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10468-10477, 2020.
[115] Steven N Evans and Daniel Lanoue. Recovering a tree from the lengths of subtrees spanned by a randomly chosen sequence of leaves. Advances in Applied Mathematics, 96:39-75, 2018.
[116] Peter D Grünwald, Paul MB Vitányi, et al. Algorithmic information theory. Handbook of the Philosophy of Information, pages 281-320, 2008.
[117] Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
[118] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
[119] Poojan Oza and Vishal M Patel. C2ae: Class conditioned auto-encoder for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2307–2316, 2019.
[120] Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto Iida, and Takeshi Naemura. Classification-reconstruction learning for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4016-4025, 2019.
[121] Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. Adversarial reciprocal points learning for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11): 8065-8081, 2021.
[122] Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In European Conference on Computer Vision, pages 507-522. Springer, 2020.
[123] Yu Shu, Yemin Shi, Yaowei Wang, Tiejun Huang, and Yonghong Tian. P-odn: Prototype-based open deep network for open set recognition. Scientific reports, 10(1):7146, 2020.
[124] ZongYuan Ge, Sergey Demyanov, Zetao Chen, and Rahul Garnavi. Generative openmax for multi-class open set classification. arXiv preprint arXiv:1707.07418, 2017.
[125] Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822, 2021.
[126] Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In European Conference on Computer Vision, pages 613–628, 2018.
[127] Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Counterfactual zero-shot and open-set visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15404-15414, 2021.
A.1 Notation and Definitions
Let us first formalize our notation and definition for the rest of the section. Some definitions might overlap with the notations in the main paper. However, we repeat them here for ease of access.
Probabilistic Notations. We denote the input random variable with  $X$  and the category random variable with  $C$ . The category code random variable, which we define as the embedding sequence of input  $X^i$ , is denoted by  $z^i = z_1^i z_2^i \dots z_L^i$ , in which superscript  $i$  shows the  $i$ th sample, while subscript  $L$  shows the digit position in the code sequence.
Coding Notations. Let  $\mathcal{C}$  be a countable set, we use  $\mathcal{C}^*$  to show all possible finite sequences using the members of this set. For instance:  $\{0,1\}^{*} = \{\epsilon ,0,1,00,01,10,11,\dots \}$  in which  $\epsilon$  is empty word. The length of each sequence  $z$ , which we show with  $l(z)$ , equals the number of digits present in that sequence. For instance, for the sequence  $l(01010) = 5$ .
Shannon Information Theory Notations. We denote the Shannon entropy or entropy of the random variable  $X$  with  $H(X)$ . It measures the randomness of values of  $X$  when we only have knowledge about its distribution  $P$ . It also measures the minimum number of bits required on average to transmit or encode the values drawn from this probability distribution [50, 51]. The conditional entropy of a random variable  $X$  given random variable  $Z$  is shown by  $H(X|Z)$ , which states the amount of randomness we expect to see from  $X$  after observing  $Z$ . In addition,  $I(X;Z)$  indicates the mutual information between random variables  $X$  and  $Z$  [50, 51], which measures the amount of information we can obtain for one random variable by observing the other one. Note that contrary to  $H(X|Z)$ , mutual information is symmetric.
Algorithmic Information Theory Notations. Similar to Shannon's information theory, Kolmogorov Complexity or Algorithmic Information Theory[59-61] measures the shortest length to describe an object. Their difference is that Shannon's information considers that the objects can be described by the characteristic of the source that produces them, but Kolmogorov Complexity considers that the description of each object in isolation can be used to describe it with minimum length. For example, a binary string consisting of one thousand zeros might be assigned a code based on the underlying distribution it has been drawn from. However, Kolmogorov Complexity shows that we can encode this particular observation by transforming a description such as "print 0 for 1000 times". The analogon to entropy is called complexity  $K(\mathbf{x})$ , which specifies the minimum length of a sequence that can specify output for a particular system. We denote the algorithmic mutual information for sequences  $\mathbf{x}$  and  $\mathbf{z}$  with  $I_{alg}(\mathbf{x} : \mathbf{z})$ , which specifies how much information about sequence  $\mathbf{x}$  we can obtain by observing sequence  $\mathbf{z}$ .
A.2 Maximizing the Algorithmic Mutual Information
Lemma 2 For each category  $c$  and for  $X^i$  with  $C^i = c$ , we can find a binary decision tree  $\mathcal{T}_{\mathrm{c}}$  that starting from its root, reaches each  $X^i$  by following the decision tree path. Based on this path, we assign code  $c(X^i) = c_1^i c_2^i \dots c_M^i$  to each  $X^i$  to uniquely define and retrieve it from the tree.
Proof of Lemma 2. Since the number of examples in the dataset is finite, we can enumerate samples of category  $c$  with any arbitrary coding. We then can replace these enumerations with their binary equivalent codes. We start from a root, and every time we encounter 1 in digits of these codes, we add a right child node, and for 0, we add a left child node. We then continue from the child node until we reach the code's end. Since the number of samples with category  $c$  is limited, this process should terminate. On the other hand, since the binary codes for different samples are different, these paths are unique, and by the time we traverse a path from the root to a leaf node, we can identify the unique sample corresponding to that node.
As mentioned in the main paper, using this Lemma, we can find at least one supertree  $\mathbf{T}$  for the whole data space that addresses all samples in which samples of the same category share a similar prefix. We can define a model that provides the binary code  $z^i = z_1^i\cdots z_L^i$  for data input  $X^i$  with category  $c$  based on the path it takes in these eligible trees. We define these path encoding functions valid encoding as defined in Definition 2:
Definition 2 A valid encoding for input space  $\mathcal{X}$  and category space  $\mathcal{C}$  is defined as an encoding that uniquely identifies every  $X^i\in \mathcal{X}$ . At the same time, for each category  $c\in \mathcal{C}$ , it ensures that there is a sequence that is shared among all members of this category but no member out of the category.
Since there is no condition on how to create these trees and their subtrees, many candidate trees can address the whole data space while preserving a similar prefix for the members of each category.
However, based on our inspirations for how the brain does categorization, we assume the ground truth underlying tree  $\mathbf{T}$  has a minimum average length path from the root to each node. In other words, each sample  $x$  has the shortest description code  $z$  to describe that data point while maintaining its validity. If we use a model to learn this encoding, the optimal model tree should be isomorph to the underlying tree  $\mathbf{T}$ ,
Lemma 3 For a learned binary code  $z^i$  to address input  $X^i$ , uniquely, if the decision tree of this encoding is optimal, it is isomorph to the underlying tree  $T$ .
Proof of Lemma 3. Since the underlying tree has the minimum Kolmogorov complexity for each sample, we can extract the optimal lengths of each sample by traversing the tree. Evans and Lanoue [115] showed that a tree can be recovered from the sequence of lengths of the paths from the root to leaves to the level of isomorphism. Based on our assumption about the underlying tree  $\mathbf{T}$ , the optimal tree can not have a shorter length for any sample codes than the underlying tree. On the other hand, having longer codes contradicts its optimality. Hence the optimal tree should have similar path lengths to the underlying ground truth tree. Therefore, it is isomorphic to the underlying tree.  $\square$
Since the optimal tree with the valid encoding  $\tilde{z}$  is isomorph to the underlying tree, we will have the necessary conditions that Theorem 1 provides.
Theorem 1 For a learned binary code  $z^i$  to address input  $x^i$ , uniquely, if the decision tree of this encoding is isomorph to underlying tree  $\mathbf{T}$ , we will have the following necessary conditions:
1.  $I_{alg}(z : x) \geq I_{alg}(\tilde{z} : x) \quad \forall \tilde{z}, \tilde{z}$  is a valid encoding for  $x$
2.  $I_{alg}(z:c)\geq I_{alg}(\tilde{z}:c)\quad \forall \tilde{z},\tilde{z}$  is a valid encoding for  $x$
Part one: From the way  $\mathbf{T}$  has been constructed, we know that  $K(x|\mathbf{T})\leq K(x|\mathcal{T})$  in which  $\mathcal{T}$  is an arbitrary tree. From the complexity and mutual information properties, we also have  $I_{\mathrm{alg}}(\mathrm{z}:x) = K(\mathrm{z}) - K(x|\mathrm{z})$  [116]. Since  $\tilde{z}$  and  $\mathbf{Z}$  have isomorph tree structures, then  $K(\tilde{z}) = K(\mathrm{z})$  hence:  $I_{\mathrm{alg}}(\mathrm{z}:x)\geq I_{\mathrm{alg}}(\tilde{\mathrm{z}}:x)$ .
Part two: In any tree that is a valid encoding, all samples of a category should be the descendants of that node. Thus, the path length to corresponding nodes should be similar in both trees. Otherwise, the length of the path to all samples of this category will not be optimal. We can use the same logic and deduce that the subtree with the category nodes as its leaves would be isomorph for both embeddings. Let's denote the path from the root to category nodes with  $\mathbf{z}_c$  and from the category node to its corresponding samples with  $\mathbf{z}_x$ . If we assume these two paths can be considered independent, we will have  $K(x) = K(\mathbf{z}_c\mathbf{z}_x) = K(\mathbf{z}_c) + K(\mathbf{z}_x)$ , which indicates that minimizing  $K(x)$  in the tree implies that  $K(c)$  also should be minimized. By applying the same logic as part one, we can deduce that  $I_{\mathrm{alg}}(\mathbf{z}:c)\geq I_{\mathrm{alg}}(\tilde{\mathbf{z}}:c)$ .
A.2.1 Shannon Mutual Information Approximation
Optimization in Theorem 1 is generally not computable [58-61]. However, We can approximate these requirements using Shannon mutual information instead. Let's consider two functions  $f$  and  $g$ , such that both are  $\{0,1\}^* \to \mathbb{R}$ . For these functions,  $f < g$  means that there exists a constant  $\kappa$ , such that  $f \leq g + c$ , when both  $f < g$  and  $g < f$  hold, then  $f \pm g$  [116].
Theorem 2 [116] Let  $P$  be a computable probability distribution on  $\{0,1\}^* \times \{0,1\}^*$ . Then:
$$
I (X; Z) - K (P) <   \sum_ {x} ^ {+} \sum_ {z} p (x, z) I _ {\text {a l g}} (\mathrm {x}: \mathrm {z}) <   I (X; Z) + 2 K (P) \tag {9}
$$
This theorem states that the expected value of algorithmic mutual information is close to its probabilistic counterpart. This means that if we maximize the Shannon information, we also approximately maximize the algorithmic information and vice versa.
Since Shannon entropy does not consider the inner regularity of the symbols it codes, to make each sequence meaningful from a probabilistic perspective, we convert each sequence to an equivalent random variable number by considering its binary digit representation. To this end, we consider  $Z^{i} = \sum_{k=1}^{m} \frac{z_{k}^{i}}{2^{k}}$ , which is a number between 0 and 1. Note that we can recover the sequence from the value of this random variable. Since the differences in the first bits affect the number more, for different error thresholds, Shannon's information will focus on the initial bits more. In dealing with real-world data, the first bits of encoding of a category sequence are more valuable than later ones due to the hierarchical nature of categories. Furthermore, with this tweak, we equip Shannon's model with a knowledge of different positions of digits in a sequence. To replace the first item of Theorem 1 by its equivalent Shannon mutual information, we must also ensure that  $z$  has the minimum length. For the moment, let's assume we know this length by the function  $l(X^{i}) = l_{i}$ . Instead of  $Z^{i}$ , we can consider its truncated form  $Z_{l_{i}}^{i} = \sum_{k=1}^{l_{i}} \frac{z_{k}^{i}}{2^{k}}$ . This term, which we call the address loss function, is defined as follows:
$$
\mathcal {L} _ {\mathrm {a d r}} = - \frac {1}{N} \sum_ {i = 0} ^ {N} I \left(X ^ {i}; Z _ {l _ {i}} ^ {i}\right) \quad s. t. \quad Z _ {l _ {i}} ^ {i} = \sum_ {k = 1} ^ {l _ {i}} \frac {z _ {k} ^ {i}}{2 ^ {k}} \text {a n d} \forall k, z _ {k} ^ {i} \in \{0, 1 \}. \tag {10}
$$
We can approximate this optimization with a reconstruction or contrastive loss.
A.2.2 Approximation with Reconstruction Loss
Let's approximate the maximization of the mutual information by minimizing the  $\mathcal{L}_{MSE}$  of the reconstruction from the code  $z$ . Suppose that  $D(X)$  is the decoder function, and it is a Lipschitz continuous function, which is a valid assumption for most deep networks with conventional activation functions [117]. We can find an upper bound for  $\mathcal{L}_{MSE}$  using Lemma 3.
Lemma 3 Suppose that  $D(X)$  is a Lipschitz continuous function with Lipschitz constant  $\kappa$ , then we will have the following upper bound for  $\mathcal{L}_{MSE}$ :
$$
\mathcal {L} _ {M S E} (X) \leq \kappa \frac {1}{N} \sum_ {i = 0} ^ {N} 2 ^ {- 2 l _ {i}}
$$
Proof of Lemma 3. Let's consider the  $\mathcal{L}_{MSE}$  loss for the reconstruction  $\hat{X}^i$  from the code  $Z^i$ . We denote reconstruction from the truncated category code  $Z_{l_i}^i$  with  $\hat{X}_{l_i}^i$ .
$$
\mathcal {L} _ {M S E} (X) = \frac {1}{N} \sum_ {i = 0} ^ {N} \| \hat {X} _ {l _ {i}} ^ {i} - X ^ {i} \| ^ {2}
$$
If we expand this loss, we will have the following:
$$
\begin{array}{l} \mathcal {L} _ {M S E} (X) = \frac {1}{N} \sum_ {i = 0} ^ {N} \| D \left(Z _ {L \left(X ^ {i}\right)} ^ {i}\right) - X ^ {i} \| ^ {2} \\ = \frac {1}{N} \sum_ {i = 0} ^ {N} \| D \left(\sum_ {k = 0} ^ {l _ {i}} \frac {z _ {k} ^ {i}}{2 ^ {k}}\right) - X ^ {i} \| ^ {2} \\ \end{array}
$$
Let's assume the optimal model can reconstruct  $X^i$  using the entire code length  $Z^i$ , i.e.  $X^i = D\left(\sum_{k=0}^{m} \frac{z_k^i}{2^k}\right)$ . Now let's replace this in the equation:
$$
\mathcal {L} _ {M S E} (X) = \frac {1}{N} \sum_ {i = 0} ^ {N} \| D (\sum_ {k = 0} ^ {l _ {i}} \frac {z _ {k} ^ {i}}{2 ^ {k}}) - D (\sum_ {k = 0} ^ {m} \frac {z _ {k} ^ {i}}{2 ^ {k}}) \| ^ {2}
$$
Given that  $D(X)$  is a Lipschitz continuous function with the Lipschitz constant  $\kappa$ , then we will have the following:
$$
\begin{array}{l} \mathcal {L} _ {M S E} (X) \leq \kappa \frac {1}{N} \sum_ {i = 0} ^ {N} \| \sum_ {k = 0} ^ {l _ {i}} \frac {z _ {k} ^ {i}}{2 ^ {k}} - \sum_ {k = 0} ^ {m} \frac {z _ {k} ^ {i}}{2 ^ {k}} \| ^ {2} \\ \leq \kappa \frac {1}{N} \sum_ {i = 0} ^ {N} \| 2 ^ {- l _ {i}} \| ^ {2} \\ = \kappa \frac {1}{N} \sum_ {i = 0} ^ {N} 2 ^ {- 2 l _ {i}} \quad \square \\ \end{array}
$$
Lemma 3 indicates that to minimize the upper bound on  $\mathcal{L}_{MSE}$ , we should aim for codes with maximum length, which can also be seen intuitively. The more length of latent code we preserve, the more accurate the reconstruction would be. This is in direct contrast with the length minimization of the algorithmic mutual information. So, the tradeoff between these two objectives defines the optimal final length of the category codes.
A.2.3 Approximation with Contrastive Loss
One of the advantages of contrastive learning is to find a representation that maximizes the mutual information with the input [64]. More precisely, if for input  $X^i$ , we show the hidden representation learning  $Z^i$ , that is learned contrastively by minimizing the InfoNCE loss, [64] showed that the following lower bound on mutual information exists:
$$
I \left(X ^ {i}; Z ^ {i}\right) \geq \log (N) - \mathcal {L} _ {N}. \tag {11}
$$
Here,  $\mathcal{L}_N$  is the InfoNCE loss, and  $N$  indicates the sample size consisting of one positive and  $N - 1$  negative samples. Equation 11 shows that contrastive learning with the InfoNCE loss can be a suitable choice for minimizing the  $\mathcal{L}_{adr}$  in Equation 10. We will use this to our advantage on two different levels. Let's consider that  $Z^i$  has dimension  $d$ , and each latent variable  $z_k^i$  can take up  $n$  different values. The complexity of the feature space for this latent variable would be  $\mathcal{O}(n^d)$ , then the number of structurally different binary trees for this feature space would be  $\mathcal{O}(C_{n^d})$ , in which  $C_i$  is the  $i$ th Catalan number, which asymptotically grows as  $\mathcal{O}(4^i)$ . Hence the number of possible binary taxonomies for the categories will be  $\mathcal{O}(4^{n^d})$ . So minimizing  $n$  and, to a lesser degree,  $d$ , will be the most effective way to limit the number of possible binary trees. Since our model and the amount of training data is bounded, we must minimize the possible search space while still providing reasonable performance. On the other hand, the input feature space  $X^i$  with  $N$  possible values and dimension  $D$  has  $\mathcal{O}(N^D)$  possible states, and to cover it completely, we can not arbitrarily decrease  $d$  and  $n$ . Note that for a nearly continuous function  $N \to \infty$ , the probability of a random discrete tree to fully covering this space would be near zero.
A.3 Category Code Length Minimization
In the main paper, we indicate the code length loss  $\mathcal{L}_{\text{length}}$ , which we define as  $\mathcal{L}_{\text{length}} = \frac{1}{N} \sum_{i=0}^{N} l_i$ . To minimize this loss, we define a binary mask sequence  $\mathrm{m}^i = m_1^i m_2^i \cdots m_L^i$  to simulate the subscript property of  $l_i$ . We discussed minimizing the  $L_p$  Norm for the weighted version of the mask, which we denote with  $\bar{\mathrm{m}}^i = (m_1^i 2^1)(m_2^i 2^2) \cdots (m_L^i 2^L)$ . This will ensure the requirements because adding one extra bit has an equivalent loss of all previous bits.
$$
\mathcal {L} _ {\text {l e n g t h}} \approx \frac {1}{N} \sum_ {i = 0} ^ {N} \| \bar {\mathrm {m}} ^ {i} \| _ {p}. \tag {12}
$$
Lemma 4 Consider the weighted mask  $\bar{\mathfrak{m}} = (m_12^1)(m_22^2)\dots (m_L2^L)$  where  $m_j$ s are 0 or 1. Consider the norm  $\| \bar{\mathfrak{m}}\| _p$  where  $p\geq 1$ , the rightmost 1 digit contributes more to the norm than the entire left sequence.
Proof of Lemma 4. Let's consider the loss function for mask  $\bar{\mathfrak{m}} = (m_12^1)(m_22^2)\dots (m_L2^L)$  and let's denote the rightmost 1 index, with  $k$ , for simplicity we consider the  $\| \bar{\mathfrak{m}}\| _p^p$
$$
\parallel \bar {\mathrm {m}} \parallel_ {p} ^ {p} = \sum_ {j = 0} ^ {L} (m _ {j} 2 ^ {j}) ^ {p} = \sum_ {j = 0} ^ {k - 1} (m _ {j} 2 ^ {j}) ^ {p} + (m _ {k} 2 ^ {k}) ^ {p} + \sum_ {j = k + 1} ^ {L} (m _ {j} 2 ^ {j}) ^ {p}
$$
given that  $m_j = 0, \forall j > k$  and  $m_k = 1$ , we will have:
$$
\parallel \bar {\mathrm {m}} \parallel_ {p} ^ {p} = = \sum_ {j = 0} ^ {k - 1} (m _ {j} 2 ^ {j}) ^ {p} + 2 ^ {k p} + 0
$$
now let's compare the two subparts of the right-hand side with each other:
$$
\sum_ {j = 0} ^ {k - 1} \left(m _ {j} 2 ^ {j}\right) ^ {p} \leq \sum_ {j = 0} ^ {k - 1} \left(2 ^ {j}\right) ^ {p} = \frac {2 ^ {k p} - 1}{2 ^ {p} - 1} <   2 ^ {k p} \quad \square
$$
Hence  $\mathcal{L}_{\text{Length}}$  tries to minimize the position of the rightmost 1, simulating the cutting length subscript.
A.3.1 Satisfying Binary Constraints.
In the main paper, we stated that we have two conditions, Code Constraint:  $\forall z_k^i$ ,  $z_k^i = 0$  or  $z_k^i = 1$  and Mask Constraint  $\forall m_k^i$ ,  $m_k^i = 0$  or  $m_k^i = 1$ . We formulate each constraint in an equivalent Lagrangian function to make sure they are satisfied. For the binary code constraint we consider  $f_{\mathrm{code}}(z_k^i) = (z_k^i)(1 - z_k^i) = 0$ , which is only zero if  $z_k^i = 0$  or  $z_k^i = 1$ . Similarly, for the binary mask constraint, we have  $f_{\mathrm{mask}}(m_k^i) = (m_k^i)(1 - m_k^i) = 0$ . To ensure these constraints are satisfied, we optimize them with the Lagrangian function of the overall loss. Consider the Lagrangian function for  $\mathcal{L}_{\mathrm{total}}$ ,
$$
\mathbf {L} (\mathcal {L} _ {\text {t o t a l}}, \eta , \mu) = \mathcal {L} _ {\text {t o t a l}} + \eta \mathcal {L} _ {\text {c o d e} _ {\text {c o n d}}} + \mu \mathcal {L} _ {\text {m a s k} _ {\text {c o n d}}}
$$
This lagrangian function ensures that constraints are satisfied for  $\eta \rightarrow +\infty$  and  $\mu \rightarrow +\infty$ . Note that our method uses a tanh activation function which has been mapped between 0 and 1, to produce  $m_{k}$  and  $z_{k}$ , so the conditions are always greater or equal to zero. For an unbounded output, we can consider the squared version of constraint functions to ensure that constraints will be satisfied. This shows how we reach the final unconstrained loss function in the paper.
To acquire the train and test splits, we follow [27]. We subsample the training dataset in a ratio of  $50\%$  of known categories at the train and all samples of unknown categories. For all datasets except CIFAR100, we consider  $50\%$  of categories as known categories at training time. For CIFAR100 as in [27]  $80\%$  of the categories are known during training time. A summary of dataset statistics and their train test splits is shown in Table 6.
CIFAR10/100[65] are coarse-grained datasets consisting of general categories such as car, ship, airplane, truck, horse, deer, cat, dog, frog and bird.
ImageNet-100 is a subset of 100 categories from the coarse-grained ImageNet [66] dataset.
CUB or the Caltech-UCSD Birds-200-2011 (CUB-200-2011) [67] is one of the most used datasets for fine-grained image recognition. It contains different bird species, which should be distinguished by relying on subtle details.
FGVC-Aircraft or Fine-Grained Visual Classification of Aircraft [68] dataset is another fine-grained dataset, which, instead of animals, relies on airplanes. This might be challenging for image recognition models since, in this dataset, structure changes with design.
SCars or Stanford Cars [69] is a fine-grained dataset of different brands of cars. This is challenging since the same brand of cars can look different from different angles or with different colors.