SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kanishk Jain<sup>1</sup>, Shyamgopal Karthik<sup>2</sup>, Vineet Gandhi<sup>1</sup>
$^{1}$ IIIT Hyderabad  $^{2}$ University of Tübingen
We investigate the problem of reducing mistake severity for fine-grained classification. Fine-grained classification can be challenging, mainly due to the requirement of domain expertise for accurate annotation. However, humans are particularly adept at performing coarse classification as it requires relatively low levels of expertise. To this end, we present a novel approach for Post-Hoc Correction called Hierarchical Ensembles (HiE) that utilizes label hierarchy to improve the performance of fine-grained classification at test-time using the coarse-grained predictions. By only requiring the parents of leaf nodes, our method significantly reduces avg. mistake severity while improving top-1 accuracy on the iNaturalist-19 and tieredImageNet-H datasets, achieving a new state-of-the-art on both benchmarks. We also investigate the efficacy of our approach in the semi-supervised setting. Our approach brings notable gains in top-1 accuracy while significantly decreasing the severity of mistakes as training data decreases for the fine-grained classes. The simplicity and post-hoc nature of HiE renders it practical to be used with any off-the-shelf trained model to improve its predictions further. The code is available at: https://github.com/kanji95/Hierarchical-Ensembles
Over the past decade, large-scale datasets have been instrumental in driving the rapid progress of computer vision/image recognition. However, in settings that require experts to annotate samples, collecting large amounts of labeled data can be prohibitively expensive. Fine-Grained Visual Classification (FGVC) is one such example, where one would need a domain expert to be able to identify the category for a particular sample. However, it can be cheaper to obtain coarse labels for the same samples in these settings. For instance, in Figure 1 while it is evident to spot and identify a butterfly or a bird, one would need a Lepidopterist or an Ornithologists to categorize them as Junonia Genoveva or Water Ouzel.
Therefore, one popular research direction in recent times has been to utilize the availability of a larger set of images with coarse labels to improve the performance (i.e. top-1 accuracies) of the neural network models on various fine-grained classification benchmarks [1, 2]. Using coarse labels in a classification setting introduces the concept of a label hierarchy, where all the labels in a dataset are connected through some taxonomy. This taxonomy could either be defined from the biological taxonomy, as with many species recognition datasets or could be derived from language-based ontologies such as WordNet [3]. Another mainstream usage of label hierarchies in recent times is to use them to reduce the severity of mistakes committed by various classification models [4, 5, 6, 7, 8]. For instance, mistaking a car for a bus is a better mistake than mistaking a car for a lamppost. This originated from the extensive work on cost-sensitive classification [9, 10]. However, defining pairwise costs can be a tedious process and scales quadratically with the number of classes. Therefore, the recent works have all focused on using the label hierarchy and defining costs automatically based on graph distances (e.g. using the height of the least common ancestor between the predicted and the ground truth class as a proxy for cost). The research in this direction aims to invent methods that at
Coarse Label
Fine Label
Linnet
Water Ouzel
Figure 1: While it is straightforward to identify the images based on their coarse labels, it is challenging to distinguish them based on fine-grained labels without domain expertise.
Indigo Bunting
CommonBuckeye
Junonia Genoveva
least retain the accuracies of the backbone baseline models while reducing the severity of mistakes committed by it.
While these two research directions might seem entirely different, they work with the same inputs (i.e., images, labels, and the label hierarchy). Therefore, one would ideally want a method that can use additional coarsely labelled samples to improve the model's accuracy and utilise the label hierarchy to reduce the severity of the mistakes. However, these two goals have been tackled independently. Post-hoc correction methods [4, 7], learning hierarchy-aware features [6, 8, 11] as well as structured embedding spaces [5, 12] have been proposed to reduce the severity of mistakes. Coarse labels have been used for self-training/pseudo-labelling [13] and with an additional objective to regularise/enhance the training [2] in the semi-supervised learning frameworks.
In this work, we propose a simple method that is able to bring significant improvements to both of these problems. Our fundamental insight is that models trained for different granularities focus on different aspects. This is also corroborated by recent work [11, 8], which shows that when training a model jointly, coarse-level label prediction exacerbates fine-grained feature learning. Therefore, we train two separate models for the coarse and fine-grained labels. At inference, we employ a modified decision rule, which computes the argmax over the normalized score of coarse and fine grained predictions. We refer to the proposed strategy as a Hierarchical Ensemble (HiE). We find that employing HiE not only reduces the severity of mistakes, but is able to improve top-1 accuracies, which are far more pronounced in the semi-supervised setting. More formally, we make the following contributions:
1. We introduce a simple post-hoc mechanism called Hierarchical Ensembles for incorporating cues from multiple models trained for different granularities of a hierarchy tree.
2. We demonstrate that HiE brings considerable reductions in mistake severity while improving the top-1 accuracy. HiE significantly outperforms two competent baselines and six other methods from the prior art and achieves state-of-the-art performance on both iNaturalist-19 and tieredImageNet-H benchmarks.
3. We illustrate that HiE brings consistent gains when plugged with existing semi-supervised methods exploiting additional coarse labeled data to improve the fine-grained classification. On iNaturalist-19 dataset, HiE combined with Moco [14] and hierarchical loss [2], recovers  $86\%$  of the underlying performance while using only  $0.5\%$  of the fine-grained annotations.
Learning from Taxonomic Labels: There has been a long history of utilizing label hierarchies for better classification performance. Sila and Freitas [15] perform a comprehensive survey of hierarchical classification methods applied to various tasks in different application domains. Broadly, the methods that train hierarchy-aware features can be categorized into three categories a) Label-Embedding methods, b) Hierarchical architectures, and c) Hierarchical loss functions.
The label-embedding methods project the categories into a semantic embedding space [12, 16] instead of viewing them as one-hot encodings and are widely used, especially in zero-shot learning [17]. The semantic embedding space can be derived either from side-information such as class attributes [16],
taxonomies [6], or from natural language [12]. These can be further enhanced by learning these embeddings on a hypersphere [5] or hyperbolic spaces [18, 19, 20].
On the other hand, hierarchical loss functions try to modify the training objective to incorporate the label hierarchy [21, 6, 22, 8, 23]. The most common approach here is to introduce a loss at every level in the hierarchy [21, 6, 8]. The challenge in this direction has been to ensure that the coarse-grained task does not hamper the accuracy of the fine-grained classifier [11, 8].
There have also been attempts to modify the architecture based on the label hierarchy. For instance, popular ideas revolve around having branches at different layers of the model [24], predicting the conditional probabilities at each node in the hierarchy [25, 26], or partitioning the feature space using the levels in the hierarchies [11].
Most closely related to our work are the works that attempt to incorporate the label hierarchy information post-hoc during inference. Here, traditional Conditional Risk Minimization (CRM) [27] was applied in this setting by [4] and recently revisited by [7]. Our proposed approach is also applied during inference; however, it is orthogonal to CRM, and we demonstrate that CRM can be used to improve our results further. Concurrent to our work, [28] proposed a similar approach to utilize subclasses to improve superclass recognition performance in a post-hoc manner with vision-language models [29].
Fine-Grained Visual Classification: Contrary to the popular image recognition benchmarks, fine-grained visual classification focuses on recognizing the differences between similar-looking categories [30, 31, 32]. Here, methods often focus on learning discriminative features that use local information since most categories share a similar global structure [33, 34, 35]. However, the most related to our work are the ones that use class taxonomies to improve fine-grained classification performance. This is most frequently done in the semi-supervised setting [1], where one has access to an additional set of weakly labeled samples. Here, approaches have explored incorporating a hierarchical loss on the coarse labels [2] or using the coarse labels to filter pseudo-labels [13] in addition to the standard semi-supervised learning techniques of consistency regularization [36], pseudo-labeling [37, 38], and contrastive learning [14]. Our proposed approach is complementary to these works and can be applied to all of these works to bring additional improvements.
We consider the fine-grained classification task with a label hierarchy  $\mathcal{H}$  of  $L$  levels defined over the class labels. The leaf nodes of  $\mathcal{H}$  correspond to fine-grained classes at level  $L$ , and the class labels at level  $l < L$  are considered coarse-grained labels. The goal is to use the information from coarse annotations to improve the performance on fine labels. Formally, given a dataset  $\mathcal{X} = \{(x_i,y_i^l)\mid i = 1,2,\ldots ,N\}$  consisting of  $N$  training images and their respective ground truth labels at level  $l$ , where label  $y_{i}^{l}\in \mathcal{Y}^{l} = \{1,2,\dots,N_{l}\}$ , and  $N_{l}$  corresponds to the number of classes at level  $l$ , the task is to train a classifier  $f_{\theta}^{L}:X\to p(\mathcal{V}^{L})$ , on fine-grained class labels by making use of the information available through coarse-grained labels  $\mathcal{Y}^{l < L}$ .
Suppose we have two classifiers,  $f_{\theta}^{L}$  and  $f_{\phi}^{L - 1}$  trained on hierarchy levels  $L$ ,  $L - 1$  with  $\theta$  and  $\phi$  being their parameters, respectively, then for a sample  $x \sim D$ , we have,  $\hat{y}_L = f_\theta^L (x)$  and  $\hat{y}_{L - 1} = f_{\phi}^{L - 1}(x)$  as logits of two classifiers for the input  $x$ . We further define:
$$
Q = \left[ q _ {1}, q _ {2}, \dots , q _ {N _ {L}} \right] = \operatorname {s o f t m a x} \left(\hat {y} _ {L}\right)
$$
$$
R = \left[ r _ {1}, r _ {2}, \dots , r _ {N _ {L - 1}} \right] = \operatorname {s o f t m a x} \left(\hat {y} _ {L - 1}\right)
$$
Figure 2: Consider a four class hierarchy tree and corresponding fine-grained predictions  $(q_{i})$  and coarse level predictions  $(r_i)$  obtained using independent models. Our work aims to improve the fine grained classification by leveraging the predictions at the coarse level. In the given example, the class prediction changes from 'rose' to 'bus' after the post-hoc correction.
We reweigh the fine-grained predictions  $\mathcal{Q}$ , given the softmax probabilities  $\mathcal{R}$  obtained from the coarse-grained classifier:
$$
P (i | x, \theta , \phi , \mathcal {H}) = \frac {P (i | x ; \theta) P \left(i _ {\text {p a r e n t}} | x ; \phi\right)}{\sum_ {j = 1} ^ {N _ {L}} P (j | x ; \theta) P \left(j _ {\text {p a r e n t}} | x ; \phi\right)}, \quad \text {w h e r e} i _ {\text {p a r e n t}} = \mathcal {H} (i) \tag {1}
$$
The denominator is the normalization term, ensuring  $\sum_{i=1}^{N_L} P(i|x,\mathcal{H}) = 1$ . In the studied setting,  $P(i|x,\mathcal{H})$  simplifies to:
$$
P (i | x, \mathcal {H}) = s _ {i} = \frac {q _ {i} \cdot r _ {i _ {\text {p a r e n t}}}}{\sum_ {j = 1} ^ {N _ {L}} q _ {j} \cdot r _ {j _ {\text {p a r e n t}}}} \tag {2}
$$
The main modification we make in the classification strategy at the fine grained level is in the decision rule, which now selects the class that maximizes the normalized score:
$$
\underset {i} {\operatorname {a r g m a x}} P (i | x, \mathcal {H}) \tag {3}
$$
In Figure 2, we illustrate a four-class example, comparing standard cross-entropy predictions with those from our proposed method. We would like to point out that the proposed technique can be interpreted as a Hierarchical Ensemble. Existing literature in neural networks-based ensemble learning has often focused on classifiers trained at a single granularity [39] and has largely overlooked ensembles with hierarchical context. In this work, we highlight the practical benefits of hierarchical ensembles for improving the fine-grained classification task by choosing the simplest possible method of multiplying the predictions at different levels of hierarchy.
We show that if we make a correct prediction at the coarse level, the proposed Hierarchical Ensemble (HiE) is guaranteed to improve the downstream predictions at the fine-grained classification task.
Theorem 3.1. Assuming,  $Q = [q_{1}, q_{2}, \dots, q_{N_{L}}]$  and  $R = [r_{1}, r_{2}, \dots, r_{N_{L-1}}]$  are the predictions obtained at the fine and coarse grained labels for a given input  $x$ , such that  $\sum_{i=1}^{N_{L}} q_{i} = 1$  and  $\sum_{i=1}^{N_{L-1}} r_{i} = 1$ . For the ground truth labels  $g$  and  $g_{parent}$  at the fine grained and the coarse grained levels respectively:
Now assuming that the coarse label is correctly predicted by the coarse prediction network i.e.  $\operatorname{argmax}(R) = g_{parent}$ , we wish to prove that:
$$
\frac {q _ {g} \cdot r _ {g _ {p a r e n t}}}{\sum_ {j = 1} ^ {N _ {L}} q _ {j} \cdot r _ {j _ {p a r e n t}}} \geq q _ {g}.
$$
Proof. The denominator iterates over the fine grained predictions and multiplies them with their parent's prediction scores. This is equivalent to iterating over the coarse label predictions and multiplying with the sum of prediction scores for all its children. By rewriting the denominator, we obtain:
$$
\sum_ {j = 1} ^ {N _ {L}} q _ {j}. r _ {j _ {p a r e n t}} = \sum_ {j = 1} ^ {N _ {L - 1}} r _ {j} \sum_ {i \in j _ {c h i l d}} q _ {i}
$$
Assuming,  $\sum_{i\in j_{child}}q_i = z_j$
$$
\sum_ {j = 1} ^ {N _ {L - 1}} r _ {j} \sum_ {i \in j _ {\text {c h i l d}}} q _ {i} = \sum_ {j = 1} ^ {N _ {L - 1}} r _ {j}. z _ {j} = R ^ {T} Z
$$
Invoking Holder's inequality, using  $a = \infty$  and  $b = 1$ ,  $(1 / a + 1 / b = 1)$ , we obtain:
$$
R ^ {T} Z \leq \| R \| _ {\infty} \| Z \| _ {1}
$$
Since,  $\| R \|_{\infty} = \max(R) = r_{g_{parent}}$  and  $\| Z \|_1 = \sum_{i=1}^{N_L} q_i = 1$ . We can say:
$$
R ^ {T} Z \leq \| R \| _ {\infty} \| Z \| _ {1} \leq r _ {g _ {p a r e n t}}
$$
Given the above equation, we can conclude that:
$$
\frac {r _ {g _ {p a r e n t}}}{\sum_ {j = 1} ^ {N _ {L}} q _ {j} \cdot r _ {j _ {p a r e n t}}} \geq 1.
$$
When correct predictions are made at both the coarse and fine-grained levels, the method will help increase the confidence of the prediction. A more interesting case occurs when the coarse-level prediction is correct; however, the fine-grained classifier makes a wrong prediction. The hierarchical ensemble can shift the prediction to the correct sub-tree, reducing the mistake severity and potentially improving the top-1 accuracy.
Like prior works [6, 8, 7], we evaluate our approach on the iNaturalist-19 [40] and tieredImageNet-H [41] datasets. The iNaturalist-19 dataset is a species classification dataset containing plants, animals, fungi, etc., and has an 8-level hierarchy. The tieredImageNet-H dataset contains classes from a wide range of categories, including musical instruments, animal breeds, tools, and plant species. It has a 13-level hierarchy derived from the WordNet hierarchy. However, rather than using the full hierarchy of categories, for our method, we only use the leaf classes and their parent categories during training and inference. The iNaturalist-19 dataset includes 1010 fine-grained "species" classes and 72 coarse-grained classes for the "genus" taxonomy. The tieredImageNet-H dataset has 608 leaf classes and 201 parent classes. We train classifiers for each level independently, using the same hyperparameter settings.
In addition, we also evaluate the performance of our approach in a semi-supervised setting. Specifically, we want to test the performance on fine-grained classification tasks using a large number of coarsely annotated examples and a limited number of finely annotated examples. We use the iNaturalist-19 dataset for this setting; the coarse labels for all the images are used, while for fine-grained labels, we sample a limited number of images per class label during training. We used the "species" and "genus" taxonomies for the fine and coarse level labels and performed experiments for cases when the number of images per fine label is 100, 50, 25, and 10. Similar to the supervised setting, we train separate classifiers on coarse and fine labels.
To align with prior efforts [7, 8], all of our experiments are performed using the ResNet-18 backbone. We randomly crop a portion of images and resize them to  $224 \times 224$  resolution for both datasets. For the iNaturalist-19 dataset, we initialize the ResNet-18 weights with a pre-trained ImageNet model and train the classifiers using a customized SGD optimizer for 100 epochs, with different learning rates for the backbone network and the fully connected layer (0.01 and 0.1, respectively). For the tieredImageNet-H dataset, we train the ResNet-18 from scratch (since it is derived from the ImageNet
Method	Top-1 Error(↓)	Mistakes severity(↓)	Hier dist@1(↓)	Hier dist@5(↓)	Hier dist@20(↓)
Without CRM
Cross-Entropy	36.44 ± 0.061	2.39 ± 0.007	0.87 ± 0.004	1.97 ± 0.002	3.25 ± 0.002
Cross-Entropy-H	37.63 ± 0.054	2.37 ± 0.005	0.89 ± 0.009	1.86 ± 0.001	2.96 ± 0.001
HiE-Self	36.36 ± 0.047	2.36 ± 0.014	0.86 ± 0.005	1.44 ± 0.003	2.31 ± 0.004
Barz & Denzler [5]	62.63 ± 0.278	1.99 ± 0.008	1.24 ± 0.005	1.49 ± 0.005	1.97 ± 0.005
YOLO-v2 [25]	44.37 ± 0.106	2.42 ± 0.003	1.08 ± 0.004	1.90 ± 0.003	2.87 ± 0.010
HXE α=0.1 [6]	41.48 ± 0.204	2.41 ± 0.009	1.00 ± 0.006	1.77 ± 0.011	2.69 ± 0.021
HXE α=0.6 [6]	45.45 ± 0.014	2.24 ± 0.006	1.02 ± 0.003	1.70 ± 0.005	2.55 ± 0.005
Soft-labels β = 30 [6]	41.67 ± 0.134	2.32 ± 0.010	0.97 ± 0.006	1.50 ± 0.006	2.23 ± 0.005
Soft-labels β = 4 [6]	74.70 ± 0.212	1.82 ± 0.005	1.36 ± 0.004	1.49 ± 0.003	1.96 ± 0.004
Chang et al. [11]	37.23 ± 0.175	2.28 ± 0.006	0.85 ± 0.004	1.75 ± 0.005	3.02 ± 0.008
HAF [8]	36.40 ± 0.092	2.28 ± 0.012	0.83 ± 0.002	1.62 ± 0.002	2.55 ± 0.003
HiE (Ours)	35.33 ± 0.037	2.15 ± 0.003	0.76 ± 0.001	1.33 ± 0.001	2.19 ± 0.003
With CRM
Cross-Entropy [7]	36.51 ± 0.083	2.33 ± 0.001	0.85 ± 0.002	1.32 ± 0.001	1.86 ± 0.002
Cross-Entropy-H	37.70 ± 0.038	2.33 ± 0.011	0.88 ± 0.003	1.33 ± 0.002	1.87 ± 0.004
HiE-Self	36.45 ± 0.057	2.34 ± 0.019	0.85 ± 0.003	1.32 ± 0.003	1.86 ± 0.008
Barz & Denzler	75.61 ± 0.098	4.62 ± 0.032	3.49 ± 0.025	3.66 ± 0.024	3.84 ± 0.018
YOLO-v2	45.17 ± 0.046	2.43 ± 0.001	1.10 ± 0.001	1.50 ± 0.001	1.99 ± 0.002
HXE α=0.1	41.47 ± 0.220	2.38 ± 0.011	0.99 ± 0.008	1.41 ± 0.006	1.93 ± 0.005
HXE α=0.6	45.60 ± 0.017	2.21 ± 0.008	1.01 ± 0.003	1.40 ± 0.004	1.40 ± 0.004
Soft-labels β = 30	41.99 ± 0.126	2.31 ± 0.009	0.97 ± 0.007	1.40 ± 0.005	1.91 ± 0.005
Soft-labels β = 4	77.34 ± 0.262	2.06 ± 0.012	1.60 ± 0.007	1.72 ± 0.008	2.14 ± 0.007
Chang et al. [11]	37.31 ± 0.145	2.24 ± 0.008	0.84 ± 0.002	1.30 ± 0.002	1.84 ± 0.002
HAF [8]	36.48 ± 0.095	2.25 ± 0.012	0.82 ± 0.003	1.29 ± 0.004	1.84 ± 0.002
HiE (Ours)	35.42 ± 0.025	2.14 ± 0.015	0.75 ± 0.005	1.23 ± 0.005	1.79 ± 0.005
Table 1: Results comparing top-1 error(%) and hierarchical metrics for iNaturalist-19. Results in the Top block are reported without using CRM [7] technique and Bottom block are reported using CRM. Methods highlighted with aqua green are the best performing methods in top-1 error  $(\%)$ . We use dark green for the top performers and light green for the runners-up in each metric.
dataset) and use the AMSGrad variant of the Adam optimizer with a learning rate of  $1e^{-5}$  for 120 epochs. We conduct five runs for each experiment in tables 1 and 2 and report the mean and standard deviations.
Evaluation Metrics: Like previous works [6, 7], we validate our approach using the top-1 accuracy, avg. mistake severity and hierarchical distance@k metrics. Avg. mistake severity metric computes the average height of the lowest common ancestor (LCA) between the predicted class and the ground truth class for the incorrectly classified samples. Since this metric considers only the incorrect predictions, it is possible to reduce the mistake severity by making more low cost mistakes [7], as a consequence of which the top-1 accuracy decreases. To overcome this issue, hierarchical distance@k is used, which measures the average height of top-k predictions with the ground truth class for all samples. For  $k > 1$ , this metric captures the quality of ranking imposed by the network in the label space.
4.1 Making better mistakes
In this section, we demonstrate that the proposed hierarchical ensembles significantly reduce the severity of mistakes compared to the current state-of-the-art methods. We compare our method with [5], YOLO-v2 [25]; both approaches proposed by [6] with varying parameters - soft-labels and HXE. We also compare against the method proposed by [11] for multi-granular classification and HAF [8] imposing consistency on prediction heads across label hierarchy. We contrast the performance of the methods mentioned above to the cross-entropy baseline.
We also propose two custom baselines called Cross-Entropy-H and HiE-Self. Cross-Entropy-H obtains coarse class predictions by marginalizing the prediction of its fine-grained children classes. It then employs an additional hierarchical loss term [2] over the obtained coarse predictions. HiE-Self applies a hierarchical ensemble using the coarse predictions obtained post marginalization over fine-grained classes (instead of training a separate network for the coarse level).
We also experiment with the CRM technique proposed by [7]. Since CRM is a post-hoc approach that reweighs the probability distribution of samples obtained from any trained model, it can be applied to
pre-trained models of all the approaches. Therefore, in each of the Tables 1-2, we group the results to report evaluation metrics with and without using CRM at test time.