SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Method	Top-1 error(↓)	Mistakes severity(↓)	Hier dist@1(↓)	Hier dist@5(↓)	Hier dist@20(↓)
Without CRM
Cross-Entropy	30.64 ± 0.030	7.07 ± 0.010	2.17 ± 0.006	5.70 ± 0.003	7.25 ± 0.003
Cross-Entropy-H	32.87 ± 0.042	7.13 ± 0.031	2.35 ± 0.003	5.70 ± 0.009	7.14 ± 0.012
HiE-Self	30.78 ± 0.054	7.05 ± 0.028	2.19 ± 0.004	5.35 ± 0.009	6.92 ± 0.007
Barz & Denzler [5]	39.73 ± 0.240	6.80 ± 0.019	2.70 ± 0.022	5.48 ± 0.271	6.21 ± 0.005
YOLO-v2 [25]	33.37 ± 0.082	7.02 ± 0.004	2.34 ± 0.016	5.85 ± 0.011	7.43 ± 0.016
HXE α=0.1 [6]	30.72 ± 0.036	7.00 ± 0.019	2.15 ± 0.005	5.62 ± 0.008	7.08 ± 0.015
HXE α=0.6 [6]	34.50 ± 0.007	6.73 ± 0.014	2.32 ± 0.003	5.48 ± 0.001	6.78 ± 0.003
Soft-labels β = 30 [6]	30.53 ± 0.194	7.05 ± 0.009	2.15 ± 0.013	5.66 ± 0.002	7.14 ± 0.008
Soft-labels β = 4 [6]	38.99 ± 0.105	6.60 ± 0.024	2.57 ± 0.004	5.13 ± 0.002	6.21 ± 0.001
Chang et al. [11]	33.46 ± 0.026	6.99 ± 0.010	2.34 ± 0.006	5.75 ± 0.005	7.34 ± 0.010
HAF [8]	30.50 ± 0.010	7.03 ± 0.024	2.14 ± 0.008	5.62 ± 0.011	6.99 ± 0.009
HiE (Ours)	29.81 ± 0.081	6.95 ± 0.013	2.07 ± 0.014	5.30 ± 0.001	6.86 ± 0.001
With CRM
Cross-Entropy [7]	30.56 ± 0.020	7.01 ± 0.007	2.14 ± 0.006	4.93 ± 0.002	6.11 ± 0.001
Cross-Entropy-H	32.93 ± 0.029	7.05 ± 0.006	2.32 ± 0.004	5.01 ± 0.008	6.14 ± 0.003
HiE-Self	30.79 ± 0.031	7.04 ± 0.008	2.17 ± 0.004	4.98 ± 0.005	6.14 ± 0.002
Barz & Denzler	83.55 ± 0.000	11.94 ± 0.000	11.92 ± 0.000	11.91 ± 0.000	11.91 ± 0.000
YOLO-v2	33.98 ± 0.099	6.99 ± 0.011	2.38 ± 0.012	5.05 ± 0.001	6.17 ± 0.001
HXE α=0.1	30.80 ± 0.079	6.95 ± 0.021	2.14 ± 0.005	4.94 ± 0.003	6.11 ± 0.002
HXE α=0.6	34.68 ± 0.003	6.69 ± 0.007	2.32 ± 0.001	4.99 ± 0.005	6.13 ± 0.003
Soft-labels β = 30	30.69 ± 0.125	6.99 ± 0.007	2.15 ± 0.008	4.95 ± 0.001	6.11 ± 0.001
Soft-labels β = 4	82.72 ± 0.079	7.54 ± 0.001	6.24 ± 0.005	6.94 ± 0.005	7.25 ± 0.002
Chang et al. [11]	33.73 ± 0.033	6.93 ± 0.015	2.34 ± 0.002	5.02 ± 0.007	6.15 ± 0.001
HAF [8]	30.63 ± 0.007	6.97 ± 0.024	2.14 ± 0.008	4.95 ± 0.004	6.11 ± 0.001
HiE (Ours)	29.89 ± 0.082	6.93 ± 0.013	2.07 ± 0.013	4.93 ± 0.001	6.11 ± 0.001
Table 2: Results comparing top-1 error(%) and hierarchical metrics on the test set of tieredImageNet-H. The Top block reports results without using CRM [7] and the Bottom block are reported using CRM. Methods highlighted with aqua green are the best performing methods in top-1 error  $(\%)$ . We use dark green for the top performers and light green for the runners-up in each metric.
Apart from HAF, all the other methods from the prior art trade-off the top-1 accuracy with the mistake severity (Tables 1&2). The results contrast with the problem's main objective, i.e., improve the hierarchical metrics by maintaining or improving the top-1 error. In some of the instances (e.g. HXE  $\alpha = 0.6$ , Soft-labels  $\beta = 30$ ), the avg. mistake severity is reduced by making additional low-severity mistakes (indicated by an increase in top-1 error and  $Hier~dist@1$ ). The multi-head approach by [11] brings minor gains, and HAF further improves upon it by introducing consistency loss among different head predictions.
The proposed minimalist approach of Hierarchical Ensembles (HiE) significantly outperforms all the methods; on both datasets, with and without CRM. On the three metrics of top-1 error, avg. mistake severity, and  $\text{Hier\_dist} @ 1$ , the performance achieved by HiE without CRM betters all other methods post-CRM. The gains achieved by HiE on  $\text{Hier\_dist} @ 5$  and  $\text{Hier\_dist} @ 20$  metrics are more pronounced, for instance, improving over HAF without CRM on iNaturalist-19 dataset by  $17\%$  and  $14\%$  respectively. These two ranking metrics compare the ordering of the classes provided by each of these classifiers. The results illustrate that HiE is able to reliably align the predictions with the hierarchy. The results are interesting given that HiE only employs the bottom two levels of the hierarchy, in contrast to other methods which utilize the entire hierarchy tree.
Compared with the baselines, on the iNaturalist-19 dataset, the Cross-Entropy-H brings minor gains on  $\text{Hier dist}@5$  and  $\text{Hier dist}@20$  metrics; however, the top-1 error increases by over a percent. On tieredImageNet-H, Cross-Entropy-H fails to give any improvements. In contrast, the HiE-Self baseline is able to retain the original top-1 performance while bringing noticeable gains on  $\text{Hier dist}@k$  metrics. It illustrates that hierarchical ensembles are helpful even when applied on a single network. However, training separate networks for coarse and fine-grained levels allows us to learn complementary features which are then combined using our HiE approach.
Methods	#Images/Label	without HiE	with HiE
Top-1 Acc	Avg. Mistake Severity	Hierarchical Distance@1	Top-1 Acc	Avg. Mistake Severity	Hierarchical Distance@1
w/o CRM	w CRM	w/o CRM	w CRM	w/o CRM	w CRM	w/o CRM	w CRM	w/o CRM	w CRM	w/o CRM	w CRM
CrossEntropy	53.90	53.93	2.55	2.48	1.17	1.14	56.95	56.82	2.07	2.06	0.89	0.89
Pseudo-Label	51.25	51.09	2.52	2.46	1.23	1.20	54.78	54.61	2.04	2.03	0.92	0.92
MoCo	54.72	54.51	2.41	2.36	1.09	1.07	57.07	56.91	2.05	2.04	0.88	0.88
MocoST	55.77	55.59	2.42	2.37	1.07	1.05	57.99	57.78	2.07	2.06	0.87	0.87
CrossEntropy-H	100	55.76	55.71	2.48	2.42	1.10	1.07	58.50	58.32	2.08	2.07	0.86	0.86
PseudoLabel-H	56.18	55.98	2.47	2.41	1.08	1.06	58.80	58.79	2.09	2.08	0.86	0.86
ST-H	54.82	54.71	2.47	2.41	1.12	1.09	57.51	57.45	2.07	2.07	0.88	0.88
Moco-H	56.05	55.85	2.47	2.42	1.09	1.07	58.56	58.49	2.09	2.09	0.87	0.87
MocoST-H	55.74	55.49	2.47	2.41	1.10	1.07	58.32	58.17	2.08	2.08	0.87	0.87
CrossEntropy	45.47	45.60	2.70	2.61	1.47	1.42	50.39	50.29	2.00	1.99	0.99	0.99
Pseudo-Label	45.97	45.84	2.57	2.51	1.39	1.36	50.59	50.43	1.99	1.98	0.98	0.98
MoCo	47.94	47.75	2.55	2.50	1.33	1.31	51.72	51.61	2.01	2.00	0.97	0.97
MocoST	49.31	49.20	2.54	2.48	1.29	1.26	52.87	52.73	2.02	2.00	0.95	0.95
CrossEntropy-H	50	54.87	54.83	2.50	2.43	1.13	1.10	57.69	57.50	2.08	2.07	0.88	0.88
PseudoLabel-H	55.17	54.97	2.48	2.40	1.11	1.08	57.97	57.84	2.07	2.07	0.87	0.87
ST-H	52.42	52.27	2.50	2.43	1.19	1.16	55.74	55.65	2.04	2.05	0.90	0.91
Moco-H	54.76	54.58	2.48	2.42	1.12	1.10	57.65	57.45	2.07	2.08	0.88	0.88
MocoST-H	54.53	54.26	2.48	2.42	1.13	1.11	57.37	57.21	2.07	2.08	0.88	0.89
CrossEntropy	36.47	36.63	2.87	2.77	1.82	1.76	43.28	43.22	1.93	1.91	1.10	1.09
Pseudo-Label	39.13	39.20	2.72	2.66	1.66	1.61	44.96	44.91	1.95	1.93	1.07	1.06
MoCo	39.48	39.55	2.74	2.68	1.66	1.62	45.06	45.00	1.94	1.92	1.06	1.06
MocoST	41.22	41.12	2.72	2.63	1.60	1.55	46.37	46.30	1.96	1.95	1.05	1.05
CrossEntropy-H	25	53.26	53.18	2.51	2.44	1.17	1.14	56.43	56.21	2.06	2.05	0.90	0.90
PseudoLabel-H	53.30	53.22	2.51	2.43	1.17	1.14	56.57	56.44	2.06	2.05	0.89	0.89
ST-H	48.29	48.19	2.59	2.52	1.34	1.30	52.69	52.52	2.01	2.01	0.95	0.96
Moco-H	53.01	52.97	2.50	2.43	1.17	1.14	56.29	56.25	2.06	2.05	0.90	0.90
MocoST-H	53.21	53.02	2.49	2.44	1.17	1.14	56.34	56.12	2.05	2.06	0.90	0.90
CrossEntropy	24.79	25.17	3.12	2.98	2.34	2.23	33.33	33.32	1.82	1.81	1.22	1.20
Pseudo-Label	27.36	27.67	3.01	2.91	2.19	2.11	35.08	35.12	1.87	1.86	1.22	1.21
MoCo	27.11	26.90	3.01	2.95	2.19	2.16	34.95	34.93	1.83	1.82	1.19	1.18
MocoST	28.59	28.56	2.95	2.88	2.11	2.05	36.03	36.00	1.84	1.82	1.17	1.16
CrossEntropy-H	10	51.36	51.31	2.54	2.47	1.23	1.20	54.92	54.80	2.05	2.04	0.92	0.92
PseudoLabel-H	51.35	51.28	2.56	2.49	1.24	1.21	55.02	55.02	2.04	2.04	0.92	0.92
ST-H	40.61	40.65	2.77	2.66	1.64	1.58	46.67	46.60	1.96	1.94	1.04	1.03
Moco-H	51.23	51.27	2.57	2.50	1.25	1.22	55.13	55.03	2.06	2.06	0.92	0.93
MocoST-H	50.33	50.26	2.56	2.50	1.27	1.24	54.11	54.04	2.04	2.05	0.94	0.94
Table 3: Results of the proposed HiE approach on the semi-supervised setting on the iNaturalist-19 dataset. Best results on each metric are emphasized in bold.
4.2 Semi Supervised Learning
We additionally validate our approach on fine-grained classification in the semi-supervised setting. We assume that coarse annotations are available for all the samples; however, fine-grained annotations are available only for a few samples in each class. The studied setting differs from cases [13], which divide the fine-grained classes into two categories: base classes with abundant annotated samples and novel classes where only a few annotated samples are available. This framework [13] also limits the evaluation only to the novel classes. Our setting imposes firm limits on difficult-to-obtain fine-grained labels. In the extreme setting, we assume only ten annotated training samples are available for each fine-grained class. We also evaluate the performance across all the fine-grained classes together.
We train a cross-entropy baseline exclusively using the annotated fine-grained samples. We compare its performance with representative semi-supervised methods, including Pseudo-Label [37], Self-Training (ST) with distillation [42], Self-Supervised learning (MoCo) with distillation [14] and a combination of Self-Supervised learning and Self Training (MoCo+ST). We also include a variation of all these methods (Pseudo-Label-H, MoCo-H, ST-H, MoCoST-H) with hierarchical supervised loss as described in [2] with "genus" supervision. The hierarchical supervised loss is applied to the coarse predictions obtained by marginalizing the fine-grained class predictions. The proposed hierarchical ensemble is a complementary approach, and we compare the performance of the above methods with and without applying the HiE.
The results are illustrated in Table 3. As expected, the performance of the cross-entropy baseline steeply drops with the reduction in training data. The performance of semi-supervised methods (Pseudo-Label, ST, MoCo, MoCo-ST) improves over the cross-entropy baseline; however, they fail to retain the overall performance. The semi-supervised methods become more effective with the reduced number of labeled examples (providing better proportional gains over the cross-entropy baseline). Incorporating hierarchical supervised loss brings remarkable gains in top-1 accuracy across all experiments. For instance, in the lowest data regime (using only ten fine-grained annotated samples for training), hierarchical supervised loss brings twofold improvement over the cross-entropy baseline. The performance gains showcase the efficacy of utilizing coarse taxonomic labels in the studied setting.
Employing the proposed Hierarchical Ensemble brings consistent gains across all the methods on the three studied evaluation metrics. The combined approach of Moco-H/PseudoLabel-H with HiE
Model	Vanilla	CRM	HiE
Top 1 Acc.	Mistake Severity	Hier. Dist@1	Top 1 Acc.	Mistake Severity	Hier. Dist@1	Top 1 Acc.	Mistake Severity	Hier. Dist@1
Mobilenet_V3	45.86	2.89	1.56	45.83	2.79	1.51	46.66	2.69	1.44
ResNet18	63.63	2.39	0.87	63.65	2.31	0.84	64.65	2.15	0.76
ResNet50	69.48	2.23	0.68	69.49	2.19	0.67	70.34	2.03	0.60
ResNet101	70.86	2.14	0.62	70.87	2.11	0.61	71.66	1.96	0.56
EfficientNet-B0	67.69	2.25	0.73	67.68	2.21	0.71	68.75	2.01	0.63
DenseNet121	67.86	2.27	0.73	67.89	2.23	0.72	68.97	2.04	0.63
DeiT	64.71	2.38	0.84	64.74	2.32	0.82	65.73	2.14	0.73
ViT	66.19	2.36	0.80	66.27	2.29	0.77	67.56	2.06	0.67
SwinT	75.08	2.10	0.52	75.09	2.06	0.51	75.74	1.88	0.46
Table 4: Results of the proposed HiE approach with different architectures on species taxonomy of the iNaturalist-19 dataset.
Hier. Levels	Top-1 Acc.	Mistake Severity	Hier. Dist @ 1	Hier. Dist @ 5	Hier. Dist @ 20
{7}	63.63	2.39	0.87	1.97	3.25
{6,7}	64.65	2.15	0.76	1.33	2.19
{5 - 7}	64.72	2.09	0.74	1.26	2.00
{4 - 7}	64.48	2.12	0.75	1.26	1.91
{3 - 7}	64.47	2.12	0.75	1.26	1.89
{2 - 7}	64.46	2.13	0.76	1.27	1.89
{1 - 7}	64.44	2.14	0.76	1.27	1.89
Table 5: Results of the proposed HiE approach on fine-grained level (species) of iNaturalist-19 dataset when incorporating multiple hierarchical levels. Best results are highlighted in bold.
achieves a new state of the art in all settings. Using ten annotated examples for each fine-grained class, Moco-H with HiE achieves a top-1 accuracy of 55.13, which is higher than the performance of cross entropy baselines using a hundred annotated samples per fine-grained class. The performance gains of HiE are more evident on the avg. mistake severity and Hier dist@1 metrics, bringing profound gains over just using the hierarchical supervised loss. For instance, in 100 sample experiments, the Hier dist@1 drops from 1.17 to 1.10 (6% improvement) by applying hierarchical supervised loss. In contrast, applying HiE brings 24% improvement, reducing Hier dist@1 to 0.89.
Finally, we observe that in Table 1, a cross-entropy baseline with Resnet-18 backbone trained on the entirety of the iNaturalist-19 dataset, i.e., 187385 examples with fine-grained labels achieves a top-1 accuracy of  $63.56\%$ . In the semi-supervised setting using coarse-grained labels for entire data and just using 10 examples with fine-grained annotations for each class (10100 training samples), Moco-H with HiE is able to recover the significant portion of the underlying performance (achieving top-1 accuracy of 55.13), reducing the requirement of expert annotations to only  $0.5\%$  of the total number of samples.
4.3 Performance across different architectures
We also showcase the generalizability of the proposed HiE approach across different vision architectures in Table 4. We utilize commonly used vision architectures like MobileNet V3 [43], Resnet variants [44], EfficientNet [45], DenseNet121 [46], and tiny variants of DeiT [47], ViT [48] and SwinT [49]. All networks are trained with the same hyper-parameter setting. We compare the proposed HiE against the vanilla and CRM-based post-hoc approach. HiE outperforms other approaches by significant margins across different architectures and on all metrics. Furthermore, similar performance gains are observed irrespective of model capacity, avg. gain of  $0.9\%$  in top-1 accuracy across resnet variants.
4.4 Incorporating multiple hierarchical levels
For all the above experiments, we only use two levels of hierarchies. As an additional ablation study, we validate the effectiveness of HiE in the presence of multiple hierarchical levels in Table 5. Specifically, we train a separate classifier for each hierarchical level and cascade the predictions top-down. We observe that incorporating the hierarchical information from two levels (levels 5, 6) above the leaf level gives the best performance; the gains taper off gradually as we incorporate information from higher levels. The reason for this dip in performance is that the higher levels do not have much additional information as they cover too many classes (e.g., on iNaturalist: kingdom
(level 1), phylum (level 2), and class (level 3) have 3, 4 and 9 classes respectively) which does not help much in improving the fine-grained predictions further.
The main limitation of HiE is the requirement of training an additional network for the coarse level. However, existing approaches employ a full hierarchy with a separate classification head for each hierarchical level and additional losses to enforce the hierarchical structure, making their training process complex and difficult to converge. Instead, our approach trained with vanilla cross-entropy loss works with partial hierarchy and trade-offs the extra compute for adaptability, reproducibility and simplicity in training. A future avenue for exploration would be to learn disentangled coarse and fine-grained features imitating HiE in a unified architecture. Another limitation is the assumption regarding the availability of underlying label hierarchy. To overcome this limitation, we can exploit large-language models [50] to obtain label hierarchy using raw class labels.
We proposed using Hierarchical Ensembles (HiE) of independently trained networks over coarse and fine-grained levels of the label hierarchy. In terms of mistake severity, our proposed post-hoc correction consistently outperforms state-of-the-art methods in deep hierarchy-aware image classification by large margins in terms of decrease in avg. mistake severity and hierarchical distance@  $k$ , while simultaneously improving the top-1 accuracy. In the semi-supervised paradigm, we show that HiE delivers consistent performance gains when used in conjunction with off-the-shelf semi-supervised learning algorithms. We show that, comparable performance to a fully supervised baseline can be attained, even using merely 10 annotations for each fine-grained class on a large fine-grained image classification dataset encompassing 1010 classes.
[1] Jong-Chyi Su, Zezhou Cheng, and Subhransu Maji. A realistic evaluation of semi-supervised learning for fine-grained classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12966-12975, 2021.
[2] Jong-Chyi Su and Subhransu Maji. Semi-supervised learning with taxonomic labels. arXiv preprint arXiv:2111.11595, 2021.
[3] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41, 1995.
[4] Jia Deng, Alexander C Berg, Kai Li, and Li Fei-Fei. What does classifying more than 10,000 image categories tell us? In European conference on computer vision, pages 71–84. Springer, 2010.
[5] Björn Barz and Joachim Denzler. Hierarchy-based image embeddings for semantic image retrieval. In WACV, 2019.
[6] Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A Lord. Making better mistakes: Leveraging class hierarchies with deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12506-12515, 2020.
[7] Shyamgopal Karthik, Ameya Prabhu, Puneet K Dokania, and Vineet Gandhi. No cost likelihood manipulation at test time for making better mistakes in deep networks. *ICLR*, 2021.
[8] Ashima Garg, Depanshu Sani, and Saket Anand. Learning hierarchy aware features for reducing mistake severity. In European Conference on Computer Vision, pages 252-267. Springer, 2022.
[9] Pedro Domingos. Metacost: A general method for making classifiers cost-sensitive. In KDD, 1999.
[46] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[47] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[48] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[49] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[50] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.