Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification
Authors: Kanishk Jain
Shyamgopal Karthik
Vineet Gandhi
Venue: NeurIPS 2023 poster (2023)
Keywords: Hierarchical Classification
Fine-grained Classification
Ensembles
Mistake Severity
TL;DR: We leverage label hierarchy to reduce mistake severity of neural networks for the task of fine-grained classification.

Abstract: We investigate the problem of reducing mistake severity for fine-grained classification. Fine-grained classification can be challenging, mainly due to the requirement of knowledge or domain expertise for accurate annotation. However, humans are particularly adept at performing coarse classification as it requires relatively low levels of expertise. To this end, we present a novel approach for Post-Hoc Correction called Hierarchical Ensembles (HiE) that utilizes label hierarchy to improve the performance of fine-grained classification at test-time using the coarse-grained predictions. By only requiring the parents of leaf nodes, our method significantly reduces avg. mistake severity while improving top-1 accuracy on the iNaturalist-19 and tieredImageNet-H datasets, achieving a new state-of-the-art on both benchmarks. We also investigate the efficacy of our approach in the semi-supervised setting. Our approach brings notable gains in top-1 accuracy while significantly decreasing the severity of mistakes as training data decreases for the fine-grained classes. The simplicity and post-hoc nature of HiE renders it practical to be used with any off-the-shelf trained model to improve its predictions further.

Test-Time Amendment with a Coarse Classifier for FraieGr Classification
Kanishk Jain', Shyamgopal Karthik?, Vineet Gandhi' III Hyderabad Uniiersity of Tübingen
Abstract
We investigate the problem of reducing mistake severity for fine-grained classifica- tion. Fine-grained classification can be challenging, mainly due to the requirement domain expertise for accurate annotation. However, humans are particularly it requires relatively low levels of expertise. To this end, we present a novel approach for Post-Hoc Correction called Hierarchical Ensembles (HiF) that utilizes label hierarchy to improve the performance of fine-grained classification at test-time using the coarse- -rriindd predictions. By only requiring the parents of leaf nodes, method significantly reduces avg. mistake severity while improving top-1 accuracy on the iNaturalist- 19 and tieredImageNet-Zl datasets, achieving a new state-of-the-ee on both bench- marks. We also investigate the efficacy of our approach in the semi-supervised setting. Our approach brings notable gains in top-/ accuracy while significantly decreasing the severity of mistakes as training data decreases for the fine fine-graied classes. The simplicity and post-hoc nature HiE renders it practical to used with any off-the-shelf trained model to improve its predictions further. The code is available at: https: htt.a//ooo/oeeee/////eeerrr caa.errrrrrrcaaaaaaeeeeelleeee
adept performing coarse classification
Introduction
Over the past decade, large-scale datasets have been instrumental in driving the rapid progress of computer vision/image recognition. However, in settings that require experts to annotate samples, collecting large amounts of labeled data can be prohibitively expensive. Shne-Granned Visual Classifi- cation (FGVC) is one such example, where one would need a domain expert to be able to identify the category for particular sample. However, it can be cheaper obtain coarse labels for the same samples these settings. For instance, in Figure while it is evident to spot and identify a butterfly or Ornithologists categorize them Junonia Genoveva
bird, one would need Lepidopterist or Water Ouzel.
Therefore, one popular research direction in recent times has been to utilize the availability of larger set of images with coarse labels to improve the performance (i.e top- accuracies) of the neural network models on various fin-grained classification benchmarks 2 Using coarse labels in classification setting introduces the concept label hierarchy. where all the labels a dataset connected through some taxonomy. This taxonomy could either be defined from the biological taxonomy, as with many species recognition datasets or could be derived from language-hased ontologies such WordNet 3 Another mainstream usage of label hierarchies in recent times is to use them to reduce the severity of mistakes committed by various classification models 5 For instance, mistaking car for a bus is better mistake than mistaking car for lamppost. This originated from the extensive work on cost-sensitive classification [0]. However, defining pairwise costs can be a tedious process and scales quadratically with the number of classes. Therefore, the recent works have all focused on using the label hierarchy and defining costs automatically based on graph distances (e.g. using the height of the least common ancestor between the predicted and the ground truth class as proxy for cost). The research in this direction aims to invent methods that at
6 7 g
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Coarse Label
Bird
Butterfiy
Image
Fine Label
Linnet
Water Ouzel
Indigo Bunting
Peacock Pansy
Common Buckeye
Junonia Genoveva
While it is straightforward identify the images based on their coarse labels, it is challenging to distinguish them based on fine- fine-grained labels without domain expertise.
Figure 1:
least retain the accuracies of the backbone baseline models while reducing the severity of mistakes committed by it.
While these two research directions might seem entirely different, they work with the same inputs (i.e., images, labels, and the label hierarchy). Therefore, one would ideally want method that can use additional coarsely labelled samples to improve the model's accuracy and utilise the label hierarchy reduce the severity of the mistakes. However, these two goals have been tackled independently. Post-hoc correction methods 4 77. learning hierarchy-awaid features [6 g M as well as structured embedding spaces [ 12 have been proposed to reduce the severity of mistakes. Coarse labels have been used for self- www.-rraiinngssuuubeeee lselllllabellig [13] and with an additional objective regularise/enhance the training (2) in the supervupe learning frameworks.
In this work, we propose a simple method that is able to bring significant improvements to both of these problems. Our fundamental insight is that models trained for different granularities focus different aspects. This is also corroborated by recent work [T] 8 which shows that when training model jointly, coarse-level label prediction exacerbates fine-grained feature learning. Therefore, we train two separate models for the coarse and fine- ane-ggained labels. At inference, we employ modified decision rule, which computes the argmax over the normalized score of coarse and fine grained predictions, We refer to the proposed strategy as a Hierarchical Ensemble (HiE). We find that employing HiE not only reduces the severity of mistakes, but is able improve top-/ accuracies, which are far more pronounced in the semi- semi-supervised setting. More formally, we make the following contributions:
1. We introduce a simple post-hoc mechanism called Hierarchical Ensembles for incorporating cues from multiple models trained for different granularities of hierarchy tree.
2. We demonstrate that HiE brings considerable reductions in mistake severity while improving the top-1 accurucy. HiE significantly outperforms two competent baselines and six other methods from the prior art and achieves state-of-the-art performance on both iNaturalist- 19 and tieredlmageNNt-- -H benchmarks.
3. We illustrate that HiE brings consistent gains when plugged with existing semi-supervine methods exploiting additional coarse labeled data to improve the fine-grained classification. On iNaturalist- /9 dataset, HiE combined with Moco [14 and hierarchical loss 22, recovers 86% of the underlying performance while using only 0.5% of the fine-graing mne-ggained annotations.
Related Work
Learning from Taxonomic Labels: There has been long history of utilizing label hierarchies for better classification performance. Sila and Freitas [15] perform comprehensive survey of hierarchical classificatton methods applied to various tasks in different application domains. Broadly, the methods that train hierarchy-awar features be categorized into three categories Label-Embedding methods, b) Hierarchical architectures, and c) Hierarchical loss functions.
The label-embedding methods project the categories into a semantic embedding space 12 16 instead
of viewing them as one-hot encodings and are widely used, especially in zero-shot learning The semantic embedding space can be derived either from idd-information such as class attributes [16).

---

taxonomies [6], or from natural language [2]. These can be further enhanced by learning these
embeddings on hypersphere [5] hyperbolic spaces 18 19 20
On the other hand, hierarchical loss functions try to modify the training objective to incorporate the
label hierarchy [21 6 22 23 The most common approach here is introduce loss at every level
in the hierarchy 22 6 8J. The challenge in this direction has been to ensure that the coarse-grained task does not hamper the accuracy of the fine-grained classifier
There have also been attempts to modify the architecture based on the label hierarchy. For instance, popular ideas revolve around having branches at different layers of the model [24], predicting the conditional probabilities at each node in the hierarchy [25] 26], or partitioning the feature space using the levels in the hierarchies
Most closely related to our work are the works that attempt to incorporate the label hierarchy information post-hoc during inference. Here, traditional Conditional Risk Minimization (CRM) (27 was applied in this setting by 4 and recently revisited by 77]. Our proposed approach is also applied during inference; however, it is orthogonal to CRM, and we demonstrate that CRM can be used to improve our results further. Concurrent to our work, (28) proposed a similar approach to utilize subclasses to improve superclass recognition performance in post-hoc manner with vision-language models [29].
Fine-Graines Visual Classification: Contrary to the popular image recognition benchmarks, fine- grained visual classification focuses on recognizing the differences between similar-looking cate- gories (30 31 32). Here, methods often focus on learning discriminative features that use local information since most categories share a similar global structure [33 34 35 However, the most related to our work are the ones that use class taxonomies to improve fine- fine-grained classification performance. This is most frequently done the semi-supervised setting []. where one has access to additional set of weakly labeled samples. Here, approaches have explored incorporating hierarchical loss on the coarse labels (2] or using the coarse labels to filter pseudo-labels [3 in
addition to the standard aseudo-labeling (37 338. and contrastive learning [14]. Our proposed approach is complementary to these works and can be applied to all of these works to bring additional improvements.
supervisd learning techniques of consistency regularization [36].
3 Methodology
3.1 Problem Formulation
We consider the fine- fine-grained classification task with a label hierarchy of L levels defined over the class labels. The leaf nodes of H. correspond to fine-grained classes at level L, and the class labels level < L are considered coarse-graine labels. The goal is use the information from coarse annotations to improve the performance on fine labels. Formally, given dataset 1.2..... consisting of N training images and their respective ground truth labels at level 1, where label v y = 21, 2, ..., Ni), and N; corresponds the number of classes level 1, the task is train classifier J X p()'4, on fine-grained class labels by making of the information available through coarse- graiiee labels yKL
X' ((.,, v/)
3.2 Proposed Method
Suppose we have two classifiers, fb and fb- trained on hierarchy levels I., I- with @ and ø being their parameters, respectively, then for sample D, have, ΦL fx) and ΦL-1 f/ (x) as logits of two classifiers for the input We further define:
Q [91, 42, .... gNz] softmax(01)
R= ==(ri, 12,, "N,....N---1 -N = (oftmax(01-1)
3

---

Root
9. pprrent
purent
2j== parent
Vahice
Fooer
0.65 0.35 0.23
5: 0.43
0.85
0.35
0.08 0.05
5,- 0.09
Bus
Truck
Rose
Orchid
0.35 0.19
= 0.35
0.35 0.07
94 0.13
9L 0.35
4:- +
0.08
0.5S
4 0.02
Figure 2: Consider four class hierarchy and corresponding fine-grained predictions (q:) and coarse level predictions (rz) obtained using independent models. Our work aims to improve the fine grained classification leveraging the predictions the coarse level. In the given example, the class prediction changes from 'rose' to 'bus' after the post-hoc correction.
We reweigh the fine- fine-grained predictions Q, given the softmax probabilities obtained from the coarse-grained classifier:
P(ilar; O) (x;; Φ)
0,0,,0,0,70) = ΣS P) (parent xz; g)
where Eparent H(ii)
(1)
The denominator is the normalization term, ensuring Em P(ip,, 1. In the studied setting, P(ilæ, H.) simplifies to:
T Tiparant
Pil2, H)
Nu E/ qj Typarens
The main modification we make in the classification strategy at the fine grained level is in the decision rule, which selects the class that maximizes the normalized score:
argmax P(ilæ, H)
(3)
In Figure 2 illustrate a four-class example, comparing standard cross-entropy predictions with those from our proposed method. We would like to point out that the proposed technique can be interpreted as a Hierarchical Ensemble. Existing literature in neural networks-bar based ensemble learning has often focused on classifiers trained at single granularity [39 and has largely overlooked ensembles with hierarchical context. In this work, we highlight the practical benefits of hierarchical ensembles for improving the fine-graines classification task by choosing the simplest possible method of multiplying the predictions at different levels of hierarchy.
We show that if we make a correct prediction at the coarse level, the proposed Hierarchical Ensemble (HiE) guaranteed improve the downstream predictions the fine-grained classification task.
Theorem 3.1. Assuming, Q [q Q2 ... ANz) and R [ri, T2, ... NNN-il are the predictions obtained at the fine and coarse grained labels for given input such that ΣN qi = and NN--' r, =1 For the ground truth labels g and Iparent at the fine grained and the coarse grained levels respectively:
Now assuming that the coarse label is correctly predicted by the coarse prediction network i.e. argmax R) -pparens we wish to prove that:
2g Tgparent
2 9g-
Σj Gj ppaaont
No
Proof. The denominator iterates over the fine grained predictions and multiplies them with their parent's prediction scores. This is equivalent to iterating over the coarse label predictions and multiplying with the sum of prediction scores for all its children. By rewriting the denominator, we obtain:

---

Ny Σ
N1-1
Σ
Σ qi iEDehiid
T Iparent
jml
j=1
Assuming, Eieronia 91 2z
NL-1 qi
Σ j=1
Tj 13.25 25 =R
/=1
SEjehiid
Invoking Holder's inequality, using a = 0o and b 1, (1/a + 1/b = 1), we obtain:
R |
Since, ||||/s = max(R)- = T Aparent and || = ΣN,, qi 1. can say:
Rsh Z < R |||| T Oparent
Given the above equation, we can conclude that:
gparent
2 1.
Ej=1 95 pparannt
When correct predictions are made at both the coarse and fine-grained levels, the method will help increase the confidence of the prediction. A more interesting case occurs when the coarse-level prediction is correct; however, the fine-grained classifier makes a wrong prediction. The hierarchical ensemble can shift the prediction to the correct sub-tree, reducing the mistake severity and potentially improving the top-1 accuracy.
4 Experiments & Results
Like prior works 6 we evaluate our approach on the iNaturalist-19 [40 and tieredInageNer--l 4l datasets. The iNaturalist- 19 dataset is species classification dataset containing plants, animals, fungi, etc., and has an 8-level hierarchy. The tieredlmageNet-H// dataset contains classes from a wide range of categories, including musical instruments, animal breeds, tools, and plant species. It has 13-level hierarchy derived from the WordNet hierarchy. However, rather than using the full hierarchy of categories, for method, we only use the Icaf classes and their parent categories during training and inference. The iNaturalist-I5 19 dataset includes 1010 fine- aneegrraied "species" classes and 72 coarse-grainet classes for the "genus" taxonomy. The tieredImageNet-H dataset has 608 leaf classes and 201 parent classes. We train classifiers for each level independently, using the same hyperparameter settings.
In addition, we also evaluate the performance of our approach in sumi-supervised setting. Specif- ically, we want to test the performance on fine-grained classification tasks using a large number of coarsely annotated examples and a limited number of finely annotated examples. We use the iNaturalist-19 19 dataset for this setting; the coarse labels for all the images are used, while for fine- grained labels, we sample a limited number of images per class label during training. We used the "species" and "genus" taxonomies for the fine and coarse level labels and performed experiments for cases when the number of images per fine label 100, 50, 25, and 10. Similar to the supervised setting, we train separate classifiers coarse and fine labels.
To align with prior efforts 8 all of our experiments are performed using the ResNet-18 18 backbone. We randomly crop portion of images and resize them to 224 x 224 resolution for both datasets. For the iNaturalist-19 dataset, we initialize the ResNet-15 18 weights with a pre-trained ImageNet model and train the classifiers using a customized SGD optimizer for 100 epochs, with different learning rates for the backbone network and the fully connected layer (0.01 and 0. respectively). For the tieredlmageNet-HH dataset, train the ResNet- 18 from scratch (since it is derived from the ImageNet
5

---

Top-1 Emvor()
Mistakes severity(L)
Hier dist@((()))
Hier diat@ S(L)
Hier 2
Method
Without
Cmos-Fatrrpps
36.44 0.061
2.39 4: 0.007
0.87 :: 0.004
1.97 + 0.002
3.25 :: 0.002
Cross Chus-Entrupy-
37.63 1: 0.054
2.37 1 0.005
0.89 1 0.009
1.86 1: 0.001
2.96 1: 0.001
HiE Self
36.36 0.047
2.36 1: 0.014
0.86 1. 0.005
1.44 1: 0.003
2.31 1: 0.004
+: 0.005
YOLO-v2 [5]
44.37 = 0.106
2.42 = 0.003
1.08 + 0.004
1,90 = 0.003
2.87 = 0.010
HXT o=0.1
41.48 0.204
2.41 + 0.009
1.00 0.006
1.77 + 0.011
2.69 + 0.021
HXE om0.6
2.24 - 0.006
1.02 : 0.003
0.005
- 0.005
Soft labels 8
30
41.67 0.134
2.32 1 0.010
0.97 1 0.006
1.50
0.006
2.23 1 0.005
Soft-labels 3
6
74.70 0.212
1,82 + 0.005
1.36 - 0.004
1.49 + 0.003
1.96 0.004
Chang et |1]
37.23 0.175
= 0.006
- 0.004
0.005
3.02 = 0.008
HAF X
36.40 0.092
0.012
0.83 0.002
0.003
HiF (Ours)
35.33 + 0.037
2.15 + 0.003
0.76 + 0.001
1.33 + 0.001
2.19 = 0.003
With CRM
Cross-Eatrops
+ 0.001
+ 0.002
0.001
1.86 + 0.002
Choss-Eatropye
31.70 0.038
2.33 + 0.011
0.88 + 0.003
1.33 + 0.002
1.87 + 0.004
HiE-Saff
36.45 0.057
2.34 + 0.019
0.85 + 0.003
1.32 0.003
1,86 + 0.008
Barz.
+
YOLO-v2
45.17 0.046
2.43 1 0.011
1.0 1: 0.001
1.50 0.001
1.99 + 0,002
HXE a=0.1
41.47 = 0.220
2.38 + 0.011
0.99 + 0.008
1.41 = 0.006
1.33 + 0.005
HXE a=0.6
45.60 0.017
2.21 + 0.008
1.01 + 0.003
1.40 + 0.004
1.40 0.044
Soft-labels
0.126
0.009
0.007
0.005
0.005
Soft-lbels @
77.34 0.262
2.06
0.012
1.60 1 0.007
1.72 +: 0.008
2.14 + 0.007
Chang et al. 11
37.31 0.145
2.24 1 0.008
0.84 +: 0.002
1,30 + 0.002
1.84 + 0.002
HAP [
HiE (Ours)
35.42 0.025
2.14 + 0.015
0.75 + 0.005
1.23 0.005
1.79 + 0.005
Table 1: Results comparing top-/ error( () and hierarchical metrics for iNaturalist-19 Results in the Top block are reported without using CRM [ technique and Bottom block are reported using CRM. Methods highlighted with aqua green the best performing methods in top-1 error (%). We use dark green for the top performers and light green for the runners-up in each metric.
dataset) and the AMSGrad variant of the Adam optimizer with learning rate of 55 for 120 epochs. We conduct five runs for each experiment in tables [ and and2aa and report the mean and standard deviations.
Evaluation Metrics: Like previous works 66 77. we validate our approach using the top-1 accuracy, avg. mistake severity and hierarchical distance @ k metrics. Arg. mistake severity metric computes the average height of the lowest common ancestor (LCA) between the predicted class and the ground truth class for the incorrectly classified samples. Since this metric considers only the incorrect predictions, is possible to reduce the mistake severity by making more low cost mistakes (7]. as a consequence of which the top-/ accuracy decreases. To overcome this issue, hierarchical distance@k k is used, which measures the average height of top-k predictions with the ground truth class for all samples. For 1, this metric captures the quality of ranking imposed by the network in the label space.
4.1 Making better mistakes
In this section, demonstrate that the proposed hierarchical ensembles significantly reduce the severity of mistakes compared to the current state-of-the-art methods. We compare our method with [5). YOLO-v2 255; both approaches proposed by (6) with varying parameters soft-labels and HXE. We also compare against the method proposed by [1 for multi-granular classification and HAF g imposing consistency on prediction heads across label hierarchy. We contrast the performance the methods mentioned above to the cross-entropy baseline.
We also propose two custom baselines called Cross-Entropy-H and HiE-Self. Cross-Entropy-HI obtains coarse class predictions by marginalizing the prediction of its fine-grained children classes. then employs additional hierarchical loss term [2] over the obtained coarse predictions. HiE-Self applies a hierarchical ensemble using the coarse predictions obtained post marginalization over fine-grained classes (instead of training separate network for the coarse level).
We also experiment with the CRM technique proposed by 77] Since CRM is post-hoc approach that reweighs the probability distribution of samples obtained from any trained model, it can be applied to

---

pre-trained models of all the approaches. Therefore, in each of the Tables 12 we group the results to report evaluation metries with and without using CRM at test time.
Top-1 emoot()
Mistakes severity())
Hier dist dist
Hier diste dist@
Hier dist 20(U)
Method
Without CRM
Eoss-Enlropy
0.010
: 0.006
0.003
Cross-Batropy-H
32.87 0.042
7.13 = 0.031
2.35 + 0.003
5.70 = 0.009
7.14 = 0.012
HiF-Self
30.78 0.054
7.05 + 0.028
2.19 0.004
5.35 1: 0.009
6.92 + 0.007
Barz. Denzler
39.73 0,240
= 0.019
2:70 0.022
+ 0.271
0.005
YOLO-v2 25
33.37 0.082
7.02 1 0.004
2.34 134.1.0011 0.016
5.85 1. 0.011
7.43 1: 0.016
HXE a=0.1
30.72 0.036
7,00 + 0.019
2.15
5,62 + 0,008
7.08 + 0.015
HXE a=0.6
34.50 0.007
6.73 = 0.014
2.32
0.001
6.78 0.003
Soft-labels
0.009
0.13
5.66
7.14 0.008
Soft-labels 8
P
38.99 0,105
6.60 + 0.024
157
5.13 0.002
6.21 0.000
Chang et al.
11
33.46 0.026
6.99 1 0.010
2.34 0.006
5.75 1 0.005
7.34 1 0.010
HAF g
0.024
0.00
0.009
HRE (Ours)
29.81 1: 0.081
6.95 + 0.013
2,07 0.014
5.30 4 0.001
6.86 + 0.001
With CRM
Cross Ents--y
1:
2.14
Chos--Ettroy-11
32.93 0.029
7.05 1 0.006
2.32 1: 0.004
5.01 1: 0,008
6.14 1: 0.003
HiE-Self
30.79 + 0.031
7.04 + 0.008
2.17 + 0.004
4.98 + 0.005
6.14 = 0.002
Barz Denzler
83.55 0.000
11.94 0.000
11.92 0.000
11.91 0,000
11.91 + 0.000
YOLO-v2
33.98
+ 0.011
+ 0.012
5.05 0.001
0.001
HXF 1=0.1
30.80 0.079
6.95 + 0.021
2.14 114:0.000 0.005
4.94 1: 0.003
6.11 0.002
HXE a=0.6
34.68 0.003
6.69 1 0.007
2.32 1: 0.001
4.99 1 0,005
6.13 0.003
labels
0.007
Soft-lablls 8
82.72 0.079
7.54 + 0.001
6.24 0.005
6.94 = 0.005
7.25 + 0.002
Chang et al. (II)
33.73 0.033
6.93 0.015
2.34 0.002
5.02 + 0.007
6.15 - 0.001
HAF [S]
HiF (Ours)
29 89 0.082
6.93 0.013
2.07 0.013
4.3 3 0.001
6.11 0.001
Table 2: Results comparing top-1 error(%) and hierarchical metrics the test set of tieredImageNee- H. The Top block reports results without using CRM 7 and the Bottom block are reported using CRM. Methods highlighted with aqua green the best performing methods in top-1 error (%). We use dark green for the top performers and light green for the runners-up each metric.
Apart from HAF, all the other methods from the prior art trade-off the top-/ accuracy with the mistake severity (Tables [ The results contrast with the problem's main objective, i.e. improve the hierarchical metrics by maintaining or improving the top-/ error. In some of the instances (e.g. HXE = 0.6, Soft-labels 3 = 30), the avg. mistake severity reduced by making additional low-severity mistakes (indicated by increase in top-l error and Hier dist@ 1). The multi-head approach by
[1 brings minor gains, and HAF further improves upon it by introducing consistency loss among different head predictions.
The proposed minimalist approach Hierarchical Ensembles (HiE) significantly outperforms all the methods; on both datasets, with and without CRM. On the three metrics of top-/ error, avg. mistake severity, and Hier disr@ 1, the performance achieved by HiE without CRM betters all other methods post-CRM. The gains achieved by HiE on Hier dist @ 5 and Hier dist @ 20 metrics are more pronounced, for instance, improving over HAF without CRM on iNaturalist- 19 dataset by 17% and 14% respectively. These two ranking metrics compare the ordering of the classes provided by each of these classifiers. The results illustrate that HiE is able to reliably align the predictions with the hierarchy. The results are interesting given that HiE only employs the bottom two levels of the hierarchy, in contrast to other methods which utilize the entire hierarchy tree.
Compared with the baselines, on the iNaturalist- 19 dataset, the Cross Entropy-H brings minor gains on Hier d and Hier disr@ metrics; however, the top- error increases by over a percent. On tieredlmageNer-ff Cross-Entroon -H fails to give any improvements. In contrast, the HiE-Self baseline able to retain the original top-/ performance while bringing noticeable gains on Hier dist@k & metrics. It illustrates that hierarchical ensembles are helpful even when applied on a single network. However, training separate networks for coarse and fine- fine-grained levels allows us to learn complementary features which are then combined using HiE approach.
7

---

Without HEE
with HIE
Marhooh
Alngel shol
Top: Ace
Arg.
Mistake Severit
Hienachicca Distanett)
Top-1 Aoe
Avg. Mistake Sevvcity
Heranchcc Distanelll1
CRM
CRM
CCRM
CRM
CRM
vRM
CR3
CCRM
CRM
Crouttomayy
$3.50
53.95
2.55
2,45
1.17
1.14
50.95
50.43
2.07
2.05
1189
1189
Picudl Label
$1.25
51,00
2.52
2,45
1.23
1.20
54.78
54.61
2.04
2,83
0.91
0.92
Malla
3:36
1.01
57007
2.09
2.04
MocoST
237
1.05
2.05
56.18
55.06
2.43
2,41
1.08
1706
58.00
$4.79
2.00
2.0%
0.56
0.56
ST.
54.82
54.71
2.47
2
1.12
1,00
57.51
57.45
2.07
2.07
0285
0.85
MixnoH
55.05
55.85
2.47
2,42
1.09
1.07
58.56
$5.40
2.00
2.00
0.87
0.87
MocoSTial
55.74
55.49
2.47
2.41
1.30
1.07
55.32
55.17
2.144
2.05
02:87
0,57
Pseudo
45,97
2.51
MaDa
47.04
47-75
2.53
3,50
1.35
1.31
51.73
2.00
0.97
0.97
MasoST
49.31
40:30
3.54
2,45
1.29
1.26
50.87
2.00
2.00
0.05
0.95
Crestattggy it
30
54.87
54.85
2
1.83
1.10
57.09
1255
1235
Pacubal.abed-ff
2,40
1.06
Mico-H
1.10
Ma
54.53
$4.26
2.48
2,42
1,05
1.11
57.37
57.21
2.00
2.05
0.85
0.89
Choofrtrpyy
36.47
35.63
2.87
2.73
1.82
1.76
63.25
43.22
1.03
1.91
1.10
1,0
Pecul--Laael
2,66
1.66
1.61
1.05
MoseST
1.55
Creclintney-ll
24
53.36
$1:18
7.44
1.14
2.05
10:00
0.90
PicsboLabed Ht
53.30
53.22
2.51
2,43
1.17
1.14
$6.57
55.44
2.06
2.05
0.89
0.89
STH
48.29
48,10
2.59
2.53
1.34
1.30
53.69
51.52
2.01
2.01
0,95
0.95
Mirnoll
2,41
1.14
2.05
M
53.21
53.00
2.49
2.44
1.17
1.14
56.34
55.12
2.05
0,90
0,90
Croolvnroyy
34.39
25:17
511
7.9%
7.34
3.35
51.15
51.12
1.87
1.81
1:23
1.20
Psoudo Label
27.36
23.67
3.01
2.11
2,19
2.13
35.08
35.12
1.5
1.85
1.22
1.21
Malo
21:11
26-90
500
2.05
2.89
2.16
$4.55
$4.55
188
1.83
1.19
1.16
MocoST
28.59
28.56
295
2.85
2.01
2,05
36.03
36.00
188
188
1.17
1.15
Crostantgg
1.311
ST-H
8061
2.6
1.5%
45.00
196
1104
1.03
Mocn-3
51.23
$1.27
2.57
2.50
1.25
1.22
55.03
55.08
2.06
2.05
0.92
0.93
Moo0ST-11
50.35
50.76
2.56
2,50
1.27
1.24
54.11
54.08
2.04
2.05
0.94
0.94
Table 3: Results of the proposed HiE approach on the semi-supervised setting on the iNaturalist- 19
dataset. Best results on each metric are emphasized in bold.
4.2 Semi Supervised Learning
We additionally validate our approach on fine-grained classification in the semi-supervised setting. We assume that coarse annotations are available for all the samples; however, ane--grined annotations available only for a few samples in each class. The studied setting differs from cases [13], which divide the fine-ge greiger classes into two categories: base classes with abundant annotated samples and novel classes where only few annotated samples are available. This framework [13] also limits the evaluation only to the novel classes. Our setting imposes firm limits on lufiicult-to-obtiil fine-grained labels. In the extreme setting, assume only ten annotated training samples are available for each fine-grained class. We also evaluate the performance across all the fine- one-grained classes together.
We train cross-entropy baseline exclusively using the annotated fine-grained samples. We compare its performance with representative semi-supervised methods, including Pseudo-Label (37), Self-
Training (ST) with distillation [2]. Superviue learning (MoCo) with distillation [4 and
combination of Superviuee learning and Self Training (MoCo+ST). We also include a variation of all these methods (Pseudo-Labcl-H. MoCo-H, ST-H, MoCoST-H) with hierarchical supervised loss as described in [2] with "genus" supervision. The hierarchical supervised loss is applied to the coarse predictions obtained by marginalizing the fine-grained class predictions. The proposed hierarchical ensemble is a complementary approach, and we compare the performance of the above methods with and without applying the HiE.
The results are illustrated in Table 3 As expected, the performance of the cross-entropy baseline steeply drops with the reduction in training data. The performance of semi-supervised methods (Pscudo-Label. ST, MoCo. MoCo-ST) improves over the cross-entropy bascline; however, they fail to retain the overall performance. The semi-supervised methods become more effective with the reduced number of labeled examples (providing better proportional gains over the cross-entropy baseline). Incorporating hierarchical supervised loss brings remarkable gains in top-1 accuracy across all experiments. For instance, in the lowest data regime (using only ten fine mme-graine annotated samples for training), hierarchical supervised loss brings twofold improvement over the cross-entropy baseline. The performance gains showcase the efficacy of utilizing coarse taxonomic labels in the studied setting.
Employing the proposed Hierarchical Ensemble brings consistent gains across all the methods on the three studied evaluation metrics. The combined approach of
with HiE
8

---

Model
Vanilla
CRM
HIE
Top
Severty
Ace.
Seveniy
Diate
Top
Severity
Hier. Divor
Mobilent V3
45,86
2,80
1.56
45.83
2,79
1.51
46,66
2,69
1.44
ResNet18
63.83
2,39
63.65
2.31
64.65
2.15
0.76
ResNe:50
1.23
69.09
2.19
0.67
70.34
0.00
ResNe1101
70.86
2.14
0.62
70.87
2.11
0.1
71.66
1,96
0.56
DefcientNet
DenscNer121
67.86
2.27
0.75
67.89
2.25
0.72
68.97
2004
0163
DelT
66.73
VIT 19.19
2,36
0.80
66.27
2.29
0.77
67.56
2,06
0.67
SwinT
75.08
2,10
0.52
73.09
2,06
0.51
73.74
1.88
0.46
Table 4: Results of the proposed HiE approach with different architectures on species taxonomy of the iNaturalist- 19 dataset.
Hier. Levels
Top-1 1 Acc.
Mistake Severity
Hier. Dist @
Hier. Dist @
Hier. Dist 20
[7]
63,63
2.39
0.87
1.97
3.25
[6, 7
64.65
2.15
0.76
1.33
2.19
[5- 7|
64.72
2,00
[4-
64,48
2.12
0.75
1.26
1.91
[3-7] 64.47
2.12
0.75
1.26
1.89
(2-
64.46
2.13
0.76
1.27
1.89
(1-7) 64.44
2.14
0.76
1.27
1.89
Table 5: Results of the proposed HiE approach on fine-grained level (species) iNaturalist- dataset when incorporating multiple hierarchical levels. Best results highlighted in bold.
achieves state of the art in all settings. Using ten annotated examples for each tie-grained class, Moco-H with HiE achieves a top-I accuracy of 55.13, which is higher than the performance of cross entropy baselines using a hundred annotated samples per fine-grained class. The performance gains of HiE are more evident on the avg. mistake severity and Hier disr@) metrics, bringing profound gains over just using the hierarchical supervised loss. For instance, in 100 sample experiments, the Hier disr@1 drops from 1.17 to 1.10 (6% improvement) by applying hierarchical supervised loss. In contrast, applying HiE brings 24% improvement, reducing Hier dist@) to 0.89.
Finally, we observe that in Table cross-entropy baseline with Resnet-18 backbone trained on the entirety of the iNaturalist- 19 dataset, i.e., 187385 examples with fine-graines labels achieves top-1 accuracy 63.56%. In the semi-supervised setting using coarse- coarse-graing grained labels for entire data and just using 10 examples with fine- fine-grained annotations for each class (10100 training samples), Moco-H with HiE able to recover the sigmificant portion of the underlying performance (achieving top-l accuracy of 55.13), reducing the requirement of expert annotations to only 0.5% of the total number samples.
4.3 Performance across different architectures
We also showcase the generalizability of the proposed HiE approach across different vision ar- chitectures in Table We utilize commonly used vision architectures like MobileNet V3
4
Resnet variants [44] EfficientNet 45). DenseNet121 [46), and tiny variants of DeiT (47). VIT 48
and Swin'T [49]. All networks are trained with the same hyper-parameter setting. We compare the proposed HiE against the vanilla and CRM-based post-hoc approach. HiE outperforms other approaches by significant margins across different architectures and on all metrics. Furthermore, similar performance gains observed irrespective of model capacity, avg, gain of 0.9% in top-1 accuracy across resnet variants.
4.4 Incorporating multiple hierarchical levels
For all the above experiments, we only use two levels of hierarchies. As an additional ablation study, we validate the effectiveness of HiE the presence of multiple hierarchical levels in Table 5) Specifically, we train a separate classifier for each hierarchical level and cascade the predictions top-down. We observe that incorporating the hierarchical information from two levels (levels 6) above the leaf level gives the best performance; the gains taper off gradually we incorporate information from higher levels. The reason for this dip performance is that the higher levels do not have much additional information as they cover too many classes (c.g., on iNaturalist: kingdom

---

(level 1), phylum (level 2), and class (level 3) have 3, 4 and 9 classes respectively) which does not help much in improving the fine-grained predictions further.
Limitation
The main limitation of HiE is the requirement of training an additional network for the coarse level. However, existing approaches employ a full hierarchy with separate classification head for each hierarchical level and additional losses enforce the hierarchical structure, making their training process complex and difficult to converge. Instead, our approach trained with vanilla cross-entropy loss works with partial hierarchy and trade-offs the extra compute for adaptability, reproducibility and simplicity in training. A future avenue for exploration would be to learn disentangled coarse and fine-grained features imitating HiE unified architecture. Another limitation the assumption regarding the availability of underlying label hierarchy. To overcome this limitation, we can exploit large-language models [50) to obtain label hierarchy using raw class labels.
6 Conclusion
We proposed using Hierarchical Ensembles (HiE) of independently trained networks over coarse and fine-grained levels of the label hierarchy. In terms of mistake severity, our proposed post- hoc correction consistently outperforms state-of-the-art methods deep hierarchy-aware image classification by large margins in terms of decrease in avg. mistake severity and hierarchical distance@ k, while simultaneously improving the top-/ accuracy. In the semi-supervised paradigm, show that HiE delivers consistent performance gains when used in conjunction with off-the fie-the-shlff semi-supervise learning algorithms. We show that, comparable performance to a fully supervised baseline be attained, even using merely 10 annotations for each fine-gras tne-grained class a large fine-grained image classification dataset encompassing 1010 classes.
References
[1] Jong-Chyi Su, Zezhou Cheng, and Subhransu Maji. A realistic evaluation of semi-supervised learning for fine-grained classification. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 12966-12977 2021.
[2] Jong-Chyi Su and Subhransu Maji. Semi-supervised learning with taxonomic labels. arXiv preprint arX1v:211..1555,, 2021.
[3] George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11):39-41, 1995.
[4] Jia Deng, Alexander C Berg, Kai Li, and Li Fei-Fei. What does classifying more than 10,000 image categories tell us? In European conference on computer vision, pages 71-84. Springer, 2010.
[5] Björn Barz and Joachim Denzler. Hierarchy-based image embeddings for semantic image retrieval. In WACV, 2019.
[6] Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A Lord. Making better mistakes: Leveraging class hierarchies with deep networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12506-12515, 2020.
[7] Shyamgopal Karthik, Ameya Prabhu, Puneet K Dokania, and Vineet Gandhi. No cost likelihood manipulation at test time for making better mistakes in deep networks. ICLR, 2021.
[8] Ashima Garg. Depanshu Sani, and Saket Anand. Learning hicrarchy aware features for reducing mistake severity. In European Conference on Computer Vision, pages 252-267. Springer, 2022.
[9] Pedro Domingos. Metacost: general method for making classifiers cost-sensitive. In KDD,
1999.
10

---

[10] Bianca Zadrozny and Charles Elkan. Learning and making decisions when costs and probabili- ties are both unknown. In KDD, 2001.
[11] Dongliang Chang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song. and Jun Guo. Your" flamingo" is my" bird": Fine- tine-grained, or not. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11476 1885, 2021.
[12] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances in neural information processing systems, 26, 2013.
[13] Cheng Perng Phoo and Bharath Hariharan. Coarsely-labeled data for better few-shot transfer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9052-9061, 2021.
[14) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick, Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[15] Carlos N Silla and Alex A Freitas A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 22(1):31-72, 2011.
[16] Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for zero-shot classification. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 69-77, 2016.
[17] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251-226: 2018.
[18] Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical represen- tations. In Advances in neural information processing systems, pages 6338-66 6647, 2017.
[19] Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempit- sky. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 6418-6428, 2020.
[20] Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong- Wah Ngo. Tat-Seng Chua, and Yu- Yu-Gang Jiang. Hyperbolic visual embedding learning for zero-shot recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9273-9281, 2020.
[21] Hui Wu, Michele Merler, Rosario Uceda-Sosa. and John R Smith. Learning to make better mis- takes: Semantics-aware visual food recognition. In Proceedings of the 24th ACM international conference on Multimedia, pages 172-176, 2016.
[22] Vivien Sainte Fare Garnot and Loie Landrieu. Leveraging class hierarchies with metrie-guided prototype learning. In BMVC, 2021.
[23] Jack Valmadre. Hierarchical classification at multiple operating points. In NeurIPS, 2022.
[24] Alsallakh Bilal, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren. Do convolutional neural networks learn class hierarchy? IEEE transactions on visualization and computer graphics, 24(1):152-162, 2017.
[25] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7263-7271, 2017.
[26] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.1000 2021.
[27] Richard O Duda, Peter E Hart, et al. Pattern classification and scene analysis, volume 3. Wiley New York, 1973.
[28] Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shot image classification with hierarchical label sets. In ICML, 2023.
11

---

[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. pages 8748-8888888 8763, 2021.
[30] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8769-8778, 2018.
[31] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 200-2011 dataset. 2011.
[32] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint 1rXiv:130..515,, 2013.
[33] Anelia Angelova and Shenghuo Zhu. Efficient object detection and segmentation for fine- grained recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8118818, 2013.
[34] Di Lin, Xiaoyong Shen, Cewu Lu, and Jiaya Jia. Deep lac: Deep localization, alignment and classification for fine-graiie grained recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1666-1674, 2015.
[35] Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell. Part-based r-cnns for fine-grained category detection. In European conference on computer vision, pages 834-849. Springer, 2014.
[36] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596 608, 2020.
[37] Mong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, 2013.
[38] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-awar pscudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.0633 2021.
[39] Xiaofang Wang. Dan Kondratyuk, Eric Christiansen, Kris M Kitani, Yair Alon, and Elad Eban. Wisdom of committees: An overlooked approach to faster and more accurate models. arXiv preprint arXiv:20/2.011988 2020.
[40] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEFF conference computer vision and pattern recognition, pages 8769-8778, 2018.
[41] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenen- baum, Hugo Larochelle, and Richard S Zemel. Meta-leaming for semi-supervised few-shot classification. arXiv preprint 18X3::880300067 2018
[42] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprini arXiv: 1503.005000..3331.. 2015.
[43] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 04861, 2017.
[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[45] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114, PMLR, 2019.
12

---

[46] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[47] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference machine learning, pages 10347-10357. PMLR, 2021.
[48] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[49] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[50] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. NeurIPS, 2020.
13


Reviews:
[{"review_id": "Is8EPeugam", "reviewer": "Reviewer_cuDf", "summary": "This paper focuses on label hierarchy problems and proposes using Hierarchical Ensembles (HiE) of independently trained networks over coarse and fine-grained levels. The reported experimental results show that the proposed method can achieve comparable performance to a fully supervised baseline, even using merely 10 annotations for each fine-grained class on a large fine-grained image classification dataset encompassing 1010 classes.", "questions": "Please give a detailed motivation for this paper and a detailed derivation of Theorem 3.1", "limitations": "The subject of the paper is Test-Time Amendment, thus limiting the potential performance improvement.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The topic of this paper is well-introduced and the proposed approach is straightforward.\n\n2. The proposed HiE utilizes label hierarchy to improve the performance of fine-grained classification at test time using coarse-grained predictions.\n\n3. The proposed methods and experimental results are well presented and the manuscript is overall well organized.\n\n4. Performance of the proposed method is promising according to the comparison experiments with other state-of-the-art studies.\n", "weaknesses": "1. The paper lacks a general investigation of hierarchical architectures and hierarchical embeddings.\n\n2. The proof of the Theorem 3.1 is not convincing, thus making the overall scheme lack theoretical support.\n\n3. The experimental setup of semi-supervised learning is not shown clearly, so it is not possible to judge whether the comparison of the proposed method with other methods is fair.\n\n4. Some errors, such as why Figure 2 is 4 levels.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback. We address the reviewer's concerns and questions below:\n\n**Motivation of the paper:** The primary motivation behind the paper is that training independent coarse and fine-grained help to learn complimentary features. We provide the thought process behind Eqn1 in the global response above (at the top). We show that a minimal post-hoc approach improves upon complex efforts exploring hierarchical architectures, hierarchical loss functions and hierarchical embeddings on two important problems of reducing mistake severity and semi-supervised learning. The proposed approach also provides significant advantages in terms of adaptability, reproducibility and simplicity of training. \n\n**Detailed derivation of theorem:**  The detailed derivation of theorem 3.1 is provided below.\n\nWe show that if we make a correct prediction at the coarse level, the proposed Hierarchical Ensemble (HiE) is guaranteed to improve the downstream predictions at the fine-grained classification task.\n\n**Theorem 3.1:** Assuming, $Q = [q_1, q_2 , ..., q_{N_L}]$ and  $R = [r_1, r_2 , ..., r_{N_{L-1}}]$ are the predictions obtained at the fine and coarse grained labels for a given input $x$, such that $\\sum_{i=1}^{N_L} q_i = 1$ and $\\sum_{i=1}^{N_{L-1}} r_i = 1$. Let $g$ and $g_{parent}$ be the ground truth labels at the fine grained and the coarse grained levels respectively. \n\nNow assuming that the coarse label is correctly predicted by the coarse prediction network i.e. $\\mathrm{argmax} (R) = g_{parent}$, we wish to prove that: \n$$\n\\frac{q_g \\mbox{ . } r_{g_{parent}}}{\\sum_{j=1}^{N_L} q_j \\mbox{ . } r_{j_{parent}}} \\ge q_g    \\quad (1)\n$$\n\nWhere L.H.S is the prediction for the ground truth class using HiE and R.H.S is the direct prediction of the fine grained network for the ground truth class. \n\n**Proof:** The denominator iterates over the fine grained predictions and multiplies them with their parent's prediction score. This is equivalent to iterating over the coarse label predictions and multiplying with the sum of prediction scores for all its children. By rewriting the denominator, we obtain:\n\n$$\n\\sum_{j=1}^{N_L} q_j \\mbox{ . } r_{j_{parent}} = \\sum_{j=1}^{N_{L-1}} r_j \\sum_{i\\in j_{child}} q_{i}  \\quad (2)\n$$\n\nAssuming, $\\sum_{i\\in j_{child}} q_{i} = z_j$\n$$\n\\sum_{j=1}^{N_{L-1}} r_j \\sum_{i\\in j_{child}} q_{i} = \\sum_{j=1}^{N_{L-1}}  r_j \\mbox{ . } z_j = R^T Z       \\quad (3)\n$$\n\nInvoking Holder's inequality, using $a=\\infty$ and $b=1$, ($1/a + 1/b = 1$), we obtain: \n$$\nR^T Z \\le \\lVert R \\rVert_{\\infty}  \\lVert Z \\rVert_{1}      \\quad (4)\n$$\n\nSince, $\\lVert R \\rVert_{\\infty} = \\mathrm{max} (R) = r_{g_{parent}}$ and $\\lVert Z \\rVert_{1}  = \\sum_{i=1}^{N_L} q_i = 1$. We can say:\n$$\nR^T Z  \\le \\lVert R \\rVert_{\\infty} \\lVert Z \\rVert_{1}  \\le r_{g_{parent}}   \\quad (5)\n$$\n\nGiven Eqn 5, we can conclude that: \n$$\n\\frac{r_{g_{parent}}}{\\sum_{j=1}^{N_L} q_j \\mbox{ . } r_{j_{parent}}} \\ge 1   \\implies  \\frac{q_g \\mbox{ . } r_{g_{parent}}}{\\sum_{j=1}^{N_L} q_j \\mbox{ . } r_{j_{parent}}} \\ge q_g \\quad (6)   \n$$\n\n\n**Hierarchical Architectures and Embeddings:** We would like to point out that we sufficiently discuss hierarchical architectures and embeddings in the related work. We compare against four hierarchical architectures [8,11,25,6] and a hierarchical embeddings-based method [5]. We also reproduce two baselines (Cross-Entropy-H and HiE Self). \\\nAdditionally, based on the suggestions provided, we experiment with a unified architecture with a shared feature backbone and only different classification heads for coarse and fine-grained classes. We also experiment with applying HiE over the predictions from the separate classification heads. The results are given below:\n| Method | Top-1 | Avg. Mistakes | Hier Dist@1 | Hier Dist@5 | Hier Dist@20 |\n|------------|------:|--------------:|------------:|------------:|-------------:|\n| Baseline | 63.70 | 2.38 | 0.86 | 1.96 | 3.24 |\n| Unified Arch. | 63.32 | 2.20 | 0.81 | 1.50 | 2.62 |\n| HiE Unified| 63.10 | 2.17 | 0.80 | 1.33 | 2.26 |\n| HiE Separate | 64.61 | 2.15 | 0.76 | 1.33 | 2.19 |\n\nHowever, if the reviewer has additional suggestions for baselines/methods proposing hierarchical architectures and embeddings, we would be happy to incorporate it. \n\n**Clarification of the semi-supervised learning setting:** In the semi-supervised setting (section 4.2, lines 225-228), we assume the availability of a lot of coarsely labelled samples and a small number of fine-grained samples. We experiment with reducing the number of fine-grained annotations, from 100 annotations per class to merely 10 annotations per class. We compare against several methods and show results with-without CRM and with-without HiE.\n\n**Figure 2 caption:** We would like to clarify that Figure 2 refers to a 2 level hierarchy with 4 leaf classes. "}, {"review_id": "Rn0TzpvUdS", "reviewer": "Reviewer_tPTQ", "summary": "This paper proposes a novel approach called Hierarchical Ensembles (HiE) to improve the performance of fine-grained classification by utilizing a label hierarchy and coarse-grained predictions at test-time. The method significantly reduces mistake severity while improving top-1 accuracy on benchmark datasets, achieving state-of-the-art results. The approach is also effective in the semi-supervised setting, bringing notable gains in accuracy and reducing mistake severity as training data decreases for fine-grained classes.", "questions": "This paper makes extensive and comprehensive experiments on the new method proposed by the author, which fully proves the effectiveness of the method. The innovations of the paper can be applied to many off-the-shelf models, but there are not enough of them. If more general structures could be proposed, this paper would be able to make a greater contribution to the field dealt with.", "limitations": "The authors adequately address their proposed limitations in previous models, as their proposed method is a complementary structure that can be applied to any existing model and can effectively improve the performance of the model on the dataset.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "\tOriginality: The paper introduces a novel approach called Hierarchical Ensembles (HiE) that combines coarse-grained predictions and label hierarchy to improve the performance of fine-grained classification. This approach is unique and addresses the challenge of reducing mistake severity while improving accuracy in fine-grained classification. Therefore, the paper demonstrates originality in its proposed methodology.\n\n\tQuality: The paper achieves state-of-the-art results on benchmark datasets by significantly reducing mistake severity and improving top-1 accuracy. The approach is effective not only in the supervised setting but also in the semi-supervised setting, bringing notable gains in accuracy. The paper also compares its approach with previous baselines and demonstrates superior performance. These factors indicate the high quality of the research presented in the paper.\n\n\tClarity: The paper provides a clear and concise explanation of the proposed approach, including the motivation, methodology, and experimental results. The authors effectively communicate the problem statement, the significance of their approach, and the experimental setup. The paper also includes figures and examples to enhance clarity. Overall, the paper is well-written and easy to understand.\n\n\tSignificance: The paper addresses the challenge of fine-grained classification, which requires domain expertise and large amounts of labeled data. By utilizing coarse-grained predictions and label hierarchy, the proposed approach significantly reduces mistake severity and improves accuracy. This has practical implications in various domains where fine-grained classification is important, such as image recognition and object detection. The paper's state-of-the-art results and compatibility with existing semi-supervised methods further highlight its significance.\n", "weaknesses": "There are some problems, which must be solved before it is considered for publication. If the following problems are well-addressed, this reviewer believes that the essential contribution of this paper are important for fine-grained classification. The paper has some context inconsistency errors, for example, it refers to these semi-supervised models on line 233, but then refers to them as self-supervised models on lines 242 and 244. In addition, although the complementary method proposed in this paper can be applied to the off-the-shelf model, the overall innovation of the paper is insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and would like to address their concerns below. \n\n**Typo on lines 242-244:** Thank you for pointing out this inconsistency. We will update the paper to fix this. It should be semi-supervised on lines 242 and 244. \n\n**Overall Innovation of the work:** We would like to point out that the key innovation in our work is proposing a theoretically principled method that can be applied off-the-shelf to a variety of models to improve the accuracy as well as reduce the severity of mistakes in the setting of hierarchical classification. We present comprehensive experiments, study and compare against several methods exploring hierarchical architectures, hierarchical loss functions and hierarchical embeddings. We find that, albeit minimal, the proposed approach achieves state-of-the-art performance, while providing advantages in terms of adaptability, reproducibility and simplicity of training. We believe that these findings are of significant value to the community."}, {"review_id": "hg7pa4pcJE", "reviewer": "Reviewer_yfvr", "summary": "This method introduces a novel approach to achieve state-of-the-art fine-grained image classification. It addresses the issue of mistake severity by developing the Hierarchical Ensemble (HiE) loss, which effectively penalizes incorrect predictions of both course and fine-grained labels. The HiE loss combines the probabilities of predicting the course label and the fine-grained label, providing a joint probability measure.\n\nThe authors of this method provide a proof that accurately predicting the course label probability enhances overall accuracy. To validate their approach, they conducted extensive experimentation on two hierarchical datasets, comparing their method against various existing related works and three additional baselines. The evaluation metrics used include top-1 error, mistake severity, and hierarchical distance. The results clearly demonstrate that their method outperforms comparable approaches across all these measures.\nMoreover, the authors conducted further experiments in a semi-supervised setting to assess the performance of their method when only 10% of labels are available. Remarkably, their approach exhibits significant performance improvements even under such limited label availability, highlighting its effectiveness and robustness.\n\nIn addition to the aforementioned experiments, the authors explored the impact of using different pretrained backbone models in their method. They conducted experiments with various hierarchical depths to evaluate the method's adaptability and generalization across different classification hierarchies. The results of these additional experiments further support the superiority and versatility of their proposed method.\n\nOverall, this research presents a compelling method for fine-grained image classification, showcasing its state-of-the-art performance, reduced mistake severity, and its ability to improve accuracy even with limited labeled data. The comprehensive experimentation and analysis conducted by the authors demonstrate the effectiveness and versatility of their approach in various scenarios, making it a valuable contribution to the field of image classification.\n", "questions": "\n", "limitations": "I believe the authors appropriately discussed the limitations of their method.\n\nThe collection of course-grained labels in addition to fine-grained labels seems very costly but this is outside of the scope of this method. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The text is very well written and easy to understand. It’s very clear what the problem is and how their method can be used to increase performance. \n\nThe authors conducted a comprehensive range of experiments to thoroughly evaluate the performance of their model.\n\nThe method demonstrates robustness by effectively utilizing any pretrained backbone model.", "weaknesses": "Since I’m not as familiar with this method, it would have been helpful to have a clearer description of these metrics within the paper, as I had to refer to the related works to gain a better understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their positive feedback. \n\n**Evaluation Metrics:** Thank you for the suggestion. We will update the paper to discuss the evaluation metrics in more detail and make it self-contained.  "}, {"review_id": "ClN5C4XAtp", "reviewer": "Reviewer_39mn", "summary": "The paper considers the problem of fine-grained classification and proposes to use label hierarchy information at test time to improve the performance of the fine-grained classifier. The overarching goal is to improve the top-1 accuracy while at the same time reducing the severity of the mistakes (e.g., misclassifying species from the same kind or family is more acceptable than making a totally unrelated prediction).\n\nThe paper presents a post-hoc correction technique called Hierarchical Ensembles (HiE). The key idea is to train 2 classifiers - 1 fine-grained and 1 coarse-grained. At inference time, the predictions of the coarse-grained classifier are used to re-weight the probabilities of the fine-grained one. The authors perform experiments on 2 standard benchmark datasets: iNaturalist-19 and tieredImageNet-H and claim to achieve new state-of-the-art results on both of them. They also show the promise of their method in a semi supervised learning setup where the coarse grained labels are known for all samples in the dataset but the fine-grained (being more expensive to collect) only for few samples per class.", "questions": "Q1: Could you please clarify how you formally derive the intuition for the decision rule defined in Eqs. (2) and (3)?\n\nQ2: Would it help if you use the logits from $\\hat{y}_{L}$ and\n\n$\\hat{y}_{L-1}$ directly instead of the softmaxed probabilities in $Q$ and $R$?\n\nQ3: To complete the discussion from L143-147, you can also consider the case when the fine-grained prediction is correct but the coarse-level prediction is wrong.\n\nQ4: I am curious how often there is a mismatch between the fine-grained and the coarse classifier. And what is the distribution / the average of the depth of their LCA nodes.\n\nQ5: L220-221: \"... training a separate network at the coarse label allows explicit disentanglement of features at the coarse and fine-grained levels ...\". I don't believe that the current set of experiments is sufficient to make such a strong claim.\n\nQ6: Does the $\\pm$ in the tables refer to standard deviation? How many runs were executed?\n\nQ7: Sec. 4.4: Following the comments in the Weaknesses section above, it might be useful to formally clarify what is meant by \"cascade the predictions top-down\" (L279-280). Also, do you need to incorporate consecutive hierarchical levels (e.g., would it be possible and sufficient to take levels 5 and 7, omitting level 6)?\n\nQ8: Sec 5: What do preliminary experiments suggest if you unify the architecture of the fine-grained and the coarse-grained classifier (and, e.g., only fine-tune separate classification heads for the two classifiers)?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is written well and is easy to read.\n* The paper extends and builds up on prior works and seems to exceed their results.\n* The Semi Supervised Learning setup is interesting.\n* The conducted experiments and ablations are sensible.", "weaknesses": "My main concern is with the methodology (Section 3). I understand the final decision rule (Eqs. (2) and (3)) but its derivation and motivation is confusing to me.\n\nL118: \"Assuming conditional independence between the estimated logits...\": $i ⫫ i_{parent} \\mid x$ means $P(i, i_{parent} \\mid x) = P(i \\mid x)P(i_{parent} \\mid x)$. Thus, to the best of my knowledge, the denominator in Eq. (1) should not be there. Moreover, I believe that the conditional independence assumption is unrealistic for the considered setup. It does not feel right that the fine-grained prediction/probability is independent of its parent. This affects the derivations in the rest of the section. \n\nMinor:\n\n* The following related, in my opinion, citation is missing:\\\nRidnik et al., ImageNet-21K Pretraining for the Masses, NeurIPS 2021 (Datasets and Benchmarks)\\\nThey train ImageNet-21K classifiers using the WordNet hierarchy.\n* For deep hierarchies, it may still be costly to collect labels from the penultimate hierarchy level. It would be useful if the authors include the set of labels from the last two levels for each of the datasets and refer to them from the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and thoughtful comments. They were indeed helpful to improve the paper. We take this opportunity to address your concerns:\n\n* **Q1: Clarify how you formally derive the intuition for the decision rule defined in Eqs. (2) and (3)?** We agree with your comments and we recognize that bypassing the intermediary thinking process resulted in a lack of clarity. We carefully redo the derivation of the proposed decision rule in the global response above (at the top). Furthermore, we would like to point out that if we simply consider Eqn1 as a modified scoring function, the proof and the strong empirical results still hold. \n\n* **Q2: Would it help if you use the logits?** Using logits instead of softmax probabilities for HiE improves over the baseline, however, the improvement is lower compared to that using softmax. We provide the comparison below.\n| | Top-1 | Avg. Mist | Hie Dist@1 | Hie Dist@5 | Hie Dist@20 |\n|:-------:|:-----:|:---------:|:----------:|:----------:|:-----------:|\n| Logits | 64.46 | 2.16 | 0.77 | 1.35 | 2.43 |\n| Softmax | 64.61 | 2.15 | 0.76 | 1.33 | 2.19 |\n\n* **Q3: L143-147 Consider the case when the fine-grained prediction is correct but the coarse-level prediction is wrong** We provide the mistake mismatch distribution on the iNaturalist dataset below. There are a few examples where fine-grained predictions are correct, while coarse-grained predictions are wrong (3.46%). In such cases, HiE may hamper the performance. However, that is only a small fraction compared to the more interesting scenario of coarse-correct and fine-incorrect (25.19%). We will add this discussion to the paper.\n| Before HiE | Fine Correct | Fine Incorrect |\n|------------------|--------------|----------------|\n| Coarse Correct | 60.24 | 25.19  |\n| Coarse Incorrect | 3.46 | 11.11 |\n\n* **Q4: Mismatch between the fine-grained and the coarse classifier. And what is the distribution / the average of the depth of their LCA nodes** We provide the distribution in response to Q3. We provide the average LCA, between the predicted fine-grained and coarse-grained labels below (for each of the four cases)\n| Before HiE | Fine Correct | Fine Incorrect |\n|------------------|--------------|----------------|\n| Coarse Correct |  1.00 | 1.65 |\n| Coarse Incorrect | 4.07 | 3.07 |\n\n* **Q5: L220-221: \"training a separate network at the coarse label allows explicit disentanglement of features at the coarse and fine-grained levels\"** We thank the reviewer for pointing this out. Indeed, we claim that the two models learn complementary features which can be combined using our HiE method. We will update the paper to avoid using the term explicit disentanglement.\n\n* **Q6: Does the ± in the tables refer to standard deviation? How many runs were executed?** Yes, it indicates the standard deviation over 5 runs.\n\n* **Q7: Sec. 4.4: Cascade and skipping Hierarchy** In the Cascade process, we apply HiE on level 2 using level 1 for coarse predictions. The updated level 2 predictions are then used as coarse predictions to apply HiE on level 3. The process repeats till the leaf node. \\\nOur approach does not necessarily require consecutive hierarchical levels. HiE results on level 7 of the iNaturalist dataset, directly using level 5 (omitting level 6) for coarse prediction are given below. It leads to noticeable gains over the baseline, however, the performance remains marginally lower compared to using level 6 for coarse classification. \n| Method | Top-1 | Avg. Mistakes | Hier Dist@1 | Hier Dist@5 | Hier Dist@20 |\n|------------|------:|--------------:|------------:|------------:|-------------:|\n| Baseline | 63.70 | 2.38 | 0.86 | 1.96 | 3.24 |\n| HiE (Level 6) | 64.61 | 2.15 | 0.76 | 1.33 | 2.19 |\n| HiE (Level 5) | 64.49 | 2.16 | 0.77 | 1.36 | 2.09 |\n\n* **Q8: Sec 5 Preliminary experiments on unified architecture** Compared to the baseline, unified architecture with a common backbone for feature extraction and separate classification heads for coarse and fine-grained level results in decreased performance for fine-grained classification (63.32% vs 63.70%) and improved performance on coarse classification (86.15% vs 85.30%). Applying HiE between the two classification heads improves the hierarchical metrics but results in lower top-1 accuracy.\n| Method | Top-1 | Avg. Mistakes | Hier Dist@1 | Hier Dist@5 | Hier Dist@20 |\n|------------|------:|--------------:|------------:|------------:|-------------:|\n| Baseline | 63.70 | 2.38 | 0.86 | 1.96 | 3.24 |\n| Unified Arch. | 63.32 | 2.20 | 0.81 | 1.50 | 2.62 |\n| HiE Unified| 63.10 | 2.17 | 0.80 | 1.33 | 2.26 |\n| HiE Separate | 64.61 | 2.15 | 0.76 | 1.33 | 2.19 |\n\n**Missing Citation:** Thank you for pointing out this reference. We will add this to the paper. We would also like to point out that the method in [Ridnik et al., ImageNet-21K Pretraining for the Masses] is similar to [25], which we compare against in our experiments."}]

Author rebuttal:
We describe the steps clarifying the derivation of Eqn 1 in the paper. We use slightly different notations for the sake of improved clarity. 

>          (X)
>         /   \
>        /     \
>       /       \
>     (C)       (F)
>     Graphical model for separate classifiers trained on coarse and fine-grained labels.
>     (X) is the input image, (C) and (F) are coarse and fine-grained labels, respectively.

Considering the given graphical model, by the product rule, we have:
$$
P(C, F, X) = P(C, F | X) \cdot P(X)   \quad  (1)
$$

By factorizing the graphical model we obtain,
$$
P(C, F, X) = P(C | X, \phi) \cdot P(F | X, \theta) \cdot P(X)  \quad  (2)
$$

From Eq. (1) and Eq. (2), we have,
$$
P(C, F | X) = P(C | X, \phi) \cdot P(F | X, \theta)   \quad  (3)
$$

From Eq. (3), we have conditional independence between the predictions of coarse and fine-grained classifiers, i.e.: $C \mathrel{\unicode{x2AEB}} F | X.$


Now, assuming access to a label hierarchy $\mathcal{H}$ between coarse and fine labels, we define the following score function between the coarse classifier's prediction for the $j^{th}$ coarse class $c_j$ and the fine classifier's prediction for the $i^{th}$ fine class $f_i$:
$$
S(f_i, c_j ; x ,\theta, \phi, \mathcal{H}) = P(f_i | x, \theta) \cdot P(c_j | x, \phi) \cdot \mathbb{1}_{\mathcal{H}}(c_j = \text{parent}(f_i))   \quad  (4)
$$

Where $\mathbb{1}\_{\mathcal{H}}$ is the indicator function manifesting the label hierarchy. Assuming that each fine-grained label is assigned only a single coarse label and $c_{i_{\text{parent}}}$ = $\text{parent}(f_i)$, we can simplify the above equation to:
$$
S(f_i, c_{i_{\text{parent}}} ; x ,\theta, \phi) = P(f_i | x, \theta) \cdot P(c_{i_{\text{parent}}} | x, \phi) , \text{ for } i=1,2,...,N_L,    \quad  (5)
$$

We normalize Eq. (5) to make it a valid probability density function:
$$
P(f_i, c_{i_{\text{parent}}} | x ,\theta, \phi) = \frac{P(f_i | x, \theta) \cdot P(c_{i_{\text{parent}}} | x, \phi)}{\sum_{j=1}^{N_{L}} P(f_j | x, \theta) \cdot P(c_{j_{\text{parent}}} | x, \phi)} , \text{ for } i=1,2,...,N_L,    \quad  (6)
$$

Eqn (6) is equivalent to Eqn 1 in the paper.


Meta-review: The paper proposes an approach called Hierarchical Ensembles (HiE) that utilizes label hierarchy and coarse-grained predictions to improve the performance of fine-grained classification at test-time. The paper claims to achieve state-of-the-art results on two benchmark datasets, iNaturalist-19 and tieredImageNet-H. The main concern was about the derivation of the method described in Section 3. Reviewer 39mn pointed out this issue and discussed with authors during the rebuttal phase and the authors provided a detailed explanation of their reasoning. Overall, based on the reviews and comments, I recommend an ACCEPT to this paper. The paper demonstrates its superiority over previous baselines on two benchmark datasets and also shows its applicability in the semi-supervised setting. The authors addressed most of the concerns and questions raised by the reviewers in their rebuttal, but they should also incorporate their clarifications and corrections in the final version.
Decision: Accept (poster)