No Cost Likelihood Manipulation at Test Time for Making Better Mistakes in Deep Networks
Authors: Shyamgopal Karthik
Ameya Prabhu
Puneet K. Dokania
Vineet Gandhi
Venue: ICLR 2021 Poster (2021)
Keywords: Hierarchy-Aware Classification
Conditional Risk Minimization
Post-Hoc Correction

Abstract: There has been increasing interest in building deep hierarchy-aware classifiers that aim to quantify and reduce the severity of mistakes, and not just reduce the number of errors. The idea is to exploit the label hierarchy (e.g., the WordNet ontology) and consider graph distances as a proxy for mistake severity. Surprisingly, on examining mistake-severity distributions of the top-1 prediction, we find that current state-of-the-art hierarchy-aware deep classifiers do not always show practical improvement over the standard cross-entropy baseline in making better mistakes. The reason for the reduction in average mistake-severity can be attributed to the increase in low-severity mistakes, which may also explain the noticeable drop in their accuracy. To this end, we use the classical Conditional Risk Minimization (CRM) framework for hierarchy-aware classification. Given a cost matrix and a reliable estimate of likelihoods (obtained from a trained network), CRM simply amends mistakes at inference time; it needs no extra hyperparameters and requires adding just a few lines of code to the standard cross-entropy baseline. It significantly outperforms the state-of-the-art and consistently obtains large reductions in the average hierarchical distance of top-$k$ predictions across datasets, with very little loss in accuracy. CRM, because of its simplicity, can be used with any off-the-shelf trained model that provides reliable likelihood estimates.

Published as a conference paper at ICLR 2021
No COST LIKELIHOOD MANIPULATION AT TEST TIME FOR MAKING BETTER MISTAKES IN DEEP NETWORKS
Shyamgopal Karthik'
Ameya Prabhu
Puneet K. Dokania University of Oxford & Five Al Limited
CVTT, HIT Hyderabad, India
University of Oxford
Vineet Gandhi CVTT, IIT Hyderabad, India
ABSTRACT
There has been increasing interest in building deep hierarchy- aware classifiers that aim to quantify and reduce the severity of mistakes, and not just reduce the number of errors. The idea is to exploit the label hierarchy (e.g.. the WordNet ontology) and consider graph distances as a proxy for mistake severity. Surprisingly, on examining mistake-severity distributions of the top-1 prediction, we find that cur- rent ofte--of-the-ar hierarchy-awanl deep classifiers do not always show practical improvement over the standard cross-entropy baseline in making better mistakes. The reason for the reduction in average mistake-severity can be attributed to the increase in low-severity mistakes, which may also explain the noticeable drop their accuracy. To this end, we use the classical Conditional Risk Minimization (CRM) framework for hierarchy aware classification. Given a cost matrix and a reliable estimate of likelihoods (obtained from trained network), CRM simply amends mistakes at inference time; it needs no extra hyperparameters, and re- quires adding just few lines of code to the standard cross-entropy baseline. It significantly outperforms the ofute-of-thh-art and consistently obtains large reduc- tions in the average hierarchical distance of top-k predictions across datasets, with very little loss in accuracy. CRM, because of its simplicity, can be used with any off-the-shell trained model that provides reliable likelihood estimates.
INTRODUCTION
The conventional performance measure of accuracy for image classification treats all classes other than ground truth as equally wrong. However, some mistakes may have much higher impact than others in real-world applications. An intuitive example being an autonomous vehicle mistaking car for bus a better mistake than mistaking a car for a lamppost. Consequently, it is essential to integrate the notion of mistake severity into classifiers and one convenient way to do so is to use taxonomic hierarchy tree of class labels, where severity defined by a distance on the graph (e.g., height of the Lowest Common Ancestor) between the ground truth and the predicted label (Deng et al., 2010; Zhao et al., 2011). This similar to the problem of providing a good ranking classes in retrieval setting. Consider the case of an autonomous vehicle ranking classes for a thin, white, narrow band (a pole, in reality). A top-3 prediction of {pole, lamppost, tree) would be a better prediction than (pole, person, building). Notice that the top-k class predictions would have least incorrect predictions here, and the aim is to reduce the severity of these mistakes, measured by the average hierarchical distance of each of the top k predictions from the ground truth. Silla & Freitas (2011) survey classical methods leveraging class hierarchy when designing classifiers across various application domains and illustrate clear advantages over the flat hierarchy classification. especially when the labels have well-defined hierarchy.
There has been growing interest in the problem of deep hhearchy--wware image classification (Barz & Denzler, 2019; Bertinetto et al., 2020). These approaches seek to leverage the class hierarchy
"shyamgopal karthikkresearchh wiw.iiiiiii fift.ac.in

---

Published as a conference paper at ICLR 2021
inherent in the large scale datasets (e.g., the ImageNet dataset is derived from the WordNet seman- tic ontology). Hierarchy is incorporated using either label embedding methods, hierarchical loss functions, hierarchical architectures. We empirically found that these models indeed improve the ranking of the top-k predicted classes ensuring that the top alternative classes are closer in the class hierarchy. However, this improvement is observed only for k 1.
While inspecting closely the top-1 predictions of these models, we observe that instead of improving the mistake severity, they simply introduce additional low-severity mistakes which in turn favours the mistake-severity metric proposed in (Bertinetto et al., 2020). This metric involves division by the number of misclassified samples, therefore, in many situations (discussed in the paper), it can prefer a model making additional low-severity mistakes over the one that does not make such mistakes. This is at odds with the intuitive notion of making better mistakes. These additional low-severity mistakes can also explain the significant drop in their top-1 accuracy compared to the vanilla cross- entropy model. We also find these models to be highly miscalibrated which further limits their practical usability.
In this work we explore a different direction for hierarchy-aware classification where we amend mistake severity at test time by making post-hoc corrections over the class likelihoods (e.g.. softmax in the case of deep neural networks). Given a label hierarchy, we perform such amendments to the likelihood by applying the very well-known and classical approach called Conditional Risk Mini- mization (CRM). We found that CRM outperforms state-of-the-ar deep hierarchy-awan classifiers by large margins at ranking classes with little loss in the classification accuracy. As opposed to other recent approaches, CRM does not hurt the calibration of a model as the cross-entropy likelihoods can still be used for the same. CRM is simple, requires addition of just a few lines of code to the standard cross-entropy model, does not require retraining of a network, and contains no hyperpa- rameters whatsoever.
We would like to emphasize that we do not claim any algorithmic novelty as CRM has been well explored in the literature (Duda & Hart, 1973, Ch. 2). Almost decade ago. Deng et al. (2010) had proposed a very similar solution using Support Vector Machine (SVM) classifier applied on handcrafted features. However, this did not result in practically useful performance because of the lack of modern machine learning tools at that time, We intend to bring this old, simple, and extremely effective approach back into the attention before we delve deeper into the sophisticated ones requiring expensive retraining of large neural networks and designing complex loss functions. Overall, our investigation into the seeaarchy-awar classification makes the following contributions:
We highlight a shortcoming in one of the metrics proposed to evaluate hierarchy-awand aware classification and show that can easily be fooled and give the wrong impression of making better mistakes
We revisit an old post-hoc correction technique (CRM) which significantly outperforms prior art when the ranking of the predictions made by the model are considered.
We also investigate the reliability of prior in terms of calibration and show that these methods are severely miscalibrated, limiting their practical usefulness.
2 RELATED WORKS
2.1 COST-SENSITIVE CLASSIFICATION
Cost-sensitive classification assigns varying costs to different types of misclassification errors. The work by Abe al. (2004) groups cost- sest-sensitive classifiers into three main categories. The first cat- egory specifically extends one particular classification model to be cost-sensitive, such as support vector machines (Tu & Lin, 2010) or decision trees (Lomax & Vadera, 2013). The second category makes the training procedure cost-sensitive, which is typically achieved by assigning the training examples of different classes with different weights (rescaling) (Zhou & Liu, 2010) or by chang- ing the proportions of each class while training using sampling (rebalancing) (Elkan, 2001). The third category makes the prediction procedure cost-sensitive (Domingos, 1999; Zadrozny & Elkan, 2001a). Such direct cost-sensitive decision-making making is the most generic: it considers the underlying classifier as a black box and extends to any number of classes and arbitrary cost matrices. Our work comes under the third category of post-hoc amendment. We study cost-sensitive classification in
2

---

Published as a conference paper at ICLR 2021
large scale setting (e.g.. ImageNet) and explore the use of taxonomic hierarchy to obtain the misclassification costs.
2.2 HIERARCHY AWARE CLASSIHCATION
There is a rich literature around exploiting hierarchies to improve the task of image classification. Embedddng- -ased methods define each class as a soft embedding vector, instead of the typical one- hot. DiVi (Frome et al., 2013) learn a transformation over image features to maximize the cosine similarity with their respective word2vec label embeddings. The transformation is learned using ranking loss and places the image embeddings in a semantically meaningful space. Akata et al. (2015); Xian et al. (2016) explore variations of text embeddings, and ranking loss frameworks. Barz & Denzler (2019) project classes on a hypersphere, such that the correlation of class embeddings equals the semantic similarity of the classes. The semantic similarity is derived from the height of the lowest common ancestor (LCA) a given hierarchy tree.
Another line of work directly alters the loss functions the algorihmns/archieeture Zhao et al. (2011) propose a weighted (hierarchy-aware) multi-class logistic regression formulation. Verma al. (2012) optimize context-sensitive loss to learn separate distance metric for each node in the class taxonomy tree. Wu et al. (2016) combine losses at different hierarchies of the tree by learning separate, fully connected layers for each level post shared feature space.Bilas Bilal al. (2017) add branches at different depths of AlexNet architecture to fuse losses at different levels of the hierarchy. Brust & Denzler (2019) use conditional probability chains to derive novel label encoding and corresponding loss function.
Most deep learning-based methods overlook the severity of mistakes, and the evaluation revolves around counting the top-k errors. Bertinetto et al. (2020) has revived the interest in this direction by jointly analyzing the top-k accuracies with the severity of errors. They propose two modifications to cross-entropy to better capture the hierarchy: one based on label embeddings (Soft-labels) and the other, which factors the cross-entropy loss into the individual terms for each of the edges in the hierarchy tree and assigns different weights them (Hierarchical cross-entropy or HXE)
Our method uses models trained with vanilla cross-entropy loss and alters the decision rule to pick the class that minimizes the conditional risk where the condition is being imposed using the known class-hierarchy. On similar lines, Deng et al. (2010) study the effect of minimizing conditional risk on the mean hierarchical cost. They leverage the ImageNet hierarchy for cost and compute posteriors by fitting a sigmoid function to the SVM's output or taking the percent of neighbours from class for Nearest Neighbour classification. Our work investigates the relevance of CRM in the deep learning era and highlights the importance of looking beyond mean hierarchical costs and jointly analyzing the role of accuracy and calibration.
2.3 CALIBRATION OF DEEP NEURAL NETWORKS
Networks are said to be well-calibrated if their predicted probability estimates representative of the true correctness likelihood. Calibrated confidence estimates are important for model inter- pretability and its use in downstream applications. Platt scaling (Platt al., 1999), Histogram binning (Zadrozny & Elkan, 2001b) and Isotonic regression (Zadrozny & Elkan, 2002) are three common calibration methods. Although originally proposed for the SVM classifier, their variations are used in improving the calibration of neural networks (Guo et al., 2017). Calibrated probabil- ity estimates are particularly important when cost-sensitive decisions are be made (Zadrozny & Elkan, b001b) and are often measured using Expected Calibration Error (ECF) and Maximum Cali- bration Error (MCE) (Niculescu-Mizil & Caruana, 2005; Naeini et al., 2015; Mukhoti et al., 2020).
We desire models with high accuracy that have low calibration error and make less severe mistakes. However, there often a compromise. Studies cost-sensitive classification (Jan et al., 2012) reveal a trade-off between costs and error rates. Reliability literature aims to obtain better calibrated deep networks while retaining top-k accuracy (Seo et al., 2019). We further observe that methods like Soft-labels Hierarchical cross-entropy successfully minimize the average top-k hierarchical cost, but result in poorly calibrated networks. In contrast, the proposed framework retains top-k accuracy and good calibration, while significantly reducing the hierarchical cost.
3

---

Published as a conference paper at ICLR 2021
root
LCA6J)
1.05
1.15
2
0
1
0.38
1.22
Flower
2
0.02
1.58
Bus
Truck
Rose
Orchid
p=0.35
p=0.25
p=0.38
p=0.02
(a)
Figure 1: (a) Consider four class hierarchy tree and corresponding leaf predictions obtained using a cross-entropy baseline. The risk computation is shown beside the tree. argmax p(y|x) predicts the class "rose", while the argmin R(y k|x) predicts the class "bus". (b) Two nodes and j and the subtree (shaded gray) originating at their lowest common ancestor 5CA(i,
3 APPROACH
The K-class classification problem comes with a training set {(x; y:) where label y; E )' (1, 2,....., K). The classifier is a deep neural network ʃa X' p()) parametrized by 0 which maps the imput samples to probability distribution over the label space V. The p(y|x) is typically derived using softmax function the logits obtained for input Given ply(x), the network minimizes cross-entropy with the ground truth class over samples from the training set, and uses SGD to optimize 0, forming the standard hierarchy-agnostic cross-entropy baseline. The decision rule is naturally given by argmax ply k((x).
The classical CRM framework (Duda & Hart, 1973) can be adapted image classification by taking the trained model with a given 0 and incorporating the hierarchy information at deployment time. A symmetric class-relationship matrix C is created using the given hierarchy tree (which can either be drawn from the WordNet ontology or an application specific taxonomy), where C.j is the height of the lowest common ancestor Ay,, Yj) between classes and j. The height of node is deñied as the number of edges between the given node and the furthest leaf. C,2, is zero when i j and is bounded by the maximum height of the hierarchy tree.
Given an input x, the likelihood p(y) x) is obtained by passing the sample through the network fo(x). The only modification we make in the decision rule, which now selects the class that minimizes the conditional risk R(y ke|x), given by:
argmin R(y kx|) argmin Σ Ckj ply j(x)
For the ease of the reader, we illustrate a four-class example Figure la, comparing predictions obtained using the standard cross coosspy baseline (leaf nodes), and the prediction using CRM (Eq. (1)) for a given class-relationship matrix. Given the probability each class p(y[x), argmin It(y x) is the Bayes optimal prediction. It is guaranteed to achieve the lowest possible overall cost, i.e. lowest expected cost over all possible examples weighted by their probabilities (Duda & Hart, 1973, Ch. 2).
Depending on the cost-matrix and p(y[x), the top-1 prediction of the CRM applied on cross-entropy might differ from the top-1 prediction of the cross-entropy baseline. However, because of the over- confident nature of recent deep neural networks, we observe that the top-1 probability of p(y x) is greater than 0.5 for significant number of test samples. Below prove that in such situations where maxp(y x) is higher than the sum of other probabilities, the post-hoe correction (CRM) does not change the top-1 prediction irrespective of the structure of the tree. Since the second highest probability is guaranteed to be less than 0.5 by definition, our correction can effectively re-rank the classes. Experimentally we find it to significantly reduce the hierarchical distance k.
Theorem I. If max(ply x)) > 0.5, then argmin, Σ OLj ply : -ilx) and argmax ply (x) are identical irrespective of the tree structure and both lead to the same top-1 prediction.
4

---

Published as a conference paper at ICLR 2021
Proof. Consider the tree illustrated in Figure 1b; two leaf nodes (class labels) i.j and the subtree (Tiy) rooted at their Lowest Common Ancestor. Assuming the height of the 5) = h and argmax ((ygax i, the risk R(y j|x) R(j) is given as:
RG) = h p(1) +
E
Cz . p(k)p()))+ +
Σ
CJa= : p(k))
RET
VkETe
Ignoring the cost of other nodes inside Tij, we get R(j) 2 h.p(i) + EvKRTE Cj,R p(k). Similarly, for the risk of class é:
R(1) - h- (1 (1-p(1))+ +
Σ
Cak p(k)
YkgTi
Outside the subtree roted at Tij Ci = Cj.t Vk and therefore without loss of generality we can say that R(i) RGG). if p(i) 0.5.
4 EXPERIMENTS
We evaluate our method on two large-scale hierarchy-awarr benchmarks: (i) tieredImageNet-H for a broad range of classes and (ii) iNaturalistH for fine-grained classification, both of which are complex enough to cover large number of visual concepts. We closely follow the experimental pipeline from Bertinetto et al. (2020) including the train/validationneest splits, hyperparameters for training models, and evaluation metrics.
expertiees
Experimental Details: All models are trained using a ResNet-18 architecture (pre-trained on Im- ageNet) using an Adam optimizer for 200K updates using mini-batch of 256 samples, a learning rate of 10-5, and standard data augmentation of flips and randomly resized crops. We train all the hierarchy-aware models Hierarchical cross-entropy (HXE) (Bertinetto al., 2020), Soft-labels (Bertinetto et al., 2020), YOLO-v2 (Redmon & Farhadi, 2017). VeViSE (Frome et al., 2013), and Barz & Denzler (2019)- along with cross-entropy baseline. We pick the epoch corresponding to the lowest loss on the validation set along with two epochs preceding and succeeding it and report the average of the results obtained from these five checkpoints on the test set. Unlike Bertinetto et al. (2020) we do not preprocess the dataset to downsample the images to 224x224 as it notice- ably reduces the accuracy. Instead, we use the RandomRes i zedCrop () augmentation to crop the images to 224x224 resolution. This accounts for small, but significant improvement in performance across models, thus leading to stronger baselines.
Metrics: We primarily focus on two major metrics; (i) top-1 error, and (ii) average hierarchical distance@ k. which is the mean LCA height between the ground truth and each of the most likely classes. These metrics capture different views of the problem: top-1 error treats all classifier mis- takes the same, whereas average hierarchical distance@) captures a notion of mistake severity, i.e., better or worse mistakes. Average hierarchical distance@ k captures the notion of ranking/ordering the predicted classes closer to the ground truth class. This metric, also used in Bertinetto et al. (2020), is natural extension of the hierarchical distance @ I proposed by Russakovsky et al. (2015) in the original ImageNet evaluation. We also investigate the average mistake-severity metric sug- gested in Bertinetto et al. (2020) which computes the hierarchical distance between the top-1 predic- tion and the ground truth for all the misclassified samples. Note that LCA a log-scaled distance: an increment of 1.0 signifies an error of an entire level of the tree, In the simple case of full binary tree, an increase by one level implies that the number of possible leaf nodes doubles.
4.1 HIERARCHICAL DISTANCE OF TOP-1 PREDICTIONS
Hierarchy- classification methods typically seek to make better mistakes (less costly in terms of hierarchical distance). It is essential that the evaluation metric correctly measures this goal, ie. higher value of the evaluation metric should reflect that the model indeed makes better mistakes. Below we discuss a shortcoming of the average mistake-severin metric proposed in Bertinetto et al. (2020) which considers the mistake severity averaged only over the incorrectly classified samples,
5

---

Published as a conference paper at ICLR 2021
Aug. M scabe seuarity vn hop-1 errr on temolimashatt
Demniuttoon of mitsvs made on tenss-imngelee
D Dietance va top-l ercer tand-imagafie
Iro Moookiu Craave oake
Dareer 4t GPM
Lake Soonting
bwild
Top-
Harchaal Detanoe
Arg. Misi sewerlay top-1 andor on siltt
Dieriier on miteaes rrade or Pattrrllett
Tietzene vs tops Levor on Wonus iet9
Getackall Disara
Tax Ere
(a)
(b)
(c)
Figure 2: (a) Trade-off between hierarchical distance@) I and the top-1 error. The mean here is taken only over the misclassified samples. (b) Histogram of the severity of top-1 mistakes (height of the LCA). The number in the bracket is the mean mistake severity. (c) Trade-off between hierarchi- cal distance@1 and the top-1 error. The mean is taken over all the test samples (the hierarchical @istance@1 zero for correctly classified samples). The top and bottom row correspond to tiered- ImageNet and 1 datasets respectively.
and show that it can be misleading in the sense that a model can show improved performance over this metric, while just making additional low-severity mistakes.
In Figure 2a, we evaluate different approaches only on the set of incorrectly classified samples (hence different test sets for different models as the mistakes will be different). It seems to indicate that recently proposed methods are able to achieve good trade-off between top-1 error and mistake severity. We select models that show a marked trade-off in terms of the mistake-severity metric- Soft-labels with 3 = 4 and HXE with a = 0.6 and analyze the frequency of mistakes at different levels of severity (illustrated in Figure 2b). Surprisingly, we observe that in these regimes, HXE and Soft-labels largely do not make better mistakes; they mostly make additional low-severity mistakes. This behaviour is better demonstrated in the histograms shown in Appendix A.1 (Figure 4). For example, in the case of Soft-labels on iNaturalist99 dataset, it is evident from Figure 4 that as / decreases, the number of less-severe mistakes increases, whereas, the high-severity mistakes remain more or less the same. Similar observations can be made for HXE. This behaviour is not captured in Figure 2a as the metric here involves division by the number of mistakes made by the model. More precisely, say the high severity mistakes made by two models are exactly the same (dz, > 0) over the same number of mistakes (m > 0). Now, if the second model makes additional n > mistakes with overall distance severity of d; > 0, then it is straightforward to observe that du 2 di if da 2 di This implies that the metric would prefer model making additional low-severity mistakes as long as the impact of the severity due to these additional mistakes is less than the overall impact by the high-severity ones.
m+n
We avoided this shortcoming by using the hierarchical distance@) I computed over all the samples (Russakovsky et al., 2015). As shown in Figure 2c, the best best-performing ones in Figure 2a now show the highest hierarchical distance@ as we account for the additional number of low-severity mistakes made by them. Note, distance@) was also used in Bertinetto et al. (2020), however, they also proposed the above mentioned sistaie-severrty metric and performed analyses over it, which, as discussed and showed empirically, casily mislead us towards choosing classifier that just makes additional low severity mistakes while not improving the overall mistake severity at all.
In this more reliable evaluation set-up, we observe CRM (ours) marginally reduces mistake sever- ity compared to cross-entropy. We would like to emphasize that cross-entropy provides near best results. Overall, our experiments suggest that existing methods reduce the average mistake-severity metric by largely making additional low-severity mistakes. This also explains why such models provide lower test accuracy (top-1). Resolving this issue, see that prior art significantly out-

---

Published as a conference paper at ICLR 2021
H Hearchical Distanook on tired inagelict
Dicaade top-) entr on Geee-liragehet
Dieraace vs 3op-) entr dened-lir Iragehht
Seccring
topt Sereciia
X.00
6:10
6:411
0.13
6:1
Herarchicll Distanookk Naturnist10
vtance top ener Naturulst10
Distance top-i error Naturall it.1
1911 Aotuu
20g Hireere DIESTOOR
110 7:0
Tope
Topd
(a) Ranking performance on the given hierarchy
Herurchical Distonoli on tiead imagelist
Jondklmag:Nu(/anmeen
Hizad cittnce BirosimaaeNuiaanknnn
Dicarade
Ave Oue 5 Marac oa 402
Arc tup-22 Seceet
6.0
6.0
0:00
Ileeorciccal Distzneekk on Noturiss13(Rantom))
Harachicl dolancs (endan
Hiouchial (aarnaa)33Rrrooo)
Chec
Ave 100 Socein S01 CoCTi
Avc 1op-23
Top-1 Eivor
(b) Ranking performance randomly hierarchy.
Figure 3: Left: Average hierarchical distance@k & with varying k. Middle: Average hierarchical
distance@5 top-1 error. Right: Average hierarchical distance@20 vs top-1 error.
performs the cross-entropy baseline either in making better mistakes (distance@) 1) or in the top-1 accuracy.
4.2 HIERARCHICAL DISTANCE TOP-K PREDICTIONS
We now compare the ordering of classes provided by each of these classifiers. Ranking predictions give us significant insight into how reliably the predictions align with the hierarchy. We measure the quality of ranking using the average hierarchical distance @k,, for various values of and present them in Figure 3a (left). We find that CRM significantly outperforms all the competing methods, giving the best hicrarchically aligned models on the hierarchical distance@* k. Note, for > 1, recent approaches also provide improvement in distance @h compared to the cross-entropy model.
A better ranking classes often comes with a significant trade-off with top-1 accuracy. We plot the hierarchical distance @ k with top-1 accuracy for A 5 and & 20 in Figure 3a (middle and right) to better understand this trf Interestingly, we observe that CRM improves ranking with almost loss in top-1 performance and outperforms other methods by substantial margin.

---

Published as a conference paper at ICLR 2021
ticredImageNet-H
iNaturalist-H
Loss Function
ECE
MCE
ECE
MCE
pre
post T
pre T
post
pre T
post T
pre T
post T
Soft-labels (3=15)
7.05%
4.79%
18.55%
13.03%
34.64%
16.63%
54.86%
26.87&
Soft-labels (3=10)
29.55%
6.36%
39.95%
22.07%
29.55%
19.87%
39.95%
33.29%
Soft-labels( 3=5)
58.99%
10.92%
83.53%
26.86%
24.53%
17.20
88.32%
55.68%
Soft-labels ((==4)
57.16%
11.12%
86.92%
27.44%
19.29%
11.46%
19.29%
56.06%
HXB (a=0.2)
1.53%
1.53%
5.84%
5.84%
4.37%
1.50%
7.73%
3.61%
HXE (o=0.4)
2,44%
2.44%
5.48%
5.48%
1.13%
1.13%
2.62%
2.62%
IIx5(==0.5)
3.95%
2.61%
7.84%
5.40%
2.46%
2.46%
6.77%
6.77%
HXE (a=0.6)
6.25%
3.28%
10.75%
6.58%
5.24%
5.24%
11.20%
11.20%
Label Smoothing
9.61%
2.33%
15.43%
6.13%
4.93%
1.11%
7.35%
3.35%
Cross-Entropy
1.61%
1.61%
4.27%
4.27%
4.32%
1.42%
8.18%
3.26%
Table 1: ECE and MCE for the various models on the tiered-ImageNet and iNaturalist19 datasets
before and after temperature scaling. The optimal temperature was found on the validation set and results reported on the test set.
interesting extension analyze how dependent these approaches are on a given hierarchy, and how modifying the hierarchy might impact their behaviour. To test this, randomly shuffle the classes at the leaf nodes of given tree structure and compare ranking performance in Figure 3b. We observe that even though CRM does not explicitly reduction in the hierarchical distance compared to all the previous methods. High accuracy of CRM in this case is because of the fact that it is post-hoc and for highly confident models such as deep networks, its top-1 accuracy remains largely unchanged (refer Theorem 1). On the other hand, models depending the tree-structure during training (directly indirectly) will try fit to the structure, which can be harmful in situations where the tree structure is not very reliable. For example, if the structure implies that 'cat" closer 'person' than a 'dog', then the models incorporating such information while learning the feature space might not be able to learn robust classifier and might potentially end-up making more mistakes, as also validated in our experiments.
the hierarchy while training, it provides drastic
4.3 IMPACT OF LABEL HIERARCHY ON THE RELIABILITY
In order for models to be useful in safety-critical scenarios, they should calibrated that they are not wrong with high confidence. To this end, we analyse the reliability of the output probabilities of Softlahels, HXE, label smoothing, and CRM (which is the vanilla cross-entropy likelihood) using widely accepted metrics such as ECE (Expected Calibration Error) and MCE (Maximum Calibra- tion Error) in Table I. Softlabels and HXE, for example, show clear trends of increasing degrada- tion in calibration on better class ranking (as measured by distance @ k), i.e., the more they attempt to adhere to the hierarchy, the less reliable their probability estimates become. We additionally experiment with improving calibration in all the above models using temperature scaling. We ob- serve that it reduces miscalibration as measured by the ECE and MCE scores, but most models still remain far worse than the cross-entropy baseline. Changes in ECE/MCE were unnoticeable when using the probability estimates corresponding to CRM predictions (taking p(y)a (x) corresponding to argmin R(y)x) instead of maximum cross-entropy prediction.
These experiments clearly suggest that while the focus should turn into developing models that make better mistakes, we should also make sure that such models are reliable by understanding how incorporating the label hierarchy during training might impact the likelihood estimates.
CONCLUSION
We proposed using Conditional Risk Minimization (CRM) as a tool to amend likelihood a post- hoc fashion obtain hierarchy-aware classifiers, an approach that is different from the three domi- nant paradigms: hierarchy-awand losses, hierarchy-aware architectures, and label embedding meth- ods. We illustrated issue with the cestake-severity metric that, otherwise, could give wrong impression of improvement while the model might just be making additional mistakes to fool the metric.
8

---

Published as a conference paper at ICLR 2021
In terms of better ranking predictions, our proposed post-hoc correction consistently outperforms state-of-the-art methods in deep dierarchy-awars image classification by large margins in terms of decrease in hierarchical distance @ k, with little to loss in top-1 accuracy. We find the direction of post-hoc corrections promising as can simultaneously deliver calibration, accuracy, and better class ranking efficiently with surprisingly little trade-offs in either.
Overall, the literature on hierarchy-aware image classification has shown the WordNet hierarchy's effectiveness in improving performance. However, previous works assumed that all the edges in the tree are equally important. A future avenue for exploration would be to learn the weights of these edges in order to compute more effective measure of mistake severity.
ACKNOWLEDGEMENTS
SGK and AP would like to thank Saujas Vaduguru, Aurobindo Munagala and Arjun P for detailed feedback on the manuscript. PKD would like to thank Sina Samangooei for constructive com- ments. This work was supported by the following grants/organisations: Early Career Research Award, ECR/2017/001222, from Science and Engineering Research Board (SERB), Department of Science & Technology, Government of India; EPSRC/MURI grant EP/N019474/1; Facebook (DeepFakes grant); and Five AI Ltd., UK.
REFERENCES
Naoki Abe, Bianca Zadrozmy, and John Langford. An iterative method for multi-class cost-sensitive learning. In KDD, 2004.
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In CVPR, 2015.
Bjöm Barz and Joachim Denzler. Hierarchy- -bsed image embeddings for semantic image retrieval. In WACV, 2019.
Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A Lord. Making better mistakes: Leveraging class hierarchies with deep networks. In CVPR, 2020.
Alsallakh Bilal, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren. Do convolutional neural networks learn class hierarchy? IEEE transactions on visualization and computer graphics, 2017.
Alexandee Brust and Joachim Denzler. Integrating domain knowledge: using hierarchies to improve deep classifiers. In ACPR, 2019.
Jia Deng, Alexander C Berg, Kai Li, and Li Fei-Fei. What does classifying more than 10,000 image categories tell us? In ECCV, 2010.
Pedro Domingos. Metacost: A general method for making classifiers cost-sensitive. In KDD, 1999.
Richard O Duda and Peter E Hart. Pattern recognition and scene analysis, 1973.
Charles Elkan. The foundations of cost-sensitive learning. In IJCAI, 2001.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio. Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: deep visual-semantic embedding model. In NeurIPS, 2013.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017.
Te-Kang Jan, Da-Wei Wang, Chi-Hung Lin, and Hsuan-Tien Lin. A simple methodology for soft cost-sensitive classification. In KDD, 2012.
Susan Lomax and Sunil Vadera. A survey of cost-sensitive decision tree induction algorithms. ACM Computing Surveys (CSUR), 2013.

---

Published as a conference paper at ICLR 2021
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H S Torr, and Puneet K Dokania. Calibrating deep neural networks using focal loss. NeurlPS, 2020.
Mahdi Pakdaman Naeini, Gregory F Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI, 2015.
Alexandru Niculescu-Mizi and Rich Caruana. Predicting good probabilities with supervised learn- ing. In ICML, 2005.
John Platt al. Probabilistiic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 1999.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In CVPR, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)2211-222 2015.
Seonguk Sco, Paul Hongsuck Seo, and Bohyung Han. Learning for single-shot confidence calibra- tion in deep neural networks through stochastic inferences. In CVPR, 2019.
Carlos N Silla and Alex Freitas A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 2011.
Han-Hsing Tu and Hsuan-Tien Lin. One-sided support vector regression for multiclass cost-sensitive classification. In JCML, 2010.
Nakul Verma, Dhruv Mahajan, Sundararajan Sellamanickam, and Vinod Nair. Learning hierarchical similarity metrics. CVPR, 2012.
Hui Wu, Michele Merler, Rosario (ceda-Sosa, and John R Smith. Learning to make better mistakes: Semantics-aware visual food recognition. In ACM MM, 2016.
Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for zero-shot classification. In CVPR, 2016.
Bianca Zadrozny and Charles Elkan. Learning and making decisions when costs and probabilities are both unknown. In KDD, 2001a.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In ICML, 2001b.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass proba- bility estimates. In KDD, 2002.
Bin Zhao, Fei Li, and Eric P Xing. Large-scale category structure aware image categorization. In NeurlPS, 2011.
Zhi-Hur Zhou and Xu- Ying Liu. On multi-class cost-sensitive learning. Computational Intelligence, 2010.
10

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 VARIATION OF MISTAKE SEVERITY WITH HIERARCHY ALIGNMENT
Distribution mistakes made tieresklmogeNet
Distibution of mistakes made tirre--mageeeet
Este-4(6.85)
Alane 665-333 5(5-7)
Beta-516
Rata-110(6.00 96)
Alax 4(5.77)
EctamL5(77001
103
Alah-e 3(5.a))
Number Sampke
B017-3307.000
Norber of Samplo
Alahe- A25.930
Crocs strooy (3)
Cress Eocropo
CRM(Ouss(0.97)
CRMA)Oun 97)
300
0
Hararchica Datares
Dtanno
mistalies
INsturalec19
INcturolat39
Ecurmef 821
Alskum 001
20100)
Bess-1002 (3)
Alpooun 03)
20000
Ecue-1522 (7)
12000
Alpha-0 3(2
Nurbe of Samples
10000
Ailhowo 102 30]
12000
Beta-5022 42)
Cress Emeragy2 39)
CHMM)Ous)12239
Bera-20002 ci)
10000
Croas
0000
CirM)(Oam)/2.34)
2000
2000
Imppen
Hecrrisca Disconc
Recrriscal Disonce
Figure 4: Mistake distribution for all values of Softlabels across 3 (left) and HXE across (right) as well baselines including cross-entropy and CRM for reference across tieredImagenet-H (top) and iNaturalist- 19 (bottom).
We first analyze the change of the distribution of mistakes across Softlabels (left column) and HXE (right column) models in Figure 4 with results in ticredImageNet-H (top row) and iNaturalist-H (bottom row). As we try align the model better with the hierarchy by decreasing 3 and increas- ing a, we observe the same trends with better alignment to hierarchy their mistakes are equally bad compared to cross-ent cnnsopy near the right end (high severity), and they increasingly make more mistakes in the left end (lower severity), lowering the average over mistakes but not always making better mistakes. This gives evidence that the models have the tendency to decrease their mistake severity (shown in the legend) by largely making lots of additional mistakes and not making better mistakes in the large and small a regimes.
A.2 VARIATION OF CLASS RANKING WITH HYPERPARAMETERS
We similarly analyze the variation of ranking classes measured by average hierarchical distance@k for Softlabels and HXE with different hyperparameters. We present our results in Figure 5, by varying Softlabel with different values on the left and HXE with different a values on the right. We observe that the previously chosen values 3 4 and 0.6 perform the best in ranking in all cases except = 4 in iNaturalist where = 10 performs the best, which we updated Figure 3. We observe there that CRM still outperforms these methods by large margins.
11

---

Published as a conference paper at ICLR 2021
Heraeciial hiee-lmapylee
Hanedical vorying
Mereeraa Denrerch
Cruedntis
Suh Lifefiizee
IACAla 04
21
10
20
Heerarchica Distancelkk verying INsturalst19
Haorcicd D Datarcafk for varying iNeeamllen1
2.5
Tre Micerte Ditenoo
25
(RM(0an)
2.8
1.0
1.0
(a) Ranking performance on the original hierarchy
Heerectical Detencalli
Hanneiccl vorying tinee--mmppket
Hove of Ontereed
Conu Enropge
Crow Brtns
IXLAge 41
Suh 11
KCEA)
Lawather 22
(RM)(D
Hearchical D stanceSk for varing on Naturalist1
Hibrrrhical Distonolts varying on Naturalszto
brg Mo Didandds
10E4)4um))
25-
35
(b) Ranking performance on a randomly class-shuffled hierarchy.
Figure 5: Average hierarchical distance @ l with varying k across different hyperparameters of Soft- labels (left) and HXE (right). We observe our selected hyperparameters (alpha=0.6) and (beta=4) perform the best among others.
A.3 VARIATION OF CALIBRATION ACROSS HIERARCHY
We can additionally calculate ECE across levels in the hierarchy by sequentially shrinking leaf nodes from the maximum depth (corresponding flat classification). The ECE depth i (from root
12

---

Published as a conference paper at ICLR 2021
plat showing the change in ECE across difffrrnt levels in the tree
10
ECE for every level on INatura
10"
ICEE or level
EGE th lovel
30"
LatebiBotaw))
Soft LatekiBet--21
1009 pham0:5
10
7
Tree
Figure 6: Calibration across various levels in the hierarchy for different models
assigned depth 0) is defined as obtaining probabilities for nodes with most depth which are at most at level i. Their probabilities obtained by summing up probabilities of their children nodes (level i). We present the results Figure 6, where we observe that overall the calibration increases you go up the hierarchy. The cross-entropy baseline shows consistent decreasing trend, but the other models have aberrations where the calibration error increases going up the hierarchy, especially the models with high calibration errors.
A.4 INCREASING BETA FOR SOFT-LABELS INATURALIST
We also experiment with increasing for Soft-labels on the Natturalist99 dataset by trying the values of 50, 75, 100 and 200 respectively. These results are shown in Table 2. We can see that increasing Beta does not significantly improve top-1 accuracy while worsening the hierarchical distance met- rics. We additionally observe this in Figure 2, where as we increase the graph shoots up with little leftward shift.
Model
Accuracy
Hier. Distance@)
Hier. Distance
Hier. Distance @ 20
(Bttllabel(Beet=300)
0.5819 0.001
0.976 0.006
1.55411.006
2.26710.004
Soft-labels(Beta-5))
0.5880 + 0.003
1.003 0.01
1.879 0.003
2.823++0.011
(etta75)
0.8876+0.002
0.997=00005
1.916:10.005
2.909+1.017
Doft-labels(Beta-tt
0.585+0.001
1.01
1.926=00.005
2.93=0.014
Soft-labelsBett=2000)
0.507710.004
0.999=000009
1.924-10.005 1.836-10.003
2.935110.012
Cross-Entropy
0.5962+0.002
0.96:10.004
2.841110.01
Table 2: Increasing Beta for Soft-labels on 1Naturalist19 beyond 30 does not increase accuracy while significantly worsening the hierarchical distance metrics.
13


Reviews:
[{"review_id": "e9K6CxkXgn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe authors propose a model to improve the output distribution of neural nets in image classification problems. Their model is a post hoc procedure and is based on the tree structure of WordNet. The model revises the classifier output based on the distance of the labels in the tree. Intuitively, their solution is to pick the candidate label that is located in the region of the tree with a higher accumulated probability mass value. They also experimentally show that the previous evaluation metrics are inconclusive. \n\nPros:\n- The authors provide a different perspective on the evaluation procedure of the previous studies and experimentally show that it was incomplete. This is an important finding.\n- Their experiments are thorough.\n\nCons:\n- The article lacks enough novelty: The problem has been investigated before. The solution is not novel. The WordNet tree structure has been extensively used in the information retrieval community before.\n- The article is not written well: There are informal vocabulary in the paper (e.g., “something similar” or “grossly miscalibrated”). There are also typos (e.g., see the paragraph before Theorem 1). In Section 3 it is not formally stated that the tree structure is derived from WordNet (the authors mention this in Abstract section). In Section 4 the baselines are not cited!", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z2JmBvqm_d", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use conditional risk minimization (CRM) for hierarchy aware classification. The proposed method simply amends mistakes using a cost matrix with the lowest common ancestor information. The method outperforms SOTA deep hierarchy-aware classifiers by large margins at ranking classes with little loss in classification accuracy.\n\nAs the authors mentioned, CRM was already proposed several decades ago, so the novelty of this paper is limited. However, the paper does demonstrate the power of this old technique when equipped with modern deep learning tools. Also, the simplicity and intuition are much appreciated.\n\nI myself am not an expert in image classification. However, for text classification, recent studies (e.g., [1]) shows that using binary cross-entropy (i.e., viewing a multi-label classification problem as L binary classification task, where L is the number of classes) can achieve higher performance than multi-label cross-entropy. In this case, it is possible that more than one label can have p(y|x) > 0.5. I wonder whether your approach is still effective for models using binary cross-entropy. It would also be better if the authors can show some experimental results on hierarchical text classification.\n\n[1] Liu et al. Deep Learning for Extreme Multi-label Text Classification. SIGIR'17.", "extra_scores": {}, "extra_text": {}}, {"review_id": "O4LEe0Vuiet", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of hierarchy-aware classification, which utilizes a hierarchy to specify certain mistakes as being worse than others. They make two main contributions. \n\nFirst, they claim that the metric used in prior work, average mistake severity, is flawed because it rewards methods that make many \"easy\" mistakes, as opposed to fewer, \"harder\" mistakes. Based on this analysis, they claim that no prior methods actually improve over the simplest baseline of cross-entropy (i.e. doing nothing).\n\nSecond, they introduce their own method, an adaptation of the classical CRM framework. They argue that CRM:\n1) Is the only method to improve over cross-entropy under average mistake severity (the metric from prior work they argue is flawed)\n2) Beats all other metrics under their new, improved metric\n3) Improves on the calibration of the predictions\n\nStrengths:\nThe method is very simple and easy to understand, builds upon existing work, and I could probably implement it from scratch in 30 minutes on top of my existing models. While ML reviewers very frequently about a lack of complexity, I think this is a great strength.\n\nThe methods section is very clearly written, and it was quite easy to quickly understand what the method is doing.\n\nWeaknesses:\nI am very concerned about the experiment sections. \n\n1)\nTo my understanding, Figure 2/Section 4.1 are factually incorrect. In particular, it appears that the soft-labels technique does essentially the same, or better than, CRM, across all fronts. In detail,\na) \nIn Figure 2(a), the leftmost softlabel point is equal to or better than CRM (and cross-entropy)\n\nb) \nFigure 2(b) really concerns me, as it appears that the hyperparameters have been chosen to make a fairly narrow point - that it is possible to have low average hierarchical distance, and high top-1 error. I agree that that indicates a problem with the metric. \n\nHowever, the authors make the fair broader claim that CRM is the only method which beats cross-entropy, which I do not think is justified. Looking at Figure 4 in A.1, it is readily apparent that choosing different hyperparameters for existing methods would yield similar error distributions to CRM. Having these plots in the appendix, combined with claims that the authors chose the best hyperparameters, feels a bit misleading.\n\nc)\nAs in 1), soft labels is essentially on top of CRM and Cross entropy (for iNaturalist19, it looks like a higher beta value would be directly on top, it's unclear why the authors did not extend the curve further)\n\n2)\nThese results, at first blush, seem fairly impressive. For the leftmost plots, I am concerned that the authors are using subpar hyperparameters, similarly to 1)(b) above. Strangely, in this instance the results for other hyperparameters are not included in the appendix. The remaining experiments are fairly convincing, though.\n\nReccomendation\nI do not think this paper can be accepted in its current form. While I suspect that CRM is a good method that I would like to use, some of the core arguments (Figure 2/Section 4.1) in the paper appear to be fatally flawed.\n\nSmaller notes:\n- The paper could use an additional proofread, as there are often odd phrasings. I found the experiment section particularly hard to follow\n- The acronym HXE is never defined, or linked to a citation", "extra_scores": {}, "extra_text": {}}, {"review_id": "4DJy7q1xmwO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses hierarchical classification, where the classes live in a hierarchy, and the cost of a mistake is the tree distance between the nodes.\n\nThe paper tests the latest cool algorithms for hierarchical-aware loss functions, versus a very old idea: CRM. In CRM, you make your best estimate of the posterior probability of a class y given input x P(y|x), and then you make a final decision based on minimizing the expected loss.\n\nThere seems to be a belief that modifying the loss function to be hierarchy-aware is clearly better than doing boring old CRM. But there is not much evidence in favor of that hypothesis. This paper offers negative evidence for that hypothesis, with two experiments:\n\n1.  By comparing hierarchical loss to top-1 loss with modified loss functions, there is a tradeoff, and there does not seem to be an advantage in using the modified loss function.\n2.  For the top-k case, using CRM clearly dominates the proposals for modifying the loss function.\n\nThese support the use of CRM.\n\nI find this paper to be really nice -- I'd far rather have a paper with good experiments with known algorithms, where I can learn something useful; than a paper with a new algorithm with somewhat useless experiments. So I would argue for acceptance.\n\nOne thing for the authors to think about:\n\nWhen they test the calibration of the modified loss functions, they find them to be poorly calibrated. This is not surprising, since the modified loss functions are not proper scoring rules. They attempt to calibrate by using a softmax  with variable T. Wouldn't it make more sense to train exp(alpha_i x + beta_i) / \\sum_{i=1}^N exp(alpha_i x + beta_i) ? that is, a gain and offset for all classes after the first one?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The approach explore the use of Conditional Risk Minimization (CRM) as a post-hoc operation to amend a classifier decision by averaging a prior class hierarchy. The authors show that it is beneficial for ranking predictions without sacrifying top-1 accuracy.

The rebuttal period clarified some reviewers' concern on paper presentation and experiments, and all reviewers recommend acceptance after the discussion period.

Although the approach is simple and directly revisits the use of CRM for deep models, the AC considers that the contribution is meaningful, and that the proposed method provides predictions with good ranking and calibration properties. The paper also sheds light into interesting issues in state-of-the-art methods integrating class hierarchies during training.
The AC therefore recommends paper acceptance.

Decision: Accept (Poster)