No Cost Likelihood Manipulation at Test Time for Making Better Mistakes in Deep Networks

1 INTRODUCTION . The conventional performance measure of accuracy for image classification treats all classes other than ground truth as equally wrong . However , some mistakes may have a much higher impact than others in real-world applications . An intuitive example being an autonomous vehicle mistaking a car for a bus is a better mistake than mistaking a car for a lamppost . Consequently , it is essential to integrate the notion of mistake severity into classifiers and one convenient way to do so is to use a taxonomic hierarchy tree of class labels , where severity is defined by a distance on the graph ( e.g. , height of the Lowest Common Ancestor ) between the ground truth and the predicted label ( Deng et al. , 2010 ; Zhao et al. , 2011 ) . This is similar to the problem of providing a good ranking of classes in a retrieval setting . Consider the case of an autonomous vehicle ranking classes for a thin , white , narrow band ( a pole , in reality ) . A top-3 prediction of { pole , lamppost , tree } would be a better prediction than { pole , person , building } . Notice that the top-k class predictions would have at least k − 1 incorrect predictions here , and the aim is to reduce the severity of these mistakes , measured by the average hierarchical distance of each of the top k predictions from the ground truth . Silla & Freitas ( 2011 ) survey classical methods leveraging class hierarchy when designing classifiers across various application domains and illustrate clear advantages over the flat hierarchy classification , especially when the labels have a well-defined hierarchy . There has been growing interest in the problem of deep hierarchy-aware image classification ( Barz & Denzler , 2019 ; Bertinetto et al. , 2020 ) . These approaches seek to leverage the class hierarchy ∗shyamgopal.karthik @ research.iiit.ac.in inherent in the large scale datasets ( e.g. , the ImageNet dataset is derived from the WordNet semantic ontology ) . Hierarchy is incorporated using either label embedding methods , hierarchical loss functions , or hierarchical architectures . We empirically found that these models indeed improve the ranking of the top-k predicted classes – ensuring that the top alternative classes are closer in the class hierarchy . However , this improvement is observed only for k > 1 . While inspecting closely the top-1 predictions of these models , we observe that instead of improving the mistake severity , they simply introduce additional low-severity mistakes which in turn favours the mistake-severity metric proposed in ( Bertinetto et al. , 2020 ) . This metric involves division by the number of misclassified samples , therefore , in many situations ( discussed in the paper ) , it can prefer a model making additional low-severity mistakes over the one that does not make such mistakes . This is at odds with the intuitive notion of making better mistakes . These additional low-severity mistakes can also explain the significant drop in their top-1 accuracy compared to the vanilla crossentropy model . We also find these models to be highly miscalibrated which further limits their practical usability . In this work we explore a different direction for hierarchy-aware classification where we amend mistake severity at test time by making post-hoc corrections over the class likelihoods ( e.g. , softmax in the case of deep neural networks ) . Given a label hierarchy , we perform such amendments to the likelihood by applying the very well-known and classical approach called Conditional Risk Minimization ( CRM ) . We found that CRM outperforms state-of-the-art deep hierarchy-aware classifiers by large margins at ranking classes with little loss in the classification accuracy . As opposed to other recent approaches , CRM does not hurt the calibration of a model as the cross-entropy likelihoods can still be used for the same . CRM is simple , requires addition of just a few lines of code to the standard cross-entropy model , does not require retraining of a network , and contains no hyperparameters whatsoever . We would like to emphasize that we do not claim any algorithmic novelty as CRM has been well explored in the literature ( Duda & Hart , 1973 , Ch . 2 ) . Almost a decade ago , Deng et al . ( 2010 ) had proposed a very similar solution using Support Vector Machine ( SVM ) classifier applied on handcrafted features . However , this did not result in practically useful performance because of the lack of modern machine learning tools at that time . We intend to bring this old , simple , and extremely effective approach back into the attention before we delve deeper into the sophisticated ones requiring expensive retraining of large neural networks and designing complex loss functions . Overall , our investigation into the hierarchy-aware classification makes the following contributions : • We highlight a shortcoming in one of the metrics proposed to evaluate hierarchy-aware classification and show that it can easily be fooled and give the wrong impression of making better mistakes . • We revisit an old post-hoc correction technique ( CRM ) which significantly outperforms prior art when the ranking of the predictions made by the model are considered . • We also investigate the reliability of prior art in terms of calibration and show that these methods are severely miscalibrated , limiting their practical usefulness . 2 RELATED WORKS . 2.1 COST-SENSITIVE CLASSIFICATION . Cost-sensitive classification assigns varying costs to different types of misclassification errors . The work by Abe et al . ( 2004 ) groups cost-sensitive classifiers into three main categories . The first category specifically extends one particular classification model to be cost-sensitive , such as support vector machines ( Tu & Lin , 2010 ) or decision trees ( Lomax & Vadera , 2013 ) . The second category makes the training procedure cost-sensitive , which is typically achieved by assigning the training examples of different classes with different weights ( rescaling ) ( Zhou & Liu , 2010 ) or by changing the proportions of each class while training using sampling ( rebalancing ) ( Elkan , 2001 ) . The third category makes the prediction procedure cost-sensitive ( Domingos , 1999 ; Zadrozny & Elkan , 2001a ) . Such direct cost-sensitive decision-making is the most generic : it considers the underlying classifier as a black box and extends to any number of classes and arbitrary cost matrices . Our work comes under the third category of post-hoc amendment . We study cost-sensitive classification in a large scale setting ( e.g. , ImageNet ) and explore the use of a taxonomic hierarchy to obtain the misclassification costs . 2.2 HIERARCHY AWARE CLASSIFICATION . There is a rich literature around exploiting hierarchies to improve the task of image classification . Embedding-based methods define each class as a soft embedding vector , instead of the typical onehot . DeViSE ( Frome et al. , 2013 ) learn a transformation over image features to maximize the cosine similarity with their respective word2vec label embeddings . The transformation is learned using ranking loss and places the image embeddings in a semantically meaningful space . Akata et al . ( 2015 ) ; Xian et al . ( 2016 ) explore variations of text embeddings , and ranking loss frameworks . Barz & Denzler ( 2019 ) project classes on a hypersphere , such that the correlation of class embeddings equals the semantic similarity of the classes . The semantic similarity is derived from the height of the lowest common ancestor ( LCA ) in a given hierarchy tree . Another line of work directly alters the loss functions or the algorithms/architectures . Zhao et al . ( 2011 ) propose a weighted ( hierarchy-aware ) multi-class logistic regression formulation . Verma et al . ( 2012 ) optimize a context-sensitive loss to learn a separate distance metric for each node in the class taxonomy tree . Wu et al . ( 2016 ) combine losses at different hierarchies of the tree by learning separate , fully connected layers for each level post a shared feature space.Bilal et al . ( 2017 ) add branches at different depths of AlexNet architecture to fuse losses at different levels of the hierarchy . Brust & Denzler ( 2019 ) use conditional probability chains to derive a novel label encoding and a corresponding loss function . Most deep learning-based methods overlook the severity of mistakes , and the evaluation revolves around counting the top-k errors . Bertinetto et al . ( 2020 ) has revived the interest in this direction by jointly analyzing the top-k accuracies with the severity of errors . They propose two modifications to cross-entropy to better capture the hierarchy : one based on label embeddings ( Soft-labels ) and the other , which factors the cross-entropy loss into the individual terms for each of the edges in the hierarchy tree and assigns different weights to them ( Hierarchical cross-entropy or HXE ) . Our method uses models trained with vanilla cross-entropy loss and alters the decision rule to pick the class that minimizes the conditional risk where the condition is being imposed using the known class-hierarchy . On similar lines , Deng et al . ( 2010 ) study the effect of minimizing conditional risk on the mean hierarchical cost . They leverage the ImageNet hierarchy for cost and compute posteriors by fitting a sigmoid function to the SVM ’ s output or taking the percent of neighbours from a class for Nearest Neighbour classification . Our work investigates the relevance of CRM in the deep learning era and highlights the importance of looking beyond mean hierarchical costs and jointly analyzing the role of accuracy and calibration . 2.3 CALIBRATION OF DEEP NEURAL NETWORKS . Networks are said to be well-calibrated if their predicted probability estimates are representative of the true correctness likelihood . Calibrated confidence estimates are important for model interpretability and its use in downstream applications . Platt scaling ( Platt et al. , 1999 ) , Histogram binning ( Zadrozny & Elkan , 2001b ) and Isotonic regression ( Zadrozny & Elkan , 2002 ) are three common calibration methods . Although originally proposed for the SVM classifier , their variations are used in improving the calibration of neural networks ( Guo et al. , 2017 ) . Calibrated probability estimates are particularly important when cost-sensitive decisions are to be made ( Zadrozny & Elkan , 2001b ) and are often measured using Expected Calibration Error ( ECE ) and Maximum Calibration Error ( MCE ) ( Niculescu-Mizil & Caruana , 2005 ; Naeini et al. , 2015 ; Mukhoti et al. , 2020 ) . We desire models with high accuracy that have low calibration error and make less severe mistakes . However , there is often a compromise . Studies in cost-sensitive classification ( Jan et al. , 2012 ) reveal a trade-off between costs and error rates . Reliability literature aims to obtain better calibrated deep networks while retaining top-k accuracy ( Seo et al. , 2019 ) . We further observe that methods like Soft-labels or Hierarchical cross-entropy successfully minimize the average top-k hierarchical cost , but result in poorly calibrated networks . In contrast , the proposed framework retains top-k accuracy and good calibration , while significantly reducing the hierarchical cost . 3 APPROACH . The K-class classification problem comes with a training set S = { ( xi , yi ) } Ni=1 , where label yi ∈ Y = { 1 , 2 , ... , K } . The classifier is a deep neural network fθ : X → p ( Y ) parametrized by θ which maps the input samples to a probability distribution over the label space Y . The p ( y|x ) is typically derived using a softmax function on the logits obtained for an input x . Given p ( y|x ) , the network minimizes cross-entropy with the ground truth class over samples from the training set , and uses SGD to optimize θ , forming the standard hierarchy-agnostic cross-entropy baseline . The decision rule is naturally given by argmax k p ( y = k|x ) . The classical CRM framework ( Duda & Hart , 1973 ) can be adapted to image classification by taking the trained model with a given θ and incorporating the hierarchy information at deployment time . A symmetric class-relationship matrix C is created using the given hierarchy tree ( which can either be drawn from the WordNet ontology or an application specific taxonomy ) , where Ci , j is the height of the lowest common ancestor LCA ( yi , yj ) between classes i and j . The height of a node is defined as the number of edges between the given node and the furthest leaf . Ci , j is zero when i = j and is bounded by the maximum height of the hierarchy tree . Given an input x , the likelihood p ( y|x ) is obtained by passing the sample through the network fθ ( x ) . The only modification we make is in the decision rule , which now selects the class that minimizes the conditional risk R ( y = k|x ) , given by : argmin k R ( y = k|x ) = argmin k K∑ j=1 Ck , j · p ( y = j|x ) ( 1 ) For the ease of the reader , we illustrate a four-class example in Figure 1a , comparing predictions obtained using the standard cross-entropy baseline ( leaf nodes ) , and the prediction using CRM ( Eq . ( 1 ) ) for a given class-relationship matrix . Given the probability of each class p ( y|x ) , argminR ( y|x ) is the Bayes optimal prediction . It is guaranteed to achieve the lowest possible overall cost , i.e . lowest expected cost over all possible examples weighted by their probabilities ( Duda & Hart , 1973 , Ch . 2 ) . Depending on the cost-matrix and p ( y|x ) , the top-1 prediction of the CRM applied on cross-entropy might differ from the top-1 prediction of the cross-entropy baseline . However , because of the overconfident nature of recent deep neural networks , we observe that the top-1 probability of p ( y|x ) is greater than 0.5 for significant number of test samples . Below we prove that in such situations where maxp ( y|x ) is higher than the sum of other probabilities , the post-hoc correction ( CRM ) does not change the top-1 prediction irrespective of the structure of the tree . Since the second highest probability is guaranteed to be less than 0.5 by definition , our correction can effectively re-rank the classes . Experimentally we find it to significantly reduce the hierarchical distance @ k. Theorem 1 . If max ( p ( y|x ) ) > 0.5 , then argmini ∑K j=1 Ci , j · p ( y = j|x ) and argmax p ( y|x ) are identical irrespective of the tree structure and both lead to the same top-1 prediction . Proof . Consider the tree illustrated in Figure 1b ; two leaf nodes ( class labels ) i , j and the subtree ( Tij ) rooted at their Lowest Common Ancestor . Assuming the height of the LCA ( i , j ) = h and argmax p ( y|x ) = i , the risk R ( y = j|x ) = R ( j ) is given as : R ( j ) = h · p ( i ) + ∑ k∈Tij\ { i } Cj , k · p ( k ) + ∑ ∀k 6∈Tij Cj , k · p ( k ) Ignoring the cost of other nodes inside Tij , we get R ( j ) ≥ h ·p ( i ) + ∑ ∀k 6∈Tij Cj , k ·p ( k ) . Similarly , for the risk of class i : R ( i ) ≤ h · ( 1− p ( i ) ) + ∑ ∀k 6∈Tij Ci , k · p ( k ) Outside the subtree rooted at Ti , j , Ci , k = Cj , k∀k and therefore without loss of generality we can say that R ( i ) < R ( j ) , if p ( i ) > 0.5 .

Summary: The authors propose a model to improve the output distribution of neural nets in image classification problems. Their model is a post hoc procedure and is based on the tree structure of WordNet. The model revises the classifier output based on the distance of the labels in the tree. Intuitively, their solution is to pick the candidate label that is located in the region of the tree with a higher accumulated probability mass value. They also experimentally show that the previous evaluation metrics are inconclusive.