Cost-Sensitive Hierarchical Classification through Layer-wise Abstentions

1 INTRODUCTION . In many real-world machine learning tasks , the class space has a tree-structured taxonomy , e.g . the organization of species . Biologists have built datasets that include labels ranging from coarse-level domains to the most fine-grained species ( Van Horn et al. , 2018 ) . Similar hierarchical structures also exist in the classification of cells ( Keren et al. , 2018 ) , objects ( Yang et al. , 2015 ) and remote sensing images ( Li et al. , 2017 ) . When there is a label taxonomy , the ground-truth labels correspond to the leaves of the tree and predicting these labels is known as fine-grained classification . In general , we can predict at any level of the tree , and it is natural to incorporate a cost of such predictions . For example , given an image of “ owl ” , classifying it as either “ bird ” or “ animal ” are both correct , but the cost associated with them should be different , since the predicted labels have different levels of specificity . In addition , hard examples are generally easier to distinguish at a coarser level , e.g . in animal mimicry in Fig . 1 ( a ) . In many practical settings , the user is able to specify the cost of different types of errors in a classification problem . The cost can be formulated as an asymmetric matrix with the columns as the labels at the leaves and the rows as the predictions at any level.If we have a label taxonomy , as in Fig . 1 ( a ) , an example of the cost matrix is in Fig . 1 ( b ) . Cost-sensitive loss is hard to optimize since it is non-smooth and non-convex . Previous works usually rely on its approximation using convex surrogates losses ( Zhou & Liu , 2005 ; Tu & Lin , 2010 ; Chung et al. , 2016 ; Liu et al. , 2019 ; Khan et al. , 2017 ) . However , in hierarchical classification , a convex surrogate for the entire taxonomy is not easy to find . Previous work on hierarchical classification usually takes a top-down strategy to decouple the problem into O ( n ) binary classification problems for a n node tree ( Fan et al. , 2015 ; Charuvaka & Rangwala , 2015 ; Chen & Warren , 2013 ) . However , there is no guaranteed way to optimize the resulting cost-sensitive loss . The most relevant work that deals with both cost-sensitive and hierarchical learning takes a reward maximization perspective and Details can be found in Sec . 3 . ( b ) The corresponding cost-sensitive loss Ch on the full hierarchy . uses a post-training heuristic to find the desired trade-off between accuracy and specificity , given an overall accuracy ( Deng et al. , 2012 ) . However , it requires an accurate estimation of probabilities ( confidence levels ) for the leaf nodes , which is not possible in standard deep neural networks . This is because DNNs tend to be overconfident even when they are wrong ( Guo et al. , 2017 ) . Distributionally robust learning framework can be incorporated to optimize a given cost-sensitive loss directly ( Asif et al. , 2015 ) , which involves solving a linear programming problem at each inference step . However , this scales poorly as the number of classes grows since the time of solving linear programs numerically scales polynomially with the number of classes ( Spielman & Teng , 2004 ) . Instead of using a numerical solution , analytical solution to the linear program is available for certain special classes including the abstaining losses when the loss value is no greater than 0.5 ( assuming loss is in [ 0 , 1 ] ) ( Fathony et al. , 2016 ; 2018 ) . Abstaining means that the predictor rejects to predict any of the classes and instead pays a given abstaining cost , which is typically not higher than the cost of wrong prediction . However , these special cases of cost sensitive losses do not incorporate a hierarchy . Moreover , most of the previous methods only discuss the case of low-dimensional linear features and do not incorporate deep neural networks ( Asif et al. , 2015 ; Fathony et al. , 2016 ; 2018 ) . Our approach : In this paper , we take a bottom-up approach to break down the hierarchical costsensitive loss into the abstaining losses at each layer . Here , abstaining means that the predictor decides not to predict the leaf nodes and instead explores predicting labels at higher levels of the tree . The key insight is that each mistake made by the predictor is due to a series of abstaining and prediction decisions along the hierarchy , which results in a cumulative cost . Fig . 1 ( a ) demonstrates learning-to-abstain losses in each layer that are equivalent to the hierarchical cost-sensitive loss in Fig . 1 ( b ) . The corresponding prediction mechanism is shown in Fig . 2 ( b ) , where a prediction is first made on the leaves and the abstained data is evaluated on its parents in the tree . We then solve the learning-to-abstain problems in each layer using a distributionally robust learning ( DRL ) method . Applying DRL to each layer of the tree involves retraining of the neural network for different number of classes . In practice , retraining is efficient since only the heads are replaced and the rest of the representation is shared . Therefore , we train the higher-level models with a different head and parameters from the lower-level ones as the initialization ( Fig . 2 ( a ) ) . Our contribution can be summarized as follows : 1 . We propose to decompose the cost-sensitive hierarchical classification problem into learning-to- abstain sub-problems at each layer . We show the bijective correspondence when the cost-sensitive losses satisfy symmetry conditions ( Sec . 3.1 ) . 2 . We propose to solve the learning-to-abstain problems using a distributionally robust learning ( DRL ) approach that directly minimizes the abstaining loss at each of the layers . Our method also allows for efficient training of the model in different layers . We call our method Layer-wise Abstaining Loss Minimization method ( LAM ) . 3 . We conduct experiments on large-scale bird recognition with 244 nodes ( iNat2017 ( Van Horn et al. , 2018 ) ) and cell classification tasks ( Keren et al. , 2018 ) . Our results demonstrate that LAM achieves lower hierarchical cost-sensitive loss in high accuracy regions , compared to previous methods , even though it is not directly optimizing this loss . This is because the regime of high accuracy corresponds to low abstaining costs and our method is proven to correspond to the original hierarchical loss in this regime . For each layer , we can also achieve higher accuracy when the overall accuracy is kept fixed across different methods . 4 . Our method is also flexible enough to achieve a user-defined performance profile efficiently . We provide a heuristic based on per-class loss adjustment to show the potential of our method in achieving arbitrary performance profiles . 2 BACKGROUND . Hierarchical Classification Given input x 2 X and labels y 2 Y , we are interested in learning a classifier f ( x ) : x ! y . We know that y has a taxonomy as a tree structure , which means each x can be classified as a set of correct labels , from fine-grained to coarse labels . Assume the most fine-grained class among all the correct labels for x is y⇤ ( x ) , then as long as f ( x ) 2 P ( y⇤ ( x ) ) , the prediction is considered as correct , where P ( y⇤ ( x ) ) represents the “ path ” from the leaf node y⇤ ( x ) to the root of the tree . The classifier predicts a coarse label by abstaining from the more fine-grained leaf nodes . However , not all correct labels provide the same correctness . The user defines a cost matrix C that represents the importance of the different types of errors and abstentions when evaluating the performance of f ( x ) . Assume we have m most fine-grained level classes and n total labels in the tree . The matrix has m columns and n rows . The ( i , j ) -th entry C ( i , j ) in the cost matrix represents the cost the classifier suffers when the ground-truth label of x is j but the prediction is i. Cost-sensitive Learning Assume our training data is drawn from a data distribution P ( x , y ) , cost-sensitive learning aims to find a classifier that minimizes the following expected loss function : f = arg minE ( x , y ) ⇠P ( x , y ) [ C ( y , f ( x ) ) ] . Under the framework of empirical risk minimization , the expectation is approximated using finite samples . Assume , we have M data points ( xi , yi ) drawn from training set Dtrain , then the problem is converted to : f = arg min 1M P ( xi , yi ) ⇠Dtrain C ( yi , f ( xi ) ) . Conventional machine learning usually approximates the expected cost-sensitive loss using a convex surrogate . Different losses induce different classifiers . For example , the 0-1 loss is a special case of the cost-sensitive loss when there is no label taxonomy . Both log loss and hinge loss are convex surrogates of 0-1 loss . Logistic regression and support vector machines are derived by optimizing log loss and hinge loss , respectively . Learning to abstain Learning-to-abstain problems aim to minimize the abstaining losses using training data . Abstaining loss is a special case of general cost-sensitive losses . It is an asymmetric matrix with k + 1 rows and k columns if k is the total number of classes . The additional row represents the abstaining cost for each of the ground truth labels . We use Ca to denote an abstaining loss . The abstaining cost should be smaller than the cost of making a wrong prediction . Otherwise , the classifier will never abstain . Then if the first k ⇥ k block of Ca is exactly the 0-1 loss , than the entries ↵k needs to satisfy : 0 < ↵k < 1 . 2.1 DISTRIBUTIONALLY ROBUST COST-SENSITIVE CLASSIFICATION . We now introduce the background in distributionally robust learning ( DRL ) for optimizing the cost-sensitive loss . The formulation : Fathony et al . ( 2018 ) proposed a distributionally robust learning method that incorporate a minimax game to robustly optimize the expected cost-sensitive loss . Assume g ( x ) and h ( x ) are valid conditional label distributions . Formally , G : Rd 7 ! RC , { g ( x ) |x 2 X , g ( x ) 2 RC \ } ( 1 ) Here , d , C and denote the input dimension , class number , and probabilistic simplex , respectively . Instead of empirical risk minimization , the DRL approach optimizes the following : arg min G max H2⌃ Ex⇠P ( x ) [ gT ( x ) Ch ( x ) ] , ( 2 ) where g ( x ) , h ( x ) 2 RC are the conditional label distributions given an input x and G , H are the entire distribution over all the input . ⌃ is a constraint set h needs to also reside in . The idea of this formulation is that g , as a predictor player , plays a two-player adversarial risk minimization game Grünwald et al . ( 2004 ) with an adversarial player h. The predictor player minimizes the expected cost , while the adversary player maximizes the expected cost . The adversary is allowed to perturb the ground-truth labels , subject to certain feature-matching constraints to ensure data compatibility . In particular , the adversary H is constrained by the empirical training data features : ⌃ = { H| P ihy ( xi ) = P iI [ yi = y ] ( xi ) , 8y } , ( 3 ) where xi ⇠ Dtrain , hy is the y-th dimension of h and is a feature function . Eq . 3 is a necessary but not sufficient condition for H equals to the ground truth P ( Y |X ) , when P ( Y |X ) is represented using finite training data . The insights here is that given a predefined feature function , when the adversary perturbs the labels , certain aggregate function of on g should equal to the counterpart on the empirical source data . The derived prediction form : Solving Eq . 2 involves solving a two-player zero-sum game with constraints . Fathony et al . ( 2018 ) provides analytical solutions for G when the cost-sensitive loss C takes some special cases . In these cases , G would be parameterized with dual parameters ✓ . Here , we omit the derivation but show an example of the analytical solution when we incorporate Ca in Eq . 2 with ↵k 0.5 . Theorem 1 . ( Theorem 13 in ( Fathony et al. , 2018 ) ) Let ✓⇤ be the learned parameter , and f ( x ) be the potential vector for all classes where fi ( x ) = ✓⇤| i ( x ) , i 2 [ 0 , C ] . Given a new data point x , let i⇤ = arg maxi fi ( x ) ( break tie arbitrarily ) , j⇤ = arg maxj 6=i⇤ fj ( x ) , and ei⇤ 2 RC be the i⇤-th cannonical vector . Then the predictor G can be directly computed as , for each x : g ( x ) = ei⇤ 0 if fi⇤ ( x ) fj⇤ ( x ) 1 ( 4 ) g ( x ) = ( fi⇤ ( x ) fj⇤ ( x ) ) ei⇤ 1 fi⇤ ( x ) + fj⇤ ( x ) if fi⇤ ( x ) fj⇤ ( x ) < 1 . ( 5 ) The parameter learning : Learning the parameters in DRL involves computing the ( sub ) -gradients of ✓ with respect to the dual objective of Eq . 2 . In this paper , we investigate using deep neural networks as the feature function ( x ) in DRL .

Summary: The authors propose a new framework for cost-sensitive hierarchical classification. First, they decompose it into level by level learning to abstain (with different abstain costs per class) sub-problems. To solve these subproblems, authors apply deep distributionally robust learning (DRL) approach that directly minimizes the abstaining loss (based on Fathony et al. 2018). These two elements create a method named the Layer-wise Abstaining Loss Minimization method (LAM). The proposed method is compared with DARTS on two datasets and achieves attractive performance. The authors also demonstrate that this decomposition makes it easier to achieve the desired performance profile by adjusting abstaining losses of the layers.