Cost-Sensitive Hierarchical Classification through Layer-wise Abstentions

1 INTRODUCTION . In many real-world machine learning tasks , the class space has a tree-structured taxonomy , e.g . the organization of species . Biologists have built datasets that include labels ranging from coarse-level domains to the most fine-grained species ( Van Horn et al. , 2018 ) . Similar hierarchical structures also exist in the classification of cells ( Keren et al. , 2018 ) , objects ( Yang et al. , 2015 ) and remote sensing images ( Li et al. , 2017 ) . When there is a label taxonomy , the ground-truth labels correspond to the leaves of the tree and predicting these labels is known as fine-grained classification . In general , we can predict at any level of the tree , and it is natural to incorporate a cost of such predictions . For example , given an image of “ owl ” , classifying it as either “ bird ” or “ animal ” are both correct , but the cost associated with them should be different , since the predicted labels have different levels of specificity . In addition , hard examples are generally easier to distinguish at a coarser level , e.g . in animal mimicry in Fig . 1 ( a ) . In many practical settings , the user is able to specify the cost of different types of errors in a classification problem . The cost can be formulated as an asymmetric matrix with the columns as the labels at the leaves and the rows as the predictions at any level.If we have a label taxonomy , as in Fig . 1 ( a ) , an example of the cost matrix is in Fig . 1 ( b ) . Cost-sensitive loss is hard to optimize since it is non-smooth and non-convex . Previous works usually rely on its approximation using convex surrogates losses ( Zhou & Liu , 2005 ; Tu & Lin , 2010 ; Chung et al. , 2016 ; Liu et al. , 2019 ; Khan et al. , 2017 ) . However , in hierarchical classification , a convex surrogate for the entire taxonomy is not easy to find . Previous work on hierarchical classification usually takes a top-down strategy to decouple the problem into O ( n ) binary classification problems for a n node tree ( Fan et al. , 2015 ; Charuvaka & Rangwala , 2015 ; Chen & Warren , 2013 ) . However , there is no guaranteed way to optimize the resulting cost-sensitive loss . The most relevant work that deals with both cost-sensitive and hierarchical learning takes a reward maximization perspective and Details can be found in Sec . 3 . ( b ) The corresponding cost-sensitive loss Ch on the full hierarchy . uses a post-training heuristic to find the desired trade-off between accuracy and specificity , given an overall accuracy ( Deng et al. , 2012 ) . However , it requires an accurate estimation of probabilities ( confidence levels ) for the leaf nodes , which is not possible in standard deep neural networks . This is because DNNs tend to be overconfident even when they are wrong ( Guo et al. , 2017 ) . Distributionally robust learning framework can be incorporated to optimize a given cost-sensitive loss directly ( Asif et al. , 2015 ) , which involves solving a linear programming problem at each inference step . However , this scales poorly as the number of classes grows since the time of solving linear programs numerically scales polynomially with the number of classes ( Spielman & Teng , 2004 ) . Instead of using a numerical solution , analytical solution to the linear program is available for certain special classes including the abstaining losses when the loss value is no greater than 0.5 ( assuming loss is in [ 0 , 1 ] ) ( Fathony et al. , 2016 ; 2018 ) . Abstaining means that the predictor rejects to predict any of the classes and instead pays a given abstaining cost , which is typically not higher than the cost of wrong prediction . However , these special cases of cost sensitive losses do not incorporate a hierarchy . Moreover , most of the previous methods only discuss the case of low-dimensional linear features and do not incorporate deep neural networks ( Asif et al. , 2015 ; Fathony et al. , 2016 ; 2018 ) . Our approach : In this paper , we take a bottom-up approach to break down the hierarchical costsensitive loss into the abstaining losses at each layer . Here , abstaining means that the predictor decides not to predict the leaf nodes and instead explores predicting labels at higher levels of the tree . The key insight is that each mistake made by the predictor is due to a series of abstaining and prediction decisions along the hierarchy , which results in a cumulative cost . Fig . 1 ( a ) demonstrates learning-to-abstain losses in each layer that are equivalent to the hierarchical cost-sensitive loss in Fig . 1 ( b ) . The corresponding prediction mechanism is shown in Fig . 2 ( b ) , where a prediction is first made on the leaves and the abstained data is evaluated on its parents in the tree . We then solve the learning-to-abstain problems in each layer using a distributionally robust learning ( DRL ) method . Applying DRL to each layer of the tree involves retraining of the neural network for different number of classes . In practice , retraining is efficient since only the heads are replaced and the rest of the representation is shared . Therefore , we train the higher-level models with a different head and parameters from the lower-level ones as the initialization ( Fig . 2 ( a ) ) . Our contribution can be summarized as follows : 1 . We propose to decompose the cost-sensitive hierarchical classification problem into learning-to- abstain sub-problems at each layer . We show the bijective correspondence when the cost-sensitive losses satisfy symmetry conditions ( Sec . 3.1 ) . 2 . We propose to solve the learning-to-abstain problems using a distributionally robust learning ( DRL ) approach that directly minimizes the abstaining loss at each of the layers . Our method also allows for efficient training of the model in different layers . We call our method Layer-wise Abstaining Loss Minimization method ( LAM ) . 3 . We conduct experiments on large-scale bird recognition with 244 nodes ( iNat2017 ( Van Horn et al. , 2018 ) ) and cell classification tasks ( Keren et al. , 2018 ) . Our results demonstrate that LAM achieves lower hierarchical cost-sensitive loss in high accuracy regions , compared to previous methods , even though it is not directly optimizing this loss . This is because the regime of high accuracy corresponds to low abstaining costs and our method is proven to correspond to the original hierarchical loss in this regime . For each layer , we can also achieve higher accuracy when the overall accuracy is kept fixed across different methods . 4 . Our method is also flexible enough to achieve a user-defined performance profile efficiently . We provide a heuristic based on per-class loss adjustment to show the potential of our method in achieving arbitrary performance profiles . 2 BACKGROUND . Hierarchical Classification Given input x 2 X and labels y 2 Y , we are interested in learning a classifier f ( x ) : x ! y . We know that y has a taxonomy as a tree structure , which means each x can be classified as a set of correct labels , from fine-grained to coarse labels . Assume the most fine-grained class among all the correct labels for x is y⇤ ( x ) , then as long as f ( x ) 2 P ( y⇤ ( x ) ) , the prediction is considered as correct , where P ( y⇤ ( x ) ) represents the “ path ” from the leaf node y⇤ ( x ) to the root of the tree . The classifier predicts a coarse label by abstaining from the more fine-grained leaf nodes . However , not all correct labels provide the same correctness . The user defines a cost matrix C that represents the importance of the different types of errors and abstentions when evaluating the performance of f ( x ) . Assume we have m most fine-grained level classes and n total labels in the tree . The matrix has m columns and n rows . The ( i , j ) -th entry C ( i , j ) in the cost matrix represents the cost the classifier suffers when the ground-truth label of x is j but the prediction is i. Cost-sensitive Learning Assume our training data is drawn from a data distribution P ( x , y ) , cost-sensitive learning aims to find a classifier that minimizes the following expected loss function : f = arg minE ( x , y ) ⇠P ( x , y ) [ C ( y , f ( x ) ) ] . Under the framework of empirical risk minimization , the expectation is approximated using finite samples . Assume , we have M data points ( xi , yi ) drawn from training set Dtrain , then the problem is converted to : f = arg min 1M P ( xi , yi ) ⇠Dtrain C ( yi , f ( xi ) ) . Conventional machine learning usually approximates the expected cost-sensitive loss using a convex surrogate . Different losses induce different classifiers . For example , the 0-1 loss is a special case of the cost-sensitive loss when there is no label taxonomy . Both log loss and hinge loss are convex surrogates of 0-1 loss . Logistic regression and support vector machines are derived by optimizing log loss and hinge loss , respectively . Learning to abstain Learning-to-abstain problems aim to minimize the abstaining losses using training data . Abstaining loss is a special case of general cost-sensitive losses . It is an asymmetric matrix with k + 1 rows and k columns if k is the total number of classes . The additional row represents the abstaining cost for each of the ground truth labels . We use Ca to denote an abstaining loss . The abstaining cost should be smaller than the cost of making a wrong prediction . Otherwise , the classifier will never abstain . Then if the first k ⇥ k block of Ca is exactly the 0-1 loss , than the entries ↵k needs to satisfy : 0 < ↵k < 1 . 2.1 DISTRIBUTIONALLY ROBUST COST-SENSITIVE CLASSIFICATION . We now introduce the background in distributionally robust learning ( DRL ) for optimizing the cost-sensitive loss . The formulation : Fathony et al . ( 2018 ) proposed a distributionally robust learning method that incorporate a minimax game to robustly optimize the expected cost-sensitive loss . Assume g ( x ) and h ( x ) are valid conditional label distributions . Formally , G : Rd 7 ! RC , { g ( x ) |x 2 X , g ( x ) 2 RC \ } ( 1 ) Here , d , C and denote the input dimension , class number , and probabilistic simplex , respectively . Instead of empirical risk minimization , the DRL approach optimizes the following : arg min G max H2⌃ Ex⇠P ( x ) [ gT ( x ) Ch ( x ) ] , ( 2 ) where g ( x ) , h ( x ) 2 RC are the conditional label distributions given an input x and G , H are the entire distribution over all the input . ⌃ is a constraint set h needs to also reside in . The idea of this formulation is that g , as a predictor player , plays a two-player adversarial risk minimization game Grünwald et al . ( 2004 ) with an adversarial player h. The predictor player minimizes the expected cost , while the adversary player maximizes the expected cost . The adversary is allowed to perturb the ground-truth labels , subject to certain feature-matching constraints to ensure data compatibility . In particular , the adversary H is constrained by the empirical training data features : ⌃ = { H| P ihy ( xi ) = P iI [ yi = y ] ( xi ) , 8y } , ( 3 ) where xi ⇠ Dtrain , hy is the y-th dimension of h and is a feature function . Eq . 3 is a necessary but not sufficient condition for H equals to the ground truth P ( Y |X ) , when P ( Y |X ) is represented using finite training data . The insights here is that given a predefined feature function , when the adversary perturbs the labels , certain aggregate function of on g should equal to the counterpart on the empirical source data . The derived prediction form : Solving Eq . 2 involves solving a two-player zero-sum game with constraints . Fathony et al . ( 2018 ) provides analytical solutions for G when the cost-sensitive loss C takes some special cases . In these cases , G would be parameterized with dual parameters ✓ . Here , we omit the derivation but show an example of the analytical solution when we incorporate Ca in Eq . 2 with ↵k 0.5 . Theorem 1 . ( Theorem 13 in ( Fathony et al. , 2018 ) ) Let ✓⇤ be the learned parameter , and f ( x ) be the potential vector for all classes where fi ( x ) = ✓⇤| i ( x ) , i 2 [ 0 , C ] . Given a new data point x , let i⇤ = arg maxi fi ( x ) ( break tie arbitrarily ) , j⇤ = arg maxj 6=i⇤ fj ( x ) , and ei⇤ 2 RC be the i⇤-th cannonical vector . Then the predictor G can be directly computed as , for each x : g ( x ) = ei⇤ 0 if fi⇤ ( x ) fj⇤ ( x ) 1 ( 4 ) g ( x ) = ( fi⇤ ( x ) fj⇤ ( x ) ) ei⇤ 1 fi⇤ ( x ) + fj⇤ ( x ) if fi⇤ ( x ) fj⇤ ( x ) < 1 . ( 5 ) The parameter learning : Learning the parameters in DRL involves computing the ( sub ) -gradients of ✓ with respect to the dual objective of Eq . 2 . In this paper , we investigate using deep neural networks as the feature function ( x ) in DRL .

Summary: The paper studys cost-sensitive hierarchical classification problems. The novelty is very limited, the experiments are vey tirial. I recommend to reject.