Cost-Sensitive Hierarchical Classification through Layer-wise Abstentions
Authors: Alycia Lee
Anthony L Pineci
Uriah Israel
Omer Bar-Tal
Leeat Keren
David A. Van Valen
Anima Anandkumar
Yisong Yue
Anqi Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: cost-sensitive learning
hierarchical classification
learning to abstain

Abstract: We study the problem of cost-sensitive hierarchical classification where a label taxonomy has a cost-sensitive loss associated with it, which represents the cost of (wrong) predictions at different levels of the hierarchy. Directly optimizing the cost-sensitive hierarchical loss is hard, due to its non-convexity, especially when the size of the taxonomy is large. In this paper, we propose a \textbf{L}ayer-wise \textbf{A}bstaining Loss \textbf{M}inimization method (LAM),  a tractable method that breaks the hierarchical learning problem into layer-by-layer learning-to-abstain sub-problems.  We prove that there is a bijective mapping between the original hierarchical cost-sensitive loss and the set of layer-wise abstaining losses under symmetry assumptions. We employ the distributionally robust learning framework to solve the learning-to-abstain problems in each layer. We conduct experiments on large-scale bird dataset as well as on cell classification problems. Our results demonstrate that LAM achieves a lower hierarchical cost-sensitive loss in high accuracy regions, compared to previous methods and their modified versions for a fair comparison, even though they are not directly optimizing this loss. For each layer, we also achieve higher accuracy when the overall accuracy is kept fixed across different methods. Furthermore, we also show the flexibility of LAM by proposing a per-class loss-adjustment heuristic to achieve a performance profile. This can be used for cost design to translate user requirements into optimizable cost functions.

Under review as a conference paper at ICLR 2022
COST-SENSITIVE HIERARCHICAL CLASSIFICATION THROUGH LAYER-WISE ABSTENTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
We study the problem of cost-sensitive hierarchical classification where a label taxonomy has cost-sensitive loss associated with it, which represents the cost of (wrong) predictions at different levels of the hierarchy. Directly optimizing the cost-sensitive hierarchical loss is hard, due to its non-convexity, especially when the size of the taxonomy is large. In this paper, we propose Layer-wise Abstaining Loss Minimization method (LAM), a tractable method that breaks the hierarchical learning problem into layer-by-layer learning- learning-to-abstain sub-problems. We prove that there is a bijective mapping between the original hierarchical cost-sensitive loss and the set of layer-wise abstaining losses under symmetry assumptions. We employ the distributionally robust learning framework to solve the learning- to-abstain problems in each layer. We conduct experiments dataset as well as on cell classification problems. Our results demonstrate that LAM achieves a lower hierarchical cost-sensitive loss in high accuracy regions, compared to previous methods and their modified versions for fair comparison, even though they are not directly optimizing this loss. For each layer, we also achieve higher accuracy when the overall accuracy is kept fixed across different methods. Furthermore, we also show the flexibility LAM by proposing a per- class loss-adjustment heuristic to achieve performance profile. This can be used for cost design to translate user requirements into optimizable cost functions.
large-scale bird
INTRODUCTION
In many real-world machine learning tasks, the class space has a tree-structured taxonomy, e.g. the organization of species. Biologists have built datasets that include labels ranging from coarse-level domains to the most fine-grained species (Van Horn et al. 2018) Similar hierarchical structures also exist in the classification of cells (Keren al. 2018). objects (Yang et al., 2015) and remote sensing images (Li et al., 2017).
When there is label taxonomy, the ground- truthh labels correspond to the leaves of the tree and predicting these labels is known as fine-grained classification. In general, we can predict at any level of the tree, and it is natural to incorporate cost of such predictions. For example, given an image of "owl", classifying it as either "bird" or "animal" are both correct, but the cost associated with them should be different, since the predicted labels have different levels of specificity. In addition, hard examples are generally easier to distinguish at coarser level, e.g. in animal mimicry in Fig. (a). In many practical settings, the user able to specify the cost of different types of errors in classification problem. The cost can be formulated as an asymmetric matrix with the columns as the
labels at the leaves and the rows as the predictions at any level.If we have a label taxonomy, in Fig.
i (a), an example of the cost matrix is in Fig. (b).
Cost-sensitive loss is hard to optimize since it is non-smooth and non-convex. Previous works usually rely on its approximation using convex surrogates losses (Zhou & Liul 2005 Tu & Lin 2010 Chung
et al. 2016 Liu et al., 2019; Khan et al., 2017). However, in hierarchical classification, convex surrogate for the entire taxonomy is not easy find. Previous work hierarchical classification usually takes a top-down strategy to decouple the problem into O(n) binary classification problems for n node tree (Fan et al. 2015 Charuvaka & Rangwala 2015 Chen & Warren, 2013). However, there is no guaranteed way to optimize the resulting cost-sensitive loss. The most relevant work that deals with both cost-sensitive and hierarchical learning takes reward maximization perspective and

---

Under review as a conference paper at ICLR 2022
Animal
0..3
0.4
0
1
Bird
1
0
2
8
1
1
0
1
1
C₁
0.5
1.5
1.5
3
0
2
1
@
1.5
0.5
0.5
Owl
Butterfly
Dragonfly
L0.5
0.5
0.5~
L0.8
0.9
0.9
(a)
(b)
Figure 1: (a) An example of hierarchical species classification on animal mimicry images. The hierarchy is example of the label taxonomy. The two loss matrices optimized in the layer-by- layer
problems the proposed method. Bold labels are the index of the nodes in the tree. The "a" in the figure represents the abstaining option each layer in our proposed method. Details can be found in Sec. (b) The corresponding cost-sensitive loss C on the full hierarchy.
uses post-training heuristic to find the desired trade-off between accuracy and specificity, given an overall accuracy (Deng et al., 2012). However, it requires accurate estimation of probabilities (confidence levels) for the leaf nodes, which is not possible standard deep neural networks. This because DNNs tend to be overconfident even when they are wrong (Guo al. 2017).
Distributionally robust learning framework can be incorporated to optimize given cost-sensitive loss directly (Asif et al. 2015), which involves solving a linear programming problem at each inference step. However, this scales poorly as the number of classes grows since the time of solving linear programs numerically scales polynomially with the number of classes (Spielman & 'eng 2004). Instead of using a numerical solution, analytical solution to the linear program is available for certain special classes including the abstaining losses when the loss value is greater than 0.5 (assuming loss is in [0. 11) (Fathony et al. 2016 2018). Abstaining means that the predictor rejects to predict any of the classes and instead pays a given abstaining cost, which is typically not higher than the cost of wrong prediction. However, these special cases of cost sensitive losses do not incorporate hierarchy. Moreover, most of the previous methods only discuss the case low-dimensional linear features and do not incorporate deep neural networks (Asif et al. 2015 Fathony et al. 2016: 2018 8).
Our approach: this paper, we take bottom-up approach to break down the hierarchical cost- sensitive loss into the abstaining losses at each layer. Here, abstaining means that the predictor decides not to predict the leaf nodes and instead explores predicting labels at higher levels of the tree. The key insight is that each mistake made by the predictor is due series of abstaining and prediction decisions along the hierarchy, which results in a cumulative cost. Fig. (a) demonstrates aaaastngto losses in each layer that equivalent the hierarchical cost-sensitive loss in Fig. (b). The corresponding prediction mechanism is shown in Fig. 2 (b). where a prediction is first made the leaves and the abstained data is evaluated its parents in the tree. We then solve the soeniing-t--abstale problems in each layer using distributionally robust learning (DRL) method. Applying DRL each layer the tree involves retraining of the neural network for different number of classes. In practice, retraining is efficient since only the heads are replaced and the rest of the representation is shared. Therefore, we train the higher-level models with a different head and parameters from the lower-level ones as the initialization (Fig. ( (a)).
Our contribution can be summarized as follows:
1. We propose to decompose the cost-sensitive hierarchical classification problem into learning-to- abstain sub-problems each layer. We show the bijective correspondence when the cost-sensitive losses satisfy symmetry conditions (Sec. 3.I..
2. We propose to solve the learning-to-abeet (DRL) approach that directly minimizes the abstaining loss at each the layers. Our method also allows for efficient training of the model in different layers. We call our method Layer-wise Abstaining Loss Minimization method (LAM).
problems using a distributionally robust learning
3. We conduct experiments large-scale bird recognition with 244 nodes (iNat2017 (Van Horn et al. 2018)) and cell classification tasks (Keren et al. (0018). Our results demonstrate that LAM achieves lower hierarchical cost-sensitive loss in high accuracy regions, compared to previous
2

---

Under review as a conference paper at ICLR 2022
Head Blaer 0)
Predicted
Test Data
Layero
Training Data
Model
Abstained
Layer
NN
Head [lave11]
Layer1 Model
Abstained
Layer
Figure 2: (a) The training mechanism of the proposed approach. For the training of the non-leaf layers, we relabel the data using the corresponding nodes and initialize the backbone models using parameters from leaf layers, except for the last layer of the neural network. (b) The prediction procedure over the hierarchy. The leaf model is first employed to make predictions on all the samples, then the models on the father layers are employed make predictions on the abstained samples collected from the previous layers.
methods, even though it is not directly optimizing this loss. This is because the regime of high accuracy corresponds to low abstaining costs and our method is proven to correspond to the original hierarchical loss in this regime. For each layer, we also achieve higher accuracy when the overall accuracy is kept fixed across different methods.
4. Our method is also flexible enough to achieve user-defined performance profile efficiently. We provide a heuristic based on per-class loss adjustment to show the potential of our method in achieving arbitrary performance profiles.
BACKGROUND
Hierarchical Classification Given input E X' and labels E V, we are interested in learning a classifier f(x) x + y. We know that y has taxonomy as tree structure, which means each can be classified as set of correct labels, from fine-grained to coarse labels. Assume the most fine-grained class among all the correct labels for æ is y (æ), then as long as f(x) E Ply (æ)). the prediction is considered correct, where Pyy (x)) represents the "path" from the leaf node y (x) to the root of the tree. The classifier predicts a coarse label by abstaining from the more fine-grad ane-grained leaf nodes. However, not all correct labels provide the same correctness. The user defines cost matrix C that represents the importance of the different types of errors and abstentions when evaluating the performance of (x)) Assume we have most fine- fine-grained level classes and n total labels in the tree. The matrix has m columns and rows. The (i,j) -th entry (ii,j) j) in the cost matrix represents the cost the classifier suffers when the ground-truth label of æ is j but the prediction
Cost-sensitive Learning Assume our training data is drawn from a data distribution P(x:, u). cost-sensitive learning aims to find a classifier that minimizes the following expected loss function: arg min E( ((m.)))((mm.))) [C(y, fx)))] Under the framework of empirical risk minimization, the expectation approximated using finite samples. Assume, we have M data points (æ,,, v:) drawn from training set Dtrain then the problem is converted to: f arg min J (o ) Puiin Cly, fxx)). Conventional machine learning usually approximates the expected cost-sensitive loss using a convex surrogate. Different losses induce different classifiers. For example, the 0-1 loss is a special case of the cost-sensitive loss when there is no label taxonomy. Both log loss and hinge loss convex surrogates 0-1 loss. Logistic regression and support vector machines are derived by optimizing log loss and hinge loss, respectively.
Learning abstain Learning-to-abbnn abbasttiin problems aim minimize the abstaining losses using training data. Abstaining loss is special case of general cost-sensitive losses. is an asymmetric matrix with & + rows and k columns if & is the total number of classes. The additional row represents the abstaining cost for each of the ground truth labels. We use C2 to denote an abstaining loss. The abstaining cost should be smaller than the cost of making wrong prediction. Otherwise, the classifier will never abstain. Then if the first x & block of C, is exactly the 0-1 loss, than the entries ag needs to satisfy: 0 < og < 1.
0
1
1
0
1
C.
1
1
0
Q22
Cra
3

---

Under review as a conference paper at ICLR 2022
2.1 DISTRIBUTIONALLY ROBUST COST-SENSI CLASSIFICATION
We now introduce the background in distributionally robust learning (DRL) for optimizing the cost-sensitive loss.
The formulation: Fathony et al. (2018) proposed a distributionally robust learning method that incorporate minimax game to robustly optimize the expected cost-sensitive loss. Assume g(x) and h (x) are valid conditional label distributions. Formally,
G
Rd + RC & {g(x)) E (x) E R n
(1)
Here, d. C and denote the input dimension, class number, and probabilistic simplex, respectively. Instead of empirical risk minimization, the DRL approach optimizes the following:
arg min max Pm) (r))xC((()))
(2)
G HEΣ
where g(x), (ax) E RC are the conditional label distributions given an input and G. H the entire distribution over all the input. Σ is a constraint set h needs to also reside in. The idea of this formulation is that g. as predictor player, plays two-player adversarial risk minimization game Grünwald et al. 2004) with an adversarial player h. The predictor player minimizes the expected cost, while the adversary player maximizes the expected cost. The adversary is allowed perturb the ground-trun truth labels, subject to certain maature-matching constraints to ensure data compatibility. In particular, the adversary H constrained by the empirical training data features:
Σ [ ((x,) E,4y y)0((;;) Vy).
where x; ~ Drrain- hg is the y-th dimension of h and Φ is a feature function. Eq. 3 is a necessary but not sufficient condition for H equals the ground truth X(Y when P(Y is represented using finite training data. The insights here that given a predefined feature function Φ, when the adversary perturbs the labels, certain aggregate function of Φ on g should equal the counterpart on the empirical source data.
The derived prediction form: Solving Eq. involves solving two-player zero-sum game with constraints. Fathony et al. (2018) provides analytical solutions for G when the cost-sensitive loss C takes some special cases. In these cases, G would be parameterized with dual parameters 6. Here, we omit the derivation but show an example of the analytical solution when we incorporate C. in Eq. 2 with ag S 0.5.
Theorem 1. (Theorem 13 in ( (Fathony et al. 2018 Let 0t be the learned parameter, and f(x) be the potential vector for all classes where f((x) = 0 Tx))), i [0, C1. Given a new data point æ, let i = arg max, (ix) (break tie arbitrarily), j = arg max (z),, and e,, E RC be the i -th cannonical vector. Then the predictor G can be directly computed as, for each x:
e,-
g(x)=
f:) (x)) 2
(4)
0
(fe- (x) fz (z) ex) 1 1- fo. (x) + fy- (x)
g(x)
if -x) <.
(5)
The parameter learning: Learning the parameters in DRL involves computing the (sub)-gradients
of 9 with respect to the dual objective of Eq. 2 In this paper, we investigate using deep neural networks as the feature function ((x) in DRL.
3 COST-SENSITIVE HIERARCHICAL CLASSIFICATION
In this section, we describe our proposed distributionally robust learning method for cost-sensitive learning in hierarchical classification. We first break the cost-sensitive loss on the full taxonomy into abstaining losses in the learning-to-abstann use deep DRL to solve the Saaming-to-abster problems. We provide details on the gradient (Sec. B.2p. Finally, we propose the algorithms for training and evaluating our cost-sensitive hierarchical classification method (Sec. 3.3
problems in each layer (Sec. B.T). We then propose to
3.1 COST-SENSITIVE LOSS AS AN ACCUMULATIVE LOSS
4

---

Under review as a conference paper at ICLR 2022
We take bottom-up approach and observe that the prediction on the taxonomy takes a series of decisions. In Fig. [ if an "owl" is classified as a "insect", it means the classifier abstains layer and then makes a wrong prediction in the "bird" vs. "insect" layer. Therefore, we can break the overall classification problem into smaller layer-by-layer learning-to-absin problems, Here we analyze whether there one-on-one correspondence between a Cn and set of C. in hierarchical classification.
Ch(5,3)
the leaf
Figure 3: An example of label taxonomy.
Given a tree taxonomy with m leaves and total number of nodes in the label space, we assume P(y) represents the "path" from the leaf node to the root of the tree. We also assume (yy) represents the index of the layers of each node. Assume the leaf nodes has D = O. Then we have the following:
Lemma 1. Assume an abstaining loss C. is associated with each of the layers with ag as the abstaining loss for each of the nodes, E [1, -
(except the root node), hierarchical cost- sensitive loss C₁ is uniquely constructed by Ch(i.j) = EkEPG)nk> ag + 1[i E P())..
Lemma 2. Given a tree laxonomy, assume the overall cost-sensitive loss is Cv. Let L denote the set of leaf nodes in the tree. D(y) is the depth of a node. There is unique C. for each layer such that Ci ENEP(YNSS Og + I(i P()), Cn satisfies the following
1. Vy E L, ifi P(y), (1,1,) < minkgP(y) (k;):)))
2. Vi & L, Ck(i,)) 11+ D())
3. Vi $ L, (i.i,) C,(i,)"), for Vj.j' satisfying i E P('';
4. Vi $ L. (i.)))) = C.(ii")), for Vj.j' satisfying i P U P(j')
Remarks: Lemma T trivial by construction. Lemma 2 can be proved easily by induction. Con- straints in lemma 2 defines the structure the cost-sensitive loss needs to possess. Without loss of generality, this lemma shows that our method is applicable a large set of cost-sensitive losses. Here we explain each of the conditions as follows:
The cost of mistake in any layer is higher than correct prediction. This naturally required in any meaningful cost-sensitive losses.
The abstaining cost is not greater than the cost of making a mistake in any layer. This makes sure that there is no harm to abstain, compared to predicting wrong label. Otherwise, no abstention will be made and all predictions and mistakes will be made in the leaf nodes.
3. If two leaf nodes share the same relative position in the tree, i.e., they are the children of the same father, there is symmetry in their cost if they are abstained on. For example, in Fig. 3 if a predictor makes abstention decisions on the leaf layer but predicts node suffers the same cost when the ground truth is node 2 or 3: (5, 3) (5, 2).
The symmetry also holds if mistake made in the non-leaf layers. Following the same logic, in Fig. B if predictor makes abstention decisions the leaf layer but predicts node 4, it suffers the same cost when the ground truth is node 3: Cn(4, = Cx,4,2) 2).
3.2 DISTRIBUTIONALLY ROBUST LEARNING FOR LEARNING TO-AASAIN PROBLEMS
We present the distributionally robust learning method for the layer-by-layer Ceaning-to-abbsie problems. We first review the subb))gradien of the the original DRL formulation. We then discuss how to incorporate deep neural networks and how parameters in the neural networks optimized. Finally, we present an algorithm for optimizing the abstaining loss in each layer in Alg. [ Parameter learning in DRL: According Fathony al. (2018). the subdifferential of the objective with respect to the parameter 0 can be fully characterized by the following. For each data sample:
8g, (x, 3) = conv hīp(()) - 1( ())(() h(x)) E H.}
(6)
where H" is the of optimal solution of the following:
hea gEa { [g 0T Σ, )))(1)) - E, T(u = 5)6((ar)} }
h = arg max min
5

---

Under review as a conference paper at ICLR 2022
In general, need solve linear program to compute h² When 0.5 in Ca. there exists analytical form of the solution such that the gradient can be computed efficiently.
Incorporation of deep neural networks: Instead of low-dimensional features, incorporate deep neural networks to learn representation ((x, 0, w) for constraining the adversarial in the DRL framework. The last layer of the network is a linear layer parameterized by 0, the other layers parameterized with w. Formally, (xx θ, w) = 8 p(x, w). Then after we update 0 using Eq. 7 and Eq. 6 the sub-gradient of <(x, w) can be computed as follows:
& (r, y) = conv{ 0: 1000(h" - (1)) (n*()))1(1))
h(x)) }
where 12y is the one-hot encoding of y. Then we use Eq. 8 to back-prop to compute gradients of w the networks. We present the algorthim Alg. 1
Algorithm 1 Training for deep DRL for opti- mizing the layer-by-layer abstaining loss
Algorithm Training and testing of LAM
Training data Dunin test data Drests cost-sensitive loss Cb. label taxonomy with T layers, ' DNN: Mo: Mr-1- 2: Decompose C₁ into T layer-by-layer ab- staining loss C C..... CT' pretrain My using Co.
Input:
1: Input: Training data, abstaining loss for this layer Ca, DNN 9T p(x, w), SGD op- timizer learning rate y, epoch number T.
2: Initialization: epoch +- 0
3: While epoch 7
4:
For each data sample (x, y)
For E --
5:
Compute h" following Eq. 7
Use pretrained Mo to initialize MI,..,, ...,M
6: Compute the gradient of Tol-
lowing 6
5:
M, = Algorithm
IC Durin):
7: Back proppapppag to compute
6:
gradients for w.
//Making predictions:
8: 0 - 0 ys (x.1);;
8:
For / E [0, 1]:
9: Update w by SGD2(7) using
9:
(Dam, Y:) = Predict (M, Diest: Ci):
derived gradients;
10:
Diest Dami:
10: epoch + epoch +1
11: Output: Trained models Mo. ... MT
11:
Output: Trained networks HT (i w).
predictions for the test data.
3.3 THE OVERALL ALGORITHM
We now present the training and prediction techniques for the whole taxonomy.
Training:
We run Alg. for each layer of the taxonomy using the corresponding Ca in that layer. However, since the only difference in the data is the relabeling of the labels using nodes in different layers, initialize the networks in the father layers using pre-trained networks in the leaf layer to achieve efficient training.
Making predictions: After the models for each of the layers trained, the prediction follows bottom-up procedure. We first use the model the leaf layer to make predictions on all the samples, the abstained samples prediction also requires the loss C, since it involves solving the game, like in Theorem [ This process goes on until we reach the root of the tree. Eventually, every data sample assigned to node in the taxonomy. We summarize the whole algorithm in Alg. 2
collected for predictions using the father layer's model. Note that the
4 EXPERIMENTS
In this section, evaluate the effectiveness of our LAM algorithm in two real-world tasks: bird classification and cell classification. We also compare one-layer LAM with previous baselines on real-world image classification datasets: CIFAR10 and SVHN.
For the bird data, we use a subset of the Aves supercategory data from iNaturalist (iNat) 2018 Competition. The Aves dataset contains images of bird species in the Aves kingdom with labels on the hierarchical taxonomy of the animal in the image: from kingdom to order, family, genus, and species (ground-truth). Thus, there five layers in the Aves data tree with the Aves kingdom the

---

Under review as a conference paper at ICLR 2022
overal
Overal Absiah Alpha
0.5
Indd Ina
Ina Ina
1.00
kod
lod
04
2.00
300
500
05
03
071
05
06
04
82
05
06
Agrs
4p12
(a) Error rate for different abstaining loss (b) Abstaining rate for different abstaining loss
Figure 4: In the learning-to-abstain to-abstain problems, the abstaining loss affects the error rate and abstaining rate. The larger the abstaining loss is, the more aggressive the classifier is, thus the larger the error rate and the smaller the abstaining rate.
root and species as leaf nodes. The iNat data was pre-split into training and validation images, where the training data exhibits long-tailed distribution across classes, and the validation set exhibits uniform distribution across classes. We include more dataset details in the appendix.
Our cell data comes from the triple-negative breast cancer (TNBC) dataset introduced in Keren et al. (2018). This data comes from multiplexed imaging, technique that quantifies the abundance of biomarkers within a sample while still capturing the spatial organization of the tissue. Table I gives the cell types in the dataset and their respec- tive counts in the training and validation sets. Note that in this data, not all the leaf nodes are in the same level. We then add dummy intermediate nodes to the taxonomy, which does not affect the cost structure. More dataset details can be found in the appendix.
Table I: Distribution of cell types Cell Type
Train
Val
Tumor
13,264
7,489
Immune-I hyppheyyss
2,658
672
Immune-Neer -entrop-eu
153
318
ummune-Myeloid
4,115
1051
Immune-Other Not-Immune
1323
598
2,930
967
CIFARIO 10 and SVHN are both non-hierarchical or one layer datasets. We use these datasets to validate advantage one-layer learning- mouming-to-abeln to-abstain problems over previous baseline Deep Gambler (Liu ct al. 2019). To be consistent, we use the same training, validation, and test split as in Liu et al. (2019).
4.1 COMPETING METHODS
We use the same abstaining loss for each layer and implement LAM models with a 00.1,0.2, 0.9] in Aves classification. In cell classification, we set (0.05, 0.1, 0.. 0.3). We compare our method with DARTS (Deng et al. 2012). DARTS uses the information gain as the reward for each node in the tree. After model is trained on the leaf nodes, hyperparameter that adjusts the balance between accuracy and specificity is tuned using binary search. Again, is not clear what cost-sensitive loss is optimized. In order make the comparison fair, implement two versions of DARTS. 1) DARTS (reward): We use the original reward but set the overall accuracy to be the same every DRL model. 2) DARTS (abstain loss): We modify the reward in DARTS using a "negative cost" so that each node in the tree is associated with a reward that is either -a for nodes in the leaf layer or summation of previous abstaining losses for non-leaf layers. Then we tune the modified version of DARTS using the same overall accuracy achieved by our method. Also, sometimes DARTS cannot achieve the exact required accuracy. We then compare to the closest similar accuracy we can find using DARTS. Note that this correspondence between abstaining losses and rewards is only possible when we use the same a for all the nodes in the same layer. We also compare our method against Deep Gamblers (DG) (Liu et al.) (219). DG minimizes abstaining by portfolio theory-in an empirical
loss function based on the doubling rate of gambling-inspiee risk minimization framework. We include more details in the appendix.
4.2 RESULTS AND ANALYSIS
How abstaining loss affects the learning-to-ale to-asstiin performance? We plot the overall error rate and abstaining rate of LAM models using different abstaining losses in the Aves classification task's 'ss leaf layer in Fig. We see that the larger the abstaining cost is, the more aggressive the classifier is, thus the larger the error rate and the lower the abstaining rate. This validates the working mechanism of the abstaining cost. Imagine the abstaining cost is closer to the cost of making wrong prediction.
7

---

Under review as a conference paper at ICLR 2022
The classifier does not gain much by rejecting to make prediction. On the other hand, if the abstaining cost is small, the classifier would favor abstentions more.
Ouneal snnobuttin arminry Ferioel
Oueraal Wesahttiin acrurtry Level
1000
fand siyho 01 (woard DMTSS (4bilan loo)
ford a (ho 044
DARTS jabstem brel
CARTT (wekrr)
8082
CARTS (ebelat) loe))
2063
300
0.50
8900
6.48
0.0
852
0.6
65
Aboclite ALCEHOCy
(a)
(b)
(c)
Figure 5: (a) (b) Cumulative non-abstained accuracy for each layer on bird classification. We compare LAM with abstaining cost 0.1, 0.4 with DARTS that achieves a similar overall accuracy; (c) The cost comparison of 5 different LAM models with their corresponding DARTS models achieving the same overall accuracy in the cell classification task.
How does the overall algorithm perform? We show the comparison with DARTS of non-abstained accuracy on the Aves classification in Fig. (a) and (b), which are the correct samples over all the predicted samples. We can see that, by achieving the same overall accuracy in the whole laxonomy, our method achieves higher non-abstained accuracy consistently in each of the layers. We also compare the cost-sensitive loss achieved by our method and the modified version of DARTS in Fig. (c). We can see that our cost is much lower in the high accuracy domain. We also show the comparison with DG of cost-sensitive loss on CIFARIO and SVHN in Fig. (a) and (b). We see that LAM (with only one-layer) achieves better cost-sensitive loss when the same accuracy is obtained. In (c) and (d), we compare cumulative overall aceuracy of equivalent LAM and DG models on Aves classification. The overall accuracy LAM achieves is significantly larger than DG at every level at which a model is trained-when we set DG's loss to be the same as LAM. Note that level 4 denotes the singular root node of the Aves hierarchy, and all inputs abstained on in level 3 are automatically deemed correct in level 4.
Can our method be used to achieve a specific performance profile? Since our method can assign different abstaining losses to each class (different ag)), we show how we can achieve a similar performance profile by using a loss-adjustment heuristic. Here, the user-defined performance profile means the tuple of overall accuracy, error rate and abstaining rate. In Fig. (d), we use the leaf layer in the Aves taxonomy and start with a 0.2 LAM. Our goal is achieve similar performance profile achieved by DARTS with overall accuracy 0.9 in the leaf layer. After four iterations of auss-adjus,ing, we achieve a very similar performance profile with DARTS.
The loss- adjjstmnjttt method is a binary search that takes as inputs the error per class profiles of the LAM model (in the first iteration, a = 0.2 LAM) and target model (DARTS accuracy 0.9). In every iteration, the LAM model is retrained using an alpha vector that is updated based on a comparison of the error per class between the LAM and target model: if, for a class k. the error rate is less than the error rate of the same class achieved by the target model, then ag for that class is increased to be (ackecureet + 1).2. An increase in corresponds to a lower abstaining and higher error rate as seen in Fig. 4. If the error rate is greater than the error rate achieved by the target model, then a is updated as (Qus,cument) 2.2. The updated a values are compiled into an updated vector retrain the LAM model on. "lookback" condition: at iteration i, if the overall accuracy, error, and abstain rates of the previous iteration are all closer in magnitude to the target rates than the current iteration, we stop and use the LAM model trained at iteration i 1 as the closest model that achieves the target profile.
Figure 6: Binary search heuristic to recover DARTS performance.
This iterative process stops via
5 RELATED WORK
Deep Cost-Sensitive Learning Traditional cost-sensitive learning works usually focus on transform- ing data samples (Domingos) 1999). changing the prediction procedure (Langford & Beygelzimer
8

---

Under review as a conference paper at ICLR 2022
0l44400 Can der di dos AOUSEE
SAVt AU3 Cort wwniiaiionns v. Mocki
Ueerii Accursy Per Love!
Lat C5 26 mos 2.08
resatt 1.01
1070
300
(a)
(c)
(d)
Figure 7: (a) (b) Average cost-sensitive loss vs. corresponding absolute accuracy on CIFARIO 10 and
SVHN. We compare LAM with abstaining costs 0.1, 0.15, 0.2, .... 0.85 with DG of equivalent reward. (c)(d) Cumulative overall accuracy for each layer on bird classification. We compare LAM with abstaining cost 0.3 and 0.4 with DG of equivalent reward.
2005 Zadrozny et al., ).03). and modifying the existing learning models to include cost-sensitiv
information (Kukar et al.) 1998 Tu & Lin, 2010 Zhou & Liu. 2005; Cao et al. 2013 Gangrade
al. 202T). "The investigation of the cost-sensitive perspective in deep learning is still limited.
Existing works usually optimize convex surrogate (Chen & Warren 2013 Chung et al. 2016 Khan
ct al. | Charoenphhhee et al, 202T). Learning to abstain can be regarded as a special case of cost-sensitive learning. There are several different strategies. Besides the naive dropout-based method
(Gal & Ghahramani] 2016), Geifman & EI- Yaniv/ (2017) 2019) proposed to change the architecture to have network for deciding abstain not, While Liu al. (2019) utilized tunable weight to
incorporate the rejection option in the loss function. In our paper, we investigate cost-sensitive loss on tree taxonomy and decompose the loss into tractable layer-by-layer abstaining losses, which can be optimized using DRL.
Deep Hierarchical Classification For learning with hierarchical classes, most of the work takes the top-down strategy (Babbar et al. 2013 2016; Zheng & Zhao 2020 Guo & Zhao, 2021). which suffers from error propagation. Others seek add an adjustment component (Fan al. 2015) or hierarchy-
related regularization (Gopal & Yang 2013| Charuvaka & Rangwala (015)) to the model to make reasonable hierarchical decisions. Deng al. (2012) is a post- proceesooss method that uses posterior probabilities obtained on validation set to tune parameters according to information-gal gain reward and target accuracy. These methods heavily depend on an accurate probabilistic classifier, which is usually not trained under using hierarchical information. Direct improvements over DARTS have included Bayesian methods learning (Jia et al. 2013) and integrating the uncertainty of predicting the parent nodes (Wang et al. 2017). well an approach that uses modified top-down and flatten methods to recognize novel classes (Lee et al. 2018). Recently, Zheng& & Zhao (202T) proposes to decompose the hierarchical problem in similar way. However, it stills uses the post-hoc thresholding method for each of the layers.
Distributionally robust learning Distributionally robust learning methods have been developed to robustly minimize the general loss functions (Fathony et al., 2016 2017] 2018). In the most general case, the minimax game is solved using linear programming, which is hard to scale large number of classes. More advanced techniques have been developed to provide analytical solutions (Fathony
al., (2188) and deal with combinatorially large size of the game (Wang et al. 2015) to optimize the general performance metric. Recently, this framework has been further developed to incorporate deep neural works for optimizing the logarithmic loss under covariate shift (Wang et al. (202T) and make general losses differetiable (Fathony & Kolter (2020). Our work studies the relation between the general cost-sensitive loss and the learning-to-atee to-asstain losses and proposes feasible solutions using the DRL framework to deal with real-world tigh-dimensional data.
6 CONCLUSION
In this paper, we propose to break the overall cost-sensitive learning problem into learning-to tobastin sub-problems each layer in the tree taxonomy. We provide a distributionally robust learning method for optimizing the abstaining losses in each layer. We show the bijective correspondence between the hierarchical cost-sensitive loss and the set of abstaining losses. We evaluate our method on world datasets from applications in bird classification and cell classification. Our method outperforms the competing methods by achieving lower cumulative cost and higher accuracy in each layer. We also provide heuristic for adjusting abstaining losses to achieve desired performance profile.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Kaiser Asif, Wei Xing, Sima Behpour, and Brian D Ziebart. Adversarial cost-sensitive classification. In Proceedings of the Thirty-First Conference Uncertainty in Artificial Intelligence, pp. 22-101, 2015.
Rohit Babbar, loannis Partalas, Eric Gaussier, and Massih-Reza Amini. On flat versus hierarchical classification in large-scale taxonomies. In 27th Annual Conference Neural Information Processing Systems (NIPS 26). pp. 1824-1832, 2013.
Rohit Babbar, loannis Partalas, Eric Gaussier, Massih- Reza Amini, and Cécile Amblard. Learning taxonomy adaptation in large-scale classification. The Journal of Machine Learning Research, 17 (1):3350-3386, 2016.
Peng Cao, Dazhe Zhao, and Osmar Zaiane. An optimized cost-sensitive svm for imbalanced data learning. In Pacific-Asia conference on knowledge discovery and data mining, pp- 280-292. Springer, 2013.
Nontawat
Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning, pp. 1507-1517. PMLR, 2021.
Anveshi Charuvaka and Huzefa Rangwala. Hiercost: Improving large scale hierarchical classification with cost sensitive learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 675-690. Springer, 2015.
Jianfu Chen and David Warren. Cost-sensitive learning for large-scale hierarchical classification. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp.
2013.
Yu-An Chung, Hsuan-Ties Tien Lin, and Shao-Wen Yang. Cost-aware pre-training for multiclass cost- sensitive deep learning. In Proceedings of the Twenty-Fift International Joint Conference on Artificial Intelligence, pp. 1411-1417, 2016.
Jia Deng, Jonathan Krause, Alexander C. Berg, and Li Fei-Fei. Hedging your bets: Optimizing accuracy-sppa -ppeciitspeeiiicityyy trade-offs in large scale visual recognition. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3450-3457, 2012. doi: 10.1 10.1109/CVPR.2011
6248086.
Pedro Domingos. Metacost: A general method for making classifiers cost-sensitive. In Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 155-164, 1999.
Jianping Fan, Ji Zhang, Kuizhi Mei, Jinye Peng, and Ling Gao. Cost-sensitive learning of hierar- chical tree classifiers for large-scale image classification and novel category detection. Pattern Recognition, 48(5):1673-1687 2015.
Rizal Fathony and Zico Kolter. Ap-perf: Incorporating generic performance metrics in differentiable learning. In International Conference on Artificial Intelligence and Statistics, pp. 4130-4140. PMLR, 2020.
Rizal Fathony, Anqi Liu, Kaiser Asif, and Brian Ziebart. Adversarial multiclass classification: A risk minimization perspective. In Advances Neurul Information Processing Systems, pp. 559-567, 2016.
Rizal Fathony, Mohammad Ali Bashiri, and Brian D Ziebart. Adversarial surrogate losses for ordinal regression. In NIPS, 2017.
Rizal Fathony, Kaiser Asif, Anqi Liu, Mohammad Ali Bashiri, Wei Xing, Sima Behpour, Xinhua Zhang, and Brian D Ziebart. Consistent robust adversarial prediction for general multiclass classification. arXiv preprint arXiv:1812.07522 2018.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
10

---

Under review as a conference paper at ICLR 2022
Aditya Gangrade, Anil Kag, and Venkatesh Saligrama. Selective classification via one-sided predic- tion. In International Conference on Artificial Intelligence and Statistics, pp. 2179-2187. PMLR, 2021.
Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. arXiv preprint arXiv:1705.08500, 2017.
Yonatan Geifman and Ran HI- El-iv..v. Selectivenet: A deep neural network with an integrated reject option. In International Conference on Machine Learning. pp. 2151-2159. PMLR, 2019.
Siddharth Gopal and Yiming Yang. Recursive regularization for large-scale classification with hierarchical and graphical dependencies. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 257-265, 2013.
Noah F Greenwald, Geneva Miller, Erick Moen, Alex Kong, Adam Kagel, Christine Camacho Fullaway, Brianna J McIntosh, Ke Leow, Morgan Sarah Schwartz, Thomas Dougherty, al. Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. bioRxiv, 2021.
Peter D Grünwald, A Philip Dawid, et al. Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory. Annals of statistics, 2004.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017.
Shunxin Guo and Hong Zhao. Hierarchical classification with multi-path selection based on granular 2021.
computing. Artificial Intelligence Review,
Yangqing Jia, Joshua T Abbott, Joseph I. Austerweil, Tom Griffiths, and Trevor Darrell. Visual concept learning: Combining machine vision and bayesian generalization on concept hierarchies. In Advances Neural Information Processing Systems, pp. 1842-1850, 2013.
Leeat Keren, Marc Bosse, Diana Marquez, Roshan Angoshtari, Samir Jain, Sushama Varma, Soo- Ryum Yang, Allison Kurian, David Van Valen, Robert West, et al. A structured tumor-immune microenvironment in triple negative breast cancer revealed by multiplexed ion beam imaging. Cell, 174(6):1733-137, 2018.
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE transactions on neural networks and learning systems, 29(8)33573-3587, 2017.
Matjaz Kukar, Igor Kononenko, et al. Cost-sensitive learning with neural networks. In ECAL, volume 15, pp. 88-94. Citeseer, 1998.
John Langford and Alina Beygelzimer. Sensitive error correcting output codes. In International Conference on Computational Learning Theory, pp. 158-172. Springer, 2005.
Kibok Lee, Kimin Lee, Kyle Min, Yuting Zhang, Jinwoo Shin, and Honglak Lee. Hierarchical novelty detection for visual object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1042,, 2018
Haifeng Li, Xin Dou, Chao Tao, Zhixiang Hou, Jie Chen, Jian Peng, Min Deng, and Ling Zhao. Rsi-cb: A large scale remote sensing image classification benchmark via crowdsource data, arXiv preprint arXiv: 10450, 2017.
Ziyin Liu, Zhikang Wang, Paul Pu Liang Russ R Salakhutdinov, Kouis-Philiper Morency, and Masahito Ueda. Deep gamblers: Learning to abstain with portfolio theory. Advances in Neural Information Processing Systems, 32: 02:10633-1063, 2019.
Daniel A Spielman and Shang-Hus Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. Journal of the ACM (JACM), 51(3)3885-43, 2004.
Hsing Tu and Hsuan-Tieen Lin. One-sided support vector regression for multiclass cost-sensitive classification. In ICML, 2010.
11

---

Under review as a conference paper at ICLR 2022
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8769-8778, 2018.
Haoxuan Wang, Anqi Liu, Junchi Yan, Zhiding Yu, Yisong Yue, and Anima Anandkumar. Deep distributionally robust learning for calibrated uncertainties under domain shift. arXiv preprint arXiv:2010.05784, 2021.
Hong Wang, Wei Xing, Kaiser Asif, and Brian Ziebart. Adversarial prediction games for multivariate losses. Advances Neural Information Processing Systems, 28:2728-22 2736, 2015.
Yu Wang, Qinghua Hu, Yucan Zhou, Hong Zhao, Yuhua Qian, and Jiye Liang. Local bayes risk minimization based stopping strategy for hierarchical classification. In 2017 IEEE International Conference on Data Mining (ICDM), pp. 515-524, IEEE, 2017.
Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine-grained categorization and verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3973-3981, 2015.
Bianca Zadrozny, John Langford, and Naoki Abe. Cost-sensitive learning by cost-proportionate example weighting. Third IEEE international conference on data mining, pp. 435-442 IEEE, 2003.
Weijie Zheng and Hong Zhao. Cost-sensitive hierarchical classification for imbalance classes. Applied Intelligence, 50(8):2328-2338 2020.
Weijie Zheng and Hong Zhao. Cost-sensitive hierarchical classification via multi-scale information entropy for data with an imbalanced distribution. Applied Intelligence, pp. 1-13, 2021.
Zhi-Hua Zhou and Xu- -ing Liu. Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Transactions knowledge and data engineering, ((1))663777, 2005.
12


Reviews:
[{"review_id": "rbA32hlrxZR", "reviewer": "Reviewer_5LjP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=========Positives==============\n+ It presents a formal approach for the hierarchical classification problems which otherwise somewhat unclear to formalize, have varying performance metrics, and based on ad-hoc methods. \n\n\n===========Negatives=========\n- The significance of the work seems somewhat limited. Even though it is claimed to work on large-scale datasets, this is much smaller than classification problems for hierarchical classification consisting of thousands of labels  [1,2,3]. These datasets also have fat-tailed distribution of instances among labels, and the authors need to check the feasibility of the proposed approach on such settings as well, or discuss the limitations appropriately.\n- The main premise of the proposed approach relies on abstention as an option. What if that is not an option, and one needs to make a hard choice as part of the problem formulation. \n- The paper relies heavily on existing works such as Distributionally robust cost sensitive classification has been discussed in [4], and the main contribution seems to be extending to hierarchical classification with abstention option. This does not seem enough to be a significant contribution. The formal part of the contribution in Lemma 2 needs to be proven completely instead of leaving it out mentioning simply by use of induction. \n- The experimental evaluation is performed against a 2012 approach (DARTS). Is there no work for the last ~10 years which relevant and can be used for comparison. If so, these must be used, otherwise it again seems to go back to the point about the significance of the work and potential impact in the ICLR community.\n\n[1] On Flat versus Hierarchical Classification in Large-Scale Taxonomies, NeurIPS 2013\n[2] Recursive regularization for large-scale classification with hierarchical and graphical dependencies, KDD 2013\n[3] Learning taxonomy adaptation in large-scale classification, JMLR 2016\n[4] Consistent Robust Adversarial Prediction for General Multiclass Classification, (https://arxiv.org/abs/1812.07526)", "summary_of_the_paper": "The paper presents an approach for hierarchical cost sensitive classification in which abstentions are allowed. It shows a bijection between original cost sensitive problem and the set of layer wise abstaining losses. It is based on using the existing distributionally robust cost sensitive classification and extending to it to the hierarchical setup. The proposed methodology is demonstrated on birds and cell classification datasets, which are claimed to be large-scale. It is compared to a relatively old DARTS method from 2012.", "summary_of_the_review": "The paper needs to further motivate the significance, evaluate on truly large-scale settings, discuss related works and compare with more recent methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "b3_W2Xo_R0x", "reviewer": "Reviewer_9Y3K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper has clear motivation.\n+ The organization of this paper is good, and it's easy to follow.\n+ The proposed method seems to achieve attractive results compared to DARTS.\n+ The decomposition into level-wise learning to abstain seems to help find the desired \"performance profile\" - making it easier by reducing the number of parameters.\n\nWeaknesses:\n- I lack comparison with some other methods. Authors claim that _\"LAM achieves a lower hierarchical cost-sensitive loss in high accuracy regions, compared to previous methods and their modified versions for a fair comparison\"_ but there is really only one baseline method - DARTS. It would be nice to see also the results for some other methods and some simple baseline to get a better idea about the benefits of the proposed approach. As mentioned in the introduction Bayse-optimal solution based on probability estimates from standard DNN would be a nice baseline. \n- The proposed prediction procedure over the hierarchy is quite simple and can be applied to any algorithm that can abstain. Comparison with other algorithms within the same prediction procedure would show the benefit of deep DRL.\n- The authors use two datasets (Aves, cell classification), but most of the plots present the results only for Aves. The number of datasets is already small. At least, I would like to see all the results for these two. \n- Deep DRL is directly based on the work of Anthony et al. 2018.\n- Proposed method can be only applied to hierarchies where all the leaves are on the same level.", "summary_of_the_paper": "The authors propose a new framework for cost-sensitive hierarchical classification. First, they decompose it into level by level learning to abstain (with different abstain costs per class) sub-problems. To solve these subproblems, authors apply deep distributionally robust learning (DRL) approach that directly minimizes the abstaining loss (based on Fathony et al. 2018). These two elements create a method named the Layer-wise Abstaining Loss Minimization method (LAM). The proposed method is compared with DARTS on two datasets and achieves attractive performance. The authors also demonstrate that this decomposition makes it easier to achieve the desired performance profile by adjusting abstaining losses of the layers.", "summary_of_the_review": "I believe this is a solid work but with a limited scope of contribution. I'm not that familiar with some of the related work and used datasets, but I think additional comparisons are needed to assess the attractiveness of the proposed approach correctly. That is why in my opinion, this work is now marginally below the threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_Yw6l64qfb", "reviewer": "Reviewer_jXQe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studys cost-sensitive hierarchical classification problems, and aims to propose an efficient method. But the paper contains many issues.\n1, The paper claims that \"Cost-sensitive loss is hard to optimize since it is non-smooth and non-convex\". But how did the paper address this issue? what is the time cost of the proposed methods? The experiments also do not report the time cost the proposed method. Since the paper focuses on the time complexity issues, but the proposed method did not show any benefit of time both from theory and experiments. This is a main major issue.\n2, The distributionally robust learning approach is uesed as the optimizer. But the time cost of DRO is also very high. How can you use DRO for fast optimization?what is the time cost？can you report the time in experiments？can the proposed method converge？can you provide both theory and experiments support for convergency？\n3, The data sets used in the paper is very small and trivial. Given these data sets, many existing cost-sensitive hierarchical classification methods can be efficiently optimized. Can you test your method on imagenet dataset and larg-scale NLP data sets？\n4.There many methods aim to devide the cost-sensitive learning problem into some sub-problems. What is the advantage and disadvantage of the proposed method over existing methods? ", "summary_of_the_paper": "The paper studys cost-sensitive hierarchical classification problems. The novelty is very limited, the experiments are vey tirial. I recommend to reject.", "summary_of_the_review": "1, The paper claims that \"Cost-sensitive loss is hard to optimize since it is non-smooth and non-convex\". But how did the paper address this issue? what is the time cost of the proposed methods? The experiments also do not report the time cost the proposed method. Since the paper focuses on the time complexity issues, but the proposed method did not show any benefit of time both from theory and experiments. This is a main major issue.\n2, The distributionally robust learning approach is uesed as the optimizer. But the time cost of DRO is also very high. How can you use DRO for fast optimization?what is the time cost？can you report the time in experiments？can the proposed method coverge？can you provide both theory and experiments support for convergency？\n3, The data sets used in the paper is very small and trivial. Given these data sets, many existing cost-sensitive hierarchical classification methods can be efficiently optimized. Can you test your method on imagenet dataset and larg-scale NLP data sets？\n4.There many methods aim to devide the cost-sensitive learning problem into some sub-problems. What is the advantage and disadvantage of the proposed method over existing methods? ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "N8Jq52DulS", "reviewer": "Reviewer_MMjf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.  The strengths:\n1.1) using a learning to abstention for CSHC and providing a LAM via efficient optimization of DRL framework layer by layer;\n1.2) Comparing DARTS, LAM achieves total boost in performance.\n1.3) The strategy is flexible to great extent.\n2. The weaknesses:\n2.1) Insufficient review on existing works.\nThere have existed related works devoting to this topic, however, the authors missed them, e.g., \n[m1] Cost-sensitive learning of hierarchical tree classifiers for large-scale image classification and novel category detection.\n[m2]Classification with rejection based on cost-sensitive classification.\n[m3]Selective Classification via One-Sided Prediction,\nWhile, some of the works the authors have  reviewed are NOT hierarchical classification topic, e.g., (Liu et al., 2019）\n2.2) Inappropriate descriptions such as\n For ”most of the previous methods only discuss the case of low-dimensional linear features and do not incorporate deep neural networks“ ，will which new challenges  yield when ”incorporate deep neural networks“ ?\n2.3) Lack of the reason why to select DRL for modelling seems NOT to be given! e.g., for its consistency like \"Consistent algorithms for multiclass classification with an abstain option\"!\n2.4) what relation between optimizations at each layer and on all layer is mathematical formulation?\n2.5) Insufficient experiments\nWhy to just use a DARTS (2012) as a baseline, the authors should clarify the reason of just comparing the baseline. In addition, due to the use of DRL in this paper, a strategy different DARTS,  it is unclear whether the boost of performance is from such DRL! ", "summary_of_the_paper": "The submission study the problem of cost-sensitive hierarchical classification (CSHC) with given label taxonomy via learning to abstentions each layer within the hierarchy. Indeed, \n1. CSHC subject has had many researches while using abstention or reject decision has also had many works in flat classification, briding both seems relatively few for which the authors present LAM to achieve a so-called new method. \n2. Using DRL framework to solve the learning to abstain problems in each layer makes optimization almost decouplable layer by layer, the rationale behind its strategy should attribute to the proved bijective correspondence between the hierarchical cost-sensitive loss and the set of abstaining losses.\n3. LAM achieves better performance on limited benchmarks.", "summary_of_the_review": "In my opinion, incorporating \"learning to abstention\"  to CSHC is somewhat interesting and new, in particular,  the authors provide the bijective correspondence between the hierarchical cost-sensitive loss and the set of abstaining losses to make the layer-by-layer efficient optimization possible, while the proposed LAM achieves better performance comparing DARTS despite of insufficiency.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors tackle the problem of cost-sensitive hierarchical classification. They decompose the problem into level-wise learning-to-abstain sub-problems, and apply the distributionally-robust learning (DRL) technique to minimize the abstaining loss. The proposed approach is compared with a few competitors on several data sets. The reviewers find the key idea in the work, namely leveraging DRL as the key technique to solve the decomposed problem, to be somewhat interesting and new. Some of the reviewers find the motivations clear, while others believe that the paper could use a better positioning to connect the motivation with the significance of the technical contributions. 

While the authors have extended the discussions on related works and added some additional experiment results during the rebuttal, the reviewers generally agree that the improvements were not sufficient to warrant acceptance. Most importantly, the few baseline competitors and the small-scale data sets make it hard to convince the readers about the validity of the proposed approach. In particular, the scalability of the approach to larger-scale data sets remains questionable, and the spectrum of baseline competitors, both in terms of breadth and recency, is not sufficient. Some reviewers suggest the authors include time/efficiency/convergence analysis of the proposed approach. Furthermore, the authors are encouraged to clarify the significance of contributions, explain the choice of DRL, and deepen the discussions on related works in future revisions.
Decision: Reject