Hierarchical Selective Classification
Authors: Shani Goren
Ido Galil
Ran El-Yaniv
Venue: NeurIPS 2024 poster (2024)
Keywords: Hierarchical Selective Classification
Hierarchical Uncertainty
Selective Classification
Non-Bayesian Uncertainty Estimation
CLIP
Primary area: other
TL;DR: We extend selective classification to a hierarchical setting, showing outstanding results for various models.

Abstract: Deploying deep neural networks for risk-sensitive tasks necessitates an uncertainty estimation mechanism. This paper introduces *hierarchical selective classification*, extending selective classification to a hierarchical setting. Our approach leverages the inherent structure of class relationships, enabling models to reduce the specificity of their predictions when faced with uncertainty. In this paper, we first formalize hierarchical risk and coverage, and introduce hierarchical risk-coverage curves. Next, we develop algorithms for hierarchical selective classification (which we refer to as "inference rules"), and propose an efficient algorithm that guarantees a target accuracy constraint with high probability. Lastly, we conduct extensive empirical studies on over a thousand ImageNet classifiers, revealing that training regimes such as CLIP, pretraining on ImageNet21k and knowledge distillation boost hierarchical selective performance.

Hierarchical Selective Classification
Shani Goren
Ido Galil
Technion shani. wowgggrgrnnaa...ooo com
Technion, NVIDIA
.ggmmllll.ggi.ii.......... igagmail.comm @galil@nvidia com
idogalil.
Ran El-Yanie Technion, NVIDIA
www.sset.cchhn.cccccmmmcm ac. ic, relyanivenvidia.com
Abstract
Deploying deep neural networks for risk-sensitive tasks necessitates an uncertainty estimation mechanism. This paper introduces hierarchical selective classification (HSC), extending selective classification to a hierarchical setting, Our approach leverages the inherent structure of class relationships, enabling models to reduce the specificity of their predictions when faced with uncertainty. In this paper, we first formalize hierarchical risk and coverage, and introduce hierarchical risk livercoee curves. Next, we develop algorithms for performing HSC (which we refer to as "in- ference rules"), and propose an efficient algorithm that guarantees target accuracy constraint with high probability. We also introduce Calibration-Cooeeraa curves, which analyze the effect of hierarchical selective classification on calibration. Our extensive empirical studies on over thousand ImageNet classifiers reveal that train- ing regimes such as CLIP, pretraining on ImageNet2lk, and knowledge distillation boost hierarchical selective performance. Lastly, we show that HSC improves both selective performance and confidence calibration. Code is available at fhttps:// com http:hawww..ae.iioaaaaaaaaaaaaea www.eetieeeeilssiaatttti..e
Introduction
Deep neural networks (DNNs) have achieved incredible success across various domains, including computer vision and natural language processing. To ensure the reliability of models intended for real-world applications
must incorporate uncertainty estimation mechanism, such as selective classification [19], which allows model abstain from classifying samples when is uncertain about their predictions. Standard selective classification, however, has an inherent shortcoming: for any given sample, it is limited to either making a prediction or rejecting it, thereby ignoring potentially useful information about the sample, in case of rejection.
To illustrate the potential consequences of this limitation, consider a model trained for classifying different types of brain tumors from MRI scans, including both benign and malignant tumors. If a malignant tumor is identified with high confidence, immediate action is needed. For a particular hypothetical scan, suppose the model struggles to distinguish between subtypes of malignant tumors, assigning a confidence score of 0.33 to each of them (assuming those estimates are well-calibrated and sum 1). In the traditional selective classification framework. if the confidence threshold is higher than 0.33, the model will reject the sample, failing to provide valuable information to alert healthcare professionals about the patient's 's condition, even though the model has enough information to conclude that the tumor is malignant with high certainty. This could potentially result in delayed diagnosis and treatment, posing significant risk the patient's 's life. However, a hierarchically-en aware selective model with an identical confidence threshold would classify the tumor malignant with
*The first two authors have equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

99% confidence. Although this prediction is less specific, it remains highly valuable as it can promptly notify healtheare professionals, leading to early diagnosis and life-saving treatment for the patient. This motivates us to propose hierarchical selective classification (HSC), an extension of selective classification to a setting where the classes are organized in hierarchical structure. Such hierarchies are typically represented by tree-like structure, where each node represents a class, and the edges reflect a semantic relationship between the classes, most commonly an 'is-a' relationship. Datasets with an existing hierarchy are fairly common, as there are many well-established predefined hierarchies available, such as WordNet [33] and taxonomic data for plants and animals. Consequently, popular datasets like ImageNet [1]. which is widely used in computer vision tasks, and iNaturalist 124], have been built upon these hierarchical structures, providing ready-to-ure to-eus trees. The ImageNet dataset, for example, is organized according the WordNet hierarchy. A visualization of small portion of the ImageNet hierarchy is shown in Figure] T
In cases where a hierarchical tree structure is not provided with the dataset, it can be automatically generated using methods such leveraging 28 57)
LLMss 7
image Labrador
Dog (0.91)
Hunting Doy (0155)
Sparting Dog (0.39
Hound (0.20)
Leaf Priciction: "Geldes
Retriever (0:33)
Spaniel (0.11)
Hierarchical
netrisvee"
Thizan Redbone
Selective Classifier
Hound Coonhound (0.04) (0.05)
Golden Labrador Brittany Spaniel
Retriever (0.29)
(0.13)
(0.10)
Figure 1: A detailed example of HSC for the output of ViT-L/16-384 on a specific sample. The base classifier outputs leaf softmax scores, with internal node scores being the sum of their descendant leaves' scores, displayed in parentheses next to each node. The base classifier incorrectly classifies the image 'Golden Retriever' with low confidence. A selective classifier can either make the same incorrect leaf prediction if the confidence threshold is below 0.29, or reject the sample. A hierarchical selective classifier with the Climbing inference rule (see Section 3 climbs the path from the predicted leaf to the root until the confidence threshold 0 is met. Setting @ above 0.29 yields hierarchically correct prediction, with smaller 0 values increasing the coverage. Algorithm for determining the optimal threshold is introduced in Section 4
The key contributions of this paper are as follows:
(1) We extend selective classification to a hierarchical setting. We define hierarchical selective risk and hierarchical coverage, leading us to introduce hierarchical risk-coverage curves.
(2) We introduce hierarchical selective inference rules, i.e., algorithms used to hierarchically reduce the information in predictions based on their uncertainty estimates, improving hicrarchical selective performance compared to existing baselines. We also identify and define useful properties of inference rules.
(3) We propose novel algorithm to find the optimal confidence threshold compatible with any base classifier without requiring any fine-tuning, that achieves a user-defined target accuracy with high probability, which also be set by the user, greatly improving over the existing baseline. (4) We conduct a comprehensive empirical study evaluating HSC on more than 1,000 ImageNet classifiers. We present numerous previously unknown observations, most notably that training approaches such as pretraining on larger datasets, contrastive language-image pretraining (CIIP) 35). and knowledge distillation significantly boost selective hierarchical performance. Furthermore, define hierarchical calibration and discover that HSC consistently improves confidence calibration.
2 Problem Setup
Selective Classification: Let X' be the input space, and y be the label space. Samples are drawn from an unknown joint distribution P(X' x y) over X' V. A classifier / is a prediction function X + D, and & =f() is the model's prediction for z. The true risk of classifier f w.r.t. P

---

is defined as: R(f[P)- = (xle(f(X), Y),, where E: x x 3 - R is a given loss function, for instance, the 0/1 loss. Given set of labeled samples S. = {(x,, == the empirical risk of f is: R( = m
((f())). v.). Following the notation of [2I], we use a confidence score function <(x,, f(x. to quantify prediction confidence. We require k to induce a partial order over instances in X. In this work, we focus on the most common and well-known K function, softmax For a classifier / with softmax as its last layer: <(ar, oll = fz(x). Softmax response has been shown to be a reliable confidence score in the context of selective classification
response T0 40).
[8 [9] as well as in hierarchical classification [46), consistently achieving solid performance.
A selective model [9] [3] is a pair ( where f is a classifier and X - {0. 1) is a selection
function, which serves as a binary selector for f. The selective model abstains from predicting instance z if and only if (xx = O. The selection function 9 can be defined by confidence threshold 0: (z K. f)= [x((,, f] > 0). The performance of selective model measured using selective risk and coverage. Coverage is defined as the probability mass of the non-rejected instances in X': g(f Ex]g(X)). The selective risk of (f.g) is: R(f.g) B=I((((x),Y)9xx]]
(1f.)) Risk and coverage can be evaluated over labeled set Sm. with the empirical coverage de- fined as: 2(. ()m) = I (x;), and the empirical selective risk: g(.. =
((((a)))) The performance profile of a selective classifier can be visualized by a 6(S.9) 9.))
risk-coverage curve (RC curve) [I3] a curve showing the risk as a function of coverage, measured on a set of samples. The area under the RC curve, namely AURC, was defined by [2T] for quantifying selective performance via a single scalar. See Figure 2a for an example of an RC Curve.
Hierarchical Classification: Following the notations of 12 and [46]. a hierarchy H = (V, F)) is defined by tree, with nodes V and edges E, the root of the tree is denoted r E V. Each node e E V represents a semantic class: the leaf nodes C S V are mutually exclusive ground-truth classes, and the internal nodes are unions of leaf nodes determined by the hierarchy. The root represents the semantic class containing all other objects. A sample x that belongs to class y also belongs to the ancestors of y. Each node has exactly one parent and one unique path to it from the root node. The set of leaf descendants of node e is denoted by [(v), and the set of ancestors of node v, including itself, is denoted by A(v). A hierarchical classifier f X' -> V labels sample x X' node v E V, at any level of the hierarchy, as opposed to flat classifier, that only predicts leaves.
Given the hierarchy, it is correct to label image as either its ground truth leaf node or any of its ancestors. For instance, a Labrador is also a dog, a canine, a mammal, and an animal. While any of these labels is technically correct for classifying a Labrador, the most specific label clearly preferable as it contains the most information. Thus, it is crucial to observe that the definition of hierarchical correctness incomplete without considering the amount of information held by the predictions. The correctness of a hierarchical classifier f on a set of samples Sm, is defined by: 1 Σi 1[f(z)) E A(y,)). Note that when H is flat and does not contain internal nodes, the hierarchical accuracy reduces to the accuracy of a standard leaf classifier. In the context of classification tasks, the goal usually to maximize accuracy. However, a trivial way to achieve 100% hierarchical accuracy would be to simply classify all samples as the root node. For this reason, mechanism that penalizes the model for predicting less specific nodes must be present as well. In section
3 we define coverage, which quantifies prediction specificity. We aim for trade-off between the accuracy and information gained by the prediction, which can be controlled according to user's requirements. A review of related work involving hierarchical classification can be found in section 6
Hierarchical Selective Classification: Selective classification offers binary choice: either to predict or completely reject sample. We propose hierarchical selective classification (HSC). a hierarchical to extension of selective classification, which allows the model retreat less specific nodes in the hierarchy in case of uncertainty. Instead of rejecting the whole prediction, the model can now partially reject a sample, and the degree of the rejection is determined by the model's confidence. For example, if the model is uncertain about the specific dog breed of an image of a Labrador but is confident enough to determine that it is a dog, the safest choice by a classic selective framework would be to reject it. In our setting, however, the model can still provide useful information that the object in the image is a dog (see Fi. hierarchical selective classifier fa A (J. consists of f, a base classifier, and gp X V. hierarchical selection function with confidence threshold 0. 9 determines the degree of partial rejection by selecting a node in the hierarchy with confidence higher than 0. Defining " now becomes non-triviat For instance, of can traverse the hicrarchy tree, directly choose a single node,
3

---

or follow any other algorithm, as long as the predicted node has sufficient confidence. For this reason, we refer to g8" as a hierarchical inference rule. In section we introduce several inference rules and discuss their properties.
Hierarchical selective classifiers differ from the previously discussed hierarchical classifiers by requir- ing a hierarchical selection function. In contrast to non-selective hierarchical classifiers, which may produce predictions at internal nodes without a selection function, a hierarchical selective classifier can handle uncertainty by gradually trading off risk and coverage, controlled by ge This distinction is crucial because hierarchical selective classifiers provide control over the full trade-off between risk and coverage, which is not always attainable with non-selective hierarchical classifiers.
To our knowledge, controlling the trade-off between accuracy and specificity was only previously explored by [12). who proposed the Dual Accuracy Reward Trade-off Search (DARTS) algorithm, which aims to obtain the most specific classifier for a user-specified accuracy constraint. Our approach differs from theirs in that guarantee control over the full trade-off, while some coverages cannot be achieved by DARTS. As mentioned in section 2 it is considered correct label an image as either its ground truth leaf node or any of its ancestors. Thus, we employ a natural extension of the 0/1 loss to define the true risk of hierarchical classifier fdt with regard to P: R (f P) = 1 (X) & A(Y)), and the empirical risk over a labeled set of samples Sm: (S - 1- 1 (x:) & A(vi)] An
additional risk penalizing hicrarchical mistake severity is discussed in Appendix A
To ensure that specific labels are preferred, it is necessary to consider the specificity of predic- tions. Hierarchical selective coverage, which we propose a hierarchical extension of selec- tive coverage, measures the amount of information present in the model's predictions. A natural quantity for that is entropy, which measures the uncertainty associated with the classes beneath a given node. Assuming uniform prior on the Ieaf nodes, the entropy of node v E V is 3 [v)() At the root, the entropy reaches its maximum (v)) = - Eo' E ((e) [[(e]] log( [
value, (r)) = lo), ), while at the leaves the entropy is minimized, with (v)) = 0 for any leaf node y E C. This allows us to define coverage for single node v. regardless of gø We define hierarchical coverage (from now on referred to as coverage) as the entropy of v relative to the H(v)
H(r) The root node has zero coverage, as it does not contain any information. The coverage gradually increases until it reaches at the leaves. We can also define true coverage for a hierarchical selective model: o" (f") = Ep (X)] The empirical coverage of classifier over a labeled set of samples Sm is defined as the mean coverage over its predictions: o"( (S.) = J En "ff (x;)). For a hierarchy comprised of leaves and a root node, hierarchical coverage reduces to selective coverage, where classifying sample as the root corresponds with rejection.
entropy of the root node: af (v)
The hierarchical selective performance of a model can be visualized with a hierarchical RC curve. The area under the hicrarchical RC curve, which we term hAURC, extends the non- non-hierarchical AURC, by using the hicrarchical extensions of selective risk and coverage. For examples of hicrarchical RC curves see Figure 2a and Figure 2b
Confidence calibration refers to the task of predicting probability esti- mates that accurately reflect the true likelihood of correctness. As reducing hierarchical coverage may improve accuracy, it may also improve calibration over the samples the model has not abstained from. To capture this relationship. we introduce the concept of Calibration-Cover curves), which, analogous RC curves, plot the Expected Calibration Error (ECE) 34 of coverage. see Figure for for an example.
Hierarchical Calibration:
Curves (CC
function
3 Hierarchical Selective Inference Rules
To define hierarchical selective model p# = (F.95)) given a base classifier f, an explicit definition of o is required. o determines the internal node in the hierarchy 1 retreats to when the leaf prediction of f is uncertain. o requires obtaining confidence for internal nodes in the hierarchy. Since most modern classification models only assign probabilities to leaf nodes, we follow [12] by setting the probability of an internal node to be the sum of its leaf descendant probabilities: f (x) = (v) fy(x). Unlike other works that assign the sum of leaf descendant probabilities to internal nodes, our algorithm first calibrates leaf probabilities through lemperature scaling. Since the probabilities of internal nodes heavily rely the leaf probabilities supplied by f. calibration is

---

Algorithm 1 Climbing Inference Rule
Input:
Classifier S, class hierarchy H (V, E), sample æ; X, confidence threshold 0.
Output:
Predicted node ü.
Perform temperature scaling 23
Obtain (xx) Vo E
é f arg maxxayaale (x,) yEt while (a(ara) do
0 4 parent( (ū)
end while
beneficial for obtaining more reliable leaf probabilities, and since the internal nodes probabilities are sums of leaf probabilities, this cumulative effect is even more pronounced. Further, [[8] found that applying temperature scaling also improves ranking and selective performance, which is beneficial to our objectíve. In this section, properties. We propose the Climbing inference rule (Algorithm I]. which starts at the most likely leaf and climbs the path to the root, until reaching an ancestor with confidence above the threshold, A visualization of the Climbing inference rule is shown in Figure []
introduce several inference rules, along with useful theoretical
We compare our proposed inference rules the following baselines: The first the selective inference rule, which predicts 'root' for every uncertain sample (this approach is identical to standard iiierarchicllly-inoraaa
selective classification). The second hierarchical baseline, proposed by [46], selects the node with the highest coverage among those with confidence above the threshold. We refer to this rule as "Max-Cover Cooe--Cveeeee"" (MC), and its algorithm detailed in Appendix B
Certain tasks might require guarantees on inference rules. For example, it can be useful to ensure that an inference rule does not turn a correct prediction into an incorrect one. Therefore, for an inference rule og' we define:
Monotonicity in Correctness: For any 01 S 02. base classifier f and labeled sample (z: (f gh, (x) E A(y)) = (f, (f,gø, (x) E A(y). Increasing the threshold will never cause correct prediction become incorrect.
2. Monotonicity in Coverage: For any 01 S 02. base classifier f and labeled sample (x, y):
8d [[f. 96, (x)] 2 o ""[(f. (x))), Increasing the threshold will never increase the coverage.
The Climbing inference rule outlined in Algorithm satisfies both monotonicity properties. MC satisfies monotonicity in coverage, but not in correctness. An additional algorithm that does not satisfy any of the properties is discussed in Appendix
0.32
0.13
Selective
0.12
5maller MURC (X3000) I Better
MexCovveage
Slimbing
0.10
5maller NAURC (x2000) better
hAURC 21.27
Nok
0.08
NAURC 21, 16
0.06
hAURC 19.48
0.54
0.94
0.02
0.02
NAURC 15.31
0.00
0.00
6.0
0.2
0.1
0.6
0.8
3.0
0.0
0.2
0.4
0.6
0.8
1.0
Coverage
Coverage
(b)
Figure 2: (a) hierarchical RC curve of a ViT-L/16-384 model trained on ImageNetlk, evaluated with the 0/1 loss the risk and softmax response as its confidence function The purple shaded area represents the area under the RC curve (hAURC). Full coverage occurs when the model accepts all leaf predictions, for which the risk is 0.13. Increasing the confidence threshold leads the rejection of more samples. For example, when the threshold is 0.77 the the risk is 0.04, with coverage 0.8. (b) hierarchical RC curves of different inference rules with EVA-L/14-196 14 as the base classi- fier. When the coverage is 1.0, all inference rules predict leaves. Each inference rule achieves different trade-off, resulting in distinct curves. This example represents the prevalent case, where the "hierarchically-igorre" selective inference rule performs the worst and Climbing outperforms MC.
5

---

Algorithm 2 Optimal Hierarchical Selective Threshold
Input: Hierarchical selective classifier f (f.gg), hierarchy H (V, F),, calibration set Sp = [(1,, 34) target accuracy 1 1-a. a E (0, 1), confidence level 1 - 6.
Output: Optimal threshold 0, and s.t. P([C(n, a) (1 ai) S t) 2 1 - 6, where C(n, a)
P(On+11 S ()S.))
1: Based on at n, 6, calculate
>> See Appendix
2: for (Ii; 3/4) in Sn do
3: Obtain f (T:) Yu E V
4: 0,- - (g0 (z
See Algorithm 5 in Appendix D
5: end for
6: Sort [0,]2=1 in ascending order
7: 0 < [[m+1)(1-0]
When comparing hierarchical selective models, we find it useful to measure the performance improve- ment gained by using hierarchical selective inference. We propose a new metric: hierarchical gain, defined as the improvement in hAURC between (J. "hierarchically-ignnaa"
selective model, and (f. ( ), the same base classifier with a hierarchical inference rule. This metric might also point to which models have better hierarchical understanding, as it directly measures the improvement gained by allowing models to leverage hierarchical knowledge. Note that if the selective inference rule is better than the hierarchical inference rule being assessed, the hierarchical gain will be negative. An illustrative individual example of RC curves comparison for one model is shown in Figure 2b
4 Optimal Selective Threshold Algorithm
In Section 3 we defined several inference rules that, given a confidence threshold, return a predicted node in the hierarchy. In this section, we propose an algorithm that efficiently finds the optimal thresh- old for user-defined target accuracy and confidence level. The algorithm, outlined in Algorithm 2 receives as input hierarchical selective classifier f an accuracy target a, a confidence level 1 1-8 (which refers to the interval around - not to be confused with the model's 's confidence), and a calibration set. It outputs the optimal threshold @ that ensures the classifier's 's accuracy on an unseen test set falls within
5 confidence interval around 1 a, with a resulting error margin of The threshold is calculated once on the calibration set and then used statically during inference. The algorithm does not require any retraining or fine-tuning the model's weights. For each sample (zi, yi) in the calibration set, the algorithm first calculates 0,, the minimal threshold that would have made the prediction f (x:) hierarchically correct. Then ô, the optimal threshold, is calculated in method inspired by split conformal prediction [48] Theorem 1 Suppose the calibration set Sn { b and a given test sample (Ia+1 1, Da+t exchangeable. For any target accuracy (0, and 6, define put1, 1 and as in Algorithm and C(n, a) P(O,, S S..). Then:P((C(n,.0) a)
2
Proof: See Appendix F.
Remark on Theorem 1: our algorithm can provide an even greater degree of flexibility: the user may choose to set the values of any three parameters out of a, n. O, With the three parameters fixed, we can compute the remaining parameter. The size of the calibration set is of particular interest. Although increasing n yields more stable results, our guarantees hold for calibration sets of any size. Even for a small calibration set, our algorithm provides users with the flexibility to set the other parameters according their requirements and constraints. For example: user with low budget for a calibration set who may be, perhaps, more interested in controlling can still achieve a reasonable constraint by relaxing instead of increasing the calibration set size. See Appendix E for details.
5 Experiments
In this section we evaluate the methods introduced in Section and Section The evaluation was performed on 1,115 vision models pretrained on ImageNetIk [T], and 6 models pretrained on
iNat-21 [24] (available in timm 0.9.16 [49 and torchvision 0.15.1 32 ). The reported results were

---

Table I: Comparison of mean hAURC and hierarchical gain results for the selective, Max-Coverage
(MC) and Climbing inference rules applied to 1,115 models trained on ImageNetlk, and models trained on 1Nat21.
ImageNet-1k (1,115 models)
iNat21 models)
hAURC
Hier. Gain (%)
hAURC
Hier. Gain
Inference Rule
(x 1000)
(x 1000)
(%)
Selective
42.27:10.46 39.99=00.53 36.51:1..47
13.35110.50
MC
6,92++0.26
01.96=00445
10,45=0.46 23.94.11244
Climbing
14.94:0.22
10.15:10.41
obtained on the corresponding validation sets (ImageNetIk and iNat-21). The complete results and source code necessary for reproducing the experiments are provided in the Supplementary Material.
Table compares the mean results of the Climbing inference rule to both hierarchi- cal and non-hierarchical baselines. The evaluation is based on RC curves generated for 1,115 models pretrained on ImageNetlk and 6 models pretrained on iNat2 with each inference rule applied to their output. These aggregated results show the effect of different hierarchical selective inference rules, independent of the properties or output of specific model. Compared the non-hierarchical selective baseline, hierarchical inference has a clear benefit. Allowing models to handle uncertainty whole, proves to be advantageous; the
Inference Rules:
by partially rejecting a prediction instead of rejecting average model is capable of leveraging the hierarchy to predict internal nodes that reduce risk while preserving coverage. Nonetheless, the differences remain stark when comparing the hierarchical inference rules. Climbing, achieving almost 15% hierarchical gain, outperforms MC, with more than double the gain of the latter. These results highlight the fact that the approach taken by inference rules to leverage the hierarchy can have a significant impact. A possible explanation for Climbing's 's superior performance could stem from the fact that most models are trained to optimize leaf classification. By starting from the most likely leaf, Climbing utilizes the prior knowledge embedded in models for leaf classification, while MC ignores it.
Optimal Threshold Algorithm: We compare our algorithm to DARTS 12] We evaluate the performance of both algorithms on 1,115 vision models trained on ImageNetlk, for set of target accuracies. For each model and target accuracy the algorithm was run 1000 times, each with a randomly drawn calibration set. We also evaluate both algorithms on 6 models trained on the iNat21 dataset. We compare the algorithms based on two metries: (1) target accuracy error, i.c., the mean distance between the target accuracy and the accuracy measured the test set: (2) coverage achieved by using the algorithms' output on the test set. The results presented in Table 2 and Table [ in Appendix
[ show that our algorithm consistently achieves substantially lower target accuracy errors, indicating that our algorithm succeeds in nearing the target accuracy more precisely. This property allows the model to provide better balance between risk and coverage. Our algorithm more inclined towards this trade-off, as it almost always achieves higher coverage than DARTS. This is particularly noteworthy when the target accuracy is high: while DARTS loses coverage quickly, our algorithm maintains coverage that is up to twice as high. Importantly, DARTS does not capture the whole risk-coverage trade-off. Specifically, at the extreme point in the trade-off when it aims to maximize specificity, it still falls short of providing full coverage, and its predictions do not reduce to a flat classifier's 's predictions, i.c. it may still predict internal nodes.
Table 2: Results (mean scores) comparing the hierarchical selective threshold algorithm (Algorithm 2) with DARTS, repeated 1000 times for each model and target accuracy with a randomly drawn calibration set of 5,000 samples, applied to 1,115 models trained on ImageNet (meaning we
evaluated each algorithm 1,105,000 times). 1 is set 0.9.
Target Accuracy (%)
Target Accuracy Error (%)
Coverage
DARTS
Ours
DARTS
Ours
70
14.52:11.3-03
10.79-3.1e-00
LAe-5
1.00155.0e-05
80
4.8615.2e-03
2.19-27.cc-03
0.9-555.e-05
0.e-11
90
0.73
0.02:44.0--04
0.8--4
0.874:29c-05
1.69=1.4e-02
0.02::2.e-004
0.744:2.6e-3
0.76::7.4e-05
0.63:14.2-03
0.0-::1.1e-04
0.22+:2.0e-03
0.401:3.2e-04
99.5
0.9--3
0.02-7.5e-05
0.1349.8e-04
0.26122.5-04
7

---

Our algorithm offers additional flexibility to the user by allowing the tuning of the confidence interval (1-ő), while DARTS does not offer such control. F over DARTS in detail. Appendix H compares the results of an additional baseline, conformal risk control 3 The results show that the conformal risk control algorithm exhibits a significantly higher mean accuracy error than both our algorithm and DARTS.
the superiority of our technique
Mean
mcan
mmcan 54.99%
Ours
Durs
Algorthim
meana53.52%%
mesh=100.00 100.0%
DARTS
Nean Accuracy Entr = 1.6%%
DARTS
Hoan Accuracy Ereer 5.0%
93.5
95.0
95.5
95
97
98
90
100
Acouracy (16)
Acouracy (16)
Figure 3: Individual model examples comparing the hierarchical selective threshold algorithm against DARTS, with each algorithm repeated 1000 times. The mean and median results are shown in dark green. The light green area shows the interval around the target accuracy, and the remaining area is marked in red (i.e., each repetition has a 1 - 5 probability of being in the green area and a d probability of being in the red area). The target accuracy is 95% and 5 = 0.9. In both examples, the target accuracy error of DARTS is high, and the entirety of its accuracy distribution lies outside of the confidence interval. Left: EVA-Giant/14 (4. DARTS fails meet the constraint, whereas our algorithm's 's mean accuracy is very close to the target. Right: ResNet-152 [50] While our algorithm has a near-perfect mean accuracy, DARTS rejects all samples, resulting in zero coverage.
Empirical Study of Training Regimes: Inspired by [88, which demonstrated that training methods such as knowledge distillation significantly impact selective performance, we aimed to investigate whether training regimes could also contribute to hicrarchical selective performance. The goal of this experimental section is to provide practitioners with valuable insights for selecting effective training regimes or pretrained models for HSC.
We evaluated the hAURC of models trained with several training regimes: (a) Knowledge Distillation
(KD) 42 1 37;;
or on ImtgeNet12k, a 11,821 class subset of the full ImageNet2 kk 49]; (c) Contrastive Language-Image
: (b) Pretraining: models pretrained either on ImageNet21k 43 56
29
37
pretraining (CLIP) (35 CLIP models equipped with linear-probes, pretrained on WTT-400M image- fine-tuned
text pairs by ApenAl, as well models pretrained with OpenCLIP on LATON-2B 33
either on ImageNet k or on ImageNet 2k and then ImageNet k. Note that zero-shof CLIP models (i.e.. CLIP models without linear-probes) were not included in this evaluation, and are discussed later in this section. (d) Adversarial Training [53] 44]: (c) various forms of Weakly-Supervee
[30] or Semi-Superrvsed Learning (55] 54). To ensure a fair comparison, we only compare pairs of models that share identical architectures, except for the method being assessed (c.g., a model trained with KD is compared to its vanilla counterpart without KD). Sample sizes vary according to the number of available models for each method. The hAURC results of all models were obtained by using the Climbing inference rule.
(1) CLIP exceptionally improves hierarchical selective performance, compared to other train- ing regimes. Out of the methods mentioned above, CLIP (orange box), when equipped with "linear-probe", improves hierarchical performance the most by a large margin. As seen in Figure 4 the improvement measured by the relative improvement in hAURC between the vanilla version of the model and the model itself. CLIP achieves an exceptional improvement surpassing 40%. Further, its median improvement almost double the next best methods, pretraining (purple box) and semi-superviser learning (light blue box). One possible explanation for this improvement is that the rich representations learned by CLIP lead to improved hierarchical understanding. The image-tex text aligmment of CLIP can express semantic concepts that may not be present when learning exclusively from images. Alternatively, it could be the result of the vast amount of pertaining data.
(2) Pretraining on larger ImageNet datasets benefits hierarchical selective performance, with certain models achieving up to a 40% improvement. However, the improvement rates vary signifi- cantly: not all models experience the same benefit from pretraining. Semi-supervised learning also
8

---

hAURC improvement over Baxelino 1%)
80
Knowledge Distilation
Pretraining (mageeNet2kk ImageNet122)
Contrastive Language Image Pretraining(CLIP)
Semi- Superviud
Adversarial Training
Learning
Figure 4: Comparison of different methods by their improvement hAURC, relative to the same model's performance without the method. The number of models evaluated for each method: knowledge distillation: 42, pretraining: 61, CLIP: 16, semi-supervised learning: 11, adversarial training: 8.
shows a noticeable improvement hierarchical selective performance.
(3) Knowledge Distillation achieves notable improvement, with median improvement of around 10%. Although not as substantial as the dramatic increase seen with CLIP, it still offers a solid improvement. This observation aligns with [18]. who found that knowledge distillation improves selective prediction performance as well as ranking and calibration.
(4) Linear-prohs CLIP significantly outperforms zero-shot CLIP in HSC: of models with identical backbones, where one is the original zero-shot model, and the other was equipped with linear-probe, that is, it uses the same frozen feature extractor but has added head trained to classify ImageNet1k. The zero-shot models evaluated are the publicly available models released by OpenAL. The mean relative improvement from zero-shot CLIP linear-probe CLIP is 45%, with improvement rates ranging from 32% to 53%. We hypothesize that the hierarchical selec- tive performance boost may be related to better calibration ranking abilities. Specifically, all CLIP models with linear-probes showed significantly higher AUROC than their zero-shot counterparts, indicating superior ranking. Figure 7 in Appendix shows these results in more detail.
We compared pairs
Since hAURC closely related to accuracy, we sought to determine whether its improvement could be attributed solely gains in predictive performance from the training regimes. Our analysis, detailed in Appendix D] suggests this is not the case. Additionally, we explored a training regime specifically aimed enhancing HSC performance. Our approach is described in Appendix K
HSC Improves Confidence Calibration: We plotted the Calibration- Calibration-Covvrage (CC) Curves (defined in Section@fer for 1,115 ImageNet models and compared the curves produced by the Selective inference rule those from Climbing. Tigure[Jpresent the aggregated CC curve for all 1,115 models across the two inference rules. The results indicate that HSC not only improves risk, but also improves calibration, with the Climbing inference rule nearly always outperforming Selective.
0.00
Selective
0.07
3.05
3.89
19 3.04
0.00
0U6
Coverage
Figure 5: Aggregated (mean and SEM) CC curves of 1,115 ImageNet models.

---

Related Work
In selective classification, several alternative methods for confidence thresholding have been devel- oped [3] 31 38). However, these methods generally necessitate some form of training. This work is focused on post-hoe methods that are compatible with any pretrained classifier, which is particularly advantageous for practitioners. Furthermore, despite the availability of other options, threshold-basse
rejection even with the popular Softmax confidence score continues to be widely used [[8] 21], and
can be readily enhanced with temperature scaling or other techniques [6] [5] [8].
Various works leverage hierarchy of classes to improve leaf predictions of flat classifiers
26
but comparatively fewer works explore hierarchical classifiers. [51] optimized a "win" metric com- prising a weighted combination of likelihoods on the path from the root to the leaf. focused on reducing leaf mistake severity, measured by graph distance. They introduced a hierarchical cross-entropy loss, as well as a soft label loss that generalizes label smoothing. [26] claimed that both methods in 4 result in poorly calibrated models, and proposed an alternative inference method. [1] proposed the Dual Accuracy Reward Trade-off Search (DARTS) algorithm, which attempted to obtain the most specific classifier for user- eser-specified accuracy constraint. They used information gain and hierarchical accuracy as two competing factors, which are integrated into a generalized Lagrange function to effectively obtain multi- coalliigaaularttyy decisions. Additional approaches include the Level Selector network, trained by self-supervision to predict the appropriate level in the hierarchy 446 proposed loss based [52] and performed inference using a threshold-based infer- ence rule. Other works allow non-mandatory leaf node prediction, although not directly addressing the accuracy-specificity trade-off [36] 52). The evaluation of hierarchical classifiers has received relatively little attention previous to our research. The performance profile of a classifier could be inferred from either the average value of metric across the operating range or by observing operating curves that compare correctness and exactness or information precision and recall [46], or measuring Set-valued prediction 22 and hierarchical multi-label classification [45] tackle similar problem to hierarchical classification by allowing a classifier to predict set of classes. Although both approaches handle uncertainty by predicting a set of classes, the HSC framework hard-codes the sets as nodes in the hierarchy. This way, HSC injects additional world knowledge contained in the hierarchical relations between the classes, yielding a more interpretable prediction. Conformal risk control 3 is of particular interest because it constrains the prediction sets to be hierarchical. Appendix H shows a comparison of this baseline to our method.
25
information gain 12 52
Calibration is another important aspect of uncertainty estimation [27 58 18 [6]. Fisch et al. [16 demonstrate that selectively abstaining from uncertain predictions may improve calibration.
Concluding Remarks
This paper presents HSC, an extension of selective classification to a hierarchical setting, allowing models to reduce the information in predictions by retreating to internal nodes in the hierarchy when faced with uncertainty. The key contributions of this work include the formalization of hierarchical risk and coverage, hierarchical calibration, the introduction of hierarchical risk-coverage curves and hierarchical calibration-coveraa curves, the development of hierarchical selective inference rules, and an efficient algorithm to find the optimal confidence threshold for given target accuracy. Extensive empirical studies over thousand ImageNet classifiers demonstrate the advantages of the proposed algorithms over existing baselines and reveal new findings on training methods that boost hierarchical selective performance. However, there are few aspects of this work that present opportunities for further investigation and improvement: (1) Our approach utilizes softmax- based confidence scores. exploring alternative confidence functions and their impact on hierarchical selective classification could provide further insights; (2) While we have identified certain training methods that boost hierarchical selective performance, the training regimes were not optimized for hierarchical selective classification. Future research could focus on optimizing selective hierarchical performance. (3) Although our threshold algorithm is effective, it could be beneficial to train models to guarantee specific risk or coverage constraints supplied by users, in essence constructing a hierarchical "SelectiveNet" (20
Acknowledgments
The research was partially supported by Israel Science Foundation, grant No 765/23.
10

---

References
[1] Yonathan Aflalo, Asaf Noy, Ming Lin, Itamar Friedman, and Lihi Zelnik-Manor, Knapsack pruning with inner distillation. CoRR, abs/2002.08254, 2020. URL. https: org/ abs/2002. 211//20022.002....
[2] Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution uncertainty quantification. CoRR, abs/2107.07511, 2021. URL https
07511
[3] Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Confor- mal risk control. CoRR, abs/2208.0255 022814 2022.
[4] Luca Bertinetto, Romain Müller, Konstantinos Tertikas, Sina Samangooci, and Nicholas A. Lord. Making better mistakes: Leveraging class hierarchies with deep networks. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 12503-12512. Computer Vision Foundation / IEFE, URL httptttttt//peaacceeeeeeeeeeeeee thecff..hheec..c. com/ content_CVPa. SwtPaa/200//ttm/Berriieeeee http:///erttttttteooeonnnnnnener
2020. doi: 10.1 2..1099/CPRR-220222000000 Maaea Srwiisttakes_Leeeaginnggg
Class_
oatppppleeeeoxrre../PP220OOPpPpp html
[5] Yuzhou Cao, Tianchi Cai, Lei Feng, Lihong Gu, Jinjie Gu, Bo An, Gang Niu, and Masashi Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. In NeurlPS, 2022.
[6] Luís Felipe P. Cattelan and Danilo Silva. How to fix a broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks, 2024.
[7] Catherine Chen, Kevin Lin, and Dan Klein. Constructing taxonomies from pretrained language models. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2021, Online, June 6-11, 2021, pages 4687-4700. Association for Computationnl Linguistics, 2021. doi: 10. 18658/ VI2001.NAACL-MAINN. URL bittpa://doffi org 11w118665110000 naacl-main
373
[8] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 2818-2829. IEEE, 2023. doi: 10. D..109CCVPR3277.222...2277 URL https: / /doii. org /0. 10. 1109/
CVPR52729 2023.00276
[9] C. K. Chow. An optimum character recognition system using decision functions. IRE Trans. Electron. Comput., 6(4):247-25) 1957. doi: 1L1109//EC..19555222055 URL https 10.00..109TTTE /TCC. 1957. .5222355
[10] Luigi P. Cordella, Claudio De Stefano, Francesco Tortorella, and Mario Vento. A method for improving classification reliability of multilayer perceptrons. IEEE Trans. Neural Networks, 6(5): 1400-14071 117, 1995. doi: 10.7149972,,10335. URL htttps://don org/ 1109//27
410358
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248 255. IEEE Computer Society, 2009, doi: 00.1099CCPR.22009555066 URL https
//doi.org/11 1009/ CAA00//VPR.000 2009. 5206848
[12] Jia Deng, Jonathan Krause, Alexander C. Berg, and Li Fei-Fei. Hedging your bets: Optimizing securacy-peccfficit trade-off's large scale visual recognition. In 2012 IEEE Conference on Computer Vision and Puttern Recognition, Providence, RI, USA, June 16-21, 2012, pages 3450-3457. IEEE Computer Society, 2012. doi: 20.1199CCYPR.2212222444 URL https //doi. org 10. /Ow.0.///////// 2012. 6248086

---

[13] Ran EI- Yaniv and Yair Wiener. On the foundations of noise-free selective classification. J.
Mach. Learn. Res., 1641, 2010. doi: 10.5555/1756006.88533 1859904. URL https
7/d1. acm. 0rg./00//0.5555/77660000
1859904
[14] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: exploring the limits of masked visual representation learning at scale. In IEEE/CVF Conference Vancouver, BC, Canada, June 17-24, 2023, pages 19358-19361 IEFE, 2023, doi: 10. 109/ CVPRS2729.2023.01885 URL htttp:://dol: org/ 1109 Cttp://VRRR222292 2w2.:/9222000..... 20...02222...0
Computer Vision and Pattern Recognition, CVPR 2023,
[15] Leo Feng, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir H. Abdi. Towards better selective classification. In ICLR. OpenReview.net, 2023.
[16] Adam Fisch, Tommi S. Jaakkola, and Regina Barzilay. Calibrated selective classification. Trans. Mach. Learn. Res., 2022, 2022.
[17] Maurice Funk, Simon Hosemann, Jean Christoph Jung, and Carsten Lutz. Towards ontology construction with language models. In Simon Razniewski, Jan-Christoph Kalo, Sneha Singhania, and Jeff Z. Pan, editors, Joint proceedings of the 1st workshop on Knowledge Base Construction from Pre-Trained Language Models (KBC-LM) and the 2nd challenge on Language Models for Knowledge Base Construction (LM-KBC) co-located with the 22nd International Semantic Web Conference (ISWC 2023), Athens, Greece, November 6, 2023, volume 3577 of CEUR Workshop Proceedings. CEUR-WS WS..rg, 2023. https:t::///ee ws. org/Vol-3510 /0577/pappt /pppprppppp...../ pdf
[18] Ido Galil, Mohammed Dabbah, and Ran Yaniv. What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? In ICLR, 2023.
[19] Yonatan Geifman and Ran EI- Yaniv. Selective classification for deep neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 December 4-9, 2017. Long Beach, CA, USA, pages 4878 4887, 2017. URL http:: htttppss/pproceeeinngss paper/2015 / httpa///waaa22aeeeaaaaaaaeeeeeee Abstrt-tttttttttttcec.a hww.gss.errrs.... cc/
[20] Yonatan Geifman and Ran EI- El- Selectivenet: A deep neural network with an integrated reject option. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2151-2159. mlr. www..ppeessv/v/eeee /ei wegggemiiiigge 99. html
PMLR, 2019. http://:://proceeee
[21] Yonatan Geifman, Guy Uziel, and Ran El- Yaniv.. Bias-reduced uncertainty estimation for deep neural classifiers. In 7th International Conference Learning Representations, ICLR 2019. New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL /tLlatttp://poperevviee datt/orummid-Sie SJfbSssCCc
[22] Eugen Grycko. Classification with set-valued decision functions. In Otto Opitz, Berthold Lausen, and Rüdiger Klar, editors, Information and Classification, pages 218-224, Berlin, Heidelberg, 1993. Springer Berlin Heidelberg. ISBN 978-3-642-50974--.
[23] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR, 2017. URL http:tttt//prrceeeinggss mlr. www.ssreessiiee vv00 htt0//guo7a..hmmll
[24] Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge J. Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In CVPR, pages 2288411293.. Computer Vision Foundation IEEE, 2021.
[25] Ahsan Iqbal and Juergen Gall. Level selector network for optimizing accuracy-specfficity trade-offs. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pages 1466-1474 IEEE, 2019. doi: 100/C///1C 000844 httpst /oi. org/ 10. 1109/ ICCVW. 2019 001884
12

---

[26] Shyamgopal Karthik, Ameya Prabhu, Puneet K. Dokania, and Vineet Gandhi. No cost likelihood manipulation at test time for making better mistakes deep networks. In 9th International Conference on Learning Representations, ICLR 2021, Viriual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URI. wttpttts////ppereiiiieeeeee http:////pppppaeeeeeeeee////aaaa
[27] Jocy Kuang and Alexander Wong. On calibration of modern quantized efficient neural networks. CoRR, abs/2300.13866 2023.
[28] Grace Li, Tao Long, and Lydia B. Chilton. Eliciting topic hierarchies from large language models. CoRR, abs/2310. 102x/330..1975,, 2023. doi: 10.4850//ARXIV23100197 URL https I/doi.org/10. 0tte/10.4855/////// ariiv 2310.19277
[29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9992-10008 IEEE, 2021. doi: 10. 10911//CV448922222211 2021.000200..00088 https:ts:// /to.. org/10/ 10. 1109/ 1ww11119911vvv/ ICCV48922
[30] Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens der Maaten. Exploring the limits of weakly supervised pretraining. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8- 8-14, 2018, Proceedings, Part II, volume 11206 of Lecture Notes in Computer Science, pages doi: 10. 1007/978-3-03000115
185-201. Springer, 2018.
12. httpptts::///oi org/
1007/978-3-030001121-
[31] Anqi Mao, Mehryar Mohri, and Yutao Zhong. Theoretically grounded loss functions and algorithms for score-based multi-class abstention. CoRR, abs/2310. 2023.
[32] Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In ACM Multimedia, pages 1485-1488. ACM, 2010.
[33] George A. Miller. Wordnet: A lexical database for english. Commun ACM, 38( 38(11):39-41 1995. doi: 10. 10.1145/219717.219748 https://///do http::///oooooo///0/00 1tt//100.145//197777 .197488
[34] Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI, pages 2901-2907 AAAI Press, 2015.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages $748-8763, PMLR, 2021. URL html
http: http://proceeddnng: http:aisss.......ss..eeeemmmmsee
[36] Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pages 6517-6525, 6525. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017..22 URL
https://dois org/ 10. 1109/CVPR, 2017 690
[37] Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik. Imagenet-21k pre- training for the masses. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Pro- ceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
http:///aaaatttss-noonrrrrrraeee proceedings neurips. cc/ httpiipp..ccpppp/ppppemm pptpp//e////2//ss... 98f 0af13700810191077777766666333333 Awwerttt--ttttttteeee--eeee- round1 .html
[38] Mahmoud Salem, Mohamed Osama Ahmed, Frederick Tung, and Gabriel L. Oliveira, Gumbel- softmax selective networks. CoRR, abs/2211. 2022. doi: 10.48550/ARXIV.2211
URL https: :/ //doi. org/ 10. 48550/arXi1 2211 2211 10564
13

---

[39] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5HH an open large-scale dataset for training next generation image-text models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- De- cember 9, 2022, 2022. URL http mips. cc/paprr lirfsliillsl_lele wwwees/appr///ae paaer/2022/hashh wwwp://oooee--bbatat--------e--- Dattaraattt-aaatst.....m and Serd__Bnccharreee ks html
[40] Claudio De Stefano, Carlo Sansone, and Mario Vento. To reject or not to reject: that is the question-an answer in case of neural classifiers. IEEE Trans. Syst. Man Cybern. Part C. 30(1): 84-94, 2000. doi: 10.1 1.1119995326.2245577 URL https:://oi. org/10..... 1110/53266 827457
[41] Mingxing Tan and Quoc V. Le. Efficientnetv2: Smaller models and faster training. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, JCML 2021. 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10096-10106. PMLR, 2021. URL http://proccedingg mlr (ates//v399ttaa22/ html
[42]
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMLR, 2021. URL URLhhttp:// prrtte:d//rrcceednggs html
mlr. ppress/v13//tovvvee//
[43] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit III: revenge of the vit. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, volume 13684 of Lecture Notes in Computer Science, pages 516-533. Springer, 2022. doi: 10. 1000/997--3-031-200535
Utth:tts::/ 2ettps:/ddoi.or//000
1107/978-3-001-22055-3-
[44] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, and Patrick D. McDaniel. Ensemble adversarial training: Attacks and defenses. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https:///ppenreveex net/
forum? Sorum?id-rk2ve-B RZ
[45] Chhavi Tyagi and Wenge Guo. Multi-label classification under uncertainty: tree-based conformal prediction approach. In Harris Papadopoulos, Khuong An Nguyen, Henrik Boström, and Lars Carlsson, editors, Conformal and Probabilistic Prediction with Applications, 13-15 September 2023, Limassol, Cyprus, volume 204 of Proceedings of Machine Learning Research, pages 488-512. PMLR, 2023. URL UtRp: mlr, wwwpppeeeeueeeeeee Sassss/////ttyygi23.. tyagi23a html
[46] Jack Valmadre. Hierarchical classification multiple operating points. Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleams, LA, USA, November 28 Decem- nips. cc ct/ hatp:/appee____ee files/ patpppeefflleepppprr///22222 pptp///pp22//aaaa/////
ber 9, 2022, 2022. URL -oatraaeee Conference 777856331dF@2114114444444c Aot.aaaaeecccaaaaaaaoaaaatteeeee wwbb--bbtratttt.
html
[47] Vladimir Vovk. Conditional validity inductive conformal predictors. CoRR, 2bb//209,,673, 2012.
[48] Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. In Ivan Bratko and Saso Dzeroski, editors, Proceedings of the Sixteenth International Conference on Machine Learning (ICMI. 1999), Bled, Slovenia, June 27 300 1999, pages 444-453. Morgan Kaufmann, 1999.
14

---

[49] Ross Wightman. Pytorch image models. pytorch- image-aaaaaae models 2019.
https://githin www.alt..ooouuuuuu hon.giiggcee/gggggaacgaa
[50] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. CoRR, abs/21 10.00476, 2021. URL
00476
[51] Cinna Wu, Mark Tygert, and Yann LeCun. Hierarchical loss for classification. CoRR, abs/1709.01022, 2017. URL http: httt:://arxiv..oo/ org/ www.a//.......//00
[52] Tz-Ying Wu, Pedro Morgado, Pei Wang. Chih-Hui Ho. and Nuno Vasconcelos. Solving long- tailed recognition with deep realistic taxonomic classifier. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII, volume 12353 of Lecture Notes Computer Science, pages 171-189. Springer, 2020. doi: 10. 1010009777---00-----0 58598-31 030 58598-8---8888-111
11. URL http:tts: org 10.
[53] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L. Yuille, and Quoc V. Le. Adversarial examples improve image recognition. 2020 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 316-825. Computer Vision Foundation IEEE, 2020. doi: 10. (0.11090CVPR42600. 2020.00090. UtLltttss:// thectf com wwwt:ototttttoooeco Bwtpanneeet_______e___ Seeptt__V___222__0 2UTPR_2020/P10 20l020222/hhha 21/20////l//////// Xie_ Adversarial htt.pp/eee___mprrre_memeeeeeeeee
paper html
[54] Qizhe Xie, Minh-Thang Luong, Eduard H. Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10684-10695. Computer Vision Foundation IEEE, 2020. doi: 10.110//CVR422200220000. https://openaccess. thecvf. whteerrr..ooooooeeteete////////a http://wwwm..iieeffefrerrrnnnnno
URL
Self
html
[55] I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. CoRR, 0ttss/995.005544 2019. URLhttt
/ //arxiv. arggabs//909 19000000066....0 00546
[56] Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. IEEE Trans. Pattern Anal. Mach. Intell, 46 (2):896 912, 2024. doi: 10. 0..119//PAMM.2202333322
htt.hhtt////doo orgiioorg1110. 10. 1109/
TPAMI. 2023. 3329173
[57] Qingkai Zeng, Yuyang Bai, Zhaoxuan Tan, Shangbin Feng, Zhenwen Liang, Zhihan Zhang, and Meng Jiang. lfayer Iteratively prompting large language models for taxonomy induction from limited examples. CoRR, abs/2402.077386 2024. doi: 10.48550/ARXIV.24000
07386. URL https://doi. org/10. 485444550//22 waw.////rvvvvvv.a/ 2402 2432.
[58] Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 11387-11397. PMLR, 2020.
15

---

A Hierarchical Severity Risk
The presence of a hierarchy allows us to measure the severity of incorrect predictions, instead of treating all mistakes equal. For instance, classifying Labrador Golden Retriever is much milder mistake compared to classifying it as a snake. We use the coverage the Lowest Common Ancestor (LCAA) of the predicted node i and the label define novel loss function that accounts ((LCA(0.))
mistake severity: 1 (ú, v)
o(6)
Building upon the above example, the LCA of Labrador Retriever and Golden Retriever the node 'Retriever', which has significantly higher coverage relative to the prediction 'Golden Retriever', in contrast to "Vertebrate', the LCA of Labrador Retriever and Snake, resulting the former misclassification having lower risk compared to the latter.
We provide below the results of our experiments on ImageNet, complementing Table
Table Comparison mean hAURC and hierarchical gain results for the selective, Max- Max-Coverage (MC) and Climbing inference rules applied to 1,115 models trained on ImageNet1k, using the hierarchical severity risk.
ImageNet-Ik (1,115 models)
hAURC
Hier. Gain (%)
hAURC
Hier: Gain
Inference Rule
(x 1000)
(x 1000)
(%)
Selective
24.98.10.29 24.50++0.34
MC
3.31++030 4.71::0.22
Climbing
24.04:10.31
The results align with those presented in the main body of the paper, where the risk is the the 0/1 loss. Climbing remains the inference rule with the best (lowest) hAURC and the highest hierarchical gain. In practice, the results of both losses are highly correlated.
B Max-Coverage Inference Rule Algorithm
Algorithm 3 Max-Coverage Inference Rule [46)
Input: Classifier f. class hierarchy H (V, E), sample 2; X, confidence threshold Output: Predicted node é. Obtain fx(x Ve E
è t- arg max {o(v) fo(xx) 2 2
Break ties by confidence rate
C Jumping Inference Rule
The Jumping inference rule (Algorithm 4 traverses the most likely nodes at each level of the hierarchy, until reaching 0. Unlike to leaf. Although performed better than a "hierarchically-innoan"" selective inference, still underperformed by the other algorithms.
is not confined to a single path from the root
16

---

Algorithm 4 Jumping Inference Rule
Input:
Classifier /, class hierarchy H = (V, E), sample æ; X, confidence threshold 0. Predicted node
Output:
Perform temperature scaling
Obtain (x, E
é + arg arg frgmaxfar f (2)) yEL while (a(
do 0 + arg max e V, height(v) = h}
h +
end while
(*) If the leaves in H are not all at the same depth, pad the hierarchy with dummy nodes
D Hierarchy Traversal for Threshold Finding
We demonstrate an example algorithm for the Climbing inference rule that obtains 0 for a single instance (I4, yt).
Algorithm 5 Hierarchy Traversal Climbing
Input: Calibration sample (I,, 34), classifier f. class hicrarchy E). tightness coefficient e. Output: 0; - the minimal threshold that would have made the prediction correct.
0 <
2: e < arg max (x;)
yEL
3: while U; A(y:) do
4: 0g + Jr, (x:) + c. Sparent(s)) (I:)
5: U; f- parent(v;) (v:)
6: end while
E Proof of Theorem 1
Recall that for each sample (æ;, 3/), 0 is the minimal threshold that would have made the prediction {H(x; (x;) hierarchically correct.
The following is based on 2):
To avoid handling ties, assume ( are distinct, and without loss of generality that the calibration thresholds are sorted: 0 < 01 < < 0, < 1.
To keep indexing inside the array limits: 0 A [(i++1)(a-a)
{1,
m+1
otherwise
We also assume the samples (I1, v1), .... (In+1, Bn+1) are exchangeable. Therefore, for any integer
1 E n1, we have:
P(O.+1 S 0x) n
That is, 8x41 is equally likely fall in anywhere between the calibration points. By setting
k = [(n + 1)(1 a]]:
P(0.+1 S ((m+1)(-aa)) [(n 1) - a)] n n 1
2 aa
In conformal prediction, the probability P(O,+1 < i) is called "marginal coverage" (not to be
confused with the hierarchical selective definition of coverage in section 2). The analytic form of marginal coverage given a fixed calibration set, for sufficiently large test set, is shown by 47 to be
C(n,a) a) P(On+1 < ())E Beta(n + 1-11, 1,().
17

---

where = [(n + 1)
From here, the CDF of the Bela distribution paves the way calculating the size of the calibration set needed in order to achieve coverage of 1 - a + with probability - 6:
P(]C(n,a)) - (1 ar) 2 1-6
Remarks theorem 1:
To simplify, we made an implicit assumption that users can usually allocate a limited number of samples to the calibration set, and thus would prefer the threshold's optimality guarantee to hold without imposing additional requirements on the data. However, our algorithm can provide
greater degree of flexibility: the user may choose set the values of any three parameters out of: at n, C, 5. With the three parameters fixed, the CDF of the Beta distribution can be used compute the remaining parameter. For instance, certain user may have an unlimited budget of calibration samples, while they require specific error margin. In that case, the using the Beta distribution, the algorithm computes the calibration set size required for Theorem to hold.
2. The output of the algorithm holds for a calibration set of any size n. Following the previous point, increasing does yield more stable results.
3. In the context of conformal prediction, the property stated in Theorem 1 is referred to as marginal coverage, not to be confused with the previous definition of coverage section 2 This property is called marginal coverage, since the probability is marginal (averaged) over the randomness in the calibration and test points.
F Comparison of Inference Rules Without Temperature Scaling
Table 4: hAURC and gain results for each inference rule. Mean results of 1,115 pretrained ImageNetlk models.
Inference Rule
hAURC (x 1000)
Hierarchical Gain (%)
Selective
42.27=00.66
No Temp. Scaling
Max- Mlx--Coerrage [46
39.99
6.92-10.26 8.26:10.19
Climbing (Ours)
39.24110499
Selective
41.16110.46
Temp.
Max-Coverage Climbing (Ours)
37.21
11.09-10.47
Scaling
36.51 0.47
12.61
18

---

G Threshold Algorithm Results on iNat21
Table 5: Comparison of hierarchical selective threshold algorithm (Algorithm 2 with DARTS, repeated 100 times for each model and target accuracy with randomly drawn calibration set of 10,000 samples, applied models trained on iNat21. is set 0.9.
Target Accuracy (%)
Target Accuracy Error (%) DARTS
Coverage
Ours
DARTS
Ours
70
24.64 + 7.9e-04
21.22 + 7.3e-04
0.98 + 5.73e-06
1.000 + 0.00e+00
80
14.64 8.5e-04
11.22 1.2e-03
0.98 + 1.87e-06
1.000 0.00e+00
90
4.64 + 2.4e-04
1.22 + 2.9e-03
0.98 + 3.46e-06
1.000 + 6.57e-05
95
0.32 + 2.9e-02
0.02 + 8.7e-03
0.97 + 4.75e-04
0.951 = 2.41c-04
99
0.49 + 3.0e-02
0.01 + 1.3e-03
0.39 + 2.20--22
0.687 = 2.49e-03
99.5
0.42 + 2.9e-02
0.01 + 2.5e-03
0.06 + 2.42e-02
0.413 = 2.02e-03
H Conformal Risk Control Results
F shows the mean accuracy error of the threshold algorithms introduced in section 4 compared to the additional conformal risk control (CRC) algorithm B The mean accuracy error of CRC is strikingly higher than the other algorithms, reaching climax when the target accuracy is 95%.
14
DARTS
J Lower better
12
STA (Ours)
10
Mean Accuracy
8
2
90
99 100
Target Accuracy
Figure 6: Mean accuracy error comparison between the hierarchical selective threshold algorithm (Algorithm{2) 2), conformal risk control (CRC) and DARTS, repeated 1000 times for each model and target accuracy with randomly drawn calibration set of 5,000 samples, applied 1, 15 models trained on ImageNet1k (meaning we evaluated each algorithm 1,115,000 times). 1 5 is set at 0.9.
I CLIP Zero Shot VS Linear Probe
Figure compares the improvement in hAURC and AUROC between zero-shot CLIP models and their linear- linear-probe counterparts. can be easily observed that for all the tested backbones the linear- probe models have significantly better hAURC and AUROC. We encourage follow-up research to explore further the possible connection between ranking and HSC.
19

---

Backbone
3.8
ResNet-101
ResNet-50x16
ResNct-50x4
ResNet-50x64
VIT-B/16
VIT-L/14
VIT-L14@336px
-log(hURR)
3.2
2.8
2.6
0.81
0.82
0.83
0.84
0.85
0.86
0.87
AUROC
Figure 7: Comparison between zero-shot and linear-probe CLIP models sharing the same backbone. The round markers represent the zero-shot models, and the star markers represent their respective linear-probes.
J Training Regimes Improvement in hAURC VS Accuracy
In Table 6 we compare the relative improvement in hAURC to the relative improvement in accuracy, be seen that although the training regimes with the highest and lowest improvement in hAURC have the highest and lowest improvement in accuracy, the improvement in hAURC does not determine the of improvement accuracy.
Table 6: Relative improvement in Accuracy compared to relative improvement in hAURC.
Training Regime Median Accuracy Improvement (%) Median hAURC Improvement (%)
CLIP
7,90
38.14
Pretraining SSL.
1.96
19.47
4.12
18.97
Distillation Adversarial
1.48
11.73
0.85
5.92
K Training Regime to Optimize HSC Performance
Since hAURC cannot be optimized directly, developed an alternative that could be optimized. Our best- persoperforming method entailed training models to predict the lowest common ancestor (LCA) of pairs of samples, including identical pairs (intended for testing). To achieve this, fine-tuned models using the hierarchical loss we proposed in Appendix A
All models were fine-tuned on the ImageNet validation set for 20 epochs using the SGD optimizer with warmup scheduler and batch size of 2048 on single NVIDIA A40 GPU. After fine-tuning models of various architectures (including ResNet50, ViT's, EfficientNet, and others) with this loss function and utilizing various other configurations, such as leveraging in-hatch negatives for richer training signal and multiple classification heads to prevent deterioration in the model's 's accuracy, the
20

---

improvement in hAURC we observed ranged from 3% to 5% compared to the pretrained baseline. We encourage future work explore this direction further.
L Technical Details
Most experiments consider image classification on the ImageNetlk [ validation containing 50 examples each for 1,000 classes. Additional evaluations were performed on the iNat2 1-Mmni dataset [24]. containing 50 examples each for 10,000 biological species in seven-level taxonomy. For experiments requiring calibration set, was sampled randomly from the validation set.
The evaluation was performed on 1,115 vision models pretrained on ImageNet1k [1] and 6 models
pretrained on iNat-21 [24] (available in timm 0.9.16 49 and torchvision 0.15.1 32 under the MIT license.
both available
All experiments conducted single machine with Nvidia A4000 GPU, The evaluation of all experiments on a single GPU took approximately two weeks.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the contributions of formalizing hierarchical risk and coverage, developing algorithms for hierarchical selective classification, and conducting extensive empirical studies on ImageNet classifiers.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The potential limitations of this work are discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: The paper provides a detailed description of the assumptions and the complete proofs for the theoretical results presented.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The supplementary material includes the source code necessary to replicate the findings
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The paper provides open access to the source code, along with sufficient instructions to faithfully reproduce the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Coaee//dddeSebmmmcma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: the details are provided in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports error bars and confidence intervals for the experimental results, shown in the relevant figures and tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section and Appendix detail all necessary compute information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform. in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification: The research adheres to ethical guidelines, including the responsible use of datasets and consideration of potential societal impacts of the work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: An example of societal impact was provided in the introduction, where an example of tumor classification demonstrates the works' impact, both positive and negative where hierarchical granularity is not used.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The necessary details are included in Appendix L
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The source code provided in the supplementary material is properly docu- mented to facilitate faithful reproduction.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "lXDdGY3vsq", "reviewer": "Reviewer_h9cA", "summary": "The paper proposes an extension of selective classification following a class hierarchy to reduce the specificity of model prediction when there is a high uncertainty. In particular, if the prediction confidence of a class is smaller than a predefined threshold, the proposed algorithm would proceed towards a higher class level in the hierarchical structure (the parent node), until the confidence of the considering node exceeds that threshold. The paper also formulises hierarchical risk and coverage, so that the area under curve can be used as a metric to benchmark different selective classification methods. An extensive number of pretrained classifiers on ImageNet dataset are then used to evaluate the proposed method and show promising results. The paper also include a PAC-like theoretical result, so that when finding the optimal threshold, one can select appropriate hyper-parameters to achieve their desired outcome with certain confidence level.", "questions": "Could the authors clarify if it can also be integrated into training a whole model to perform hierarchical selective classification?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper goes into details to provide an adequate background about selective classification, the definition of heirarchical risk and coverage as well as its area under curve as a metric to quantify the performance of hierarchical-based selective classification. It also links to previous studies in the same subfield. In general, the paper is well written and easy to follow.\n\nThe paper also includes a theoretical result on the guarantee of the learning algorithm when one wants to find the optimal thresholding value for their hierarchical selective classification. This simple theoretical results does strengthen the paper.\n\nThe paper also include an extensive number of experiments and ablation studies to provide insights into the newly-proposed method.", "weaknesses": "The paper relies on the setting with the following assumptions:\n- It is an inference rule. This means that the algorithm is used at test time only. If this could be even integrated into training is a plus.\n- It needs a validation set to find the optimal hyper-parameter $\\theta$, or the threshold (partly mentioned in the conclusion). It is understandable because there is no training involve here, so there is a need for that. However, in some cases, there may not be additional data available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive reply,\n\n*\"It is an inference rule. This means that the algorithm is used at test time only. If this could be even integrated into training is a plus.\", \"Could the authors clarify if it can also be integrated into training a whole model to perform hierarchical selective classification?\"*\n\nWhile we agree that incorporating a training regime to enhance HSC could be beneficial, our focus in this paper was on post-hoc methods, due to their immense importance and strength. Our method’s strength lies in its compatibility with any pretrained classifier. As newer and more advanced models become available, our plug-and-play method enables access to state-of-the-art classifiers without requiring any retraining, reducing the cost of enjoying the benefits of HSC to almost zero. This feature is highly appealing to practitioners, particularly given the high costs and complexity of training modern neural networks, and even more so with recent large multimodal models.\nFurthermore, it is worth noting that post hoc methods are well-established and widely utilized in practice. Thanks to their effectiveness, there exists extensive literature discussing post-hoc methods, temperature scaling [1] and split conformal prediction [2] are two prime examples of post-hoc practical approaches, among many others, such as [3,4].\nWith that said, we agree that training models can be beneficial for improving HSC performance. In fact, we have conducted extensive experiments on model training. Since hAURC cannot be optimized directly, we developed an alternative that could be optimized. Our best-performing method entailed training models to predict the lowest common ancestor (LCA) of pairs of samples, including identical pairs (intended for testing). To achieve this, we developed a hierarchical loss function that penalizes the model based on the hierarchical distance between the ground truth LCA and the predicted node. \nFor example: if the model is presented with an image of a Labrador and an image of a Golden Retriever, it should predict \"dog\" as the LCA. If the model instead predicts \"animal\", which is correct but less specific than \"dog\", it incurs a higher loss. We believed that this hierarchical loss function can improve the model's hierarchical understanding and, consequently, enhance HSC performance.\nAfter training models of various architectures (including ResNet50, ViTs, EfficientNet, and others) with this loss function and utilizing various other configurations, such as leveraging in-batch negatives for a richer training signal and multiple classification heads to prevent deterioration in the model's accuracy, the improvement in hAURC we observed ranged from 3\\% to 5\\% compared to the pretrained baseline. While we consider these results solid, we felt this has gone too far away from the main scope of the already packed paper and decided to leave it outside of it. We would appreciate your opinion, do you think it's preferable to include this method and results in the revision?\n\n*\"It needs a validation set to find the optimal hyper-parameter, or the threshold (partly mentioned in the conclusion). It is understandable because there is no training involve here, so there is a need for that. However, in some cases, there may not be additional data available.\"*\n\nWhile we acknowledge that collecting a calibration set (or reducing it from the training set) comes at a cost, we believe this limitation is \"soft\" in our case, since, as mentioned in the note following Theorem 1 and in the remarks in Appendix D, our guarantees hold for calibration sets of any size. Even for a small calibration set, our algorithm provides users with the flexibility to set the other parameters ($\\delta$ and $\\epsilon$) according to their requirements and constraints. For example: a user with a low budget for a calibration set who may be, perhaps, more interested in controlling $\\delta$ can still achieve a reasonable constraint by relaxing $\\epsilon$ instead of increasing the calibration set size.\nAdditionally, as we mentioned above, post-hoc methods that require an additional calibration set are widely accepted and used by the community thanks to their efficacy [1,2,3,4].\n\n[1] Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger. On Calibration of Modern Neural Networks. https://arxiv.org/pdf/1706.04599\n\n[2] Vladimir Vovk. Conditional validity of inductive conformal predictors. https://arxiv.org/pdf/1209.2673\n\n[3] Luís Felipe P. Cattelan, Danilo Silva. How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks. https://arxiv.org/abs/2305.15508\n\n[4] Ido Galil, Mohammed Dabbah, Ran El-Yaniv. What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers, ICLR 2023. https://arxiv.org/pdf/2302.11874", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mEzuDOqGZG", "reviewer": "Reviewer_vmdY", "summary": "The paper introduces a new framework for selective classification called hierarchical selective classification. In a setting where a hierarchy in the classification task is present, the authors devise a selection strategy that considers confidence at different levels of the classification hierarchy. Extensive experimental analysis is performed over 1115 ImageNet classifiers.", "questions": "I have a few questions/remarks regarding the paper.\n\n* Q1. I think the authors are not discussing an implicit (and, in my opinion, quite relevant) assumption of their strategy, i.e. the errors at different levels of the hierarchy are assumed to be the same. However, I argue this is not exactly the case in real life. For example, failing to distinguish a golden retriever from a labrador differs from failing to distinguish a dog from a spider. Can the authors elaborate on this point?\n* Q2. Can the authors discuss the points I highlighted as the main weakness?", "limitations": "The paper briefly discusses limitations. I think this section could be expanded, e.g. considering Q1.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main strengths of the paper are:\n\n1. the idea of applying selective classification in a hierarchical setting is novel;\n2. the theoretical analysis relies on conformal prediction, which guarantees the soundness of the results;\n3. the proposed framework can impact high-risk settings, as shown in the healthcare example.", "weaknesses": "Overall, I think the paper is solid. My main concern is that the empirical evaluation could be improved, especially regarding motivations and attention to detail. \nA few examples:\n* I do not fully understand why the authors focus so much on showing how different training regimes affect HSC performance. I guess this improves the overall predictive performance of the (hierarchical) classifier, which is expected to impact the HSC task positively.\n* As the authors correctly claim, the training regimes were not optimized for hierarchical selective classification. Despite the clear computation-wise motivation, I argue that including regimes optimized for HSC would make the empirical evaluation sounder.\n* a few lines are off: for instance, I would argue that line 279, i.e.,\n>CLIP achieves an exceptional improvement, surpassing 40%\n>\n does not match what is shown in Figure 4 (which shows an improvement below 40%).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback.\n\n*\"I do not fully understand why the authors focus so much on showing how different training regimes affect HSC performance. I guess this improves the overall predictive performance of the (hierarchical) classifier, which is expected to impact the HSC task positively.\"*\n\nThe methods presented in this paper are compatible with any pre-trained base classifier. As such, we were particularly interested in exploring how to guide practitioners towards choosing base classifiers with high HSC performance.\nThe experiments in this section draw inspiration from [1], who found that training regimes such as knowledge distillation significantly impact selective performance. Their findings provide valuable guidance for practitioners, helping them choose models trained using regimes that result in high selective performance. Following their findings, we set out to find if training regimes could contribute to hierarchical selective performance as well, and if our findings align with those presented in [1].\n\nWe initially shared your concern that the improvement in hAURC could be attributed to overall improvement in predictive performance resulting from the training regimes. This led us to analyze the improvement in accuracy compared to the improvement in hAURC. From the results below, it seems to us that the improvement in hierarchical performance is significantly more pronounced than the one in accuracy. Furthermore, We see examples of regimes with similar improvements in accuracy having significant differences in hierarchical performance improvements (e.g., \"Pretraining\" and \"Distillation\").\n\nWe present in the table below the mean improvement in accuracy compared to hAURC.\n\n| Training Regime | Median Accuracy Improvement (\\%) | Median hAURC Improvement (\\%) |\n|-----------------|---------------------------------|------------------------------|\n| CLIP            | 7.90                            | 38.14                        |\n| Pretraining     | 1.96                            | 19.47                        |\n| SSL             | 4.12                            | 18.97                        |\n| Distillation    | 1.48                            | 11.73                        |\n| Adversarial     | 0.85                            | 5.92                         |\n\nWe initially chose not to include this analysis in the paper due to concerns about the paper being already packed with too much information. Do you think it would be preferable to include these results in the revision?\n\n*\"As the authors correctly claim, the training regimes were not optimized for hierarchical selective classification. Despite the clear computation-wise motivation, I argue that including regimes optimized for HSC would make the empirical evaluation sounder.\"*\n\nWhile we agree that incorporating a training regime to enhance HSC could be beneficial, our focus in this paper was on post-hoc methods, due to their immense importance and strength. \nOur method’s strength lies in its compatibility with any pretrained classifier. As newer and more advanced models become available, our plug-and-play method enables access to state-of-the-art classifiers without requiring any retraining, reducing the cost of enjoying the benefits of HSC to almost zero. This feature is highly appealing to practitioners, particularly given the high costs and complexity of training modern neural networks, and even more so with recent large multimodal models.\nFurthermore, it is worth noting that post hoc methods are well-established and widely utilized in practice. Thanks to their effectiveness, there exists extensive literature discussing post-hoc methods, temperature scaling [2] and split conformal prediction [3] are two prime examples of post-hoc practical approaches, among many others, such as [1,4].\n\nWith that said, we agree that training models can be beneficial for improving HSC performance. In fact, we have conducted extensive experiments on model training. Since hAURC cannot be optimized directly, we developed an alternative that could be optimized. Our best-performing method entailed training models to predict the lowest common ancestor (LCA) of pairs of samples, including identical pairs (intended for testing). To achieve this, we developed a hierarchical loss function that penalizes the model based on the hierarchical distance between the ground truth LCA and the predicted node. \nFor example: if the model is presented with an image of a Labrador and an image of a Golden Retriever, it should predict \"dog\" as the LCA. If the model instead predicts \"animal\", which is correct but less specific than \"dog\", it incurs a higher loss. We believed that this hierarchical loss function can improve the model's hierarchical understanding and, consequently, enhance HSC performance.\n\nAfter training models of various architectures (including ResNet50, ViTs, EfficientNet, and others) with this loss function and utilizing various other configurations, such as leveraging in-batch negatives for a richer training signal and multiple classification heads to prevent deterioration in the model's accuracy, the improvement in hAURC we observed ranged from 3\\% to 5\\% compared to the pretrained baseline. While we consider these results solid, we felt this has gone too far away from the main scope of the already packed paper and decided to leave it outside of it. We would value your feedback, do you think this method and results should be included in the revision?\n\n*\"I would argue that line 279 does not match what is shown in Figure 4 (which shows an improvement below 40\\%)\"*\n\nThank you for pointing this out, we meant to say some instances of CLIP surpass 40\\%. We will report the median of the sample (38\\%) instead.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JiOwUK4W8Q", "reviewer": "Reviewer_XLCN", "summary": "The paper introduces a hierarchical selective classification technique that incorporates hierarchical risk and coverage. The authors additionally proposed an algorithm that guarantees target accuracy. Experimental results demonstrate the method's effectiveness.", "questions": "•\tRegarding results like in Table 2, would be possible to calibrate the coverage (as done on selective networks) for fair comparison? \n\n•\tHave the authors thought about the building the hierarchical tree structure as a pre-processing step? I asked that because such prior is key for wide applicability. \n\n•\tA well know problem of selective approaches, exposed on [1] is that given the non-differentiability of selection mechanism the binary function g is replaced by a relaxed function g: X → [0, 1], that way not performing selection during training, but instead assigning a soft instance weight to each training sample. The same effect is observed in the proposed method? \n\n[1] Gumbel-Softmax Selective Networks, https://arxiv.org/pdf/2211.10564.", "limitations": "•\tThe requirement for a hierarchical class tree prior to using the method is a limitation.\n\n•\tThe current experimental section is limited in terms of baseline approaches and datasets, making it challenging to gain a comprehensive understanding from the existing experiments.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Hierarchical selective classification is a new area and therefore the current method is one of the first techniques to deal with such problem. Its application to critical settings can be substantial.", "weaknesses": "•\tThe need of a prior tree among classes can limit its usage for complex scenarios. The construction of such tree can be a non-trivial step for the applicability of the approach. \n\n•\tThe main contribution looks an extension of previous methods for the hierarchical case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback on our paper.\n\n*\"The need of a prior tree among classes can limit its usage for complex scenarios. The construction of such tree can be a non-trivial step for the applicability of the approach.\"*\n\nWhile it's true that our algorithms require a tree structure,  there are many well-established predefined hierarchies available, such as WordNet and taxonomic data for plants and animals. Consequently, popular datasets like ImageNet and iNaturalist [1] have been built upon these hierarchical structures, providing ready-to-use trees. In cases where a hierarchical tree structure is not provided with the dataset, it can be generated using LLMs [2,3,4,5]. Thanks to your remark, will ensure these references are included in our paper's revision, so users are aware that our methods are applicable to datasets without published hierarchies as well.\n\n*\"The main contribution looks an extension of previous methods for the hierarchical case.\"*\n\nWe argue that our approach is not merely a simple extension, but a better alternative with potentially much better performance than classic selective classification. For instance, [6] boasts classifiers that achieve 99\\% accuracy on ImageNet with up to 47\\% coverage. However, these classifiers outright reject at least 53\\% of the samples, failing to provide any useful information to the user about them.\nIn contrast, our method transforms most of these otherwise rejected samples into valuable information.\nAs the example in our introduction illustrates, this information could, in some scenarios, be a matter of life and death. Thus, we feel this is no mere extension to the hierarchical case, but rather, a utilization of previously unused hierarchical knowledge to turn a large number of rejections into very usable information.\n\n*\"Regarding results like in Table 2, would be possible to calibrate the coverage (as done on selective networks) for fair comparison?\"*\n\nOptimizing selective classification under accuracy constraints involves approaches and algorithms different than those used for optimizing under coverage constraints. There is a well known distinction in the literature between selective algorithms optimizing for an accuracy constraint [7,8] and selective algorithms optimizing for a coverage constraint [9,10]. While it is definitely possible to construct an HSC algorithm optimized for coverage, our focus in this work is on accuracy constraints. Consequently, our comparisons are made against baselines that also target accuracy. While selective networks like SelectiveNet are notable baselines for coverage calibration, they are not directly comparable to our approach due to our focus on accuracy constraints.\n\n*\"A well know problem of selective approaches, exposed on [1] is that given the non-differentiability of selection mechanism the binary function g is replaced by a relaxed function g: X → [0, 1], that way not performing selection during training, but instead assigning a soft instance weight to each training sample. The same effect is observed in the proposed method?\"*\n\nThank you for bringing this paper to our attention. This seems like a more potent alternative to SelectiveNets.  We will make sure to reference it in our revision. \n\n*\"The current experimental section is limited in terms of baseline approaches and datasets, making it challenging to gain a comprehensive understanding from the existing experiments.\"*\n\nOur experiments were conducted on two large-scale, widely accepted datasets. We considered three algorithmic baselines, and compared *each* baseline with our algorithms on over 1,100 deep neural networks. Often, we repeated the evaluations per model 1,000 times, meaning a single comparison often consisted of over a million individual comparisons. Experiments of this magnitude require a considerable amount of resources. Existing works in the domain of hierarchical classification typically consider no more than two datasets, and evaluate only few models at most. Therefore, we hope you'll agree that our experimental section is adequately comprehensive, and that the results presented clearly highlight the benefits of our methods.\n\n[1] Grant Van Horn et al. Benchmarking Representation Learning for Natural World Image Collections. https://arxiv.org/abs/2103.16483.\n\n[2] Catherine Chen, Kevin Lin, Dan Klein. Constructing Taxonomies from Pretrained Language Models, NAACL 2021. https://arxiv.org/pdf/2010.12813\n\n[3] Maurice Funk, Simon Hosemann, Jean Christoph Jung, Carsten Lutz. Towards Ontology Construction with Language Models. https://arxiv.org/pdf/2309.09898\n\n[4] Qingkai Zeng, Yuyang Bai, Zhaoxuan Tan, Shangbin Feng, Zhenwen Liang, Zhihan Zhang, Meng Jiang. Chain-of-Layer: Iteratively Prompting Large Language Models for Taxonomy Induction from Limited Examples. https://arxiv.org/pdf/2402.07386\n\n[5] Grace Li, Tao Long, Lydia B. Chilton. Eliciting Topic Hierarchies from Large Language Models. https://arxiv.org/pdf/2310.19275\n\n[6] Ido Galil, Mohammed Dabbah, Ran El-Yaniv. What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers, ICLR 2023. https://arxiv.org/pdf/2302.11874\n\n[7] Yonatan Geifman, Ran El-Yaniv. Selective Classification for Deep Neural Networks, NeurIPS 2017. https://papers.nips.cc/paper_files/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html\n\n[8] Jia Deng; Jonathan Krause; Alexander C. Berg; Li Fei-Fei. Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition.\n\n[9] Yonatan Geifman, Ran El-Yaniv. SelectiveNet: A Deep Neural Network with an Integrated Reject Option, ICML 2019. https://proceedings.mlr.press/v97/geifman19a/geifman19a.pdf\n\n[10] Guy Bar-Shalom, Yonatan Geifman, Ran El-Yaniv. Window-Based Distribution Shift Detection for Deep Neural Networks, NeurIPS 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/4791edcba96fbd82a8962b0f790b52c9-Abstract-Conference.html", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NrxnE8rFWs", "reviewer": "Reviewer_9FfT", "summary": "The authors propose hierarchical selective classification, a method that selects the hierarchical granularity of its prediction based on uncertainty.", "questions": "* Could the authors please comment on the similarity to conformal prediction? \n* As such, how is [1] related and/or different?\n\n[1] Tyagi, Chhavi, and Wenge Guo. \"Multi-label Classification under Uncertainty: A Tree-based Conformal Prediction Approach.\" Conformal and Probabilistic Prediction with Applications. PMLR, 2023.", "limitations": "Limitations are adequately discussed apart from the necessity of assuming knowledge of the underlying class hierarchy.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written, and the proposed method is quite intuitive.\n* I like the idea that if uncertain, it makes sense to predict at a higher level of granularity.\n* The theoretical results & statements are sound.\n* Extensive experimental results showing the superiority of the proposed method to DARTS and a non-hierarchical baseline.\n* Applicability to pre-trained models", "weaknesses": "* My biggest uncertainty is the similarity of this work to conformal prediction. To me, it seems that this method is very similar to conformal prediction, where the set of possible prediction sets is restricted via this pre-defined hierarchy. While, as far as I know, it has not been explored, it decreases the perceived novelty. \n* A weakness of the setting rather than the method is that it assumes the knowledge of the underlying hierarchy. As such, the applicability is somewhat limited. The paper would benefit from a way to unsupervisedly learn this hierarchy, e.g. based on classes whose predicted probabilities are positively correlated.\n* As also touched upon in the concluding remarks, the method is post-hoc rather than being optimized during training, thus, likely not performing up to the highest possible level.\n* Minor: Line 158-159 is a worded badly, similar to \"... thus, we do A. Unlike others that do A, we do A+B\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our paper. We're glad you liked the paper and its underlying idea.\n\n*\"My biggest uncertainty is the similarity of this work to conformal prediction. To me, it seems that this method is very similar to conformal prediction, where the set of possible prediction sets is restricted via this pre-defined hierarchy. While, as far as I know, it has not been explored, it decreases the perceived novelty.\"*\n\nThank you for highlighting this reference. While we agree that this work merits discussion in our revision, we would like to emphasize a few key differences between our paper and conformal prediction in general, including [1]:\n1. [1] addresses multi-label classification, while we address hierarchical classification. In multi-label classification, prediction sets consist of lists of nodes of varying sizes. In contrast, hierarchical classification always returns a single node from the hierarchy. For large hierarchies, such as those in ImageNet (1,000 leaves) or iNat21 (10,000 leaves), uncertainty could lead to very large prediction sets in multi-label classification, rendering them impractical and uninterpretable.\n2. Unlike [1], which constructs its own tree, our approach is grounded in the use of a predefined, interpretable tree. This distinction is not just technical but fundamental to our methodology and performance.\n3. Our method builds on selective classification, specifically addressing the trade-off between risk and coverage, an aspect not covered by [1].\n\n*\"A weakness of the setting rather than the method is that it assumes the knowledge of the underlying hierarchy. As such, the applicability is somewhat limited. The paper would benefit from a way to unsupervisedly learn this hierarchy, e.g. based on classes whose predicted probabilities are positively correlated.\"*\n\nWhile it's true that our algorithms require a class hierarchy, there are many well-established hierarchies available, such as WordNet and taxonomic data for plants and animals. Consequently, popular datasets like ImageNet and iNaturalist have been built upon these hierarchical structures, providing ready-to-use trees. In cases where a class hierarchy is not provided with the dataset, it can be generated using LLMs [2,3,4,5]. Thanks to your remark, will ensure these references are included in our paper's revision, so users are aware that our methods are applicable to datasets without published hierarchies as well.\n\n*\"As also touched upon in the concluding remarks, the method is post-hoc rather than being optimized during training, thus, likely not performing up to the highest possible level.\"*\n\nWhile we agree that incorporating a training regime to enhance HSC could be beneficial, our focus in this paper was on post-hoc methods, due to their immense importance and strength. As newer and more advanced models become available, our plug-and-play method, compatible with any pretrained classifier, enables state-of-the-art performance without requiring any retraining, reducing the cost of enjoying the benefits of HSC to almost zero. This feature is highly appealing to practitioners, particularly given the high costs and complexity of training modern neural networks, and even more so with recent large multimodal models. Furthermore, it is worth noting that post hoc methods are well-established and widely utilized in practice. There exists extensive literature discussing post-hoc methods, temperature scaling [6] and split conformal prediction [7] are two prime examples, among many others, such as [8,9].\n\nWith that said, we agree that training models can be beneficial. In fact, we have conducted extensive experiments on model training. Since hAURC cannot be optimized directly, we developed an optimizable alternative. Our best-performing method entailed training models to predict the lowest common ancestor (LCA) of pairs of samples, including identical pairs (intended for testing). To achieve this, we developed a hierarchical loss function that penalizes the model based on the hierarchical distance between the ground truth LCA and the predicted node. \nFor example: if the model is presented with an image of a Labrador and an image of a Golden Retriever, it should predict \"dog\" as the LCA. If the model instead predicts \"animal\", which is correct but less specific than \"dog\", it incurs a higher loss. We believed that this hierarchical loss function can improve the model's hierarchical understanding and, consequently, enhance HSC performance.\n\nAfter training models of various architectures with this loss and utilizing various other configurations, such as leveraging in-batch negatives for a richer training signal and multiple classification heads to prevent deterioration in the model's accuracy, the improvement in hAURC we observed ranged from 3\\% to 5\\% compared to the pretrained baseline. While we consider these results solid, we felt this has gone too far away from the main scope of the already packed paper. Do you think it should be included in the revision?\n\n[1] Tyagi et al. Multi-label Classification under Uncertainty: A Tree-based Conformal Prediction Approach. Conformal and Probabilistic Prediction with Applications. PMLR, 2023.\n\n[2] Chen et al. Constructing Taxonomies from Pretrained Language Models, NAACL 2021. \n\n[3] Funk et al. Towards Ontology Construction with Language Models. \n\n[4] Zeng et al. Chain-of-Layer: Iteratively Prompting Large Language Models for Taxonomy Induction from Limited Examples.\n\n[5] Li et al. Eliciting Topic Hierarchies from Large Language Models. \n\n[6] Guo et al. On Calibration of Modern Neural Networks.\n\n[7] Vladimir Vovk. Conditional validity of inductive conformal predictors.\n\n[8] Cattelan and Silva. How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks. \n\n[9] Galil et al. What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers, ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers appreciated this paper’s contributions. In particular, this paper proposes a novel hierarchical selective classification, which is clearly distinguished to conventional selective classification and conformal prediction. Moreover, by introducing hierarchical structures into consideration, it effectively exploits information from rejected samples by the conventional selective classification. Also the hierarchical structure provides a favorable property: monotonicity in correctness and coverage, as in conformal prediction. Thus, I also vote for acceptance. 

In the final manuscript, please add promised contents discussed during the rebuttal period.
Decision: Accept (poster)