Fine-grain Inference on Out-of-Distribution Data with Hierarchical Classification
Authors: Randolph Linderman
Jingyang Zhang
Nathan Inkawhich
Hai Li
Yiran Chen
Venue: Submitted to ICLR 2023 (2023)

Abstract: Machine learning methods must be trusted to make appropriate decisions in real-world environments, even when faced with out-of-distribution (OOD) samples. Many current approaches simply aim to detect OOD examples and alert the user when an unrecognized input is given. However, when the OOD sample significantly overlaps with the training data, a binary anomaly detection is not interpretable or explainable, and provides little information to the user. We propose a new model for OOD detection that makes predictions at varying levels of granularity—as the inputs become more ambiguous, the model predictions become coarser and more conservative. Consider an animal classifier that encounters an unknown bird species and a car. Both cases are OOD, but the user gains more information if the classifier recognizes that its uncertainty over the particular species is too large and predicts “bird” instead of detecting it as OOD. Furthermore, we diagnose the classifier’s performance at each level of the hierarchy improving the explainability and interpretability of the model’s predictions. We demonstrate the effectiveness of hierarchical classifiers for both fine- and coarse-grained OOD tasks.

FINE-GRAIN INFERENCE ON OUT-OF-DISTRIBUTION DATA WITH HIERARCHICAL CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning methods must be trusted make appropriate decisions in real- world environments, even when faced with -i-f---iittiittion (OOD) samples. Many current approaches simply aim detect OOD examples and alert the user when an unrecognized input is given. However, when the OOD sample significantly overlaps with the training data, binary anomaly detection is not interpretable or explainable, and provides little information to the user. We propose a new model for OOD detection that makes predictions varying levels of granularity the inputs become more ambiguous, the model predictions become coarser and more conservative. Consider an animal classifier that encounters an unknown bird species and a car. Both cases are OOD, but the user gains more information if the classifier recognizes that its uncertainty over the particular species is too large and predicts "bird" instead of detecting it OOD. Furthermore, we diagnose the classifier's ' performance at each level of the hierarchy improving the explainability and interpretability of the model's predictions. We demonstrate the effectiveness of hierarchical classifiers for both fine- and coarse-grained OOD tasks.
INTRODUCTION
Real-world computer vision systems will encounter out-of-distribution (OOD) samples while making or informing consequential decisions. Therefore, it is crucial to design machine learning methods that make reasonable predictions for anomalous inputs that are outside the scope of the training distribution. Recently, research has focused on detecting inputs during inference that are OOD for the training distribution (Ahmed & Courville (2020] [Hendrycks & Gimpel 2017] Hendrycks et al. 2019)
Hsu et al.) 2020; Huang & 2i] aaksmmiaaraaaaan et 2017 Lee et al. 2018 [iang et al. 2018; Liu et al. 2020 Neal et al. (2018] Roady et al. 2020: Inkawhich et al.) 2022). These methods
typically threshold the model's "confidence" to produce binary decision indicating if the
sample is in-distribution (ID) or OOD. However, binary decisions based on model heuristics offer
little interpretability explainability.
The fundamental problem that there are many ways for sample to be out-of-distribution Ideally, model should provide more nuanced information about how sample differs from the training data. For example, if bird classifier is presented with a novel bird species, we would like it to recognize that the sample is a bird rather than simply reporting OOD, On the contrary, if the bird classifier shown an MNIST digit then it should indicate that the digit is outside its domain of expertise.
Recent studies have shown that fine-grained OOD samples are significantly more difficult to detect, especially when there is large number of training classes (Ahmed & Courville, 2020; Huang 202T: Roady et al. 2020 Zhang et al., 2021; Inkawhich et al. [202]). We argue that the difficulty stems from trying to address two opposing objectives: learning semantically meaningful features discriminate between IDD classes while also maintaining tight decision boundaries to avoid misclassification fine-gra -rnin OOD samples (Ahmed & Courville, 2020 Huang & Li 2021). We hypothesize that additional information about the relationships between classes could help determine those decision boundaries and simultaneously offer more interpretable predictions.
To address these challenges, we propose a new method based on hierarchical classification. The approach is illustrated in fig. [I] Rather than directly outputting distribution over all possible classes, as in a flat network, hierarchical classification methods leverage the relationships between classes to produce conditional probabilities for each node in the tree. This simplify the classification

---

In Distribution
Feature Extraction
Hiezarchical Path Probabilities
PriMero)
branbling
GrdEnch
Diga Crpoo
POO0O0o
(eltro) 8.
0.8
ladio bunling
Out uit-of-Distributiin
vertebrale
8.1
0.2
anemone fatt
0.040
sebraz
Rub
20
Lan
Inference Stopping Griterion
Dilfer
Pred
Uncertainty Threshold?
Depth=1
Depth=2
Prediction
ID:
Tzue
Goldfinch
False
Bizd
Connon Erzor Cases Undesprediction
Incorrect Classifscatiion
Oveepzediction
banbing
podftich
venabric
lohhn
puft
Pestector:
Gourdrri peher
Granndruh:
Figure 1: Method overview. Top: A ResNet50 extracts features from images and fully-comnece layers output softmax probabilities pn (1; for each set in the hierarchy H. Path wise probabilities are used for final classification. Path-wise probability and entropy thresholds generated from the training set Dirain form stopping criterion for the inference process. Bottom: Common error cases encountered by the hierarchical predictor. From left right: Standard error results from and incorrect intermediate or leaf decision, ID under- predictiorditiionn where the network predicts at a coarse granularity due to high uncertainty, OOD over ver-prediction where the OOD sample is mistaken for sibling node.
problem since each node only needs to distinguish between its children, which are far fewer in number (Redmon Farhadi 20077 Ridnik al. 2022I. It can also improve the interpretability of the neural network (Wan et al. (2021). For example, we leverage these conditional probabilities to define novel OOD metrics for hierarchical classifiers and make coarser predictions when the model is more uncertain.
By employing an inference mechanism that predicts at different levels of granularity, we can estimate how similar the OOD samples are from the ID set and at what node of the tree the sample becomes OOD. When outliers are encountered, predicting at lower granularity allows the system to convey imprecise, but accurate information.
We also propose a novel loss function for the hierarchical softmax classification technique to address the fine-grained OOD scenario. We propose hierarchical OOD metrics for detection and create hierarchical inference procedure to perform inference on and OOD samples, which improves the utility of the model. We evaluate the method's sensitivity to granularity under coarse- and fine- grain outlier datasets using ImageNet-IK ID-OOD holdout class splits (Russakovsky et al., 2015). Our in-depth analysis of the ID and OOD classification error cases illuminates the behavior of the hierarchical classifier and showcases how explainable models are instrumental for solving fine-grain OOD tasks. Ultimately, show that hierarchical classifiers effectively perform inference ID and OOD samples at varying granularity levels and improve interpretability in fine-grained OOD scenarios.
RELATED WORK
There are four main types of related works: those which focus fine-grain duito--distributtin detection; those which emphasize scalability to large numbers classes; those which leverage hierarchical classifiers; and those which utilize improved feature extactors.
2

---

Fine-grain OOD. Several recent works have identified poor OOD detection performance on fine-graie
datasets with current methods (Ahmed & Courville 2020
Roady et al. 2020; Zhang et al. (222I) Both (Zhang et al. 202T and (Ahmed & Courville 2020 highlight that selecting holdout classes
from the original dataset is a more realistic OOD) scenario that enforces detectors utilize semantically meaningful features to detect OOD samples rather than utilizing non-semantic differences between ID and OOD datasets to achieve good performance. |Raay et al. (2020) also explore the issue of OOD granularity by evaluating performance across coarse- and ane-ggained samples and outline need for new methods to address fine- -en-iiri OOD detection. These previous two works emphasize the importance of OOD granularity especially when developing solutions for real-world applications. We expand upon these works by designing OOD detection datasets at varying levels of granularity. We provide an in-depth analysis of performance across OOD granularity.
Scalable OOD. Current OOD detection techniques struggle to scale to higher resolution images or to tasks with larger numbers of ID classes (Ahmed Courville| 2020 Huang & Lij 202T Roady
ct al. 2020). Huang& & Li (202I) I) improve upon current OOD detection techniques' scalability by partitioning the classification task into smaller subsets of semantically similar categories, thereby reducing each decision's 's complexity. Grouping labels into exclusive subsets allows for an explicit "other" class to be added that is optimized to predict when the input is OOD for the current classifier. This approach uses the training data from all other subsets in similar manner to how outlier exposure uses auxiliary outlier data (Hendrycks et al.] (019). However, Hendrycks et al. (2019) utilize an entropy based optimization for the outliers instead of an explicit "other" class. Huuang & Li (202I) improve the scalability, but their approach is incapable of inference on ID and OOD data. Furthermore, Huang & Li (02T) choose 8 class groupings from WordNet without providing a systematic process of pruning the hierarchy. As the set of ID classes grows the relationships betweeen classes becomes more complex and the decision of how to group labels becomes increasingly arbitrary. We utilize the full WordNet hierarchy which provides natural boundaries between label subsets and systematically prune the hierarchy. Note that our method does not use auxiliary outlier data and developing a technique for improving uncertainty estimates via auxiliary data is left for future studies.
Hierarchical classifiers. Hierarchical classification methods have recently been exploited to improve the classifier's 's scalability to extremely large label space tasks (Ridnik al., 2021) and to enhance deep learning explainability (Wan et al. 2021). The current work seeks to utilize these properties to improve OOD detection methods. Ridnik et al. (202T) utilize the WordNet hierarchy (Fellbaum) 1998) to train a classifier on Imagenet-21K including 11,221 ID classes aiming to improve pretrained model weight available for transfer learning. Wan et al. (021) instead generate a binary decision tree from the learned weights of a standard softmax classifier using agglomerative clustering. Furthermore, Wan et al. (2021) thoroughly analyze the hierarchical classifier's behavior at each node in the tree to explain the network's 's prediction behavior. This method ensures that the hicrarchical relationships are based on visual similarity instead of human defined semantic similarity. However, for OOD detection tasks we find binary trees generated from pretrained model weights do not perform as well as shallower hierarchies for OOD tasks (table 4.. Furthermore, we build upon (Wan et al. 2021)'ss s interpretability study by utilizing the hierarchical structure to explain the common failure modes on ID and OOD data.
Another group of techniques incorporate hierarchical structures into the feature extractor (Ahmed ctt al., 2016 Yan et al., 2015) or training procedure (Alsallakh et al. 2018). In this work we do not consider specialized network architectures and training procedures because we are interested in comparing the performance of hierarchical vs. flat classification. We demonstrate in table and in section 3.5 the advantages of hierarchical classification for interpretable OOD. Exploring custom hierarchical architectures is promising area of future research to improve interpretable OOD methods.
Improved feature extractors. Vision transformers (Kolesnikov et al. 2021 and contrastive learning
Chen et al. 2020) methods learn improved feature extractors leading to gains in overall classification
accuracy as well as transfer learning outcomes. Recently, (Tack et al. 2020 Winkens et al. 2020)
utilize a SimCLR- based (Chen et al. 2020) architecture and (Ren et 2021 Vaze et al. (022) utilize ViT (Kolesnikov et al. 2021) to improve feature representations and OOD performance. Likewise, our hierarchical method will likely mutually benefit from learning improved feature extractors. However, due to the significant computational resources required to train such models we leave integrating hierarchical classifiers with contrastive learning and vision transformers for future studies.
3

---

3 METHOD
We train a hierarchical softmax classifier (section 3.1 to generate path probabilities at all nodes in the hierarchy with multi-objective loss function optimize for classification performance and OOD detection (section 3.2). Path-wise probability and entropy scoring metrics replace softmax probabilities for OOD detection incorporate model certainty along the prediction path (section (33] We employ these path-wise metrics as an inference stopping criterion to generate predictions on intermediate nodes (section B.4p. Finally, we measure hierarchical distance and accuracy to perform an in-depth analysis of model performance (section 3.5
3.1 HIERARCHICAL CLASSIFICATION
We define a hierarchy, H. as a tree-structured directed acyclic graph so that there unique path from the root node each leaf node. For notation, associate each node in the tree with an integer {0, 1, ,,...., , N} where O denotes the root node. Let par(n) E {0,..., , - 1} denote the parent of node n, let anc(n)
(0, 1} be the set of all ancestors of node n, and let ch(n) S (n + 1.....,. N) denote the set of children of node n. Finally, let y C (0. 1, 10.....,, N] denote the of internal nodes.
leaf nodes (i.e. nodes for which ch(n) = 0) and :.. y be the set
Each training data point has an input x; E IRd and label y: E V, which is associated with leaf node of the hierarchy. The training distribution, Derain { y:) }. is comprised of tuples of input images, T₁, and associated leaf nodes, Yr. For each node in the of internal nodes Z, we define Dn S Draain to be all the samples (zi, Mc) whose ancestors contain n. Likewise, define D all the examples that whose ancestors not contain n.
Given the input Tj, the network outputs probability distributions P.. [Pr.1. (chjic(n)) for each internal node n, where Pn.j 2 0 and >) Pn.j = 1. In practice, we model each pa, as a softmax function of the features in the penultimate layer of neural network. We parameterize a distribution on leaf nodes as the product of probabilities associated with each node along that path,
Pr(y: x4) = II Ppar(a).a-
(1)
aεanc(A)\(
As noted by (Wan et al. 2021), path-wise probabilities allow for errors intermediate nodes to be corrected as opposed to making "hard" decisions at each level of the tree and following the path to the prediction. Note that the path probabilities in eq. (I) form proper probability distribution such that keEy Pr(y; = & œ;) = 1. At test time, we take the network's 's prediction to be the leaf node zz).
with the highest probability, Os = arg max Pr(y,
3.2 HIERARCHICAL OOD LOss
To achieve high ID accuracy and reliable OOD detection we propose weighted multi-objective loss to optimize the hierarchical classifier. Formally, it is defined as,
Lsoft
Σ Wn Σ
I [onehot, (2), (x))
(2)
nEZ
|11 E [1...NN : anc(j))
W
(3)
N
Lother Σ
Σ
h (((c((())) p.((.)
(4)
nEZ (a.
L= a- Lsoft + Lotherr
(5)
where H(p. qi is the cross-entropy from p to q and par,, (y) is the one-hot vector for the ancestor of corresponding to node onehot,, (y) [1 (R: E anc(y)) l ch(n)..
The first objective optimizes the network for ID classification accuracy by applying cross-entropy to the network's predictions Pr(k]rr.) [Pr( k|x:)]kEy for each sample the training distribution,
4

---

The second objective (eq. (4 drives the probabilities at internal nodes that not along the path from root to ground-trun truth node to the uniform distribution, parameterized by size, (s)) with cross-entropy. This utilizes in a--distribution data outliers for all nodes in the hierarchy that are not one its ancestors.
Derain (eq. (2
As in Ridnik al. [201], find that weighting the cross-entropy contribution of each internal node improves optimization. We weight each node's loss, eq. (3). based on the proportion of the total number of classes (N) which are children of the node. This assigns higher weight nodes that are closer to the root of the tree as these decisions have the greatest effect downstream classification performance.
3.3 PREDICTION PATH ENTROPY OOD METRIC
We propose prediction path based OOD scoring functions for performing OOD detection with hierarchical classifiers. First, we propose using maximum prediction path probabilities calculated according to eq. ① which is hierarchical analog to max softmax probability for standard networks. Second, we propose using either the mean path-wise entropy, Hmean or the maximum path-wise entropy, Maxx metrics for OOD detection. Formally, these are defined as,
H mean (x:) = (nnc((:)
H[p.]
(6)
nEanc(
Hmax(T;)=
max [[pn),, mEanc()))
3.4 INFERENCE STOPPING CRITERION
Given hierarchical classifier optimized over Derain. define stopping criterion utilizing the performance statistics on the validation data. Specifically, we select true negative rate (TNR) on the IID data to decide inference stopping threshold from the micro-averaged receiver operating characteristic (ROC) curve. practice, this TNR threshold will be determined by the specific application's 's prediction fidelity requirements. Micro-averaged ROC curves are used to generate the TNR thresholds for each node in Z. We utilize path probabilities Pr(n|æ:; as the threshold score.
During inference the leaf node prediction ý determined, then the prediction path anc(ğ) is traversed from root to leaf. If any of the nodes in the path do not meet the TNR threshold, the parent node is chosen the prediction (fig. [ Both global path probability and node-wise probability and mean-, min-entropy were explored as TNR threshold metrics.
3.5 HIERARCHICAL ACCURACY AND DISTANCE
We analyze the hierarchical classifier's inference on ID and OOD samples with top-1 accuracy, as well as, average hierarchical distance. The groundtruth for OOD samples is the closest ancestor that is contained within ID hierarchy. For example, the OOD node junco in [ assigned the ID groundtruth node bird.
Furthermore, we consider the inference procedure's 's failure modes by decomposing the hierarchy distance into two parts: (1) the prediction and (2) the groundtruth distance to their closest common parent. Hierarchy distance is defined as the number of edges in the hierarchy between two nodes. By recording the groundtruth and prediction distances to the closest common parent can determine how frequently the model incorrectly predicts, overpredicts, and underpredicts for set of inputs. fig. [ (bottom) depicts common error cases that encountered and their corresponding hierarchy distances.
EXPERIMENTS
4.1 OOD DATASETS
When W, 14n 2 the form in eq. is equivalent to the entropy the leaf nodes H [y, Pr(k) D
5

---

Fine-grain OOD datasets. Some applications may face more extreme OOD examples than others. To construct OOD detection tasks with varying degrees of difficulty, we leveraged the fact that the Imagenet- K classes correspond to nouns in the WordNet hierarchy (Fellbaum 1998). We generated OOD sets by holding out subsets of Imagenet- classes in entire sub- trees of the WordNet hierarchy. Withholding large subtrees-those rooted at low depths of the hlerarchy-leads to coarse-grained OOD detection tasks, since the held-out classes are very different from the training classes. Hold- ing out small subtrees those rooted at nodes deep in the hierarchy-leate to fine-grained OOD-detection tasks. We created 2 datasets Imagenet-100 and Imagenet- K starting from 100 class subset of Imagenet-1k K classes and the full Imagenet- IK dataset. Table 2 provides summary statistics both Imagenet datasets.
Figure 2: IMAGENET- 100 pruned WordNet hierar- chy. Red edges correspond to OOD paths and blue to II).
We used the 100 class dataset, Imagenet-100, to evaluate the effects of additional fully-connected (FC) layers and hierarchy choice. We prune the WordNet hierarchy for this dataset by removing all nodes with single child and manually combining semantically similar leaf nodes into groups of Figure]]showe the hierarchy with randomly chosen coarse- and fine-grain OOD classes in red and the training classes in blue.
We used Imagenet- 1 K to evaluate the hierarchical classifier's scalability to large datasets. Again, we prune the WordNet hierarchy by first removing all nodes with single child. We then further pruned the hierarchy to a fixed number of internal nodes by iteratively pruning the node with the minimum entropy by merging its children. We use training distribution statistics to calculate the entropy for each internal node. However, we find that these simplified hierarchies do not improve ID accuracy or OOD performance.
Coarse-grain OOD datasets. For further comparison to other baseline methods and analysis of far- OOD data used coarse-grain OOD datasets for baseline comparisons. Specifically, we compared to the Naturalist (Horn et al. 2018), SUN (Xiao et al. 2010). Places365 (Zhou et al. 2018), and
Textures (Cimpoi al. 2014) subsets from (Wan et al. 2021)
4.2 MODEL TRAINING
All models were trained from scratch as the available pretrained weights were trained on the fine- grained OOD holdout classes. We used ResNet50 (He et al.) 2016) backbone for all models and trained for 90 epochs of stochastic gradient descent (SGD). We used learning rate 0. with learning rate decay steps with decay factor of 0.1 performed at epoch 30 and 60. The momentum and weight decay parameters were 0.9 and 10 respectively. We standardized the training hyperparameters to avoid performance differences due to the optimization procedure.
4.3 RESULTS
Fine-grain OOD performance We found that the hicrarchical softmax classifier (HSC) outperformed baseline methods on the Imagenet-100 100 dataset and performs within 2% of the best performing baseline for Imagenet- 1K dataset (table The hierarchical classifiers benefit from additional FC layers on top of the ResNet50 feature extractor due to the additional complexity of classifying into several disjoint sets (table 7). This also suggests that specialized hierarchical network architectures (Ahmed et al.. 2016: Yan et al. 2015 Alsallakh et al. 2018) that learn features specific to each node's 's classifier may further improve OOD performance. We assessed the effect of holdout class granularity and found that the softmax based OOD heuristics (MSP, ODIN, and prediction path probability) most sensitive to fine- grain OOD samples whereas MOS and path entropy metrics perform best on coarse-grain OOD shown in table Also, find that outlier exposure improves coarse-grain

---

Table 1: Hierarchical softmax classifier (HSC) performance on the Imagenet-100 and Imagenet- datasets. The Loofft and Lother weights (a, (3) and the OOD metric are given in parenthesis for each HSC model. OOD performance is measured by AUROC scores for the fine-, medium- and coarse- OOD classes as well as the overall OOD performance. The best performing baseline and HSC models are bolded, Each cell includes the performance statistics across 3 models trained with separate random seeds. For ensemble OOD methods (Lakshminarayanan et al. 2017) cells follow the format: "mean:.u/cesembble." Note that relative Mahalanobis Ren et al. (2021) performance is reported as outperformed the original method (Lee et al. 2018). All models are ResNet50 architectures trained for 90 epochs. All numbers are percentages.
Monez, (METHOD)
ACCURACY
PINE
MADIUM
COARSE
OVERALL
100
MSP HENDRYCRS & Gryvrs 2017
81 16,26 57V82.75 7,
72.47/0 91//73 62
92. 62/3677/94 38
90.25 67191 .94
81.26c0 53y82.75
88)/7/.3/6
93.2010. $6/99.08
MAHAAZ3ROIT INEX
81.26.0.59)
78.05(0.09)
91.
89.78/0.54)
MOS HUANG & Ei 2021
(2.41(0.02)
70.00/0.721
96.66/0.23)
93.66/0.22)
HSC
38c0 06V83.22
78(1 18//79
91.35 28V/22.35
HSC 1.0 Hnwae)
06993.25
72 11-27().88
96. 9000. /96.9866
93.92/0 20
HSC (a 1. 0.2. PRED)
1..85:0.44884.00
79. .1.. 76/880.77
95. (0.60..0//99..0000
92.29/0.15
HSC (a 1. 0.2. H
82.8550.14/84.05
79.4000677/776.35
97.2500.11/96.9
93V9500113993300
IMAGENET 1K
MSP HUNDRYCKS& & GEMPES. 2017
74.94.0 06V77.05
74. 30. (4000.244/74.90
19.3330.17,081.33
80 1000222779/82.7
77.96:0111779.57
LTANG
74.94:0.80 08/77.0
76.25(0. L11/77.62
79.84/0.11/$1..8
8L 95..155844.8
19.180.13//00.00
MOS
HOANG & Li 2021
75.00(0.43)
74.71(0.901
74.00(0.53
87.11:0.40)
77.32/0.59)
IISC
PRED)
73.79(0.13V)) 13V7613
12.73:0.47//73.12
78.33/0.27V500
77.07..24v/78.77 24/72.7//888
HSC 1.8 Heese)
73.79ku 13V76.51
84(0 643/51 .4
77, 03/0 29175,
82. 86c0 11/85 43
74.4700.317733
HSC (a+
1, 0.05. PRED)
74.46:0 06176.799
72.86.0 16//73.6
79. 48tn 201081445
82. 38r0 18//84 35
77.99( 41 41179.66
HSC 1, 0.03, Messe)
14.46:0.66/76.79
63. (6440074)/61.55 156
76.62:0. 351775.4
84.54(3.10:86...
54v73.45
OOD performance across all HSC metrics. Finally, we find that the Imagenet- trained ODIN detector is the best performer the 4 far-OOD datasets table 9 However, MOS the best performer on iNaturalist, SUN and Places when scaling the number of IT) classes to the Imagenet- K dataset.
Hierarchy selection. In table 4 evaluate the sensitivity to hierarchy depth and composition for Imagenet- 100 datasets with two human-defined semantic, WordNet (Fellbaum 1998) based hierarchies and a visually-derived binary decision tree induced from the learned weights of standard softmax classifier following the procedure from (Wan et al. (02T). The two-level hierarchy is created using the direct parents of the leaf-nodes from WordNet. We find that the performance across all OOD metries introduced in section 3.3 comparable. In particular, there is no apparent benefit to visually-derived hierarchies vs. human-defined semantic hierarchies. However, we believe that the hierarchy is a critical design choice and is likely application dependent. Specifically, the hierarchy's 's class balance, depth, and alignment with visual features are important characteristics to consider. In natural image classification domains, human-definee semantic structures may improve interpretability because they project image inputs into a human conceptual framework even though they may not perfectly represent the visual properties of the input.
Scalability to large ID datasets. In table 6 we find that HSC classifiers are able to scale to large ID datasets. However, the performance of the path- entropy based OOD detection metric H under-performs. We suspect that this can be attributed to deep, imbalanced hierarchies where some branches receive extremely low weights making optimization more difficult (eq. (3)) Indeed, find that simplifying the hierarchy with custom pruning and a 2 level hierarchy we can regain the lost OOD performance and even outperform the best baseline table [S] Therefore, well balanced hierarchies and models with sufficient capacity avoid this optimization problem.
5 ANALYSIS
Hierarchical classifiers decompose the classification problem into simpler intermediate tasks. By analyzing the model's 's confidence at each intermediate decision, we can understand where the model becomes uncertain. |Wan et al. (202I) show that through analyzing intermediate decisions we can explain the model's decision process to understand where the model makes mistakes and how behaves ambiguous labels, and we can use that insight to improve human trust in the predic-
7

---

tor. Therefore, hierarchical classifiers may greatly improve the interpretability and explainability compared to softmax classifiers.
We build off of this work by leveraging intermediate model confidence estimates to determine what level of granularity to make prediction. For ID data this corresponds to making more conservative predictions when the model is uncertain, based the intuition that correct, less specifie prediction model than an incorrect, more specific prediction. Similarly,
improves the user's confidence for OOD data the same method can indicate exactly where the sample diverges from the training distribution and therefore can predict the sample's parent class, allowing the user to make more informed decisions in the face of novel imputs.
5.1 THRESHOLDING TO REDUCE PASSIIVSS
First, we aim to understand the effects of OOD data the hierarchical classifier's performance and if thresholding is effective for detecting and predicting OOD samples. We plot separate classification decision in the hierarchy.
micro-ROC
curves (fig- 3) for synsets each corresponding The "artifact", "dog" and "bird" synsets include one or more OOD samples and the "ball" synset does not have any corresponding OOD samples (see fig. 2 Notice in Figure that when adding the activations of the OOD data ("OOD" curve) the number of false-positives increases and AUROC drops compared the ID-only curve because the OOD data being predicted more confidently than some ID data. This occurs across all synsets in the hierarchy even in the "ball" synset that does not contain any OOD descendants. However, when we employ path-wise probability based threshold 99% TNR on the training data ("THR" curves in fig. (3, the performance is recovered in all synsets. The micro-ROC curves for all synsets is displayed fig.
5.2 OOD INFERENCE PERFORMANCE
Next, we compare path-wise and node-wise thresholding methods (section 3.4 In path-wise thresholding, a global uncertainty threshold is set and the prediction is made at the deepest node that meets the threshold. In the node-wise method, threshold is set for each internal node to determine if the node is certain enough to make a prediction.
On the Imagenet- 100 dataset, achieve 73% accuracy the OOD samples while maintaining 74% ID accuracy using a path-wise probability threshold chosen at 95% TNR as witnessed by the blue line in Figure I0] fig. we plot the ID and OOD hierarchy distance over the TNRs used the prediction threshold to analyze the inference method's 's behavior. Note that the ODIN baseline is only capable of predicting a leaf-node not predicting all. This leads to the larger OOD hierarchy
ID-00D: oparating charactoristic
ort fact
dog
2.0
Average Iliererchy Distance vs. TNR Thresholds
08
4.0
ID Nide-wise
(D0 Indeopiee
ID ODIN
3.0
000
Pesiive
2.5
bd
bell
2.00
0.9
2.5
0.5
02
0.8
300
C23
030
0.75
1.00
000
020
0.0
075
1.20
Rate
Figure 4: Imagenet-100 ID and OOD average hiear- archy distance across TNR threshold values for path- wise and node-wise threshold metrics with ODIN baseline.
100 100 synset micro-ROS curves for ID data only, ID and OOD, and ID and OOD with a TNR=0.95 path threshold.
Figure 3: Imagenet-
8

---

distance across the TNR spectrum (fig. 4 grey), well as, the faster decline in ID performance compared to the node-wise thresholding method (fig. 4lbaack vs green). Figure 10 further shows the limitations of binary OOD detectors as they incapable of predicting OOD samples.
The large step changes and deviation of path-wise thresholding in Figure To and Figure Preflect that the path-wise thresholds cause the network to predict increasingly coarse nodes the confidence degrades with increasing depth (i.e. specificity, see eq. (T). Whereas the behavior of the node-wise thresholding technique changes the required certainty all nodes in the hierarchy leading smoother behavior and tighter standard deviations in the green and red lines in Figure Figure When the
distribution of OOD classes is balanced across granularity levels, the node-wise inference technique greatly outperforms the wath-wise technique due to the step wise nature of the path-wise wiih technique (figs. to
Accuracy is binary indicator of performance for each sample which is well suited to softmax and other flat classifiers that do not capture the relationships between classes. However, for inference on OOD data and prediction under uncertainty, hierarchy distance is useful differences between the prediction to the groundtruth. We show that OOD average hierarchy distance consistently decreases and the ID average hierarchy distance remains relatively constant (fig. 4, bottom). While the ID accuracy drops from 82.75% to 74.46% at the 95% TNR node-wise threshold, the average hierarchy distance decreases from 0.4045 to 0.4005 (fig. bottom right). This indicates that the increase in hierarchy distance from the ~8% drop in ID accuracy is counteracted by predicting the 177% of original error cases a lower specificity, thereby reducing the average hierarchy distance. Therefore, by allowing the hierarchical classifier to predict with less specificity, we can improve the overall prediction quality by removing uncertain leaf node predictions.
measure of the semantic
5.3 INFERENCE ERROR ANALYSIS
Error analysis is crucial to interpreting deep learning methods (Wan et al. 2021). We find that analyzing the hierarchy distance between prediction and groundtruth nodes for ID and OOD samples illuminates the common pathologies of our inference method fig. 12 Specifically, we find that both the path- and mode-wise OOD errors commonly occur from over over-prediction indicated by the concentration of samples along the prediction axis with ground truth distance of O. Conversely, the ID data is generally under-preed predicted by node indicating that the most uncertain ID samples are being predicted at coarser granularity fig. [2] In the standard ROC based metrics, the OOI) samples with a greater certainty than the lcaf-level ID predictions would be considered false-positives. We believe that analyzing the hicrarchical distance is a more appropriate compared to standard AUROC metrics for fine- -in-igrieedd OOD scenarios as it allows for predicting uncertain ID samples at a coarser level which is likely the desired behavior when the ID and OOD are very closely related.
CONCLUSION
In this work, we propose method for performing inference on OOD samples with hierarchical classifiers. We argue that in the fine-grain OOD scenario, detecting a sample as OOD is often not the optimal behavior when the sample is a descendant of an ID class. Hierarchical inference also improves ID) prediction when the model's uncertainty is large and more certain, less precise prediction is available. Hicrarchical classification enforces that intermediate predictions are made that can be directly analyzed to better interpret, explain, and validate the model's decisions prior to deployment. We provide empirical evidence to support these claims and consider hierarchical inference to be a promising direction for future research for creating trusted real-world machine learning systems.

---

REFERENCES
Faruk Ahmed and Aaron Courville. Detecting semantic anomalies. In AAAI, pp. 3154-3162, 2020.
Karim Ahmed, Mohammad Haris Baig, and Lorenzo Torresani. Network of experts for large-scale image categorization. Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), ECCV, volume 9911 of Lecture Notes in Computer Science, pp. 132116332.2555 Springer, 2016. doi: 10.1007/ URL /ttp0////////////ooo////////0//0
978-3-319-46478-777
32
Bilal Alsallakh, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren. Do convolutional neural
networks learn class hierarchy? IEEE TVCG, 24(1):152-1622 2018. doi: 10. 10.1109/TVC..2117
2744683. URL /tt.:::://o...og/2000.. htt.://rr/...11/////.... 201.9//T.V...22...7.7.. /ttp:///...//////////////////0/0
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations, In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607, 13-18 Jul 2020. URL https::/proceeddngg mll press/v119) www.che.nnhhiceoe
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, pp. 3606-3613. IEEE Computer Society, 2014. doi: 10.1109/CVPR.2014.46
org 10. 1109/CVPR. 2014 2014. .611
Christiane Fellbaum. WordNer: Electronic Lexical Database. Bradford Books, 1998.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp- 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and -istt--diittibuttoo examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The INaturalist species classification and detection dataset. In CVPR, pp. 8769-877 8778. Computer Vi- sion Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018 wtt.::tteeve.cecccooeeooocceceee content ever h018/html//e Horn The INaturalist Serctttt_peeessssee Cesiiiss_ee_______e____ 2018 pappp.ppp.... html
Yen- Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized ODIN: detecting out-of- distribution image without learning from -iit----dittibttion data. In CVPR, pp. 10948-10957, 2020.
Rui Huang and Yixuan Li. MOS: towards scaling out- distriiiitiitioonn detection for large semantic space. In CVPR, pp. 8710-8719, 2021.
Nathan Inkawhich, Jingyang Zhang, Eric K. Davis, Ryan Luley, and Yiran Chen. Improving -uitt---istiibutiin detection by learning from the deployment environment. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15:2070-2086, 2022. doi: 10. N0.1109/SSARRS.0222 3146362.
Nathan A. Inkawhich, Eric K. Davis, Matthew J. Inkawhich, Uttam K. Majumder, and Yiran Chen. Training sar-atr models for reliable operation in open-world environments. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:3954-3968 2021. doi: 202.
Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Un- terthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. In ICL.R 2021.
10

---

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NewrIPS, pp. 6402-6413, 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, pp. 7167-7177, 2018.
Shiyu Liang, Yixuan Li, and R. Srikant, Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurIPS, 2020.
Lawrence Neal, Matthew L. Olson, Xiaoli Z. Fern, Weng-Keet Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In ECCV, volume 11210, pp. 620-635. Springer, 2018.
Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. In CVPR, pp. 6517-6525, 2017.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. In ICML'21 Workshop on Uncertainty & Robustness Deep Learning, 2021.
Tal Ridnik, Emanuel Ben-Baruch, Asaf' Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. In NeurIPS, 2021.
Ryne Roady, Tyler L.. Hayes, Ronald Kemker, Ayesha Gonzales, and Christopher Kanan. Are open set classification methods effective on large-scale datasets? PLOS ONE, 15(9):1-18, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJC,, 115(3):211 1522, 2015.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via con- trastive learning on distributionally shifted instances. In H. Larochelle, M. Ranzato, R. Had- sell, M.F. Balcan, and H. Lin (eds.), NeurIPS, volume 33, pp. 11839-11852. Curran Asso- ciates, Inc., 2020. URL httpp:::/ptooeettee hetp:/eeeednnn..neeeeeeeeee nww.iis..p..... cc hctppaaae//222//11eee.. 8965876632d77 www.eeeeecccececeec00000-00-0000
pdf
Sagar Vaze, Kai Han, Andrea Vedaldi. and Andrew Zisserman. Open-set recognition: good closed-set classifier is all you need? In ICLR, 2022.
Alvin Wan, Lisa Dunlap. Daniel Ho. Jihan Yin. Scott Lee, Suzanne Petryk, Sarah Adel Bargal, and Joseph E. Gonzalez. NBDT: neural-backed decision tree. In ICLR, 2021.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R. Ledsam, Patricia MalWillamms, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, A. Taylan Cemgil, S. M. Ali Eslami, and Olaf Ronneberger. Contrastive training for improved out-of-distribttion detection. CORR, abs/2007.05566. 2020. URL.httts://arxixe org abtpl//g/abb//...ooo 05566
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, pp. 3485 3492. lEEE Computer Society, 2010. doi: 10.
URL https: http::::s///oo..oggg///// 1109/
CVPR. 5539970
Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition. In ICCV, pp. 2740 2748. IEEE Computer Society, 2015. doi: 10. 00.100//CCV22015.3...
URL https: /ww. http::///ooooogo/o//// wwt.://..../....//.............. 11w.//111./...... 2015.314.
Jingyang Zhang. Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards diitr---iistiittoonn detection in fine-grained environments, 2021. URL
https::/arxiv. http:////axxv...o/a..eo
/2o..////....../. 03917
Bolei Zhou, Ägata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. (4000))4452200 1464, 2018. doi: 00.10099PAM..222..22200999
TPATT 2211N.222772220000
11

---

A APPENDIX
A.1 COMPUTE RESOURCES
All experiments were run on an internal compute cluster with nodes containing NVIDIA RTX A5000 GPUs. Imagenet 100 experiments 8 hours for the longest running experiments. Imagenet 1K experiments were trained on 2 GPUs with data parallelization. Training for the Imagenet IK's longest running experiments lasted ~52.5 hours.
trained 1 GPU for 90 epochs which completed in
A.2 HIERARCHY STATISTICS
Table 2:
Imagenet dataset holdout set statistics. The number of leaf nodes that are held out due to trimmed branches at each level of granularity. The uniform probability used to choose the holdout nodes and the hierarchy depths for each granularity level are given for each dataset.
MAX DEPTH
DATASET
INTERNAL
DOUTS
LEAFS
MEDIUM
FINE
IMAGENET 100
28
100
IV1S
TEAFFS
6
15
5
10
100
LVLS
4-5
LEAFS
15
IMAGENET 1K
369
P=0.25
0.0622
0.0125
A.3 BALANCED IMAGENET 100 RESULTS
Figure 5: BALANCED IMAGENET 100 pruned WordNet hierarchy. Red edges correspond to 0OD paths and blue to ID.
12

---

Table Hierarchical softmax classifier (HSC) performance on the Balanced Imagenet 100 dataset. The Sofft and Lather weights (a, 3) and the OOD metric are given in parenthesis for each HSC model. OOD performance measured by AUROC scores for the fine-, medium- and coarse- OOD classes as well as the overall OOD performance. Each cell includes the performance statistics across models trained with separate random seeds. For ensemble OOD methods (Lakshminarayanan et al. 2017) cells follow the format: "meennsa/ensemble"" Note that relative Mahalanobis Ren et al. (2021) performance is reported as it outperformed the original method (Lee et al. 2018). All models are ResNet50 architectures trained for 90 epochs. All numbers are percentages.
MonEL. (METHOD)
ACCURACY
FINE
MEDIUM
AUROC
COARSE
OVERALL
BALANCED IMAGENET 100
MSP (HENDRYCKS
23.82.1
1100665973.99
01.07:0.58y773.3
92.52/0-49/993
82.04/0.21/83.72
ODIN
LIAN A.. 201K
00.85
79, 16 6/0.56/80. 37
74.35:0.57V75.84
96.099..53/996777
$6.8210.23/877..
ENERGY EI AL
20/0
80.85:0 23182.11
79. 3...0000099 5900599/80.44
74.22:0.57V75.
95. (5.9770.71/96.77
S6.79k.. 2677777
MAHAI A5TBB1 IKEN RT
2021
80.85 23)
83.07
72.66.0.5
91. (100199
85,36/0 64)
MDS
80.35c0.113
(1.49)0.65)
86.80.0.5)
74 74.23(1.05)
86.80/0.30
HSC
1,0 0. FRID)
81.19:0.3698/ 83
69 (4(4444///771 90/771 .22
71.57:1444733.1
93. 29.0. 18/99.444 .49
S1.72:0.44 (1.720..4//8..1 18
HSC
D. Howar)
81.19:0.26W0 1.83
08.12:1811/72.25
06.4660011 (1//96 69
Hean?
66.57
51.00:155y/73.44
19.4460.23://4..3
78.75:0.
IJSC (a 1.8 0. Hese)
$1.19/0. 26 .83
70.72(1 (5)/83533555
15.15/2.41/71..l
21c0.79/995 56
75.87 941/4421
HISC (a 1.8 0. ENERGY)
81.19c0. 26.88 183
75. 88(2 03//70. 26
70. 70.02:3.12V66 .97
98.28 28.0 11//98 24
S6. 10.1 20v/2. 87
HSC
0.2, PREM)
81,8300 10/82.97
73 (1 04//75 52
73.88: 19//55.66
94. 20.0 09//95 .22
84. 05, v8.. 4.77
Hawse)
10.82..9
38y74.54
HSC 1.9 D.2. Hoss)
81.83:0. 10.882.97
11.29/0.80 891/80.74
73.18:1.80 02y77.224
68.6 (4)/95 12
42.3000.4//86.38
HSC (a
1.8 0.2. Heer)
81.83:0. J0.822.9
73.72(1. 95//811.944
777.2660.85/75.20
95. 76.766..77/96..3 33
77.000.44//88.0
HICC (a L 0.2, ENERGY)
81.83:0.100 10/82.97
80.03(1.00/777 20177.799
73.50/2.09) /64.63
98. 18.0110044/88.62
87.93:0.33VS44.65
A.4 IMAGENET 100 HIERARCHY EXPERIMENTS
Table 4: ID and OOD sensitivity hierarchy selection on the Imagenet- dataset. H. type indicates whether the hierarchy is defined by human semantics or learned visual feature clustering. All numbers are percentages.
Path Entropy Max
Hierarchy H
H. Type
Accuracy
Path Predition
Mean
Min
2 Lvl WN
Semantic
82. 82.19)338)
91.73
93.43k0.08)
92.12.0.4 2/0.0))
93.08...1)
Pruned WN
Semantic 82.38(0.06)
91.33
93.9 93.92.0.20)
89, 160.66)
93.70
Binary NBDT Wan et al. 2021
Visual
81.28(o.88
91.33.0.29)
92.92.0.44
86.68o.18)
93.15
A.5 IMAGENET 1K HIERARCHY EXPERIMENTS
Table 5: Effect of hierarchy selection on Imagenet- 1K dataset. H type indicates whether the hierarchy is defined by human semantics or learned visual feature clustering. Note all models are HSC with and 3 in parentheses. All numbers are percentages.
Path Entropy
Hierarchy H.
H. Type
Aecuracy
Path Predition
Mean
Max
Min
Full (a. 9.9 0.05)
Semantic
74.34
77.73(0.55)
(4.2
76.68aa.99
62.04:o.5)
Custom Prune ta 1. 0.05)
Semantic
74. 12:000))
78.24a..7
71.3 1
75.9 75.97(0.15
60.95xa30)
2 Lvl (a 1, = 0.1)
Semantic
75.03.0.00
78.94 78.94...4
79.76..25)
79.52...20
69.91
13

---

A.6 IMAGENET 100 AND IMAGENET 1K ADDITIONAL OOD METRICS
Table Hicrarchical softmax classifier (HSC) performance on the Imagenet- 100 and Imagenet-IK datasets with additional OOD metrics. The Leoft and Lother weights (a, (3) and the OOD metric given in parenthesis for each HSC model. OOD performance is measured by AUROC scores for the fine-, medium- and coarse- OOD classes as well the overall OOD performance. The best performing baseline and HSC models are bolded. Each cell includes the performance statistics across models trained with separate random seeds. For ensemble OOD methods (Lakssminnrrayaaan et 2017) cells follow the format: "meanssa//esemmble performance is reported as it outperformed the original method (Lee al. 2018). All models are ResNe150 architectures trained for 90 epochs. All numbers are percentages.
Note that relative Mahalanobis Ren et al. (2021)
MonEL. (METROD)
ACCURACY
FINE
MEDIUM
AUTRIC
CDARSE
OVERALL
IMAGENET 100
MSP HINDRYCKS
2017
$1.26/0.52//22.25
12.4700.31173.66 62
92.62/0.57/944.8
20.2250..2//1..38
LIAGUE
$1.2500.51//52.75
72..931.875/74.23
95.90:0.47V96.71
93.20r0.36/94.00
ENERGY
LiV EIVELMM
31.2610.33//82.35
72.76(1 99//74
95.72:0.50/%8
93.02(0.399)
MARAI
4RDRE MMN FT
2021
81.2610.50
78.05(0.09)
91.34/u. 62)
89. 78.0.54)
MQ8 [HIANG
2021
82,41
70. (10
96.66/0.33)
93.6610.20 23)
(a 1. PRED)
82.38ro. 3,53
76. 76.78(3. 383/79,89
23.9330.22//55
33. 25/92.59
HSSC (a 1. =0, Hrran)
82. 3...8.3.3883.25
77. (1.2773..33/75.. 833/75.86 86
96. 90xu 11y /96. 889
93. 23.92/0.20v993 17
HSC 1. =/=00 0. Mraac)
38 3.35
76. 11 02)/75.22 21
91. 91,50,0 40V94.8
89. 46v/9 1.35
Hmm)
17/93.98
HSC
ENERUYY
12.3880.06//53.25
28c1
15
HSC (a 1. 0.2, PRED)
82,85/0, 14)/84.05
79.40.3. 76:88667
25.06:0.13/990
92, 2910 15333,33
HSCC (a 1. 0.2, Heen)
82.85 141/54.05
19.40c0.67//76.33 7655
97.23:0.11 119. 93.93
94. 08:00 13193,3993333
IICC
0.2.
82.85:0.1 14//54.05
78. 6...06/776666 02
94.57 2ww.2/9995550
91. 29/005
HSC
14/354.//4445
94.38o
HSC 8 ENURGY)
82.85 14:/8..5
(8)/71.3
98.22/0. 0998.51
17//3.71
IMAGENET IK
MSP FENDRYCKS & GIMPEL 2017
74, 94.0005/77.85
74. 213300024//74..90 9.90
79.33co 33/0.17/88 3.22
80. $0.42 19V8277
77 16/0.11//9.57
ODIN
LAN Al
2013
74 94,0 0777005777.
76.25ro 11/77 62
79,84:0
11281.82
$1.9500 15V84.07 02
79.1 13/80,98
177.05
76.20h0.06/77..6
81.45:0.17W% 17583.4
75.00:0.43)
74.7100.90)
74.00(0.53)
(7.11.0.40)
77.32:0.59)
HSC
1, 0. PRED)
1.1976633/76.5 51
72. 22.73r0.47)/7 47//73.42 42
78. 33.33300 37:80
80 30.6440.17882
77. 07.07.22/77.777 78
HSC
51
77 77.03300
11885,43
47
1.2 Mrae)
73.79/0. 11176..1
77.19c0 24y77.55
00000 22y/7.47
HSC (a 1.2 Hakdl
73.79/0.13/716 13)/16.51
52.9 S2.970235//5..0
$7.05,0 573665
69 69.41:0.21y/75.
58. 28.6440046y/055.5
HSC (a 1. O, ENERGY)
13.79/0.13//66.3
(2.36(1.52//511 1.40
78. 78.12:0.15/70 15776.77
$6. 36.27:0.99/888 09.00999/8..7
72. 22.3610.54)/1..2 1,24
IISC 1. 0.05,
74.460.063// 799669
72. 86.8656//73.6 73.69
79.40:0 1.45
$2.38:0.58y53 3:35558844.33
77.99:0.41/77.63
flee.))
HSC 1.2 0.05, Hace)
74.46. 06v776.9
73.83.0 37y/70
77 17.933c 0sy/88
87,0 30v78.800000000
83 1x/75.887
HSC (a 1. 0.06, Has)
74.46:000:/76.79
59. 13 58)/62.9 92
58, 96c0 $1./62 16
69. 4911 51y/7422
61. 50.000009990 9190o.01//65.3
IISC (a 1. 0.05, ENERGY)
74.460.06/76.79
$7. 17/466011)/54.5
78.05:0.71/777.6
88
88. 13..390.13889.8 81
73.7710.59/7319
14

---

A.7 BALANCED 100 HIERARCHY DISTANCE AND ACCURACY
Balanced 102: Meraçe Heleorcyy Jictanze vz TMR, Threcholds
Boionced 100: Heerardyy Accuracy TMR Threcholds
4.-
OCO Path year
Retivenie
35
30-
003 Pall
0.4
D03 Ridlowni
kade cane
020N
0O0
0.
48
0.2
0.5
00
TAR
Figure 6: Balanced Imagenet 100 average hicrarchy distance vs. TNR threshold values.
Figure 7: Balanced Imagenet 100 hieararchy
accuracy vs. TNR threshold values.
A.8 IMAGENET 1K HIERARCHY DISTANCE AND ACCURACY
Imagenet ak: Average Herarcit Oustance vs. TWA Threshots
Imagerat JR: farathy Accunccy is TMT Threstokd
0.001s
1830
050 ooe
1M0
Figure 8: Imagenet 1K average hierarchy dis-
Figure 9: Imagenet 1K hicararchy accuracy vs. TNR threshold values.
tance vs. TNR threshold values.
A.9 ID AND OOD INFERENCE ACCURACY VS. TNR
Accuracy vs. TWR Thresholds
B1 de
DCD OCU
00
3.0
Figure 10: Imagenet- 100 ID and OOD accuracy across TNR threshold values for path-wise wise and synset-wise threshold metrics with ODIN baseline.
15

---

A.10 SUPPLEMENTAL HIERARCHY DISTANCE CONFUSION MATRICES
Validation Set Hierarchy Distances TNR 99%
Validation Set Hierarchy Distances -
1900
2285
3062
Groundrruts Hecrarchy Distance
930
400
25
630
300
400
Groundrruth
200
200
Prodiction Hierarchy Distance
Pradiction Distance
Hierarchy Distances TNR=99%
Hierarchy Distances 95R=95%
120
Broundrruth Hisrardyy Distance
250
Oistance
0
0
0
0
0
200
24
0
0
0
80
2
0
0
150
0
0
60
0
0
0
0
0
0
0
40
0
0
0
0
20
0
0
0
0
Predictin Hierarchy Distance
Prediction Hierarchy Distance
(a) Node-wise Inference
(b) Path-wise Inference
Figure 11: Imagenet- 100 path- and nise-wise inference hierarchy distance confusion matrices on ID and OOD data.
16

---

A.11 SUPPLEMENTAL HIERARCHY DISTANCE CONFUSION MATRICES
Validation Set Hierarchy Distances TNR=99%
Validation Set Hierarchy Distances TNR=95%
1000
2783
3002
Groundruuth Hicrarchy Dizrance
Groundruth- Dotance
1012
180
0
800
466
430
19
2
0
400
89
26
37
0
28
23
15
3
660
300
400
200
200
100
Prediction Hiierercy Distance
Prediction Hierarchy Distancc
DOD Set Hierarchy Distances -9%%
OOD Hierarchy Distances 9NR=95%
Groundruth Hicrarchy Distance
230
Distemce
200
80
150
60
0
100
0
0
0
0
0
0
0
20
0
0
0
0
Hierarchy Distance
Hicrarchy
(a) Node-wise Inference
(b) Path-wise Inference
Figure 12: Imagenet- 100 path- and node-wise inference hierarchy distance confusion matrices ID and OOD data.
17

---

A.12 SUPPLEMENTAL MICRO-ROS CURVES
ID- ID-OOD Synsets: Receiver operating characteristic
000 0.11
100-221
000
000-244
0.0
strigedd instrument
fungus
Invertebrate
vertebrate
Insect
True Positive Rate
butterfly
feline
big cat
domestic cat
dog
monkcy
00-44
060-244
(00-841
10
False Positive Rate
Figure 13: Imagenet- 100 synset micro-ROC curves for ID data only, ID and OOD, and ID and OOD with TNR=0.95 prediction path threshold.
18

---

A.13 HIERARCHICAL PERFORMANCE BY DEPTH
Hierarchical classifiers provide interpretable decisions by introducing additional classification tasks at each internal node in the hierarchy. While this allows for improved interpretability and ability to perform verification, each additional classifier is another potential point of failure. We mitigate this by using "soft" decisions by using path probabilities for predictions instead of traversing the hierarchy in a top-down fashion akin to decision tree as suggested by Wan et al. (20TT). We plot percentage of classification errors that occur at each hierarchy depth (figs. 14 to 16) and compare the performance of hierarchical classifiers to flat classifier. To determine the depth of the error we follow the prediction path in a top-down fashion and record where the depth of the node where the error occurs. For example, for the Imagenet 100 dataset, if a sample's is groundtruth is "house finch" and the classifier predicts "puffer", then the error was made by the "vertebrate" " node's 's classifier and the error depth is 2 (see fig. 2). Note that this can be performed for any prediction regardless of whether the classifier can only produce Icaf node predictions. We calculate OOD error depths by assigning each OOD sample a ground truth to be the deepest internal node that is contained in the set of ID classes as we do when calculating hierarchical accuracy and hierarchical distance (section (5.2). As we are trying to evaluate the prediction accuracy and not the inference mechanism, we calculate an "oracle" accuracy for these plots. Specifically, an OOD prediction is "correct" if its path contains the OOD groundtruth path and predicted nodes that are at deeper levels than the groundtruth are ignored. For example, in the Imagenet 100 dataset "junco" is OOD and its groundtruth value is assigned to its closest ancestor that is contained in the ID set which is "bird". For the purposes of these plots, if a classifier predicts "house finch" whose path also contains "bird", the prediction is considered to be correct (see fig. 2
In figs. 14 to 16 it is clear that hierarchical and flat classifiers tend to misclassify similar difficult samples and have similar ID and OOD accuracy performance. Also, note that when the OOD accuracy performance drops in the Imagenet IK dataset (fig. [6],, the hierarchical classifiers begin to outperform the flat classifier. Since we keep the feature extractor the same between hierarchical and flat classifiers it is perhaps not surprising that they learn similar feature sets that lead to similar leaf node mistakes. The main focus of the current work is to compare the OOD performance of hierarchical versus flat models. This restricts our ability to explore additional hierarchical feature extractors as it would not allow for direct performance comparisons. An extension of the current work would explore hierarchical feature extractors to improve each node's classification performance by learning additional features that are optimized to distinguish between a given node's children. Another interesting future direction is to identify poorly performing internal nodes' classifiers during training and explicitly attempt to improve their performance through adapting the nodes' weights (eq. (3) or by developing other techniques.
Imapene:: 220 Erret
Belaneed 120 Emer Degt
HBC pe02.003
300
AC0.007
Figure 14:
Imagenet 100 errors by hierarchy depth for ID and OOD datasets. The mean ac- curacy is given in the right-most column and its scale is on the right-hand hand side.
Balanced Imagenet 100 errors by hierarchy depth for ID and OOD datasets. The mean accuracy is given the right-most column and its scale is on the right- hand side.
Figure 15:
19

---

Imagenet 1K Error Depth
p-0.3
HSc c=1, 8-0.20
Sofrrax val
HSC e-1. 2-1.3=0.3 000
0.2
0.35
0..
0.4
0.05
0.2
0.00
10
12
14
Accuiniyy
Bierarchy Dapth
Figure 16: Imagenet 1K errors by hierarchy depth for ID and OOD datasets. The mean accuracy is given in the right-most column and its scale is the right-hand side.
A.14 FULLY CONNECTED HEAD EXPERIMENTS
Table 7: Hierarchical softmax classifier (HSC) performance the Imagenet- 100 dataset when adding additions fully-connected (FC) layers to classification head. AUROC scores are provided for each OOD method: MSP (Hendrycks & Gimpel 2017), ODIN (Liang et al.) 2018), MOS (Wan al.) 2021). Note that the node-wise scaling of the HSC methods is different from table 6 causing the discripancy in HSC numbers. All numbers are percentages.
MODEL
ACCURACY
BASELINE AUROC
PATH PREDICTION
MEAN
MIN
IMAGENRT 100
90.25(0.66 62)
ODIN
81.26(0.55 53)
23.20 36)
92.41(0..0
23.66(0.22) 22)
MSP FC3
82.12(0.29) 29)
90.68/0.41)
82.12(0.29)
93.78(0.31)
MOS PC1
81.82(0.15)
93.49(0..9)
HSC
78. 78.3600.50
89 0900
92.39(0.43)
92.68c0.
HSC (a 1, 0.2)
83.05(0.12)
1(0..6))
93.93:0.27)
94.29(0.27)
HSC FC3 1.
81.90(0.08)
91. (1..33
93.86/0020)
93.46(0.18)
IISC FC3 1, 0.2)
82.73(0.24)
11.8000.46)
93.78;0.28)
94.57(0.20)
A.15
SUPPLEMENTAL OOD GRANULARITY PERFORMANCE
20

---

Table 8: Hierarchical classifier performance on the fine-grain Imagenet- IK datasets. AUROC scores are provided for each OOD method: (Ours) Hierarchical softmax classifier (HSC), MSP (Hendrycks) Gimpel, 2017). ODIN (Liang al. 2018), MOS (Wan et al.) (02T). All models are ResNet50 architectures trained for 90 epochs. FC: Number of fully-connected layers for classifier
AUROC
DATASET
MODEL
FINE
MEDIUM
COARSE
MSP
32.47.0.1
92.62(0.67)
ODIN
72.93(187)
95.90(a.47)
MOS
70.00ta.72)
96.66(0.23)
MSP FC3
72.55
(3.09(0.19)
ODIN FC3
69.05(o.75
97.08
IMAGENET 100
MOS FC3
69.78(2685)
96. 96.661..21)
HSC (PRED)
14.63(0.92)
91.02((.67)
HSC (Hmean)
73.09(0.95)
94. (4.96)).4
HSC (Hmin)
62.24(0.72)
96. (0.25))
HSC OE (PRED)
(1.11(1.98)
94. 960.6))
HSC OE (Hmean)
(0.01(1.93)
97. (0.07)
HSC OE (Inin)
(5114(1.40)
98. (
HSC FC3 (PRED)
(2.22(0.56)
94. (o.09().44)
HSC FC3 (Haaan)
71.55(0.99)
(0.83
HSC FC3 (Hiin)
59.69(1.57)
97.96 97.96(0.01)
HSC FC3 OE (PRED)
71.8610.17)
94. 46
HSC FC3 OE (Hmenn)
70.67(0.25)
96.86(0.33)
HSC FC3 OE mHinn)
68.87(0.11)
97.99 (7.99
MSP
74.30t0.24)
79.33
80.42(0.19)
ODIN
76.25(0.11)
81.95(0.15) 79.84...1)
MOS
74.7 (00.00)
74.00ko.53)
87. (0.40)
IMAGENET 1K
HSC (PRED)
72.73(0.47)
78,33
80.64(0.1)
HSC (Hmean)
64.84(0.64)
77.03.0.28
82. 12.86
HSC (Hmin)
52.97(2.35)
57.05.0.57
(0.41
HSC OE (PRED)
72.62(0.20)
79.19(0.1)
82.00(a.44)
HSC OE (Hmean)
63.6550..0)
76.60(a.9)
84.23(0225)
HSC OE (Tmin)
60.64(0.22)
59.02to.57)
69.60(.81)
A.16 PERFORMANCE ON FAR-OOD BENCHMARKS
21

---

Table 9: Coarse-grain OOD dataset baseline performance. AUROC scores are provided for each OOD method: (Ours) Hierarchical softmax classifier (HSC), MSP (Hendrycks & Gimpel 2017). ODIN (Liang et 2018). MOS (Wan et al. 2021). All models are ResNet50 architectures trained for 90 epochs. FC: Number of fully-connected layers for classifier
AUROC
ID DATASET
MODEL
INATURALIST
SUN
PLACES
TEXTURES
MSP
92.22(0)46)
93.62(0.23)
92.24(0.17)
88,60 60(0.57)
ODIN
95.60(o.34)
97.30ta.19 19)
96. 96.10.o.m 10.0.13)
94.85
MOS
93.50ca.04)
95.85x0.04)
94.72:0.12)
95.13:0.21)
MSP FC3
93.04(0.23)
94.77(0.05)
93.43.
90.07
ODIN FC3
96.52(0.10)
98. 98.00(0.07)
96.87(0.44
95.89/0.12)
IMAGENET 100
MOS FC3
(3.83(0.32)
95.89(0.00)
94.83(0.14)
94.78(0.18)
HSC (PRED)
91.22(0.29)
92.64(0.99)
91.25(0.76)
87.69t0.03)
HSC (Hmean)
91.83
93.40t0.55)
92.3310.71)
89.83.0.04)
HSC (Hmin)
86.50.0.8))
94.19(0.68)
92.6610.87
92.88.0.06)
HSC OE (PRED)
94.05(0.40)
95.73
94.59(0.03)
91.51 (0.15)
HSC OE (Hmean)
94.38(0.29)
96.3 (0.
95.45(0.02)
93.44:0.14)
HSC OE (Imin)
90.73
96.30xa.19)
95.06(0.19)
95.22.0.2
HSC FC3 (PRED)
(3.25(0.48)
95.25(0.66)
93.80xo.36)
90.82(0.21)
HSC FC3 (Hmean)
(4.09(0.47)
96.0810.29)
94.8770.88
92.79(0.17)
HSC FC3 (Hmin)
(1.93(0.4))
96.79(0.12)
95.53(0.10)
95.29(0.19)
HSC FC3 OE (PRED)
93.83(0.20)
95.68(0.12)
94.32(0.18)
90.82(0.29)
HSC FC3 OE (Hmean)
94.13co.m
96. 19t0
95.1660.1
92.69(0.22)
HSC FC3 OF (Hmin)
91. 91.20(o.76)
96.42
95.14:0.11
94.93
MSP
88.16
81.05
80.631o.18
80. 10.18
ODIN
91.05(0.33)
86. 12
84.72(0.31)
85,40ro.59)
MOS
95.22(0.35)
92.06(0.14)
90.59(o.13)
83.32(0.98)
IMAGENET 1K
HSC (PRED)
88.3600.50
80.77
80.36.0.15)
80.43x0.23)
HSC (Hmean)
89.92(0.38)
86.08:0.22)
84.75x0.10
82.93:o.29
HSC (Hmin)
80.0c 80.04.0.86
88.22(0.55)
85.6610.88)
7..77
HSC (PRED)
18.03
80.3 (0000)
79. 23)
80.3 (0.44)
HSC OE (Hmean
88.21
85.57
84. 7(.
82.51 (o.33)
HSC OE (Hmin)
72.39(3.60)
78.2 (
76.68(0.94
78.75(0.97)
22


Reviews:
[{"review_id": "ub2_4tpuat", "reviewer": "Reviewer_KPxY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": " In this paper, the authors propose a hierarchical classification framework for both fine-grained and coarse-grained OOD detection. This involves creating a tree based structure from the set of classification labels to establish a hierarchy. Once established, the authors introduce a new loss function to train the model maintaining this hierarchy. The authors also propose a new OOD detection score based on prediction path entropy.\n\nI have reviewed this paper before and the authors have addressed most of my previous concerns.\n", "summary_of_the_review": "I would recommend this paper be accepted. The AI safety community has focussed on developing OOD detection methods in a binary sense accept/ reject. Most papers develop methods which improves the OOD-AUROC marginally. We are at a state where just 1-2 points improvement using a new method won't have any practical value. This paper looks beyond the traditional binary treatment of OOD detection. This is very useful in real world deployment where just showing OOD without explanation was degrade user trust and lead to a bad experience overall. ", "strength_and_weaknesses": "Positives:\n\n- I completely agree with the authors that such an approach is highly interpretable and this is a very relevant research direction in AI safety. \n- Also, providing a less specific correct coarse prediction is more useful than an incorrect highly specific prediction, which the method provides. In practice most encountered OOD samples would be such near-OOD samples where such a response is preferred over just calling out OOD samples.\n- The paper provides a very interesting set of experiments over fine/ coarse grained OOD prediction and analyzes the result.\n- Very exhaustive comparison with existing methods.\n\nPotential improvements:\n\n- One key advantage of this approach is the explainability/ interpretability. Some metrics on this would be useful. For example: In depth case study on \"partial OOD\" samples (which is OOD in fine-grain but ID in mid-level of hierarchy) and mistakes on detecting this would be interesting.\n- Similarly, some downstream cost analysis (penalizing different kinds of mistakes at different hierarchies) could reflect how such a framework would be useful in the real world.\n- Showing some potential failure modes and discussion on them would be very useful for the community, especially to pave the direction for future work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TftqS8m0TT", "reviewer": "Reviewer_NPqs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary:\n\nThis paper examines multi-class classification in settings with out-of-distribution (OOD) data. Here, this refers to test data where the labels are not available at training time. The proposed solution is hierarchical classification, with the leaf nodes of the tree being the classes seen in training. The tree is given, so intermediate nodes may be interpretable. For classes not seen in training, the goal is to classify as deep within the tree as possible and ultimately conclude that the sample does not match any seen labels. To achieve these goals, training is done with the proposed hierarchical OOD loss. A decision to stop classifying deeper into the tree is made by a threshold on prediction path entripy measures. Experiments are conducted on a fine-grained OOD dataset and a coarse-grained OOD dataset. The difference is that fine-grained OOD datasets may exhibit deep classifications within the tree even for OOD samples, but coarse-grained OOD datasets mostly contain OOD decisions at a shallow level. Numerical performance appears reasonable.\n", "summary_of_the_review": "Summary\n\n* There are some potentially useful, if not too interesting, ideas.\n* In any case, the terminology and analysis should be improved.\n\n", "strength_and_weaknesses": "Strengths:\n\n* Seems like a reasonable approach.\n\n\nWeaknesses\n\n* The stopping condition, Section 3.4, merits further discussion on terminology (also for Section 5.1, 5.3). \n* Additionally, a histogram showing the distribution of entropy values for in distribution data (ID) and OOD might be informative on how well one might expect to determine the threshold.\n\n\nQuestion\n\n* Not really a strength or weakness, but the hypothesis seems to be that if a OOD sample is given, then at some node on the correct classification path each child should be equally likely (eqn 4). Is this really reasonable? Does this really appear to be the case when OOD (and ID) errors? Section 5.3 might be a place to expand the analysis.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1uYqq9o93ll", "reviewer": "Reviewer_Wf9S", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work authors adopt hierarchical classification for OOD detection and based on hierarchy they make predictions at varying level of granularity which could provide enhance examinability w.r.t OOD. This is indeed useful for OOD detection or open-set detection, and authors provide various qualitative & quantitative analysis based on adopted hierarchical classification.", "summary_of_the_review": "Authors present a good evaluation study by adopting hierarchical classification for OOD, but currently lacks novelty & further analysis would be required to interpret results/findings. I would encourage authors to address raised issues!", "strength_and_weaknesses": "Strengths:\n\nPaper is well written, and presents good analysis by adopting & proposing various methods and metrics within in hierarchical classification such as over-prediction vs under-prediction & average hierarchy distance, etc. This is a useful direction in general to be considered for real-world use-cases. It is interesting to observe that there is no apparent benefit to visually derived hierarchies vs human-defined semantic hierarchy in current evaluation setting.\n\nWeakness:\n\nAs the current method is fit based on likelihood in an autoregressive manner defined by hierarchy, it might important that at each level predictions are calibrated. Authors currently completely ignore calibration at intermediate levels which would result in more in-correct/over-predictions. It might be worthwhile for authors to consider calibration at each intermediate level. [3] also considers calibrated models at intermediate levels and also is evaluated for novelty detection within hierarchical classification framework, and it might be worthwhile to include this another baseline to consider.\n\nHierarchical classification is already adopted, and initial results are presented OOD setting in MOS baseline which authors included as one of baselines, and though proposed method is better in case of ImageNet 100 at fine-level its worse or on-par at coarse level especially for ImageNet1K.\n\nAlso authors it might be worthwhile to evaluate current method standard ImageNet openset detection benchmarks and also fine-grained/semantic OOD detection [1,2]\n\nReferences: [1] https://eval.ai/web/challenges/challenge-page/1041/overview or https://www.cs.cmu.edu/~shuk/open-world-vision.html [2] Vaze et al. OPEN-SET RECOGNITION: A GOOD CLOSED-SET CLASSIFIER IS ALL YOU NEED? (ICLR 2022) Proposes imagenet benchmarks & other fine-grained OOD detection tasks. [3] K. Lee et al. Hierarchical Novelty Detection for Visual Object Recognition (CVPR 2018)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0WE-W11Fcoh", "reviewer": "Reviewer_XEo4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper applies a hierarchical supervised classifier on which a measure whether an observed image should be considered \"in-distribution\" or \"out of distribution.\"  This considered both in quality -- how specific (\"fine-grained\") the classification is -- and degree, by at what point in the hierarchy to consider a sample \"OOD.\" ", "summary_of_the_review": "Based on a cursory understanding of the work, it appears to have value, but it's hard for me to put a finger on exactly the novelty and soundness of the work.  It does illustrate some interesting characteristics of hierarchical image classification, but comes to no clear conclusions. ", "strength_and_weaknesses": "The idea of learning a hierarchical classifier is valuable, in terms of giving better insights into the results, scaling to larger numbers of classes and assigning a more appropriate level of granularity to the classification. \n\nIn concept the classifier's marginal probability ascribed to the image -- the probability that it would arise in the population, irrespective of it's class should give a measure of ID versus OOD.  This would be a principled alternative to the measures used in the paper. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject