Hierarchical classification at multiple operating points
Authors: Jack Valmadre
Venue: NeurIPS 2022 Accept (2022)
Keywords: hierarchical classification
image classification
class hierarchy
inaturalist
imagenet
deep learning

Abstract: Many classification problems consider classes that form a hierarchy. Classifiers that are aware of this hierarchy may be able to make confident predictions at a coarse level despite being uncertain at the fine-grained level. While it is generally possible to vary the granularity of predictions using a threshold at inference time, most contemporary work considers only leaf-node prediction, and almost no prior work has compared methods at multiple operating points. We present an efficient algorithm to produce operating characteristic curves for any method that assigns a score to every class in the hierarchy. Applying this technique to evaluate existing methods reveals that top-down classifiers are dominated by a naive flat softmax classifier across the entire operating range. We further propose two novel loss functions and show that a soft variant of the structured hinge loss is able to significantly outperform the flat baseline. Finally, we investigate the poor accuracy of top-down classifiers and demonstrate that they perform relatively well on unseen classes.

Hierarchical classification at multiple operating points
Jack Valmadre
Australian Institute for Machine Learning, University of Adelaide jack. valmadre@@dellldde edu. au
Abstract
Many classification problems consider classes that form hierarchy. Classifiers that are aware of this hierarchy may be able to make confident predictions at a coarse level despite being uncertain at the fine-graines level. While it is gener- ally possible to vary the granularity of predictions using a threshold inference time, most contemporary work considers only leaf-node prediction, and almost no prior work has compared methods at multiple operating points. We present an efficient algorithm to produce operating characteristic curves for any method that assigns a score to every class in the hicrarchy. Applying this technique to evaluate existing methods reveals that top-down classifiers dominated by a naïve flat softmax classifier across the entire operating range. We further propose two novel loss functions and show that a soft variant of the structured hinge loss is able significantly outperform the flat baseline. Finally, we investigate the poor accuracy of top-down classifiers and demonstrate that they perform relatively well unseen classes. Code available online at https: : //gi thub. com/ vlmdr seaarr/hierclc.
Introduction
Many classification problems involve classes that can be recursively grouped into a hierarchy of progressively larger superclasses. This hierarchy can be represented by directed graph where the nodes are classes and the edges define a superset-of relation. Knowledge of the hierarchy can be useful in many different respects. For example, mistake severity can be quantified using distance in the graph [9, 2), classifiers can make predictions at coarse level to avoid an error at the fine-grained level [11, 39], classes with few labels in long-tailed distribution can benefit from the examples of similar classes [39]. and a cascade can be used to reduce inference time [26, 16, 14].
Whereas many recent works consider only leaf-node prediction, are interested in the setting where the classifier may predict any class in the hierarchy, including internal nodes. this setting, prediction involves inherent trade-off between specificity and correctness: less information but have a greater chance of being correct. The trade-off can typically be controlled using inference threshold, analogous the trade-off between sensitivity and specificity in binary classification or that between precision and recall in detection. However, while is standard to evaluate these problems using trade-off curves, most works hierarchical classification consider only a single operating point. It is important to consider the full trade-off curve in order to ensure fair comparison and enable the selection of classifiers according design specifications. This paper presents an efficient algorithm to obtain trade-off curves for existing hierarchical metrics.
general predictions contain
The proposed technique is subsequently used to compare the trade-offs realised by different methods. Given the effectiveness of deep learning, we focus on loss functions for training differentiable models. Experiments the iNat21 [36] and ImageNet- 1k [10] datasets for image classification reveal that a naive flat softmax classifier dominates the more elegant top-down classifiers, obtaining better
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

accuracy at any operating point. We further introduce a soft structured- structured-reddiction loss that dominates the flat softmax.
While it was already known that top-down approaches provide worse leaf-node predictions than a flat softmax [31, 2], we did not expect this to hold for non-leaf predictions. We hypothesise that the top-down approaches obtain worse accuracy than the "bottom-up" flat softmax because the coarse classes can be highly diverse, and thus better learned by union of distinct classifiers when the fine-grained labels are available. To support this hypothesis, we show that training a flat softmax classifier at lower level and testing at higher level provides better accuracy than training at the higher level. Finally, we consider a synthetic out-of-distribute problem where hierarchical classifier that explicitly assigns scores to higher-level classes may be expected to have an advantage.
The key contributions of this paper are as follows.
We introduce novel, efficient technique for evaluating hierarchical classifiers that captures the full trade-off between specificity and correctness using a threshold-based inference rule.
We propose two novel loss functions, simplification of Deep RTC [39] which we refer to parameter sharing (PS) softmax. While is ineffective, soft-max-margin and PS softmax achieve the best results. We conduct an empirical comparison of loss functions and inference rules using the dataset and its hierarchy of 10,000 species. The naïve softmax is found to be surprisingly effective, and the simple threshold-based inference rule performs well compared to alternative options.
and soft-max-margin and entertain a
We evaluate the robustness of different methods to unseen leaf-node classes. The top-down methods are more competitive for unseen classes, while PS softmax is the most effective.
2 Related work
There are several different types of hierarchical classification problem. In the terminology of Silla and Freitas [33], we consider problems with tree structure, Single-Path Labels (SPL) and Non-Mandatory Leaf-Node Prediction (NMLNP). Tree structure means that there is a unique root node and every other node has exactly one parent, Single-Path Labels means that sample cannot belong to two separate classes (unless one is a superclass of the other) and Non-Mandator Leaf-Node Prediction means that the classifier may predict any class in the hierarchy, not just Icaf nodes. This work assumes the hierarchy is known and does not consider the distinct problem of learning the hierarchy.
MINP with deep learning. Several recent works have sought to leverage a class hierarchy to improve leaf-node prediction. Wu et al. [38] trained softmax classifier by optimising "win" metric comprising a weighted combination of likelihoods on the path from the root to the label. Bertinetto et al. [2] proposed a similar Hierarchical Cross-Entropy (HXE) loss comprising weighted losses for the conditional likelihood of each node given its parent, as well as a Soft Label loss that generalised label smoothing [27] to incorporate the hierarchy. Karthik et al. [20] demonstrated that a flat softmax classifier is still effective for MLNP and proposed an alternative inference method, Conditional Risk Minimisation (CRM). Guo al. [18] performed inference using an RNN starting from the root node. Other works have proposed architectures that reflect the hierarchy (e.g. [41, 1, 43D). This paper seeks to compare loss functions under a simple inference procedure.
MON- methods. Comparatively few works have entertained hierarchical classifiers that can predict arbitrary nodes. Deng et al. [11] highlighted the trade-off between specificity and accuracy, and sought to obtain the most-specific classifier for a given error rate. Davis et al. [7, 8] re-calibrated flat softmax classifier within a hierarchy using held-out set and performed inference by starting at the most-likely leaf-node and climbing the tree until a threshold was met, comparing methods at several fixed thresholds. Wu al. [39] proposed the Deep Realistic Taxonomic Classifier (Deep RTC), which obtains a score for each node by summing over those of its ancestors [32] and whose loss evaluated at random cuts of the tree. They perform inference by traversing down from the root until a score threshold is crossed (by default, zero). In the YOLO- detector, Redmon and Farhadi [31] introduced a conditional softmax classifier that resembled the efficiency-driie -diven approach of Morin and Bengio [26]. The model outputs a concatenation of logit vectors that each parametrise (via softmax) the conditional distribution of children given a parent. It was noted to provide elegant degradation but the hierarchical predictions were not rigorously evaluated. Brust and Denzler [4] generalised the conditional approach multi-path labels a DAG hierarchy by replacing the softmax with sigmoid for each class given each of its parents. Inference was performed by seeking the class

---

with the maximum likelihood excluding its children, which may predict non-leaf nodes. Most of the above loss functions will be compared in the main evaluation.
Hierarchical metrics. There are many ways to measure the accuracy of hierarchical classifiers [6, 21J. For the specific case of NMLNP, it is important consider metrics for both correctness and specificity. The Wu-Palmer metrics [40] measure precision and recall using the depth of the Lowest Common Ancestor (LCA). Deng et al. [11] proposed to measure specificity using information content account for an imbalanced tree, and Zhao et al. [42] modified the Wu-Palmer metrics to use information rather than depth. These are the metrics that we adopt, although our technique can be applied construct curves for any simply metric. Examples of other metrics include the fraction of examples receiving non-root classifications [7] and the fraction of true-negative leaf nodes for correct predictions [39].
Structured prediction. The max-margin structured-prediction loss has seen historical use for hierarchical classification with linear SVMs, originally for document classification [5] and later to achieve efficient inference [34]. More recently, a soft max-margin loss [29, 15] has been used to train
deep models for snveencetto-squeene learning [13] and long-tail classification [25]. We believe this paper is the first to recognise its utility for hierarchical classification with deep learning.
3 Problem definition
3.1 Class hierarchy
Let J' be the set of classes, including both leaf nodes and their superclasses. The hierarchy is defined by a tree with edges C y², The edges define a transitive, non-strict superset relation 2 over the classes. It is helpful to think of the classes as sets of samples. A sample that belongs to class y also belongs to the ancestors (superclasses) of y: that is, X E y and y S y implies x E y'. Hierarchical classification can be seen as multi-label classification, since samples belong to multiple classes simultancously. However, samples cannot belong to arbitrary subsets of y, as belonging to one class implies belonging its superclasses. Furthermore, we consider only class hierarchies which siblings are mutually exclusive; that is, a sample & can only belong to two classes and y' if one is a superclass of the other. The problem is therefore assign a single label y E y to example, with this label signifying membership in class and all of its superclasses. While any superclass of the ground-truth label deemed be a correct classification, is preferable for the classifier predict the most-specific correct label.
We briefly introduce notation for the tree. Let r E Y be the unique root node, let TTy) E 3 be the unique parent for every node y E y (r). let C(y) S y be the children of node y, let L S ) be the set of leaf nodes, let A(y) y be the ancestors of node y. and let D(y) C y be the descendants of node y. We define the ancestors and descendants inclusively, such that D(y) n A(y) = {y}. It will also be useful introduce L(y)= L n D(y) denote the set of leaf descendants of node y and S(v) = (()))) to denote its siblings. The superset relation over classes is equivalent to the ancestor relation in the tree; that is, u 2 e iff E A(v).
We focus on models that map a sample x to a conditional likelihood p(ylæ)) over all labels in the hierarchy. If we consider y with its superset relation and mutually exclusivity of siblings as an event space, then a function p :::3 y + [0, 1] must satisfy
Vu E Y:: p(u) 2 Evec(w) P(V)
to be a probability function on y. If this holds with equality, we say that the children are exhaustive. We generally expect that the root node is uninformative p(r)
3.2 Metrics
Metrics M(,, ú) measure the quality of predicted label ý for ground-truth label y. When making non-leaf predictions, it is important to capture both correctness and specificity, either using pair of metrics or a single combined metric. The simplest such pair are the binary metrics
Correct 9) = [ 2 vl
Exact (y. 9) = [ū = vl
(2)
If the ground-truth label is not leaf node, then it is possible for the predicted label to be below it, ğ C y. This is not considered error, and we replace <- y where this occurs.
3

---

Since the Exact metric is often too striet, it is useful to introduce non non-binary measure of label specificity. Two popular choices are the depth of the node d(y) and its information content (y)
- -logp(y) = log [ log [(v) (assuming a uniform distribution over the leaf nodes [11, 42 We
prefer information as better accounts for an imbalanced tree; in perfect k-ary tree, T(y) x d(y).
Rather than directly measure the specificity of the predictions [11], the specificity measure can be
used to define precision and recall metrics [42] using the Lowest Common Ancestor (LCA):
(ycall(y, 9) [[cca(y, 1))
Precision(y, 9) / (Icaly 9))
T(y)
T
Correct and Precision are measures of correctness (decreasing with depth), whereas Exact and Recall are measures of specificity (increasing with depth). Each type should not be used without the other.
4 Inference and loss functions
We adopt the paradigm that differentiable model f with parameters é maps an input x to a real
vector 0 fa, 6) that parametrises the conditional likelihood p(-[z) = (() E [0. 1|" Learning will be performed using Stochastic Gradient Descent (SGD) to minimise the expected value of a loss function (y, 0) on training set. Inference will be performed using a function & [0. 1 + y to obtain أ = (l)). We consider the training method to be defined by the pair (f, q) and the inference method to be defined by ç. The dimension of & depends on the training method.
4.1 Inference functions
In standard flat classification, inference simply selects the most likely class E(p) arg maxyey p(y)- However, in the hierarchical setting, this would always select the uninformative root node. We can consider leaf inference {(p) = arg maxye p(y). however this will never select an internal node, We propose confidence threshold inference, taking the maximum-informatee
node
(p) = arg maxyey [yy) subject to p(y) >
When 7 2 0.5 and p satisfies (1), there is a single path in the tree whose nodes satisfy the threshold, i.e. p(y)= > > y E A(ÿ). Inference can therefore be performed by traversing down from the root and the maximiser is unique assuming that (yy is strictly increasing on the edges of the tree. While this property is elegant, paper. We refer to the special case of t = 0.5 as majority inference. Rather than consider a single threshold propose to study the operating characteristic curve in the following section.
also entertain arbitrary p and < 0.5 in the remainder of the
Two straightforward variants are plurality inference, which instead seeks the maximum-infowe informaation label which is more likely than any other non-ancestor, and information threshold inference, which reverses the roles of the terms to instead maximise p(v) subject to T(y) 2 Ç. While the latter can also generate operating curve, it does not provide adaptive specificity according to confidence.
Deng et al. [1I] proposed inference rule that maximises the expectation of transformed reward
(p) arg max (I()) + A)p(v)
(5)
with the factor À 2 0 chosen by bisection to yield the desired accuracy on a held-out set. Each value of À represents particular balance between expected reward I (v)p(t) and confidence p(y). As X > oc, the maximiser eventually becomes the root node, which maximises p(y). Although p(v) and Tιy) monotonic with respect to depth (decreasing and increasing, respectively), there is no such guarantee for the product T(y) (y)p(y), and therefore an increase in À is not guaranteed to move the call this expected information inference.
prediction towards the root. For X =
Karthik et al. [20] and Deng et al. [9] proposed conditional risk minimisation (CRM) for leaf-node prediction. This scheme selects the leaf-node label ý to minimise the expected cost E(C(Y, ū)] with Y ~ p(((z) being a random leaf node drawn from the predicted conditional likelihood
(6) If satisfies (1), then CRM can be extended to predict non-leaf nodes by performing both the optimisation and the expectation over all nodes y using the exclusive likelihood p(y) of a node and not its children
{(p) = arg mingec Evec Clu, v) p(u)
{p)) arg mingey EvEy C(u, p(u).
p(u) = p(u) Evec(w)p(())
(7)

---

If we choose Cly. B) Correct(y, () (ū) to maximise the expected reward, then CRM coincides with expected information inference when the ground-truti label is leaf node. Different choices of Cyy. ú) could be used to achieve different operating points, although it may be non-trivial to compute a complete trade-off eurve with this approach.
4.2 Loss functions and likelihood parametrisations
We first present existing loss functions before introducing our novel loss functions. We use the notation that vector œ RR has elements Vi R indexed by u E U and zv E R denotes the subvector on V _ U. The sigmoid function is denoted (xx) = 1/(1 exp(-x)) and scalar functions apply elementwise vectors. The softmax function and cross-entropy loss operate on given index
set U, defined softmaxy R + [0, 112t and CEN : .u x R² + R+ according to
[softmaxy (0),,
exp
CEe (u, 0) = - log [softmaxy(0))..
for u E U.
(8)
Ever exp 6.
It will be useful to introduce the matrix A E {0, 1) encoding the ancestor relation A.. [u 2 v). such that the linear maps x + Ar and x + A x compute sums over descendants and ancestors, respectively. (The linear maps can be computed without explicitly instantiating the matrix.) Further, let Ac E {0. 133x denote the leaf-node column subset, such that x + Agr computes the sum over leaf descendants and + A2æ computes the over the ancestors for each leaf node.
The most straightforward method is to use a flat softmax classifier with a parameter vector 0 in R The likelihoods of leaf nodes are obtained directly from the softmax, while the likelihoods of internal nodes are obtained by recursive bottom-up sum:
[softmax:(0)) (0)], V C
(0) Enec(w) (0) y $ C.
This can be succinctly expressed g(0) = Ac softmax:(0)) The NLL reduces to the familiar (convex)
cross-entropy f(y, 0) = CEc(y, 0) for leaf-node labels y E L. However, it is non-convex for general labels y V resembling a loss for Multiple Instance Learning [22]:
Ely, 0) = - log qy (0) = - log Euec(y) exp + log Exccex666 cxp
(10)
alternative loss for the same parametrisation, Hierarchical Cross Entropy (HXE). This loss considers conditional distributions given the parent, placing geometrically decreasing weight on deeper nodes with discount factor 7 = e- a E (0, 1]. (For y 1, HXE is
Bertinetto al. [2] proposed
non-convex even for leaf-node labels E L.) To define the loss, let wo, Wd(v) E y denote the ordered ancestors of y (from wo to wd(y) y) in:
d(v)
d(v)
qwp (0)
(a, 0) = _
log
log Σ exp On log Σ expp mEC(WN)
(11)
qwr-1
1=1
uEC(wn-1)
Besides the flat softmax, amother naïve baseline is the multi-label sigmoid classifier; that is, indepen- dent binary logistic regression per node. The likelihoods are parametrised by a vector f in (r) according to q(0) (f). Clearly, this is not guaranteed to satisfy (1). Since the binary problems imbalanced, we adopt the Focal Loss [23], which has two key syper-parametes::
aa p)7 logp, Σ FL.,, S w), (Ow)), FLa. - 1- - a1- ppp log(1 p). y 0. WEy
(v,
Redmon and Farhadi [31] introduced a conditional softmax classifier, which uses a separate softmax for the conditional likelihood of child nodes given each parent. The parameter vector 0 is in (r) and the likelihood of each node is obtained as recursive top-down product. Letr, ())) denote the conditional likelihood of y given its parent to obtain:
(e( (0) = softmaxc(y) (Bow)) ,
gy (0) = (0) (ntt)) (0) IILEAGAAM) (0)
(12)
This can be succinctly expressed log (f) = A (0). The conditional softmax has the same degrees of freedom as the flat softmax due to the invariance the softmax function. The loss is taken to be NI.,, which is a sum of cross-entropy losses and therefore convex for general labels E )::
Ely, 0) log (0) E'EA(wrr CES(s)) (u, 0s(w)).
(13)
5

---

Brust and Denzler [4] similarly proposed conditional sigmoid classifier for the more general problem of Multi-Pate Labels in a DAG. We specialise this tree by replacing r(0) (0) above. This ensures that parents are more likely than each child, but does not guarantee the condition in (1). Their proposed loss function is Binary Cross Entropy for children of ancestors of the label:
0y = Σ Σ
BCE((U 2 v),)) BCE(y. = {- log(1 - p), y y-
{ - log
y = 1
(14)
uE.((y) vEc(u)
Several works [32, 34, 39] have proposed to use parameter sharing, whereby the unnormalised score of each node is obtained as sum of ancestor scores. That is, cumulative path scores / E R are obtained from node scores 0 in R according to By = Aey)(()) 6u or simply / = ATO. In the context of deep learning, this approach was employed in the Deep Realistic Taxonomic Classifier (Deep RTC) of Wu al. [39]. Motivated as generalisation of "realistic classifiers", which may abstain from making prediction [37], Deep RTC performs inference by greedy top-down traversal, using a threshold on unnormalised scores as a stopping condition. To map the unnormalised scores to j0, I.. we apply the monotonic mapping (0)) (B)). To ensure that internal nodes obtain high scores even when all ground-truth labels are leaf nodes, Deep RTC is trained using Stochastic Tree Sampling, taking the expected cross-entrops at the leaf nodes K. C y of random cut of the tree:
(15) We propose to consider an ablation of Deep RTC, the Parameter Sharing (PS) softmax, which is a simple linear reparametrisation of the flat softmax using the leaf-node scores from parameter sharing 8c 8 with in R. This can be succinetly expressed q(0) Ag softmax (A),6). Unlike Deep RTC, this yields likelihoods that satisfy (1).
f(y, 8) = Ex [CEx 9x)) where (projx(v)) = A(y) n.c
Finally, we introduce two novel loss functions for parametrisation that assigns non-zero mass to internal nodes. This parametrisation uses a softmax over all nodes y with parameter vector 0 in R to first obtain "exclusive" likelihoods dy (0) (i.e. likelihood of a node and not its children), and then obtains total likelihoods by a bottom-up recursive sum:
(16) or, succinetly, q(0) A softmaxy:(0 (0). If we simply minimise the NLL., there would be little
a(0) softmaxy(1))
gu (0) (v)
+ EvEc(y) (o(
incentive to assign non-zero mass to internal nodes when most labels are leaf nodes (and the loss would be non-convex for non-leaf labels). We therefore propose the noft-max-descemee
loss
Ely, 0) = EueA(y) [C(w]]
(u 0)
(17)
where N(y) y' (A(y) U D(y)) is the set of incorrect (negative) labels for y; that is, labels which are neither ancestors nor descendants. This loss aims to ensure that each ancestor of the ground -ruth label is classified positively against all incorrect labels. We call it soft-max- -ex-eeddeee because it takes the log-sum-exp (like a soft maximum) over incorrect sub-trees. Normalisation by the number of leaf descendants is necessary prevent higher-level nodes from dominating the loss. It will later be seen that this loss is ineffective; it is included as a negative result.
Finally, inspired by recent work using logit adjustment for long-tail learning [25], we consider mott-max-margin loss [29, 15, 13], which is a soft version of the structured hinge loss [35], defined
fly. 0) Cyy(y, 0 + aC +) log (1+
Σ
exp - By + +C((y,'))) (18)
y'EA(u
where Cly. s/) gives the desired score margin between y and y'. For hierarchical classification, we adopt Cly.y') = Correct y') to seek greater separation of the ground-truth label from incorrect classes than from correct classes, and we found = 5 provide the best results. Note that the logit adjustment is only employed during training. Despite the original "hard" structured hinge loss also being convex, we were unable to use it to train deep model to high accuracy.
5 Operating characteristic curve
Given classifier that predicts a conditional distribution over the class hicrarchy, the set of predictions that can be obtained using menfidence-threshen inference with some value of T is the Pareto set; that is, the set of classes such that no other class both more confident and more informative:
H(p(-1z)) {y y Du(p(ulæ) p(y)z) A 1(u) > 1(y)))
(19)

---

1.00
1.0
0.95
Fist softmax
0.9
Multiabel focal
0.90
softmax
0.8
Gond sigmeid
RTC
Corred
softmax
0.80
0.7
Soft- Soft-max- descendiant
Sott-maa--essceniint
0.75
0.0
Majority
Plurality
0.5
0.85
Expected
0.60
0.30
0.35
0.40
0.43 0.30 0.55
0.60 0.65 0.70
0.75
0.0
0.1
0.2
0.3
0.4
0.5
Information Recall
Exact
Figure 1: Correctness-specffie trade-offs for different methods and metrics on the iNat21 validation set. Markers indicates operating points obtained by inference functions. Comparing methods using single operating points would not provide a complete perspective. The dashed line is the part of the curve with threshold below 0.5. The shaded areas depict two standard deviations to either side.
denote the elements of the Pareto set, ordered such that p(ik (z) 2 p(ū++1 1 (z) and T(ūx) I (()+1)) Further let & -p 0k: (x) denote the corresponding confidence
For single example (2r, y), let define the sequence Jo...-- -1
such that the prediction is piecewise-continuuus function of T with ğ(T) = ÿu for Sę > T 2 8K+1- The value of any metric M (y, ((T) is also a piecewise- continuous function of T, taking values žд M (v.
The cumulative metric Z(T for a set of examples {(x', "ee',")) } is a sum of piecewise- consttant functions M (y', ff (r)) and therefore prceewsse-oonstar itself, with the form Z(r) Z, for Sy > Sj+1-
The values of Z, and Sj can be obtained by introducing of # k and reordering sums:
((T) = MG' = E,6 + Ekist 51) = E, 26 + Σ (1 81.
(20)
Therefore Z,j is obtained as partial sum Z, = Z Σ Au with Zo =, 28 and the sequences Sj and A, obtained by merging the ordered lists of (si, 5i) pairs to be descending The following section will compare different methods using parametric curves of correctness and specificity metrics as a function of
The ordered Pareto set for single example can be obtained in O log DI) time. For arbitrary and I, the worst-case running time for dataset size N ON D (log N +og (V])), although each Pareto set can be computed independently in parallel. If we assume that the tree is relatively balanced, P(-1z) satisfies (1) and restrict ourselves to [0.5, 1]. then the size of the set is bounded by the maximum depth of the tree, which is O (D1)). This results in running time of ONN log ()']) to find the Pareto sets and O(N log N log ('I)) to merge the sequences. Otherwise, ensuring that all leaf nodes have equal information guarantees that the Pareto set contains at most one leaf node,
reducing the worst-case merge time by a factor of 1 [C//
6 Empirical study
We now apply this technique compare different loss functions. Most experiments consider image classification on the iNat21 dataset [36], containing 50 examples each for 10,000 biological species in seven-level taxonomy. All examples have leaf-node (full-depth) annotation. For all iNat21 experiments, we use ResNet-1 18 model [19] with input images of size 224x224 We start from the Pytorch ImageNet-pretraine
checkpoint [28] and train for 20 epochs using SGD with momentum 0.9, cosine schedule [24], batch size 64, initial learning rate 0.01 and weight decay 0.0003. Most experiments were conducted on a single machine with one Nvidia A6000 GPU. Each epoch of 1Nat21-Mini takes about 20 minutes (5-8 minibatches per second). To obtain error-bars, we used a larger, shared machine with 16 Nvidia V100 GPUs (still using one GPU to train each model). The code implemented using the Pytorch library [28] and is available under the MIT license.
7

---

Table I: Overall metries derived from operating curves on the iNat21 validation set. Average Precision (AP) and Average Correct (AC) are integrals with respect to Recall, while @@XC indicates Recall at X% Correct. All precision and recall metrics use information as the specificity measure (not depth). The error ranges are two standard deviations. *Deep RTC and PS softmax actually use the same model weights with different output parametrisations since the optimal cut probability for Deep RTC was found to be zero.
From operating curves R@90C
Majority Fi
Leaf Fi
AP
AC
R@95C
softmax
66,90 =0.16
64.71
10.18
52.44
10.42
45,88
+0.30
63.68
40.19
69.86
40.13
Multilabel focal
58.75 +0.24
$4.08
+0.20
32.64
#0.16
26.80
+0.28
65.05
+0.24
65.38
+0.29
Cond softmax [3]]
60.722#0.1
57.47
10.13
42.03
#0.12
35.71 +0220
58.54
+0.20
65.27
10.11
Cond sigmoid [4]
60.64 10.17
57.13
40.52
34.29
10.18
56.73
+0.07
65.52 10.19
Deep RTC [39]*
66.05 +0.16
60.67
+0.15
30.56 +0.79
17.64 1.00
71.65
+0.12
71.65
+0012
PS softmax* *
68,90 13113
66,86
1:012
+0.22
49.13 00.15
67.97
4:019
71.65
#0.12
66,46 00.37
64.19 +0.39
49.15
#0.36
43.29
00.56
60.99
+0.33
70.01
#0.33
Soft-max-mariin
67.63 +0.25
65.97 +0.26
56.53
+0.28
50.65
+0.41
67.92
+0027
69.89 +0.24
6.1 Operating characteristic curves
Our main experiment trains and evaluates each of the methods on the iNat21-Mini dataset to obtain operating curves. The results presented here are μ + 20 from five trials with different random seeds.
Figure presents curves for precision-vs-recal and correct-vs-exact. The most striking observation is that all methods except PS softmax, Deep RTC [39] and our soft-max-margee loss are virtually dominated by the flat softmax classifier at all operating points. For the precision-recall trade-off, PS softmax is the best at high recall and soft-max-mariin is the best at high precision. The soft-max- margin has a distinct advantage in the correct-exact trade-off and Deep RTC is more competitive. The least effective method is the binary multilabel baseline, followed by the two approaches that learn top- down conditional distributions [31, 4]. The singular operating points achieved by different inference methods are also depicted in the figure, with the operating curve always containing majority inference and leaf inference as special cases. For parametrisations p(-(x) = q()) that respect the hierarchy, specificity and correctness should be monotonic in for >> 0.5, hence dashed line is used for the segment with r < 0.5. is observed that plurality inference enables more-specific predictions than majority inference. However, both the plurality and the expected information methods lie below the curve obtained by confidence threshold inference. Further enquiry is required to understand the cause of the "knot" observed at 0.5 in the curves for the soft-max-desse max-descendan method. Critically, the curves are much more useful than single operating points for the purposes of algorithm development and selection.
Table further presents several integral and intercept metries obtained from the eurves. For com- parison, the Fj metrics for majority and leaf inference are shown. PS softmax and soft-max-marghi achieve the best results across all metrics.
Several of the proposed loss functions have additional hyper-parametert to specify. Figure 4 in the appendix examines the impact of label smoothing with flat softmax, focal loss parameters in the multilabel classifier, discount factor in HXE [2] and cut probability in Deep RTC [39]. Label smoothing and HXE include the flat softmax as a special case, and this was the optimal hyper- parameter choice in both cases. For this reason, they were excluded from our main experiment. The optimal cut probability for Deep RTC was zero, meaning it reduces to a softmax loss with logits obtained by parameter sharing. This was surprising, as it provides no incentive for the model to increase the scores of internal nodes.
We conducted additional experiments with a subset of loss functions on ImageNet-Ik Ik using the hierarchy of [2], training ResNet-50 model from scratch. Figure 2 presents operating curves for this experiment. The observations are mostly consistent: our soft-max-margin better trade-off than existing methods, Deep RTC is strong at high recall (specificity) but mediocre at high precision (correctness). and the conditional softmax is worse than the flat softmax across the full range. Unlike the previous dataset, PS softmax underperformed the flat softmax. This experiment similarly used batch size 64, weight decay 0.0003 and initial learning rate 0.01, but was only trained for 15 epochs. Figure 2 also shows the impact of using each loss function to train a linear model on top of the feature representation of CLIP [30] model with ViT-B/32 [12] architecture. The
margin method achieves a
8

---

Table 2: iNat21-Mini accuracy at
level j when training level
Coreet
ordsstrrxx Coup
sotmmx
%
j
5
Ho. SMEmaR
Saftmar--me
artmaa
3
82.7
85.2
60.7
5
86.6
64.6
50.6
Bert
information Reral
6
86.9
68.2
56.8
44.6
87.3
70.4
59.8
49.2
Figure 2: ImageNet-Ik operating curves for (left) ResNet-50 from scratch and (right) linear model with CLIP ViT-B/32 features.
Seen classes
Unseen classes
1.50
AP (wit Recall)
All
Seen
Unseen
Flat softmax
59.2
69.3
49.5
Cond softmax
57.3
63.4
51.6
Cond sigmoid
56.9
63.1
51.1
Alt sitmrs
Deep RTC
57.1
68.1
46.0
softmun
PS softmax
61.3
70.6
52.3
Soft-max-descendamt
58.3
68.1
49.2
Saluman disecndart
Soft-max-margin
59.1
70.0
48.3
52th mac margm
0.1
0.3
0.6
2.5
Information Racal
Information Boca
% AC (wrt Exact)
All
Seen
Unseen
5.0
softmax
36.7
50.8
23.5
Cond softmax
31.0
38.9
24.1
Cond sigmoid
38.4
23.4
Deep RTC
38.1
51.2
25.3
0.5
PS softmax
38.2
51.2
26.4
Soft-max max max-descendaat
32.1
44.9
22.0
Soft- ma--maxx-maaaa
38.7
54.0
23,9
0.1
0.6
0.15
3.23 Doct
3.25
Donet
Figure 3: Operating curves for the out-of-distribution experiment on iNat21, divided into examples from seen and unseen classes. The top-down methods (conditional softmax, conditional sigmoid) are much more competitive unseen classes. The soft-max-mann margin is most effective on seen classes but PS softmax is the most robust.
soft-max- -marinn still provides an improvement over the baseline, while the PS softmax model is mathematically equivalent to flat softmax (except for weight decay). Compared to training from scratch, higher recall is achieved, presumably due the strong feature representation. However, the conditional softmax is much worse, suggesting that the high-level classes are less easily separated in this representation. The linear model was trained for 20 epochs with batch size 256, initial learning rate 0.1 and weight decay 10 -5
6.2 Flat classifiers at different levels
was unexpected that the conditional softmax would fail out-perform the flat softmax any operating point. One possible explanation is that the high-level classes are better learned using union of low-level classifiers than single high-level classifier because the latter effectively ignores the fine-gra ane-ggained annotations. To investigate this hypothesis, we trained flat models at each level of the hierarchy and compared the classifiers trained at level with those trained at levels + 1, i + 2 and so on, shown in Table 2. Classifiers trained at deeper levels consistently achieved higher accuracy, although with diminishing returns. This motivated the design of our proposed loss functions, which predict scores for low-level and high-level classes.

---

6.3 Unseen classes
We hypothesised that the parametrisations which explicitly estimate scores for internal nodes might achieve better generalisation to examples from "unseen" classes, which are not present during training but still belong to an internal node. To evaluate the ability of models to correctly classify such examples, we randomly select half of the leaf nodes and remove them from the dataset and hierarchy during training. At test time, examples from these classes were retained with their labels projected onto the sub-tree.
Figure presents the operating curves and integral metries for the examples from seen and unseen classes. The conditional softmax [31] and conditional sigmoid [4] are much more competitive for unseen classes, although PS softmax remains the most effective. The soft-max-margi approach is excellent for seen classes and relatively poor for unseen classes, suggesting that it learns highly tailored model that could in fact be worse than the naïve baseline in an out-of out-of-distribution setting. The soft-max -e---max---dddede descendant approach is uncompetitive in both settings.
7
Conclusion
This work has presented new approach for evaluating hierarchical classifiers for the prediction of non-leaf nodes. It was shown that the flat softmax classifier, despite having no knowledge of the hierarchy during training, is nonetheless a highly competitive baseline. Novel loss functions were proposed, and the max-margin loss was shown to out-perform several existing methods. On the other hand, the proposed soft-max-dee max-descendant losses were shown to be ineffective. Surprisingly, top-down classifiers were found to be inferior across the entire operating range. Evidence suggests that this is due to the difficulty of learning classifier to separate high-level classes without fine- grained supervision. The top-down classifiers were shown to be more effective on a synthetic -ittrr--iistiittiionn experiment, where recognition of super-classes is required.
One potential disadvantage of the soft-max- mannxnn loss is that it requires manual design of the margin. Future work could investigate different margins, direct optimisation of the area under the curve [3] or the impact of calibration [17]. It would also be interesting to generalise parameter sharing to
DAGs, to evaluate hierarchical loss functions in long-tail setting. and to investigate other problems where the soft-max- mal--maa-aaaaaa margin loss could be applied. One notable limitation of using a fully-connected prediction layer is that it may not scale to millions of classes, where embeddings metric learning are more suitable.
Social impact. While hierarchical classifiers may mitigate some of the risks of misclassification by their ability to fail gracefully, this may enable more widespread use of automatic classification or lead human users to place more trust in the system. Caution should always be exercised when employing automatic classification in consequential settings. A class hierarchy with free structure may also encourage an oversimplified view of problem due to its inability of different classes to have a non-empty intersection.
References
[1] K. Ahmed, M. H. Baig, and L.. Torresani. Network of experts for large-scale image categorization. In European Conference on Computer Vision, pages 516-532. Springer, 2016. 2
[2] L. Bertinetto, R. Mueller, K. Tertikas, S. Samangooei, and N. A. Lord, Making better mistakes: Leveraging class hierarchies with deep networks. In CVPR, pages 12506-122 12515, 2020. 2. 8
[3] A. Brown, W. Xie, V. Kalogeiton, and A. Zisserman. Smooth-AP. Smoothing the path towards large- scale image retrieval. In European Conference Computer Vision, pages 677-694. Springer, 2020. 10
[4] C.-A. Brust and J. Denzler. Integrating domain knowledge: using hierarchies to improve deep classifiers. In Asian Conference on Pottern Recognition, pages 3-16. Springer, 2019. 2, 6, 8, 10
[5] L. Cai and T. Hofmann. Hierarchical document categorization with support vector machines. In Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management, pages 78-87, 2004. 3
[6] E. Costa, A. Lorena, A. Carvalho, and A. Freitas. A review of performance evaluation measures for hierarchical classifiers. In Evaluation Methods for Machine Learning I1: Papers from the AAAI-2007 workshop, pages 1-6, 2007.
10

---

[7] J. Davis, T. Liang, J. Enouen, and R. Ili.. Hierarchical semantic labeling with adaptive confidence. In International Symposium on Visual Computing, pages 169-183. Springer, 2019. 2,
[8] J. Davis, T. Liang, J. Enouen, and R. Iln. Hierarchical classification with confidence using generalized logits. In ICPR, pages 1874-1881. TEFE, 2021. 2
[9] J. Deng, A. C. Berg, K. Li, and L. Fei-Fei. What does classifying more than 10,000 image categories tell us? In European Conference on Computer Vision, pages 71 84. Springer, 2010. 1,4
[10] Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and Fei-Fei. ImageNet: A large-scale hicrarchical image database. In 2009 JEEE Conference on Computer Vision and Puttern Recognition, pages 248-255. leee,
2009. 1
[11] J. Deng. J. Krause, A. C. Berg. and L. Fei-Fei. Hedging your bets: Optimizing -peeficyspecifficity trade-offss in large scale visual recognition. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3450-3457 IEEE, 2012. 1, 2, 3,
[12] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 8
[13] S. Edunov, M. Out, M. Auli, D. Grangier, and M. Ranzato. Classical structured prediction losses for sequence to sequence learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pages 355-364, 2018. 3,6
[14] T. Gao and D. Koller. Discriminative learning of relaxed hicrarchy for scage-scalc visual recognition. In 2011 International Conference on Computer Vision, pages 2072-2079, IEEE, 2011.
[15] K. Gimpel and N. A. Smith. Softmax-margin CRFs: Training log -ineer models with cost functions. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 733-736, 2010. 3,6
[16] G. Griffin and P. Perona. Learning and using taxonomies for fast visual categorization. In 2008 IEFE Conference on Computer Vision and Pattern Recognition, pages 1-8. IEEE, 2008.
[17] C. Guo, G. Pleiss, Y. Sun, and K. Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages
PMLR, 2017. 10
[18] Y. Guo, Y. Liu, E. M. Bakker, Y. Guo, and M. S. Lew. CNN-RNN: A large-scale hierarchical image classification framework. Multimedia Tools and Applications, 77(8): 10251-10211 2018. 2
[19] K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[20] S. Karthik, A. Prabhu, K. Dokania, and V. Gandhi. cost likelihood manipulation at test time for making better mistakes in deep networks. In International Conference on Learning Representations, 2021. 2.
[21] A. Kosmopoulos, Partalas, E. Gaussier, G. Paliouras, and I. Androutsopoulos. Evaluation measures for hierarchical classification: a unified view and novel approaches. Data Mining and Knowledge Discovery, 29(3):820-888 265,8,00-66,.00 2015. 3
[22] O. Z. Kraus, J. L. Ba, and B. J. Frey, Classifying and segmenting microscopy images with deep multiple instance learning, Bioinformatics, 32(12):122 2016. 5
[23] T.-Y. Lin. Goyal, R. Girshick. K. He, and P. Dollár. Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, pages 2980-2988, 2017. 5
[24] I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.7
[25] A. K. Menon, S. Jayasumana, A. S. Rawat, H. Jain, A. Veit, and S. Kumar. Long-tail learning via logit adjustment. International Conference on Learning Representations, 2020. 3,6
[26] F. Morin and Y. Bengio. Hierarchical probabilistic neural network language model. In AISTATS, volume 5, pages 246-252. Citeseer, 2005. 1,2
[27] R. Müller, S. Kornblith, and G. E. Hinton. When does label smoothing help? Advances in Neural Information Processing Systems, 32, 2019. 2

---

[28] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan. T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, bigh-performance deep learning library. Advances in Neural Information Procexsing Systems, 32, 2019.
[29] P. Pletscher. C. S. Ong. and J. M. Buhmann. Entropy and margin maximization for structured output learning. Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pages 33-98. Springer, 2010. 3,
[30] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763. PMLR, 2021. 8
[31] J. Redmon and A. Farhadi. Yolo9000: Better, faster, stronger. In CVPR, pages 7263-7271, 2017. 2,5, 8,
10
[32] R. Salakhutdinov, A. Torralba, and J. Tenenbaum. Learning to share visual appearance for multiclass object detection. In CVPR 2011, pages 1481-1488, IEEE, 2011. 2,
[33] C. N. Silla and A. A. Freitas A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 22(1):31-777 72, 2011. 2
[34] M. Sun, W. Huang, and S. Savarese. Find the best path: An efficient and accurate classifier for image hierarchies. In Proceedings of the IEEE International Conference Computer Vision, pages 265-272, 2013. 6,6
[35] I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Altun. Support vector machine learning for interde- pendent and structured output spaces. In Proceedings of the Twenty- First International Conference Machine Learning, page 104, 2004. 6
[36] G. Van Hom, E. Cole, S. Beery, K. Wilber, S. Belongic, and O. Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22884-12899 2021. 1.7
[37] P. Wang and N. Vasconcelos. Towards realistic predictors. In Proceedings of the European Conference on Computer Vision (ECCV), pages 36-51, 2018.
[38] C. Wu, M. Tygert, LeCun. A hierarchical loss and its problems when classifying non- hieharchhccalll..... PLOS ONE, 14(12)e0226222, 2019.
[39] T.-Y. Wu, Morgado, Wang, C.-H. Ho, and Vasconcelos. Solving long-tailed recognition with deep realistic taxonomic classifier. In ECCV, pages 171-189. Springer, 2020. 1, 2, 3, 6, 8
[40] Z. Wu and M. Palmer. Verb semantics and lexical selection. arXiv preprint mmp-1999406033. 1994.
[41] Z. Yan, H. Zhang, R. Piramuthu, V. Jagadeesh, D. DeCoste, W. Di, and Y. Yu. HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition. Proceedings of the IEEE International Conference on Computer Vision, pages 2740 2748, 2015.
[42] H. Zhao, X. Puig, Zhou, S. Fidler, and Torralba. Open vocabulary scene parsing. In Proceedings of the IEEE International Conference on Computer Vision, pages 2002-2010, 2017. 3.4
[43] X. Zhu and M. Bain. B-CNN: Branch convolutional neural network for hierarchical classification. arXiv preprint XXXv:1170.009890 2017. 2
12


Reviews:
[{"review_id": "lUOunTrhFiE", "reviewer": "Reviewer_53HE", "summary": "Topic: the paper reviews the performance of hierarchical classification for deep learning classifiers.\n\nThey propose as scoring for internal nodes to sum up the softmax components of the leaves.\nThey propose for the above scoring two loss functions. \nThey propose in section 5 a way to compute curves over varying thresholds \\tau for node probabilities p(y), where the graphs show correct versus exact and precision vs recall, with the note that these measures are defined in the sense of hierarchical measures and not as conventional precision vs recall for flat classifiers.\n\nThey evaluate these curves for a number of methods.\nThey run an experiment with simulated unseen classes which are part of the hierarchy.\n\nThe topic is relevant.\nFor a better score, section 5 needs to be verifiable and better understandable.\n", "questions": "Questions: \n\nAre the p(y) for all methods probabilities? if yes, what is the set over which they sum up? That can be put in a table.\n\n\nHow does computing the curves in Section 5 work?\nThe ordering is a double ordering, with two criteria. How to sort with two criteria?\nWhat is varied in line 233 with z_k and the \\tau when considering the z_k? There is an unclear y in the definition of z_k which might be related to the \\tau.\nHow to merge the sequence of pairs \\delta into \\Delta?\n\nWhat is the meaning of eq(7) given the specific cost function choice C?\nWhat is the meaning of eq(17) given the specific cost function choice C? It seems to give a higher weight to nodes y' such that the prediction y is on the path from y' ? \n\n\n", "limitations": "Some discussion has been made. No issue regarding potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\nA certain novelty, with combinations of loss functions and section 5, if correct.\nIt is an overview evaluation for a number of methods.\nThe topic is interesting and relevant.\n\nWeaknesses:\n\nIt has a not so great readability.\nSome parts appear to be too brief, and it appears as if it would profit from an extended journal version.\n\n-the meaning of the choice of C(y,\\hat{y}) in line 181 is a bit unclear. One seems to sum I(y) values from ancestor nodes starting at the prediction \\hat{y}.\n-eq (7) needs a discussion\n-the changes made to deepRTC need a discussion.\n-eq (17) needs a discussion with respect to the concrete C term used.\n\n-Section 5 is hard to understand. \n\nThe ordering is a double ordering, with two criteria. How to sort with two criteria?\n\np(\\hat{y}_{k}) \\ge p(\\hat{y}_{k+1}) and I(\\hat{y}_{k}) \\le I(\\hat{y}_{k+1}). What to do if p(\\hat{y}_{k}) \\ge p(\\hat{y}_{k+1}) and I(\\hat{y}_{k}) \\ge I(\\hat{y}_{k+1}) for example ?\n\nThere is a \\Delta seemingly out of nowhere. How to merge the sequence of pairs ? Why does it work? \nThere is an unclear y in the definition of z_k which might be related to the \\tau.\n\nSection 5 reads as if one has to trust that it works as described. \n\n-a smaller issue: it could be worth to cite some pre-deep learning hierarchical papers (but that causes no reduction in scoring) like:\n\nBlaschko, Zaremba, Gretton Taxonomic Prediction with Tree-Structured Covariances\nZweig, Weinshall, Exploiting Object Hierarchy: Combining Models from Different Category Levels\nMarszalek, Schmid, Semantic Hierarchies for Visual Object Recognition\nand the like", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "__g2hw7B1YG", "reviewer": "Reviewer_CWNt", "summary": "This paper presented a new approach for evaluating hierarchical classifiers for predicting non-leaf nodes at multiple operating points. With the new evaluation metric, the paper shows that the widely used softmax classifier performs reasonably well in many cases, although a structured classifier might have an advantage in classifying unseen classes. The paper also proposes two new loss functions. One of them outperforms the softmax classifier in all examinations. The experiments are on ImangeNet and iNat classification tasks.", "questions": "Does Eq-17 have any limitations? For example, how was its accuracy in the most popular classification setting that only predicts leaf classes?", "limitations": "Lines 340-344 have the discussion and make sense to me.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and easy to follow. A large amount of literature is discussed and adequately summarized. The comparison between the paper's and previous methods is succinct but informative.\n\nRegarding significance, comparing hierarchical classifiers at multiple operating points sounds a must-have for the evaluation. Therefore I am slightly surprised that the paper mentioned that \"almost no prior work has compared methods at multiple operating points\" (in lines 5 and 6). If the above claim is valid, I believe this work provides a valuable evaluation of the problem and is a significant contribution. I like to admit that I am not an expert on this topic and will cross-check with other reviewers for the claim. To strengthen the significance, the author may explain why evaluation at multiple operating points is not popular or feasible without the method in section 5.\n\nTechnically, I do not find obvious mistakes, but the clarity of the key methods may have room to improve. For example, (1) elaborate more on why Eq-17 gives a result better than flat in Figure 1; (2) use an algorithmic block to summarize the process of section 5.\n\nRegarding the experiments, the paper uses large-scale datasets, has proper baselines and prior methods, and examines interesting aspects such as flat classifiers at different levels and unseen classes. The results provide conclusive insights and are easy to interpret. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-jBTV0Z59ma", "reviewer": "Reviewer_Kenf", "summary": "The authors consider the tasks of hierarchical classification where predictions can be done at different levels of granularity, which can be seen in datasets like ImageNet, where a super class can be spiders and its fine-grained classes are the different types of spiders.\n\nThe authors describe and compare different existing loss functions on a new metric class based on \"Operating characteristic curve\" on the  iNat21-Mini dataset. They also propose two loss functions: soft-max-descendant that scores internal nodes in the classification hierarchy, and soft-max-margin which is a soft version of the structured hinge loss.\n\nDiscussions have been made about how different loss functions behave based on the structure of the problem such as out-of-distribution and in-distribution.", "questions": "Can the authors explain the main contributions and motivation of this work? It is not clear why having \"Operating characteristic curve\" is important and why having the two loss functions \"soft-max-descendant\" and \"soft-max-margin\" are important as their performance is poor in many settings.\n\nA lot of loss functions have been compared against a new metric, what is the takeaway? What loss function should we use when there is a hierarchical classification problem, and why is the proposed metric a more reliable approach than others?", "limitations": "The authors state that \"One limitation of our general approach is that it may not easily scale to millions of classes, where embeddings can be more suitable.\" It is not clear why that is the case, why embeddings are more suitable, and what \"general approach\" really means. Please clarify. \n\nThe societal impact is poorly written,  can you clarify what this quote means? \"While hierarchical classifiers may mitigate some of the risks of misclassification due to their more logical modes of failure, this may enable more widespread use of automatic classification or lead human users to place excessive trust in the system.\" It is not clear to me what \"logical modes of failure\" mean, and I don't see why humans would place excessive trust in the system.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\n- Hierarchical classification is very interesting to the research community.\n\n\nWeaknesses:\n\n- There is no theoretical motivation or practical motivation for using \"Operating characteristic curve\" as a metric for Hierarchical classification, why is it better than the other metrics in terms of reliability, correctness, etc.? For example,  [26] proposed an improved version of the mistake-severity metric that  \"could give a wrong impression of improvement while the model might just be making additional mistakes to fool the metric\" which is a good motivation for their work. That type of motivation is missing in this work.\n\n- The document is poorly written as it is difficult to find out what the authors actually did. The authors do not clearly present their contributions. I suggest the authors summarize their contributions at the end of the introduction as a paragraph or bullet points.\n\n- The related work is poorly written, there is a large list of existing methods described but the authors do not show how relevant they are to the proposed metric/loss functions. It is important that by the end of related work paragraphs to describe what the paper proposes in relation to existing methods.\n\n- The authors spend so much text explaining previous methods and metrics without explaining why they should be there as they do not help in understanding the proposed loss functions and metrics. A simple citation and few sentence explanations of these methods would be sufficient. This amount of detail just adds confusion about finding out what the authors are actually trying to propose.\n\n- The proposed soft-max-descendant performs poorly in so many of the settings so it is not clear what the motivation is for including it in this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept