Learning Structured Representations by Embedding Class Hierarchy
Authors: Siqi Zeng
Remi Tachet des Combes
Han Zhao
Venue: ICLR 2023 poster (2023)
Keywords: structured representations
representation learning
tree embedding

Abstract: Existing models for learning representations in supervised classification problems are permutation invariant with respect to class labels. However, structured knowledge about the classes, such as hierarchical label structures, widely exists in many real-world datasets, e.g., the ImageNet and CIFAR benchmarks. How to learn representations that can preserve such structures among the classes remains an open problem. To approach this problem, given a tree of class hierarchy, we first define a tree metric between any pair of nodes in the tree to be the length of the shortest path connecting them. We then provide a method to learn the hierarchical relationship of class labels by approximately embedding the tree metric in the Euclidean space of features. More concretely, during supervised training, we propose to use the Cophenetic Correlation Coefficient (CPCC) as a regularizer for the cross-entropy loss to correlate the tree metric of classes and the Euclidean distance in the class-conditioned representations. Our proposed regularizer is computationally lightweight and easy to implement. Empirically, we demonstrate that this approach can help to learn more interpretable representations due to the preservation of the tree metric, and leads to better in-distribution generalization as well as under sub-population shifts over six real-world datasets.

Published as a conference paper at ICLR 2023
LEARNING STRUCTURED REPRESENTATIONS BY EMBEDDING CLASS HIERARCHY
Siqi Zeng
Remi Tachet des Combes"
Department of Mathematical Sciences Carnegie Mellon University Pinsburgh, PA 15213, USA sigi @ndreww. cmu. edu
remi Beachet@gmaal com
Han Zhao
Department of Computer Science
University of Illinois, Urbana- Chammpaag Urbana, IL 6 61801, USA
hanzhao@1 wwteeeiiiiiiiiiiie.ieeiiimiiiiie edu
ABSTRACT
Existing models for learning representations in supervised classification problems are permutation invariant with respect to class labels. However, structured knowl- edge about the classes, such hierarchical label structures, widely exists in many real-world datasets, e.g., the ImageNet and CIFAR benchmarks. How to learn rep- resentations that can preserve such structures among the classes remains an open problem. To approach this problem, given tree of class hierarchy, we first define tree metric between any pair of nodes in the tree to the length of the shortest path connecting them. We then provide method to learn the hierarchical relation- ship of class labels by approximately embedding the tree metric in the Euclidean space of features. More concretely, during supervised training, we propose use the Cophenetic Correlation Coefficient (CPCC) as regularizer for the cross- entropy loss to correlate the tree metric of classes and the Euclidean distance in the class-conditioned representations. Our proposed regularizer computation- ally lightweight and easy to implement. Empirically, demonstrate that this approach can help to learn more interpretable representations due to the preserva- tion of the metric, and leads better generalization in-distribution well under sub-population shifts over multiple datasets.
INTRODUCTION
In supervised learning, the cross-entropy loss is often used for classification tasks. As common practice in deep learning, in order to train model for classification, practitioners build a linear layer over the representation to obtain the logit score of each class. softmax transformation is then applied to convert the logits into a vector belonging to the probability simplex. As result, any classes without affecting the performance of the
can randomly permute the representations original classification task. However, in many real-world datasets, classification, labels are not independent from each other anymore: ImageNet (Deng et al., 2009) inherits label relationship from WordNet (Fellbaum 1998), that contains both semantic and lexical connections; iNaturalist [Van Horn et al., (217)) borrows the biological taxonomy so that each image contains seven labels that reflect the morphological characteristic of the organism. Many existing works (Deng et al.) 2014; Yan et al. 2014, Ristin et al. 2015; Guo et al. 20118) Chen et al. 2019 investigated how to leverage this hierarchical information for various purposes, but how explicitly project this knowledge onto representations remains unexplored.
move towards fine- grained
In this paper, we focus on the most common label relationship: tree hierarchy. As illustrated in Fig. lb given a tree hierarchy of classes, our goal is to learn representations in feature space such
Work done while MSR Montreal.

---

Published as a conference paper at ICLR 2023
that the Euclidean distances between different class centers approximate the distances between these classes in the tree. More concretely, we shall first define a tree metric to be the length of the short- est path connecting two subset of classes in the tree hierarchy. Based on this tree metric, we then propose a regularizer, the cophenetic correlation coefficient (CPCC) between sequences of tree met- ric and Fuclidean distance of the feature space, to ensure that the class-conditional representations inherit the tree structure of the classes, Different from the original cross-entropy loss with softmax activation, the proposed CPCC regularizer helps to break the symmetry of permutation invariance among the classes, and thus also improves the interpretability of the learned representations.
We show that the proposed CPCC regularizer is computationally lightweight with negligible over- head, and can be applied to wide range of supervised learning paradigms, including standard flat empirical risk minimization and other hierarchical objectives, including both multitask learning and cirriculum learning. For generalization, over six real-world datasets, we demonstrate that our pro- posed CPCC regularizer leads to improved generalization performance on some unseen tasks with sub-population shifts when there is only limited amount of labeled data.
2 PRELIMINARIES
In this section we first introduce the notations used throughout the paper, formulate our learning problem, and then briefly review the CPCC score to quantify the correlation of two sequences.
Notations and Setup We shall use X and Y to denote the input and target random variables, living in spaces X' and V, respectively. In this work, we mainly focus on the supervised classification setting where for each input data point
X C Rd, there is ground-truth label y E y = [k] :=
{1....,k) k), where is the number of output classes. We let fa be the joint distribution over (X, Y)
from where the data is sampled. During the learning process, the learne has access dataset D= [ y1) } of size n sampled from μ..
In the context of representation learning, a learned representation = = =f)(x) is obtained by applying feature encoder fo X - 2 parametrized by 0 to æ, where 2 TRP denotes the feature space. Upon feature vector 2, we further apply a linear predictor denote the (k-1))dimensionall probability simplex. The cross-entropy loss is our objective function. Specifically, let 9y E A be a one-hot vector with the y-th component being 1. The cross-entropy loss. (cu(-.)) between the prediction g o f(x) and the label y given by kcE(g ())).) -iE(()) 9: loolol(((( (x)).. For 2, z', || - 2' denotes the Euclidean distance between them.
2 + A.. where we use As to
2.1 CLASS HIERARCHY
In classification problems, the target label Y [k] is treated a categorical random variable that can take different nominal values. However, there is no particular ordering among these & categories, i.e., for different categories i, [b), one can only compare whether = j or not. Formally, letting d (i = 0 if i = j and d (1,j) = otherwise defines a metric (+:)) over y
However, in many real- world applications the similarity between different classes is not binary. Consider object classification in ImageNet (Deng et al. (009) as an example. Intuitively, one would think the distance between the classes corgi and chihuahua be smaller than that between corgi and panda. One way to characterize this distance between different classes is through a tree of class hierarchy, also known as dendrogram. An example is shown in Fig. la Formally. let (V, F, d) be a weighted tree, where V is the set of nodes, E. the set of weighted edges in T, and d: V V - IR specifies the distance between nodes the tree. Each node us in 'T is associated with subset of class labels S C [k), and can be recursively defined as follows:
1. For each class i E [k:], there is corresponding leaf node U/ E V. Conversely, each leaf node E V is identified with a single class label i [A)].
2. For some 5 C [k], if ug E V is not a leaf node in T, then its children form a partition of S. In other words, if us, ..., vs. are the children us. then Vi + j. S, S,= = 2 and Ut S.
3. The root node of T is tala
At a colloquial level, the tree T specifies a hierarchy of class labels that represents the structured knowledge among them. For example, shown in Fig. Ta of the MNIST dataset, the two children
2

---

Published as a conference paper at ICLR 2023
L2
L3
AAA
AAA
MNIST
(a)
(b)
la MNIST class hierarchy. The root node contains all the 10 digit classes. The two children nodes the root node correspond to the coarse classes of odd and even digits, respectively. Each leaf node in this class hierarchy corresponds to ffne class label (digit). Fig. [bb An example of class hierarchy tree along with a visualization of the data in the feature space. The CPCC score computes the correlation coefficient of the tree metric from T in the left panel and the corresponding Fuclidean distance obtained from the feature space in the right panel.
Figure 1: Fig.
of the root node correspond to the odd and even numbers, respectively. Accordingly, the distance between digits and is smaller than that between digits and 2.
2.2 COPHENETIC CORRELATION CORFFICIENT (CPCC)
In the context of clustering, Sokal Rohlf 1962) introduced the cophenetic correlation coefficient (CPCC) evaluate the correspondence between dendrograms. The CPCC is the Pearson's correlation coefficient between two sequences of pairwise distances. For a class hierarchy T and a node V. the depth (t) of is the length of the shortest path from the root of T. In the original applications of CPCC, the "dendrogrammatic ground-truth distance ((v,, Uz) between pair of nodes Us, Vy in T is defined as follows:
t(u,, vi)) == max(dt(v)) dt(;)) dt(LCA (v., oy)),
where LCA (i, vy) the least common ancestor (LCA) of it and Vy. In Fig. la the LCA of and 3 is "odd" while the LCA of and is the root node. As an example of ground-truth distance, let us consider the class hierarchy tree T and clustering classes given Fig. Tb Fig. Tb t(A,, As) 1 since they share an LCA A at L1, and t(A,, = 2 as they go up 2 levels to meet at the root node, which their LCA.
Now consider a dataset D. For node E T, since corresponds subset of classes, we use D; S D denote the subset of data points whose class label belongs to v;. The pairwise distance between D; and D, is then defined as the Euclidean distance between the center of D, and Dy: p(v; 0y) == EeED, 1 n,
EXED, x' ||: where points in each cluster. Then, the CPCC score CPCC(L, p). between distances and p is defined as:
DDj and n, |Dy] are the number of
Σ E 02) (p(u,, 0z)
CPCC(t, := (E((v))))) 1)2)1/2( (p(vi, 0z) 1/2)
where 7:= :=: Eies (vi, v;) pairwise distances for l and p.
k(k 1), p: 2 plv, 0z) 1) the averages of all the
3

---

Published as a conference paper at ICLR 2023
3 OUR METHOD
In this section we first define an alternative tree metric used to measure the distance between two nodes in a tree. Then, proceed to discuss our method that uses the proposed tree metric to learn structured representations when a class hierarchy T is available during supervised learning.
3.1 TREE METRIC
Given tree T (V, d. forming class hierarchy, we formally define the metric dr as:
Definition 3.1 (Tree Metric). The tree metric dv e') for any pair of nodes v, v' E V is the weighted length of the shortest path in connecting and v'.
Proposition 3.1. For any undirected weighted graph 9, ('s a well-defined metric over y, all edge weights of G are positive and G is connected.
Our main motivation use the tree metric dy instead of as defined in Section 2.2 two-folds.
First, while each edge in the class hierarchy T corresponds to a subset relationship, there are other kinds of structured relationships between classes that go beyond the subset relationship. For exam- ple, the benchmark dataset MetaShift (Liang & Zou 202T) also contains a graph to describe the relationship between different subsets of classes. However, in MetaShift the weighted edges do not correspond to the subset relationship in the case of a class hierarchy, but rather to a similar- ity/discrepancy measure between them. In this case, the relationship between classes corresponds to an undirected and weighted graph, where the notion of LCA does not apply any more. In fact, even in the case of a tree, the LCA is subject to change depending on which node the tree is chosen as the root node. However, the tree metric dr is invariant rotations of the tree, and applies both trees and general graphs as shown in Proposition 3.1
Second, the definition of -(:-) does not account for the weights of edges in T. This implies that all the fine-grained classes under given super-class are the same. However, in many applications, different fine-grained classes may have different proportions importance, for a given super-class. In these cases, the tree metric is more adapted since it also takes into account the edge weights.
Nevertheless, when is an unweighted class hicrarchy tree, we have the following relationship between the proposed tree metric dr d-)--) and t(--)):
Proposition 3.2. Let T be an unweighted tree with fixed root node. Then for any pair of nodes 2,, U5 V. (v,, u3) = 2t(v,, v2) - [dt(v;) - dt(v,)).
The proof is deferred App. A In particular, Proposition 3.2 states that if the depths of e and by are the same in 'T, then our tree metric reduces to twice of (w; ((10,13) u5). Multiplying a variable by a constant does not affect its correlation with others. Henceforth, what follows, we propose to use the tree metric d-(-,) in replacement of f(--)) to compute CPCC.
3.2 STRUCTURED REPRESENTATIONS BY EMBEDDING THE TREE METRIC
Now that we have defined our tree metric dr (..). we are interested in learning representations f-).), by optimizing the model parameter 6, such that the Euclidean distance between the representations of any pair of class centers i, E [k] approximates dr(r;; (v,, My) the tree Consider a dataset D = {Z,,, H= of size n for classification problem over Ag. We propose to use the CPCC between pz(-.)) and (-)) as a regularizer to the cross-entropy loss, resulting in the loss function:
L(D)
Σ lce (ce(v)(fo(z)))
CPCC(dT.
(x.N)ED
where À 0 is the regularization strength (A in the experiments). The Euclidean distance pz is computed in feature space. Concretely, we first apply the encoder fe to D and obtain a set of points in Z x Y: Dz := {(fo(r;) v.)))) =- Then, ground-truth labels, and consider the same tree structure T on Dz- Note the negative sign before coefficient X in the above formulation, as we wish to maximize the CPCC score.
partition Dz into subsets according the
In practice, at each iteration during training, since stochastic optimization methods are used, we process batch of inputs instead of the whole data set D. For each incoming batch, we track the
4

---

Published as a conference paper at ICLR 2023
number of finest classes represented in the batch before any pairwise caleulation. When all the inputs in a batch come from the same coarse class, the CPCC score is not well-defined due to the variance of dy. This can happen when the batch size is relatively small. In such cases, we fix the value of the CPCC regularizer to 0 to avoid the numerical division by zero error.
Time Complexity of the CPCC Regularizer The computation of our CPCC regularizer is lightweight. For a feature space with p dimensions, for each training iteration, there will be at most O(p min 0ipmin k²)) additional computations, where is the batch size. Such overhead is of- ten negligible when compared with the computations needed to train a neural network. In App. B we also provide a brief discussion on the convergence of optimizing the above objective functionn with SGD.
3.3 THE BENEFITS OF STRUCTURED REPRESENTATIONS
In what follows, describe two potential benefits of learning structured representations with the proposed CPCC regularizer, before providing thorough empirical validation in Section
Interpretability As we briefly discussed before, one potential drawback of the representations learned through supervised learning is the lack of interpretability. Recent work (Papyan et al.) 2020) Han et al.] 202I) have both empirically and theoretically (under certain assumptions) shown that under the cross-entropy loss, when enough training has happened, the learned representations will have reduced variance within each class, and the set of features corresponding different classes will converge to the so-called simplex equiangular tight frame (ETF). Yet, the vertices of the simplex ETF are symmetric (in the sense of being permutationninvariinn) hence the class features do not necessarily reflect the similarities/differee
between different classes, even in feature space.
By enforcing the Euclidean distances in feature space between different classes to be close to the tree metric through our CPCC regularization, we attempt to break the symmetry in learning the features. This can potentially lead to more interpretable features, as closer classes (in the sense of the tree metric) are closer to each other in feature space.
Generalization Another by-product of structured representations is potentially better generaliza- tion both in-distribution when only limited amount of labels is available, or under sub-population shifts (Santurkar et al. 2020). To see this, note that the goal of our CPCC regularizer consistent with classification accuracy: it essentially pushes data from different classes away proportionally to their distance in the tree. Consequently, for aub-population shifts, if the hierarchy correctly cap- tures coarse-fine fine relationship, future unseen fine mmn-grained classes from the same coarse category will be further away from those under a different coarse category. This may help generalize to unseen fine-grained classes in zero few-shot learning.
4 EXPERIMENTS
In this section, we apply our proposed method to: (i) study how using CPCC during training affects the representation learnt under various training objectives (Section 4.3 and (ii) see how the learned structured representations can improve generalization (Section
4.1 DATA
We conduct our experiments MNIST (Lecun et al., 1998), CIFAR100 (rizhevskyy) 2009), and BREEDS (Santurkar et al., 2020). By using this variety of datasets and hierarchies, we get a com- prehensive overview of the usefulness of CPCC regularizer. See App. for the full hierarchies.
MNIST contains handwritten digits from 9. We define odd and even digits to two coarse classes (Fig. Ta). The digits in the leaves are called fine classes below. The artificial level based on odd- and even-ness corresponds to concepts that are not visually observable. CIFARI00 comes with a predefined hierarchy: its coarse level has 20 classes, each containing 5 fine classes (e.g.. beaver, dolphin, otter, seal and whale belongs to aquatic mammals). While the hierarchies are semantically meaningful, the coarse level labels not purely defined by visual similarities. For instance, it is hard to tell the size of an animal from its image (making the coarse classes large omnivores/hertivores and mid-size mammals difficult to distinguish). BREEDS is benchmark
5

---

Published as a conference paper at ICLR 2023
built on ImageNet (Deng al., 2009). It contains manually calibrated label hierarchy, based solely on shared visual characteristics. Santurkar et al. (2020) proposed four tasks: LIVING17, ENTITYI3, ENTITY30, and NONLIVING26. For each, consider the leaf nodes, which are ImageNet classes, as our fine level classes, and define the coarse levels to be their "superclasses" at different depths. We end up with trees that only contain the root node and two levels of the initial hierarchy, and ignore intermediate relationships for CPCC regularization.
New Levels Based on the coarse and fine levels in the hierarchy, we insert mid level between the coarse and fine ones, as well as coarser level between the coarse one and the root node. This results in classes verifying Keoarser < Recarse < kmid kine- In MNIST, the mid classes are 1,3.5 (odd numbers & 5), 7,9 (odd numbers >> 5), 0,2,4 (even numbers I 5), and 6,8 (even numbers >> 5). We do not consider a coarser level (it is trivial train on the root node). CIFAR 100, each coarse level (containing 5 fine classes) is split into arbitrary groups of 2/3 fine classes, creating 40 classes in the mid level. 2 arbitrary coarse classes are merged into coarser label, creating 10 coarser labels. Since BREEDS contains 8 non-root levels in total, and all 4 datasets' coarse levels have depth 2 2, we use the original hierarchy and let the mid level be one level above the fine classes, and the coarser level be one level above the coarse classes.
Source & Target Split We split BREEDS into source (s) and target (t): s and t have the same coarser and coarse labels, but mid and fine classes are different. Following this idea, recall we split MNIST/CIFAR'S 's coarse levels into groups of 2 and 3. We take 60 fine classes as CIFAR* and the rest CIFAR 6 classes MNIST 4 in MNIST* Due this construction, there is only one mid class in each of CIFAR/MNIST* /t's coarse class. On the other hand, BREEDS's coarse classes have many mid children.
4.2 BASELINES AND METRICS
As mentioned above, we operate in fully supervised setting. We denote our neural network function h X > Aj. h = go fo can be decomposed into a feature extractor fe. and a linear classi- fier g which the softmax is applied. Training performed using the following objectives, on the fine-coarse hierarchy of MNIST, CIFAR, and BREEDS, with and without CPCC as a regularization (see App. C for more details). Our baselines include:
Flat lcr: training on the fine classes only, without leveraging any hierarchical information.
Multi-task Learning: jointly training a two-headed network to treat fine and coarse as two sepa- rate tasks. The loss function is the sum of the cross-entropies on the fine and coarse classification tasks, and we simply set the weight of the two parts to 1.
Curriculum Learning: In the spirit of curriculum learning, we first train on the coarse classes using fce and use Yeearse instead of yhne- In the second step. we remove the linear classifier and fine tune a new one the fine level labels with fer the loss function.
Sum Loss: We define a hierarchical Sum Loss as Σ ( Wh(#)) + (Cr(Vanes h(x)), W is a 1 by l2 matrix representing the relationships in the label tree: if a fine class i belongs to coarse class j, then Wj is 1, otherwise the entry is set to O.
HXE: The Hierarchical Cross Entropy (Bertinetto et al. 2020) that replaces the predicted output in ben with weighted hierarchical class conditional probabilities.
Soft: The soft labels objective (Bertinetto et al.] 2020) where labels in feE are derived from a mapping function to encode class node similarity y.
Quad: The Quadruplet Zhang et al. (016)) multi-task loss which combines fce with a generalized triplet loss to enforce different margins different levels of the hierarchy.
Metries To evaluate the representation structure learnt with the various loss functions, as well as the influence of CPCC, we use (i) silhouette scores (Rousseeuw) 1987) to measure the salience of clustering patterns at the coarse level, (ii) CPCC as metric to measure how the whole representa- tion structure is similar to the fine-coarse hierarchy, (iii) t-SNE (Van der Maaten & Hinton| 2008) for visualization of the learnt embeddings in 2D, and (iv) symmetric distance matrix to evaluate the hierarchical structure. Specifically, we calculate the Euclidean distance between the mean rep- resentation vectors for each pair of fine classes. The matrix is organized in way where fine classes from the same coarse class are grouped together, so that the coarse within-cluster distance is shown around the diagonal while other entries present coarse level between-cluster distances.

---

Published as a conference paper at ICLR 2023
Without CPCC
With CPCC
Figure 2: The matrices show the distance between fine CIFARI00 classes with and without CPCC for Flat (lighter color means smaller distance, same color palette used for both). The light diago- blocks with CPCC correspond to the coarse classes. We also show t-SNE visualization of the representations (colored by coarse labels) learnt using Flat with and without CPCC regularization. The pattern is aligned with distance matrix. When CPCC is used, fine classes from the same coarse classes tend to be closer, and coarse classes tend to be further apart.
4.3 STRUCTURE OF THE LEARNT REPRESENTATIONS
Fig. 2 show the effects training with CPCC, which matches our expectations shown Fig. Tb In the distance matrix, between-coarse cluster distance (corresponding to diagonal 5 by 5 blocks). This fact is verified qualitatively in the t-SNE plots, where coarse groups tend to be better separated. Similar patterns are observed when CPCC is paired with the other loss functions described above. We want to point that although other setups have a more structured representation some extent, these are not as perfect as when paired with CPCC (see App. A the figures).
see that the within-coarse cluster distance is much smaller than the
In Table M we see that the objectives leveraging the hicrarchical information tend increase the CPCC. This is particularly true of the multi-task and soft labels setting. But, directly optimizing the CPCC score still gives the largest gains, both on CPCC and silhouette scores.
4.4 GENERALIZATION ON DATASETS WITH SHARED HIERARCHY
Given a representation Se trained on the fine-coarse hierarchy (Section 4.2 we want to see struc- tured representations help performance in-hierarchy, i.e., the fine and coarse classes the model was trained on; as well as hiirrrhhh i.e., on new levels and/or new classes of the hierarchy.
In-hierarchy In this setting, we evaluate the models on classes and levels used during training to construct the various objectives and the tree metric (i.e., the fine and coarse classes). Results can be found in the FineAcc and CoarseAce columns of Table [] Adding our CPCC regularizer leads to better test accuracy at both levels, across objectives and datasets, with gains sometimes exceeding 1%. According to Goyal et al. (2021), ), such performance gain (especially on fine classes) is rarely observed when hierarchical information is leveraged. Overall, our findings suggest that when such information available, using CPCC regularizer is beneficial.
Out-of Out-of-bierarchy Two questions naturally arise in our hierarchical setting. The first one is how well CPCC structured representations generalize to new levels of the hierarchy. To answer this question, we report in Table I the accuracy on the new mid and coarser levels defined in Section 4.1 and not used during training. This accuracy is obtained zero-shot, via a simple marginalization (e.g.. the probability of mid class is the sum of the probabilities of all fine classes that belong to it). There too, adding the CPCC regularization results performance gains (also see App. Table[4).
The second natural question is if CPCC structured representations can generalize to classes unseen in the training hierarchy. Assume a model has learned that cats and dogs are animals. Does knowing the animal concept help it understand giraffes or horses better? To explore this, we train our models on the source split of the mid and fine classes (Section 4.1.. and evaluate their performance on the target split. We still apply "zero-shot" transfer coarse and coarser level via marginalization, but fine tuning is necessary to classify the new mid and fine classes: we freeze fo and fine tune a linear classifier Imid or gine single image from each new target label (one-shot generalization).
7

---

Published as a conference paper at ICLR 2023
Table 1: Mean % and standard deviation over 5 seeds for various datasets, objectives and metrics, with and without CPCC (overall best in bold, best for given objective with/without CPCC under- lined). BREEDS's 's results are on the source split. Regularizing with CPCC never hurts performance, and in most cases leads to consistent and sometimes significant improvements on all metrics.
Dataset
Objective
CPCC
Silhouette
FineAce
MidAcc
CoarseAce
CoarserAce
Flat
10.80 (1.49)
13.97 (0.72)
99.05 (0.23)
99.38 (0.04)
99.49 (0.08)
N/A
MNIST
FlarCCCC
99,96 (0.01)
61.33 (0.42)
99.28 (0.08)
99.38 (0.03)
99.61 (0.03)
N/A
Flat
24.38 (0.57)
5.59 (0.02)
76.82 (0.30)
80.27 (0.35)
85.59 (0.35)
(0.27)
FlaCCPCC
84,20 (0.39)
34,40 (0.11)
77.47 (0.27)
81.30 (0. 14)
86.95 (0.17)
88.17 (0.17)
MTL
39.75 (0.33)
8.09 (0.08)
76.56 (0.20)
80.17 (0.22)
85.79 (0.20)
87.11 (0.14)
MTLCPCC
84.88 (0.58)
31.58 (0.23)
76.90 (0.32)
80.91 (0.29)
87.11 (0.19)
88.39 (0.19)
Curr
23.81 (0.60)
5.25 (0.11)
76.84 (0.20)
80,40 (0.17)
85.72 (0.16)
K7.02 (0.18)
CumCPCC
85,32 (0.51)
34.08 (0.23)
77.48 (0.44)
81.42 (0.32)
87.15 (0.19)
88.44 (0.20)
CIFAR10O
SumLoss
29.85 (0.63)
4,93 (0.07)
76.78 (0.20)
80.47 (0.22)
85.88 (0.25)
87.11 (0.26)
SunLosSCPCC
84.78 (0.64)
31.16 (0.13)
77.26 (0.12)
81.17 (0.18)
86.99 (0.07)
88.20 (0.02)
HXE
(0.68)
8,31 (0.05)
76.58 (0.27)
80.17 (0.24)
85.67 (0.15)
87.02 (0.16)
HXECPCC
85.13 (0.22)
35.84 (0.18)
76.57 (0.33)
80.63 (0.24)
86.48 (0.20)
87.77 (0.20)
Soft
55.95 (0.67)
14.48 (0.11)
76.82 (0.06)
80.41 (0.07)
85,84 (0.16)
87.16 (0.07)
SoftCCCC
85.23 (0.24)
35.80 (0.16)
77.11 (0.16)
81.02 (0.13)
86.63 (0.17)
87.93 (0.14)
Quad
25.08 (0.26)
6.75 (0.06)
76.40 (0.28)
80.05 (0.27)
85.30 (0.11)
86.67 (0.14)
QuadCPCC
84.65 (0.32)
34.79 (0.23)
77.10 (0.16)
80.92 (0.12)
86.78 (0.09)
88 88.04 (0.09)
LIVINGI7
Flat
36.74 (0.92)
8.89 (0.08)
83.66 (0.51)
89.16 (0.28)
89.72 (0.26)
92.54 (0.34)
FlacCPCC
93.56 (0.46)
48.26 (0.51)
84.97 (0.68)
90,52 (0.54)
91.13 (0.61)
93.66 (0.36)
Flat
34.97 (0.44)
3.12 (0.03)
82.36 (0.38)
84.52 (0.33)
90.43 (0.34)
93.38 (0.33)
ENTITY13
FlaCCPC
90.98 (0.22)
38.68 (0.17)
83.37 (0.37)
85.53 (0.28)
91.50 (0.09)
94.22 (0.06)
ENTITY30*
Flat
FlaCPCC
24.29 (0.30)
73.55 (0.77)
29.43 (0.12)
3.07 (0.06)
80.81 (0.35)
82.00 (0.31)
82.28 (0.35)
83.57 (0.34)
87.60 (0.13)
86.00 (0.25)
89.27 (0.27)
90.72 (0.28)
NONLIVING26
Flat
26.95 (0.31)
5,90 (0. (0.13)
80.79 (0.36)
83.49 (0.30)
80.79 (0.36)
87 (0.25)
FlaCCCC
82.20 (0.44)
34.63 (0.31)
82.96 (9.35)
85.88 (0.37)
87.42 (0.36)
89.35 (0.40)
Results are shown in Table First, under this subpopulation shift, using CPCC still outperforms the original loss functions on coarse and coarser levels (zero-shot), which is consistent with results in [ Second, in one-shot generalization new mid levels, CPCC gives an often large advantage. Intuitively, as all fine classes are grouped together within coarse groups, if one data point is randomly selected, then other data points in the same coarse class will readily be assigned the same label. Without this structure in the representation, generalization is more difficult as all fine labels are evenly distributed. The only notable exception is ENTITYI3, where each coarse label has too many mid level children and grouping by coarse level hurts. Third, CPCC regularization is often harmful to one-shot fine level generalization due to coarse grouping: new fine classes are close together at the coarse level, making them hard to linearly separate. The structure of label tree matters: compared to LIVING17 and VONLIVIGG26, ENTITYY's fine level labels partition coarse labels into much more fine-grained subsets, resulting in the performance difference in BREEDS. We do observe that other hierarchical methods have some advantage compared to the flat cross entropy.
5 RELATED WORK
There were many works exploiting label hierarchy and we only refer to the most related ones. How- ever, to the best of our knowledge, none of the previous work set learning structured representations as main objective or embedded the tree metric under this context.
Background of Baseline Methods The simplest label hierarchy contains only two level, coarse and fine, which can be treated as two tasks trained jointly or sequentially. The former originates from Multi-task Learning (MTL) where part of single network is shared for multiple heads for each task during training (Caruana 1997; ZZao et al. 2020 Inoue et al.) 2020). The latter echoes with Curriculum Learning (CL) (Bengio et al. 2009), where pretraining with a easy task will help the convergence and performance on a hard task. We define coarse level classes following [Ahn jet al. (2021): ); Peterson et al. (2018); Wang & Cottrell (2015); Streteu et al. (202T) to emulate human learning behavior. Sum Loss directly modifies the loss function by marginalizing fine classes prob- ability for coarse classes. [Su & Maji (202I): Hu et al. (2018) defined a similar version to address some partially labeled tasks. Bertinetto al. 2020) provided thorough survey of different types of
8

---

Published as a conference paper at ICLR 2023
Table 2: The superscript denotes or 0-shot generalization. All models are trained on the source split 8 and evaluated on the target split t. and t have different fine/mid classes but the same coarse/coarser classes. CPCC shows an advantage on mid, coarse and coarser, but not fine, levels.
Datuset
Objective
FineAcc'
MidAcc
CoarseAce"
CoarseAAce'
MNIST
Flat
69.93 (10.46)
53.25 (4.89)
51.84 (3.71)
N/A N/A
FlarCPCC
53.25 (4.89)
55.11 (8.32)
58.75 (2.28)
Flat
28. (2.50)
30.66 (3.17)
42,90 (0.34)
47.37 (0.46)
FlatCPCC
25.73 (1.05)
32.97 (6.55)
44.58 (0.17)
48.93 (0.18)
29.59 (1.72)
(3.79)
32.86 (0.31)
(0.32)
MTLCPCC
25.75 (1.61)
32.47 (5.92)
44.43 (0.44)
48.79 (0.43)
Cur
28.77 (2.73)
30.82 (4.64)
43.88 (0.67)
48.33 (0.65)
CunCPCC
(1.18)
32.78 (4.47)
44.65 (0.47)
48.96 (0.47)
CIFARI00
SumLoss
29.31 (2.62)
30.62 (3.26)
43.15 (0.33)
56 (0.37)
SunLossCPCC
32.40 (5.98)
44.87 (0.37)
49.14 (0.39)
HXE
28.97 (2.81)
31.71 (5.51)
44,28 (0.43)
48.67 (0.46)
HXECPCC
(1.84)
32,03 (5.68)
44.38 (0.34)
48.35 (0.26)
Soft
29.35 (1.90)
32,96 (3.31)
43,99 (0.15)
48.25 (0.34)
SonCPCC
(1.80
34,10 (5.60)
44.65 (0.70)
49.08 (0.68)
Quad
27.89 (0.24)
31.28 (4.57)
42.73 (0.29)
47.32 (0.54)
QuadCPCC:
24.48
(5.57)
(0.39)
(0.42)
Flat
28.52 (3:22)
32.24 (2.60)
52.99 (0.79)
69 68 (0.38)
LIVINGI7*
FlaCCPCC
29.92 (3.40)
39.16 (3.30)
56.36 (0.69)
72.08 (0.43)
ENTITY13
Flat
16.03 (1.60)
20.66 (0.43)
58.42 (0.33)
69. 76 (0.45)
FlaCPCC
13.06 (1.64)
18.89 (1.29)
61.28 (0.18)
71.94 (0.28)
ENTITY30*
Flat
21.53 (1.49)
21.61 (1.68)
45.66 (0.20)
60.07 (0.26)
FlaCPCC
20.79 (0.78)
25.29 (1.32)
48.83 (0.12)
62.89 (0.39)
Flat
23.50 (1.74)
25.37 (2.36)
39.31 (0.21)
53.14 (0.14)
NONLIVING26
FlaCPCCC
24.04 (1.04)
27.99 (2.53)
42.49 (0.54)
56.14 (0.73)
hierarchical methods, well label embedding methods (encoding hierarchical information into labels, the Soft objective), hierarchical losses (Quad (Zhang et al. 2016). HXE). None of these use regularization method, with the exception of group overlapping lasso Zhao al., 2011). However, it was introduced for logistic regression, making it hard to to be applied to modern neural networks that use the penultimate layer as its representation.
Learning with Label Hierarchy The most common motivation of hierarchical models is to im- prove the fine-level accuracy. Interestingly, accuracy improvements are often mixed: while most works claimed gain performance improvement, Wang & Cottrell (2015) stated that this improve- ment was limited, and Goyal et al. (202T) claimed most hierarchical models lead to worse perfor- mance on non- nin-rierarl accuracy metries. Additionally, using coarse level labels often appears in a weakly supervised setting, where coarse classes are always available but fine class labels are only accessible for part of data, to reduce annotation cost at a finer level (Taherkhani et al. 2019 Lei et al. (2017: |Ristin et al., 2015). Other works built hierarchy from dataset (Murdock et al.) 2016]] Li et al. [2100] Verma et 2012 Han et al.) 20188) Zheng et al. 2017). We only name a few since they are very different from our setting where the hierarchy is defined before training.
CONCLUSION
How to include label relation into representation is an open question. In this paper, in the context of tree label hierarchies, we use the cophenetic correlation coefficient as a regularizer to embed this hierarchical relationship into representations, and outperform other baseline methods. CPCC has multiple advantages, including low time complexity. better interpretabilityy flexibility on any supervised learning paradigms, and it can be applied to any common label relation graphs. We also demonstrate that it leads to better generalization performance several downstream tasks. All these benefits show that our method provides an interesting solution to this important problem.
ACKNOWLEDGMENTSS
Han Zhao would like to thank the support from a Facebook research award and Amazon AWS Cloud Credits.

---

Published as a conference paper at ICLR 2023
REFERENCES
Seoyoung Ahn, Gregory J. Zelinsky, and Gary Lupyan. Use of superordinate labels yields more robust and human-likk like visual representations in convolutional neural networks. Journal of Vision,
(1(13):1--1,, 12 2021. ISSN 1534-7362. doi: 10.1 10.1167//ov.21.13.13. URL https: //doi. org/10. 11.110.116..1 jov. 21.13.13
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference Machine Learning, ICML '09, pp. 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 10.1145/1553374, 1553380.
URL https:://ec..rr///
1553374. 1553380
Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A. Lord. Making better mistakes: Leveraging class hierarchies with deep networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12503-12512 2020. doi:
10.1109/CVV44420022000000111
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, (0(2):223-311) 2018
Rich Caruana.
Multitask learning. Machine learning, 28(1):41-755 1997.
Hao-Yus Yun Chen, Li-Huang Tsai,
Chang, Jia- Yu Pan, Yu-Ting Chen, Wei Wei, and Da- Cheng Juan, Learning with hierarchical complement objective, 2019. (ttptttp:://are
org/ abs//b///a 1911. 07257
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 48-64, Cham, 2014. Springer International Publishing.
Christiane Fellbaum. WordNet: An Electronic Lexical Database. Bradford Books, 1998.
Palash Goyal, Divya Choudhary, and Shalini Ghosh. Hierarchical class-based curriculum loss. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, pp. 3448-24544 International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963/ijcai 2001/333. URL https://doi org ortp::ii..o0000...9990000
20213330 Main Track.
Yanming Guo, Yu Liu, Erwin M Bakker, Yuanhao Guo, and Michael S Lew. Con-rnn: a large-scale hierarchical image classification framework. Multimedia tools and applications, 77(8):00251 10271, 2018.
Xu Han, Pengfei Yu, Zhiyuan Liu, Maosong Sun, and Peng Li. Hierarchical relation extrac- tion with coarse-to-fine grained attention. In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 2236-2245, Brussels, Belgium, October- November 2018, Association for Computational Linguistics. doi: 10.18653/v//DI8-11 1247. URL
/twg///------ 1247
XY Han, Vardan Papyan, and David Donoho. Neural collapse under mse loss: dynamics the central path. In International Conference on Learning Representations, 2021.
Proximity to and
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015. URL htt.hhttss//// rxiv. o abs / 151 2. 03385
David T Hoffmann, Nadine Behrmann, Juergen Gall, Thomas Brox, and Mehdi Noroozi. Ranking info noise contrastive estimation: Boosting contrastive learning via ranked positives. In AAA! Conference Artificial Intelligence, 2022.
10

---

Published as a conference paper at ICLR 2023
Peiyun Hu, Zachary C. Lipton, Anima Anandkumar, and Deva Ramanan. Active learning with partial feedback, 2018. https::/axxxii.. http://wawwi.....aaaaaaaa/o abs//ab//// 1802. 002.8002.27777
Matheus Inoue, Carlos Henrique Forster, and Antonio Carlos dos Santos. Semantic hierarchy based convolutional neural networks for image classification. 2020 International Joint Conference Neural Networks (IJCNN), pp. 1-8, 2020. doi: 10. 109/JCNNN8605.220 202.4000552200..... 202000200224200
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-baset learning applied document recog- nition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998. doi: 10. 109/5.726791.
Jason D Lee, Max Simchowitz, Michael Jordan, and Benjamin Recht. Gradient descent only converges to minimizers. In Conference on learming theory, pp. 1246-1257. PMLR, 2016.
Jie Lei, Zhenyu Guo, and Yang Wang. Weakly supervised image classification with coarse and fine labels. In 2017 14th Conference on Computer and Robot Vision (CRV), pp. 240-247. IEEE, 2017.
Li-Jia Li, Chong Wang. Yongwhan Lim, David M. Blei, and Li Fei-Fei. Building and using semantivisual image hierarchy. In 2010 JEEE Computer Society Conference on Computer Vision and Puttern Recognition, pp. 3336-3343, 2010. doi: 10. 00.10099CCR2220055500077
Weixin Liang and James Zou. Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts. In International Conference on Learning Representations, 2021.
Amitangshu Mukherjee, Isha Garg, and Kaushik Roy. Encoding hierarchical information in neural networks helps in subpopulation shift, 2022. URL htppps wtwp://wwwwwweeeeformrmr.
id iddhJkkiffyEEy
Calvin Murdock, Zhen Li, Howard Zhou, and Tom Duerig. Blockout: Dynamic model selection for hierarchical deep networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
Katta G Murty and Santosh N Kabadi. Some np-complete problems in quadratic and nonlinear programming. Technical report, 1985.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Vivak Patel and Shushu Zhang. Stochastic gradient descent on nonconvex functions with general noise models. arXiv preprint 1XXi:2210400022: 2021.
Joshua C. Peterson, Paul Soulos, Aida Nematzadeh, and Thomas L. Griffiths. Learning hierarchical visual representations in deep neural networks using hierarchical linguistic labels, 2018. URL
https://axxiz/ org/ altpg//bs//18055 http//805500000 0001417055777777
Marko Ristin, Juergen Gall, Matthieu Guillaumin, and Lue Van Gool. From categories to sub-
categories: Large-scale image classification with partial class label refinement. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 231-239, 2015. doi: 10. ccc.oc/..e..ttooacccaaaaaaaa///2
Peter Rousseeuw. Silhouettes: A graphical to the interpretation and validation clus- ter analysis. Journal of Computational and Applied Mathematics, 20:53-65, 1987. ISSN 0377-0427. doi:
URL
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift, 2020. h/tp:///...t.pppaaaattttttaaa/a/
04859
Robert R. Sokal and F. James Rohlf. The comparison of dendrograms by objective methods. Taxon, 11(2):33-44 40, 1962. ISSN 00400262, URL htRlhttp:12 /ttp::///www... wwt.:w..s.o......... jstor. org http:/saallleeee //217700

---

Published as a conference paper at ICLR 2023
Otilia Stretcu, Emmanouil Antonios Platanios, Tom M. Mitchell, and Barnabás Póczos. Coarse-to- fine curriculum learning. CoRR, abs/2106.04072, 2021. URL
abs/
0407.
Jong-Chyi Su and Subhransu Maji. Semi-supervised learning with taxonomic labels, 2021. URL
httts://arxiv org wwg/absg///aooooooooo..oo.. /tops/211... 2111.
Fariborz: Taherkhani, Hadi Kazemi, Ali Dabouei, Jeremy Dawson, and Nasser M Nasrabadi. A weakly supervised fine label classifier enhanced by coarse supervision. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6459-6468, 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun. Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset, 2017. URL wttpt: sexxivv.orge
Nakul Verma, Dhruv Mahajan, Sundararajan Sellamanickam, and Vinod Nair. Learning hierarchical similarity metrics. 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp- 2280-2287, 2012. doi: 10.109//VP..2202..24444
Panqu Wang and Garrison W. Cottrell. Basic level categorization facilitates visual object recogni- tion, 2015. URL
04103
Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. Hd-cnn: Hierarchical deep convolutional neural network for large scale visual recognition, 2014. URL TRLhhtts:://// /ttp:://aaiivvv/aa /ttt:s///axxi...rr//bb//////
0736
Xiaofan Zhang, Feng Zhou, Yuanqing Lin, and Shaoting Zhang. Embedding label structures for fine-grained feature representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1124, 2016.
Bin Zhao, Fei Li, and Eric Xing. Large-scale category structure aware image catego- rization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Asso- ciates, Inc., 2011. URL https: [BC220C4EES0LL0222835555---::::2 pdf
Junjie Zhao, Yuxin Peng, and Xiangteng He. Attribute hierarchy based multi-task learning for fine-grained image classification. Neurocomputing, 395:150-159, 2020. ISSN 0925-2312.
doi: http::///hiiioo////.oooo/oooouoa 2018::00000000000000000000000 URL https: wttp://..c..cccc.eeett.....
com/ http://ccicnoeerrree////////2222
Yu Zheng, Jianping Fan, Zhang, and Xinbo Gao. Hierarchical learning of multi-task sparse metrics
for large-scale image classificaation, Pattern Recognition, 67:97-109, 2017. ISSN 0031-3203, doi:
/ wtwp://eeeeww.eeeeeeeeeeeeee.e..
12


Reviews:
[{"review_id": "KUp2AH8ftp", "reviewer": "Reviewer_EJjb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops a cophenetic correlation coefficient (CPCC) regularizer based on the given class hierarchy for classification.  The proposed regularizer is computationally lightweight and easy to implement. They demonstrated that the regularizer can learn more interpretable representations due to the preservation of the tree metric. Experiments six real-world datasets show that the proposed method achieves better in-distribution generalization as well as under sub-population shifts.", "summary_of_the_review": "The reviewer thinks the paper is marginally below the acceptance threshold. The main contribution in the method section is the proposed metric. However, the paper needs to provide more theoretical analysis and ablation studies to show the effectiveness of the proposed metric. ", "strength_and_weaknesses": "Strengths:\n\n[+] The idea of preserving the class hierarchy in learning representations is interesting and sounds reasonable.\n\n[+] The preliminaries give good background knowledge about the class hierarchy and the CPCC.\n\n[+] The figure demonstrates clearly the class hierarchy.\n\nWeaknesses:\n\n[-] The motivation for embedding class hierarchy is not convincing. For instance, why does the model need to know whether a digit is even or odd? The pre-defined class hierarchy in MNIST is artificial, which would not make sense to the model. Intuitively, 6(even) is more similar to 9(odd) rather than 4(even).  Moreover, the class hierarchy would not always available in practice. The title of this paper lets readers expect to see how to explore the class hierarchy rather than just using it.\n\n[-] The method is a bit short of important details about using the weights of edges in the tree graph to compute the weighted length of the shortest path.  Moreover, if there are several shortest paths between pairwise nodes in the general graph, how does the proposed tree metric deal with it?\n\n[-] The paper argues that the tree metric is invariant to rotations of the tree and the changing of the root node. In practice, if the class hierarchy is pre-defined, is it reasonable to change the root node? With the class hierarchy, when does the tree metric work better than the dendrogrammatic distance in a real-world system?  If you wish to show the benefits of the proposed metric, extensive ablation studies are needed to compare the proposed method with other baselines embedding the class hierarchy.  And it would be better to provide more deep insights about the proposed metric. \n\n[-] Is there any theoretical guarantee to make sure that the proposed regularization converges? (Especially the main body of this paper is the regularization.) Is the mini-batch training strategy equivalent to the designed strategy in Eq. (2) where the regularization is constructed on the whole dataset? \n\n[-] In the experiments, due to the use of extra information about the class hierarchy, the better performance of the model with CPCC is not that surprising. It would be better to give more deep insights through the investigation of the regularization.\n\na). It would be more convincing to investigate the effects of CPCC on different objectives.\n\nb). According to the Fine ACC and Coarse ACC in MNIST, the baselines with CPCC does not always outperform the baselines without CPCC. Here we need some explanations about the failure cases. \n\nc). Based on the Mid Acc and Coarser Acc, this paper provides a good way to show the one-shot generalization. However, it is necessary for baselines with CPCC to show the differences before and after the fine-tuning.   \n\nd). What is the effect of the number of layers in the hierarchy classes in the CPCC?\n\ne). Are there any benefits of regularization in solving the out-of-distribution problem?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Jsjc_cMN0x", "reviewer": "Reviewer_siuS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about structured representation learning for image classification at different semantic level of granularity. The author proposed to leverage class hierarchy and relationships by introducing a regularization function using the CPCC in addition to the cross-entropy loss. They introduced a measurement of distance between classes structured in a tree and enforce the network to follow the rules of this distance metric in the deep space. They perform on 6 image classification datasets with different structured representation learning paradigm such as multi-task learning, curriculum learning and others.", "summary_of_the_review": "The related work and citation seems to be accurate and recent. The figures are informative. I easily tend to accept this paper because of the quality of the explanation and the thoughrough experiment section. What would make this paper a candidate for being highlighted would have been an additional contribution. ", "strength_and_weaknesses": "Strength:\n- This paper is very well-writtent and well motivated. \n- The authors discuss the limitation of their method and provide arguments on results that do not outperform the state-of-the-art.\n- The figures are very informative and provide a good insight on the ideas that motivated this work.\n- The utilization od CPCC as a regularizer seems novel.\n- Strong experimentation showing the strength of the method on a wide variety of datasets.\n\nWeakness:\n- There is only one contribution, while it is well detailed, it could still qualify as an incremental improvement to existing work.\n- The lack of interpretability is a problem, even if it is discussed by the authors many times in the paper. For example in the MNIST dataset, the authors mentionned that the classes are sorted to the nodes as evens and odds parent classes, and that there is no visual similarity between the leaf node from the same parent. The question that arises is : Is it critical that the tree structure follow some logic ? It would have been interesting to see results on a random tree hierarchy and see if the results are the same. If they are not, then, simply breaking the class permutation invariance seems advantageous. Clearly for other datasets particularly with animals classes, visual similarity seems to be a factor in the tree hierarchy so why is it important in some cases and not others?\n- There is also a concern regarding the tree structure and current capacity of deep neural networks. The tree metric is said to be weighted with respect to edge weights in section 3.1 but the difference in magnitude between this tree distance function and the euclidean distance between deep features seems to be on completely different scales. How is this addressed in the regularization function? Is there some additionnal normalization term?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Gn2qFtrxq4", "reviewer": "Reviewer_SKmB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new method for learning structured representations from hierarchical annotations. A typical approach for learning representations in a supervised setup is to train a classifier that assigns a single label to a given image. This approach, however, might result in suboptimal representations as samples from all two classes are treated similarly, i.e. misclassifying a specific class of dog with another dog is penalized similarly to misclassifying it as a cat. The proposed method uses freely available hierarchical semantic annotation to enhance the learned representations. Their proposed method involves a new distance metric of two nodes in the hierarchical semantic tree used as an auxiliary loss function that preserves similarity between a pair of given samples on top of standard cross-entropy for classifying them. ", "summary_of_the_review": "Overall, the proposed method is simple and makes sense. But it needs to be supported by a better and fairer experimental setup.", "strength_and_weaknesses": "Strengths:\n\n- The paper follows an interesting direction, which has good potential.\n- The proposed method is simple and makes sense.\n- The paper is well-written and easy to follow.\n\nWeaknesses:\n\n- The authors aim to learn structured representations. However, there is no quantitate evaluation to show that their method results in more structure in the representation space. They only show a t-SNE plot of their method and compare it with the baseline, which is a good sign but not a thorough evaluation. They could evaluate, for instance, the OOD task, which benefits from the more structured representation [1].\n\n- The evaluation datasets are limited. It would be good to show some results on ImageNet(or at least a smaller version of it, such as Tiny Imagenet.)\n\n- The paper misses some baselines. It would be better to compare with $t(.,.)$ in table 1. Moreover, comparing with [1] would be interesting.\n\n-  I am a bit confused by the term ERM. If I am not mistaken, ERM stands for empirical risk minimization, and the authors refer to Eq 3. by ERM, which is the cross-entropy loss. But all losses in this paper minimize empirical risk. Also, why it's performance is too low in Table 1.? \n\n[1] Hoffmann et al. Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positives.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4U32jFuhux", "reviewer": "Reviewer_NCZE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents CPCC loss for learning deep representation with label hierarchy. It is based on the CPCC score, while re-defining the distance metric, as a shortest path length between two nodes. Overall proposed method sounds reasonable and looks simple. In experiments, the proposed method is evaluated on MNIST (with odd/even as coarse class), CIFAR-100 (with pre-defined coarse and fine-grained label hierarchy) and BREEDS, with a few baseline models including multi-task of coarse and fine-grained loss, curriculum learning, etc. \n\nIn experiments, authors evaluate a few metrics -- shillouette score, CPCC score, t-sne, etc -- to evaluate learned structure. W.r.t. defined metrics, the proposed method showed improved scores over baseline methods. In addition, the paper presents one-shot generalization experiments and demonstrate effectiveness on knowledge transfer.", "summary_of_the_review": "The paper is very well written and clear. The idea of using CPCC score as a loss is well motivated. However, the paper does not fully demonstrate the effectiveness of embedding label hierarchy into the representation to downstream task. Many of the metrics are about whether the model learned label hierarchy or not. It is good that one could train a model as expected, but it is unclear what it is going to be useful for. The paper demonstrates the effectiveness on one-shot generalization for mid or coarser level classification tasks, but the task is somewhat superficial and may be easily solved by combining scores of finer-grained classifiers. Moreover, there are previous methods that regularizes deep representations differently, which should have been compared.", "strength_and_weaknesses": "* Strength\n  - The paper is clearly written.\n  - The formulations are clean and well motivated.\n  - Experimental results show that the method works as expected. Results on one-shot generalization particularly highlights the effectiveness of learning with CPCC score to embed label hierarchy.\n\n* Weakness\n  - While I do not closely follow this topic of learning with label hierarchy, it seems there are multiple previous works on deep representation learning with label hierarchy. For example, [[Zhang et al., 2016](https://arxiv.org/pdf/1512.02895.pdf)], [[Bertinetto et al., 2020](https://openaccess.thecvf.com/content_CVPR_2020/papers/Bertinetto_Making_Better_Mistakes_Leveraging_Class_Hierarchies_With_Deep_Networks_CVPR_2020_paper.pdf)]. In the current writeup, the related work on learning with label hierarchy is too short and narrow. Also the baseline considered seems a bit too weak and naive.\n  - Experiments on one-shot generalization is not particularly interesting nor significant. For example, in experiment authors introduced \"coarser\" or \"mid\" level, which are both coarser level of \"coarse\" and \"fine\" class hierarchy. Even though one can train a new classifier using one data point from with \"coarser\" or \"mid\" level labels, aren't we supposed to do it in a \"zero-shot\" way by simply leveraging \"coarse\" and \"fine\" grained classifiers (i.e., for mid-level classifier, one can simply define a rule on fine-grained classifier)? Generally, I do not think this is an \"one-shot\" problem since 1) \"coarser\" and \"mid\" level labels can be derived from the \"coarse\" and \"fine\" level labels (e.g., \"mid\" level labels of odd & <= 5 is a union of 1, 3, 5 at fine-grained level labels), 2) many \"coarse\" and \"fine\" level labels are given during the model training already. \n  - In this regard, more meaningful experiments would be one-shot or few-shot generalization to unseen classes, i.e., can we learn a fine-grained representation of class \"0\" by leveraging knowledge of its superclass (\"even\") and neighbors (\"2, 4, 6, 8\")? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster