Leveraging Class Hierarchies with Metric-Guided Prototype Learning
Authors: Vivien Sainte Fare Garnot
Loic Landrieu
Venue:  (2021)
Keywords: hierarchical classification
prototypical networks

Abstract: In many classification tasks, the set of classes can be organized according to a meaningful hierarchy. This structure can be used to assess the severity of confusing each pair of classes, and summarized  under the form of a cost matrix which also defines a finite metric. We propose to integrate this metric in the supervision of a prototypical network in order to model the hierarchical class structure. Our method relies on jointly learning a feature-extracting network and a set of class representations, or prototypes, which incorporate the error metric into their relative arrangement in the embedding space. We show that this simultaneous training allows for consistent improvement of the severity of the network's errors with regard to the class hierarchy when compared to traditional methods and other prototype-based strategies. Furthermore, when the induced metric contains insight on the data structure, our approach improves the overall precision as well. Experiments on four different public datasets—from agricultural time series classification to depth image semantic segmentation—validate our approach.

Under review as a conference paper at ICLR 2021
LEVERAGING CLASS HIERARCHIES WITH METRIC-GUIDED PROTOTYPE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In many classification tasks, the set of classes can be organized according to a meaningful hierarchy. This structure can be used assess the severity of confusing each pair of classes, and summarized under the form of a cost matrix which also defines a finite metric. We propose to integrate this metric in the supervision of a prototypical network in order to model the hierarchical class structure. Our method relies on jointly learning a feature-extracting network and set of class representations, or protolypes, which incorporate the error metric into their relative arrangement in the embedding space. We show that this simultaneous training allows for consistent improvement of the severity of the network's 's errors with regard to the class hierarchy when compared to traditional methods and other prototype- based strategies. Furthermore, when the induced metric contains insight the data structure, our approach improves the overall precision as well. Experiments four different public datasets-from agricultural time series classification depth image semantic segmentation-vallen our approach.
INTRODUCTION
Most classification models focus on maximizing the prediction accuracy, regardless of the semantic nature of errors. This can lead to high performing models, but puzzling errors such as confusing a tiger and sofa. This casts doubt on what model actually actually understands from the required task and data distribution. Neural networks in particular have been criticized for their tendency to produce improbable yet confident errors, notably when attacked (Akhtar Mian 2018
The classes of most classification problems can be organized according to a hierarchical structure. Such tree-shaped taxonomy of concepts can be generated by domain experts, or automatically from class names with the WordNet graph (Miller et al. 1990) or word embeddings (Mikolov et al. 2013). A step towards more reliable and interpretable algorithms would be to explicitly model the difference of gravity between errors, as defined by a hierarchical nomenclature.
For classification task over a set K. of K classes, the hierarchy of errors can be encapsulated by cost matrix D E IR K defined such that the cost of predicting class & when the true class is / is Dik. ( 2 O, and Dik, 2015). one can define Dik,1] I] as the length of the shortest path between the nodes corresponding to classes I and L.
0 for all ks 1... K. Among many other options (Kosmopoulos et al.)
As pointed out by Bertinetto et al. (2020), the first step towards algorithms aware of hierarchical structures would be to generalize the use of cost-based metrics. For example, early iterations of the ImageNet challenge (Russakovsky et al.] 2015 Deng et al., 2010) proposed to weight errors according to hierarchy-based costs. For dataset indexed by N. the Average Hierarchical Cost (AHC) between class predictions E K.N and the true labels 2 E KCN is defined as:
AHC(y, 2) = W
1
Σ PEN
D(yns =n]
Along with evaluation metrics, loss functions should also take the cost matrix into account. While it is common to focus on retrieving certain classes through weighting schemes, preventing specific class confusions is less straightforward. The cross entropy with one-hot target vectors for example singles out the prediction with respect to the correct class, but treats all other classes equally. Beyond reducing the AHC, another advantage of incorporating the class hierarchy into the learning phase

---

Under review as a conference paper at ICLR 2021
2
5
(a) Cross entropy,
Learnt prototypes, distortion= 0.42, ER 14.2%, AHC = 0.75
(c) Guided prototypes, distortion= 0.40, ER 11.9%. AHC 0.52
C
distortion = 0.47, ER= 15.2%. AIIC = 0.81
Figure 1: Mean class representation O, prototypes O and 2-dimensional embeddings a learnt on perturbed MNIST by 3-layer convolutional net with three different classification modules: (a) cross-entropy. (b) learnt prototypes, and (c) learnt prototypes guided by tree-shaped taxonomy (constructed according to the authors' perceived visual similarity between digits). The guided prototypes
(c) embed more faithfully the class hierarchy: classes with low error cost are closer This associated with a decrease in the Average Hierarchical Cost (AHC), well Error Rate (ER), indicating that our taxonomy may contain useful information for learning better visual features.
that D may contain information about the structure of the data well. Though is not always the case, co-hyponyms (i.e. siblings) in a class hierarchy tend to share some structural properties. Encouraging such classes to have similar representations could lead to more efficient learning, e.g. by pooling common feature detectors. Such priors on the class structure may be especially crucial when dealing with a large taxonomy, as noted by Deng al. (20T0).
In this paper, we introduce method to integrate the class hierarchy into a classification algorithm. We propose new scale-free, distortion-based regularizer for prototypical network (Yang et al. 2018 Chen al., 2019). This penalty allows the network to learn prototypes organized such that their relative distances reflect their distance in class hierarchy.
The contributions of this paper follows:
We introduce scale- ceale-independent formulation of the distortion between two metric spaces, and an associated smooth regularizer.
This formulation allows us to incorporate knowledge of the class hierarchy into neural network no extra cost in trainable parameters and computation.
We show on four public datasets (CIFAR100, NYUDv2, S2-Agri, and iNaturalist-19 19) that approach decreases the average cost of the prediction of standard backbones.
As illustrated in Figure we show that our approach can also lead to a better (unweighted) precision, which we attribute to useful priors contained in the taxonomy of classes.
RELATED WORK
Prototypical Networks: Our approach builds the growing corpus of work prototypical networks. These models are deep learning analogues of nearest centroid classifiers (Tibshiraní et al. 2002) and Learning Vector Quantization networks (Sato & Yamadal 1995 Kohonen 1995). which associate to each class a representation, or prototype, and classify observations according to the nearest prototype. These networks have been successfully used for few-shot learning (Snell al. 2017; Dong & Xing 2018), zero-shot learning (Jetley et al. (015),, and supervized classification (Guerriero et al. 2018 Yang al. 2018 Mettes et al.) [219 Chen al., 2019).
In most approaches, the prototypes are directly defined as the centroid of the learnt representations of samples of their classes, and updated at each episode (Snell et al. 2017) or iteration (Guerriero et al.)
For formal definition of our scale-free distortion, see Section 3.2 computed from the means of class emheddings for the entropy.
2

---

Under review as a conference paper at ICLR 2021
2018). In the work of Mettes et al. (2019) and Jetley et al. (201)). the prototypes defined prior to learning the embedding function. In this work, we follow the approach of Yang et al. (2018) and learn the prototypes simultaneously to the data embedding function.
The idea of exploiting the latent taxonomic structure of semantic classes in order to improve the accuracy of a model has been extensively explored (Silla & Freitas) 2011), 1). from traditional Bayesian modeling (Gelman et al. [2013] Chapter 5) adaptive deep learning architectures (aan et al. 2015 Roy et al., 2020] Salakhutdinov et al., 2012 Ayub & Saini] 2011). However, for these neural networks, the hierarchy discovered by the network itself in the goal of improving the overall accuracy of the model. In our setting, the hierarchy is defined priori, and serves both to evaluate the quality of the model and guide the learning process towards a reduced prediction cost.
Hierarchical Priors:
Srivastava Salakhutdinov (2013) propose implementing Gaussian priors on the weight of neurons according to fixed hierarchy, Redmon & Farhadi (2017) implement an inference scheme based tree-shaped graphical model derived from a class taxonomy. Closest to our work, (Hou et al. (2016) propose regularization based on the earth mover distance to penalize errors with high costs.
More recently, [Bertinetto et al. (2020) highlighted the relative lack of well-suited methods for dealing with hierarchical nomenclatures in the deep learning literature. They advocate for a more widespread use of the AHC for evaluating models, and detail two simple baseline classification modules able to decrease the AHC of deep models: Soft-Labels and Hierarchical Cross- Cross-Entropy. See Section 4.3 for more details on these schemes.
Hyperbolic Prototypes: Motivated by the quality of their low-dimensional embedding of hierar- chical data structures (De Sa et al. 2018). hyperbolic spaces are at the center of recent advances
modeling hierarchical relations Nickel Kiela 2017 Khrulkov et al., 2020p. Closer to this work, (Liu al., 2020; Long et al. 2020) also propose embed a class hierarchy into the latent representation space. However, both approaches embed the class hierarchy before training the data embedding network. In contrast, training of the model allows the network and class embeddings to share their respective insights, leading to better trade-off between AHC and accuracy. In this paper, we only explore this claim in Euclidean geometry, as this setting allows for the seamless integration of our method.
argue that incorporating the hierarchical structure during the
Finite Metric Embeddings: Our objective of computing class representations with pairwise dis- tances determined by cost matrix has links with finding an isometric embedding of the cost matrix seen as a finite metric. This problem has been extensively studied (Indyk et al.) 2017] Bourgain 1985) and
the center of the growing interest for hyperbolic geometry (De Sa al. 2018). However, our goal is simply to influence the learning of prototypes with a metric rather than necessarily seeking the best possible isometry.
3 METHOD
We consider generic dataset W of N elements E X with ground truth classes E K.V The classes are organized along a tree-shape hierarchical structure, allowing us to define a cost matrix D by considering the shortest path between nodes. The matrix thus defined is symmetric, with diagonal and strictly positive elsewhere, and also respects the triangle inequality: D[k, + D(I, m) 2 [ik m] for all k. I, m in K. In other words, D defines finite metric. We denote by Ω the embedding space which, when equipped with the distance function d Ω x Ω + R,, forms a metric space as well.
3.1 PROTOTYPICAL NETWORKS
A prototypical network is characterized by an embedding function f : X' + Ω typically neural network, and a set क E R of K prototypes. ा must be chosen such that any sample v,, of a given class & has representation f(x,)) which is close me and far from other prototypes.
3

---

Under review as a conference paper at ICLR 2021
Following the methodology of Snell et al. (2017), prototypical network (f. T) associates to observation In the following posterior distribution over its class Zn:
exp T&)) p k|r,) = Σiex exp (-d (-d([(())) (In),, Vk EK.
We can then define an associated loss with the normalized negative log-likelihood of the true classes:
Lauraa (G, = N
Σ BEN
d(s(en), T..)) + log
Σ EEK
exp (-d([((())))) m))
(3)
This loss is such that the representation, f(x,) is attracted towards the prototype of the class 2n, while it is repelled by the other prototypes. Conversely, prototype тr is drawn towards the representations (a..) of samples of class k and away from the representations of other classes.
Following the insights of Yang al.) (2018), the embedding function f and the prototypes are learned sinultaneously This differs with many works prototypical networks which learn prototypes separately or define them as centroids representations. We take advantage of this joint training to learn prototypes which take into account both the distribution of the data and the relationships between classes, as described the next section.
3.2 METRIC-GUIDED PENALIZATION
We propose to incorporate the cost matrix D into a regularization term in order to encourage the prototypes to organize in the embedding space Ω in manner that is consistent with the finite metric defined by D. Since sample representations are attracted to their respective prototypes in 3 this regularization also affects the embedding network.
Scale-Free Distortion As described in De Sa et al. (2018), the distortion of a mapping k + between the finite metric space (K,, D) and the continuous metric space (S2, d) can be defined as:
|d(TKK m1) D(,,
disto( D) K(K 1)
Σ
Djk,1]
KJEKK 1nd
We argue that prototypes arrangements и with low distortion incur lower hierarchical costs. Let us first consider a misclassified sample on of true class k. Since In is misclassified, its representation f(x,) is closer another prototype than to the true prototype Ti. However, f(x,) is still closer to Ty than to most prototypes, as the loss Ldata would starkly penalize ff otherwise. Since low distortion prototypes group together prototypes of classes with small mutual error cost, the erroneous class predicted will likely be of low hierarchical cost with respect to
can assume that
However, achieving low-distortion also imposes a specific scale on the distances between prototypes in the embedding space. This scale may conflict with the second term of Ldata which encourages distances between embeddings and unrelated prototypes to be as large possible. Therefore, lower distortion may also cause lower precision. To remove this imposed scaling, we introduce scale-independent formulation of the distortion:
disto stale-frre (m, D) min disto(s σ. D),
sER,
where the scaled prototypes, whose coordinates in Ω are multiplied by scalar factor As shown in the appendix, disto scale free can be efficiently computed algorithmically.
Distortion-Basse Penalization We propose incorporate the error qualification D into the proto- types' relative arrangement by encouraging a low scale-free distortion between and D. To this end, we define the following smooth surrogate of disto scale-free,
sd( mk, mi) - [K:: 1
Laisto (T)
min SER,
Σ Agt
(6)
K(K
D(k,
K.IEK?
Laisto can be computed in elosed form function of and thus be directly used regularizer.
4

---

Under review as a conference paper at ICLR 2021
3.3 END-TO-END TRAINING
We combine Lasaa and disto in a single loss L. Lama allows jointly learn the embedding function and the class prototypes while Lassto enforces metric-consistent prototype arrangement:
[(S, σ) = Lanu (J, + XLdisto (T)
with À E R_, an hyper-parameter setting the strength the regularization.
3.4 CHOOSING A METRIC SPACE
Prototypical networks operating on = Ih typically use the squared Euclidean norm the distance function, motivated by its quality Bregman divergence (Snell et al. 2017). However, observe that defining with the Euclidean norm yields significantly better results. The non-differentiabiliy be handled by composing with Huber-like (Huber al. 1973 Charbonnier et al. 1997) function HII-I), with H defined as:
H(x) 8(V|<=//// +1-1)
(8)
and 8 E IR+, (small) hyper-parameter: The resulting metric d is asymptotically equivalent to the Euclidean norm for large distances, and behaves like smooth squared Euclidean norm for small distances. In Section 4.5 investigate the effect of this change.
4 EXPERIMENTS
4.1 DATASETS AND BACKBONES
We evaluate our approach with different tasks on public datasets with fine-grained class hierarchies: image classification on CIFARI00 (Krizhevsky al. 2009) and iNaturalist- 19 (Van Horn al.) 2018). RGB-D image segmentation on NYUDv2 (Nathan Silberman & Fergus 2012), and image sequence classification on S2-Agri (Sainte Fare Garnot et al.J 2020p. We define the cost matrix of these class sets as the length of the shortest path between nodes in the tree-shape taxonomies, represented in the Appendix. As shown in Table [ these datasets cover different settings in terms of data characteristics, as well as tree structures.
Illustrative Example MNIST: In Figure and Figure ?? of the Appendix, we illustrate the difference in performance and embedding organization for different approaches. We use small 3-layer convolutional net trained on MNIST with random rotations (up to 40 degrees) and affine transformations (up to 1.3 scaling). For plotting convenience, we set the feature's 's dimension to 2.
Image Classification CIFAR100: CIFAR100 is composed of 50 000 training images and 10 000 test images of size 32 x 32, evenly distributed across 100 classes. We use a super-class system inspired by Krizhevsky et al. (0009) and form a 5-level hierarchical nomenclature of size: 2, 8, 20, and 100 classes. We use as backbone the established ResNet-18 18 (He et al. 2016) for this dataset.
NYUDv2: NYUDv2 is RGB-D image segmentation dataset com- posed of 449 pairs of RGB images of indoor scenes and their corresponding depth maps. We use the standard split of 795 training and 654 testing pairs. We combine the and 40 class nomenclatures of Gupta et al. (2013) and the 13 class system defined by Handa et al. (2016) to construct a 3-level hierarchy. We FuseNet (Hazirbas al. 2016) backbone for this dataset.
Semantic Segmentation
Image Sequence Classification on S2-Agri: S2-Agri comprises 189 971 sequences of multi-
spectral satellite images of agricultural parcels. We define 4-level hierarchy of size 4, 12, 19, and 44 classes to organize the crop types. This class hierarchy is critical to monitoring agencies, as is related to the level of subsidy allocated farmers. We use the PSE+TAE architecture (Sainte) Fare Garnot et al. 2020) as the backbone, and follow their 5-fold cross-validation scheme for training.
Fine-Grained Image Classification on INaturalist-19 (iNat-19) iNat-19 (Van Horn al. 2018)
a fine- greeegraieedd image classification dataset comprised of 265 213 images of living organisms. The images are labeled by experts and users of the iNaturalist app, using fine-grained taxonomy. iNat-
5

---

Under review as a conference paper at ICLR 2021
contains 1 010 different classes, organized into hierarchy of levels with respective width 3, 4, 9. 34, 57, 72, and 1 1000 We use ResNet-18 pre-trained on ImageNet as backbone, We sample 75% of available images for training, while the rest is evenly split into a validation and test set.
Table Data composition and nomenclature of the four studied datasets. IR stands for the Imbalance Ratio (largest over smallest class count), nodes and leaves denote respectively the total number of classes and leaf-classes in the tree-shape hierarchy, ABF stands for the Average Branching Factor, and (D) stands for the average pairwise distance.
Data
Hierarchical Tree
Dataset
Volume (Gb)
IR
Depth
Nodes (leaves)
ABF
(D)
NYUDv2
2.8
93
57 (40)
5.0
4.3
S2-Agri
28.2
617
83 (45)
5.8
6.5
CIFARI00
0.2
1
134 (100)
3.8
7.0
iNat-19
82.0
31
1189 (1010)
6.6
11.0
4.2 HYPER- PEEEPAAARRTTTEIZATTOOO
The embedding space S is chosen as 12512 for iNat-19 and R64 for all other datasets. We define d as the Euclidean norm (see 4.5 for discussion on this choice). We evaluate approach souided-prote with À = 1 in for all datasets. We use the same training schedules and learning rates as the backhone networks in their respective papers. In particular, the class imbalance of S2-Agri is handled with a focal loss (Lin et al. 2017).
4.3 COMPETING METHODS
In the paper where they are introduced, all backbone networks presented in Section 4.1 use linear mapping between the samples embedding and the class scores, as well as the cross-entropy loss. The resulting performance of these networks serves baseline to estimate the gains in Average Hierarchical Cost (AHC) and Error Rate (ER) provided by our approach and other competing methods re-implemented.
Hierarchical Cross-Entrope (HXE) Bertinetto et al. (2020) model the class structure with hierarchical loss composed of the sum of the cross-entropies at each level of the class hierarchy. As suggested, parameter taken as 0.1 defines exponentially decaying weights for higher levels.
Soft Labels (Soft-labels) Bertinetto et al. (020) propose as second baseline in which the the one- hot target vectors are replaced by soft target vectors in the cross-entropy loss. These target vectors are defined as the softmin the costs between all labels and the true label, with a temperature
1/8 chosen 0.1, recommended in Bertinetto et al. 2020))
Earth Mover Distance regularization (XE+EMD); Hou et al. (2016) propose to account for the relationships between classes with regularization based on the squared earth mover distance. We use D as ground distance matrix between the probabilistic prediction and the true class y:
K p(an kl
LEMD(P.V) N
TEN k=1
This regularizer is added along the cross-entropy with weight of 0.5 and offset μ of 3. Hierarchical Inference (YOLO): Reemoon & Farhadi (2017) propose to model the hierarchical structure between classes into tree-shaped graphical model. First, the conditional probability that sample belongs to a class given its parent class obtained with softmax restricted to class' co-hyponyms (f.e. siblings). Then, the posterior probability of a leaf class is given by the product of the conditional probability of its ancestors. The loss is defined as the cross-entropy of the resulting probability of the leaf-classes.
(YOLO):Redmon
Hyperspherical Prototypes (Hyperspherical-prool is closer to prototypes on the hypersphere using a rank-based loss (see Section[..5) combined with a prototype- separating term. They then the squared cosine distance between the image embeddings and
The method proposed by Mettes et al. (019) it relies embedding class prototypes. They advocate to first position

---

Under review as a conference paper at ICLR 2021
CIFAR 100
INaturalist-19
Cross-Fntropy
Soft-abelle
Hypeuss serical-gr
Deeg-NCM
Learnt-proto
Gunded-proto
24.0
23.5
41
2.2
Brror Rate (CR)
Average Ilierarctical Cost (ALIC)
Error Rate (DR3
Average Ileerrccaaa Cost (AII
NYUDV2
S2-Agri
Cross-Entropy
[XX
Soft labels
XE+ EMD
YOLD
Hypens peticalgrrtt
Deep-NCM
Lesrnt-proto
Guided
Error Rattn (ER)
Average Hierarchical Cost (AHC)
Broor (ER)
Average Hieewrebiooa (AH
Figure 2: Error Rate (ER) in % and Average Hierarchical Cost (AHC) on four datasets for our proposed method (framed), the Cross-Entropy baseline (in bold), and the competing approaches. The best performances on each dataset are plotted in green. Our guided prototype approach improves both the ER and AHC across the four datasets compared the baseline. The metrics are computed with the median over 5 runs for CIFARI00, the average over 5 cross-validation folds for S2-Agri, and a single run for NYUDv2 and iNat-19. The numeric values are given in the Appendix, (* not evaluated on S2-Agri).
prototypes to train the embedding network. Note that in our re-implemennation, we used the finite metric defined by D instead of Vord2Vec (Mikolov al. 2013) embeddings position prototypes. Lastly, we do not evaluate on S2-Agri as the integration of the focal loss is non-trivial.
Deep Mean Classifiers (Deep-NCM): Guerriero al. (2018) present another prototype-bee approach. Here, the prototypes are the cumulative mean of the embeddings of the classes' samples, updated each iteration. The embedding network supervised with Ldata with d defined as the squared Fuclidean norm.
Prototype Learning(Leamtt-rronn Learnt prototypes without regularization, i.e. X 0
4.4 ANALYSIS
Overall Performance: As displayed in Figure 2 the benefits provided by our approach can be metric-guided prototype models improve the AHC by 3% on NYUDv2 and S2-Agri, and up to 9% and 14% for CIFAR100, and iNat- 19 respectively. The hierarchical inference scheme of Redmon & Farhadi (017)) performs on par or better than our methods for NYUDv2 and S2-Agri, while Soft-labels perform well on CIFARIOK 100 and NYUDv2. Yet, only the metric guided prototype approach brings a consistent reduction of the hierarchical cost across the four datasets. This suggest that arranging the embedding space consistently with the cost metric is robust way of reducing model's hierarchical error cost.
appreciated all datasets. Compared to cross-entropy,
While being initially designed to reduce the AHC, our methods also improve the ER by 3 to 4% across all datasets compared to the cross-entropy baseline. This indicates that cost matrices derived from the class hierarchy can indeed help neural networks to learn better representations.
Prototype Learning: We observe that learnt prototype approach Learnt-prott consitently outper- forms the Decp-NCM method. This suggest that defining prototypes as the centroids of their class representations might actually be disadvantageous. As illustrated on Figure Tc the positions of the
7

---

Under review as a conference paper at ICLR 2021
Table Influence of the choice of scaling in disto metric guiding regularizer, and distance function d on the performance of Guided-proto on the four datasets. For d, we compare the performance of the Euclidean norm, the ggeudo-Huberizer Euclidean norm, and the square Euclidean norm.
CIFAR 100
NYUDv2
S2-Agri
iNat-19
ER
AHC
ER
AHC
ER
AHC
ER
AHC
Guided-proto
23.6
1.052
32.5
1.440
18.9
0.685
38.9
1.721
Fixed-scale
+0.1
+0.003
0.0
0.000
+0.2
+0.001
+0.9
0.000
Rank-based guiding
-0.3
+0.004
+0.2
+0.005
+0.2
+0.006
+0.4
-0.003
Fixed-proto
+1.1
+0.031
+0.6
+0.013
+0.5
+0.025
+5.0
+0,427
Pseudo-Huber
+0.1
+0.015
-0.3
-0.017
+0.4
+0.016
+0.2
+0.003
Squared Norm
+1.0
+0.118
0.0
+0.005
+0.6
+0.022
+2.2
+0.233
embeddings follow a Voronoi partition (Fortune 1992) with respect to the learnt prototypes rather than the prototypes being at the centroid of representations. A surprising observation for us is that Learnt-proto consistently outperforms the cross-entropy, both terms of AHC and ER.
Computational Efficiency: Computing distances to prototypes is comparable in terms of com- putation time to computing a linear mapping. Consequently, both training and inference time are equivalent when using the cross-entropy guided-proio-disto, which is 2% faster.
4.5 ABLATION STUDY
Fixed-Scale Distortion: Table 2 we compare the performance of our scale-free regularizer to an 1. Across datasets, this results in
alternative version of Edisto in which the scale remains fixed an increased error rate, which we attribute to prototype distances being fixed by Ldisto The benefit of our scale-free regularizer is especially valuable for the complex class structure of iNat-19, improving the overall classification accuracy by point compared to the fixed-scale version.
Rank-based Regularization: Mettes et al.] (2019) use a rank-based loss (Burges et al.) 2005) to encourage prototype mappings whose pairwise distance follows the same order qualification of errors D. We argue that our formulation of Laisto provides a stronger supervision than only considering the order of distances, and allows the prototypes to find arrangement the embedding space. Table 2 we observe that replacing our distortion-base basedd loss by rank-based one results in slight decrease of overall performance.
external
profitable
Guided vs. fixed prototypes: : Our experiments confirm the benefit of jointly learning the proto- types and the embedding network instead of learning the prototypes first Hyperspherical-pron We also evaluate the performance of Fixed-proto in Table 2 for which we first fix the prototypes with Laisto and then train the network with Ldata- While this approach reduces the AHC compared the cross-entropy baseline in some cases, its ER is consistently higher than for guided methods. This suggests that insights from the data distribution can conversely benefit the positioning prototypes, and that they should be learned conjointly.
Choice of distance: In Table 2 we report the performance of the Guided-prote proto model on the four datasets when replacing the Euclidean norm alternately with the squared Euclidean norm and an Huberized Euclidean distance (with d = 0.1 in (8). Across our experiments, the squared-norm based model yields a worse performance. This is a notable result as it is the distance commonly used in most prototypical networks (Snell et al. 2017 Guerriero et al., 2018). The Huberized norm performs worse than the Euclidean distance all dataset, with the exception NYUDv2.
Further Ablation Study: In the Appendix we present an extended ablation study, notably show- casing the resilience of our approach to wide-range of hyper- sypeeepprrmettrs value.
8

---

Under review as a conference paper at ICLR 2021
5 CONCLUSION
We introduced a new regularizer to incorporate the class hierarchy during the training of a prototypical network. We showed that our methods consistently decreased the average hierarchical cost of three different backbone networks on different tasks and four datasets. Furthermore, our approach can reduce the rate of errors as well. In contrast to most recent literature on hierarchical classification, we showed that this joint training is beneficial compared to the staged strategy first positioning the prototypes and then training a feature extracting network. A PyTorch implementation of our framework as well as an illustrative notebook are available at https: //githu www.coo//mmeeeeeem com/nqp-a mmp-a (repository anonymized for review).
This work, along with other recent investigations (Bertinetto et al. 2020), highlights the interest of modeling hierarchical class structures in modern deep networks, Beyond the decrease in prediction costs, such hierarchies can lead to an improved overall performance. This calls for further investigation into the mechanisms by which semantic class structures can benefit the learning of expressive representations. In further research, we plan to investigate whether or claims hold in hyperbolic embedding spaces, which are known to be well-suited for embedding hierarchies.
REFERENCES
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey, IEEE Access, 2018.
Shahanaz Ayub and JP Saini. ECG classification and abnormality detection using cascade forward neural network. International Journal of Engineering, Science and Technology, 2011.
Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A Lord. Making better mistakes: Leveraging class hierarchies with deep networks. In CVPR, 2020.
Jean Bourgain. On Lipschitz embedding of finite metric spaces in Hilbert space. Israel Journal of Mathematics, 1985.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In ICML, 2005.
Pierre Charbonnier, Laure Blanc-Féraud, Gilles Aubert, and Michel Barlaud. Deterministic edge- preserving regularization in computed imaging. IEEE Transactions on Image Processing, 1997.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: deep learning for interpretable image recognition. In NeurlPS, 2019.
Christopher De Sa, Albert Gu. Christopher Ré, and Frederic Sala. Representation tradeoffs for hyperbolic embeddings. In Proceedings of Machine Learning Research, 2018.
Jia Deng, Alexander Berg, Kai Li, and Li Fei-Fei. What does classifying more than 10,000 image categories tell us? In ECCV, 2010.
Nanqing Dong and Eric Xing. Few-shot semantic segmentation with prototype learning. In BMVC, 2018.
Steven Fortune. Voronoi diagrams and Delaunay triangulations. In Computing in Euclidean Geometry. World Scientifie, 1992.
Andrew Gelman, John B Carlin, Hal S Stem, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian data analysis. CRC press, 2013.
Samantha Guerriero, Barbara Caputo, and Thomas Mensink. DeepNCM: deep nearest class mean classifiers. In ICLR, Worskhop, 2018.
Saurabh Gupta, Pablo Arbelaez, and Jitendra Malik. Perceptual organization and recognition of indoor scenes from RGB-D images. In CVPR, 2013.
Handa, Patraucean, Badrinarayanan, S Stent, and R Cipolla. Uunderstanding real world indoor scenes with synthetic data. In CVPR, 2016.

---

Under review as a conference paper at ICLR 2021
Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers. Fusenet: Incorporating depth into semantic segmentation via fusion-bass CNN architecture. In ACCV, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Le Hou, Chen-Ping Yu, and Dimitris Samaras. Squared earth mover's distance-based loss for training deep neural networks. NeurIPS workshop on learning on distributions, fustributions,uunciions. gruphs and groups, 2016.
Peter Huber et al. Robust regression: asymptotics, conjectures and monte carlo. The Annals of Statistics, 1973.
Piotr Indyk, Jiří Matoušek, and Anastasios Sidiropoulos. Low-distortion embeddings of finite metric spaces. In Handbook of discrete and compulational geomelry. Chapman and Hall/CRC, 2017.
Saumya Jetley, Bernardino Comera-Paredess Sadeep Jayasumana, and Philip Torr. Prototypical priors: From improving classification to zero-shot learning. In BMVC, 2015.
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In CVPR, 2020.
Teuvo Kohonen. Learning vector quantization. In Self-organizing maps, pp. 175-189. Springer, 1995.
Aris Kosmopoulos, Ioannis Partalas, Eric Gaussier, Georgios Paliouras, and Ion Androutsopoulos. Evaluation measures for hierarchical classification: unified view and novel approaches. Data Mining and Knowledge Discovery, 2015.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong- Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. Hyperbolic visual embedding learning for zero-shot recognition. In CVPR, 2020.
Teng Long, Pascal Mettes, Heng Tao Shen, and Cees GM Snoek. Searching for actions the hyperbole. In CVPR, 2020.
Pascal Mettes, Elise van der Pol, and Cees Snoek. Hyperspherical prototype networks. In NeurlPS, 2019.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. In ICLR Workshop. 2013.
George A Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J Miller. Introduction to wordnet: on-line lexical database. International Journal of Lexicography, 1990.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from RGBD images. ECCV, 2012.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. In NewrIPS, 2017.
Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. In CVPR, 2017.
Deboleena Roy, Priyadarshini Panda, and Kaushik Roy. Tree-CNN: hierarchical deep convolutional neural network for incremental learning. Neural Networks, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
10

---

Under review as a conference paper at ICLR 2021
Vivien Sainte Fare Garnot, Loic Landrieu, Sebastien Giordano, and Nesrine Chehata. Satellite image time series classification with pixel-set encoders and temporal self-attention. CVPR, 2020.
Ruslan Salakhutdinov, Joshua B Tenenbaum, and Antonio Torralba. Learning with hicrarchical-deep models. IEEE Transactions Pattern Analysis and Machine Intelligence, 2012.
Atsushi Sato and Keiji Yamada. Generalized learning vector quantization. NeurIPS, 1995.
Carlos Silla and Alex Freitas. survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 2011.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, 2017.
Nitish Srivastava and Russ R Salakhutdinov. Discriminative transfer learning with tree-based priors. In NeurlPS, 2013.
Robert
Tibshirani, Trevor Hastie, Balasubramanian Narasimhan, and Gilbert Chu. Diagnosis of multiple cancer types by shrunken centroids of gene expression. Proceedings of the National Academy of Sciences, 2002.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In CVPR, 2018.
Zhicheng Yan. Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition. In ICCV, 2015.
Hong-Ming Yang, Xu- Yao Zhang, Fei Yin, and Cheng- Lin Liu. Robust classification with convolu- tional prototype learning. In CVPR, 2018.


Reviews:
[{"review_id": "tgrjUQogTT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to integrate the cost of errors into a classification algorithm. The proposed loss function has two terms: one for pushing samples towards their corresponding class prototype (L_{data}), and another for forcing/guiding the pairwise distance between class prototypes to follow a predefined values (L_{distortion}).  \n\nMarginally below acceptance. \n\nI like the idea of using a priory knowledge about similarities between classes to regularize the learning process.  But,\n\n1) Out of 4 datasets, the proposed method works better than others on only two datasets for which in one case the addition of proposed term L_{distortion} has helped, but it actually degraded the results in the other dataset. So, the results are inconclusive.\n\n2) pp.3 the assumption of having a symmetric misclassification cost seems very limiting. Effectively, this assumption implies the type 1 and type 2 errors must have equal cost. The proposed method is about treating different errors differently and yet it assumes type 1 and type 2 errors are equal. Referring to the example, provided in the introduction (paragraph 1), cost of misclassifying a crossing pedestrian as a street light should be way more than the cost of classifying a street light as a crossing pedestrian. \nAnother assumption of the proposed method is triangle inequality i.e. D[k, l] + D[l,m] >= D[k,m] but this assumption is not justified except saying the cases where the assumptions do not hold, are out of scope of this paper. \n\npp.1 says, \"this error discrepancy is not taken into account ... in the evaluation metrics\". This statement is incorrect. At least in a simple binary classification task, we have type 1 and type 2 errors. We have recall and precision. They are all about putting different weights to different errors. \n\npp.2 Caption of Figure 1: tree is created based on authors perceived visual similarity. The tree should reflect the cost of class confusions. From paragraph 1 in page 1, it is inferred that the cost of a class confusion is determined by the actual consequences e.g. cost of confusing a streetlamp with a crossing pedestrian. Such cost is different than how similar the two classes are.  The notion of \"cost\" and \"similarity\" are not the same but used interchangeably in the paper. \n\npp.7 the soft-label method is the top runner in cifar-100. Its performance on the other 3 datasets may be improved if the temperature parameter is adjusted to compensate the various number of classes. Of course, same can be said about tuning hyperparameters of the proposed method.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mebuKQC2Bno", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In order to organize my review, I use the NeurIPS2020 template with slight adaptions to fit the ICLR requirements. \n\n### Summary and contributions: Briefly summarize the paper and its contributions.\n\nThe paper proposes a method to jointly learn a feature-extractor (a neural network) and a set of prototypes. Learning is performed in an end-to-end setting and the prototype distribution is regularized to follow a given distribution. For this purpose, the authors propose a cost matrix that assesses the severity of errors between classes. This matrix is organized in such a way that it yields a finite metric. The distribution of prototypes is regularized such that the distances between prototypes reflect this finite metric. In several experiments, the authors evaluate the advantages of such a setting with respect to an average cost measure.\n\n### Strengths: Describe the strengths of the work.\n\nThe work is well motivated and how the entire network setup is realized is easy to understand and straightforward.\n\n\n### Weaknesses: Explain the limitations of this work along the same axes as above.\n\nA major weakness is that the authors have not cited all the relevant previous work about prototypical networks (see my comments below). Taking all the relevant literature into account, the contribution of the paper is small. Moreover, the proposed method and observed effects raise several questions that are not sufficiently addressed. For example, the authors observe that the learning with the squared Euclidean distance does not perform as good as with the Euclidean distance, but they have not investigated why this is the case (see my further comments below).\n\n\n### Correctness: Are the claims and method correct? Is the empirical methodology correct?\n\nThe current version of the paper has some minor mathematical mistakes. However, they can be fixed and I consider them to be of little relevance for the correctness of the method. I will list them below.\nOne serious correctness point is that the authors claim (see page 4 top and page 2 contributions first bullet point) that they are inventing a principle to learn the prototypes jointly in an end-to-end setting. Considering the state of the art, this claim is incorrect.\n\n\n### Clarity: Is the paper well written?\n\nThe paper is well written and the mathematical formulations have a good quality.\n\n\n### Relation to prior work: Is it clearly discussed how this work differs from previous contributions?\n\nThe paper has not discussed all the relevant prior work. The idea to learn the prototypes jointly is not new:\n* Chen, Chaofan and Li, Oscar and Tao, Daniel and Barnett, Alina and Su, Jonathan and Rudin, Cynthia. “This Looks Like That: Deep Learning for Interpretable Image Recognition.” NeurIPS (2019).\n* Hong-Ming Yang and Xu-Yao Zhang and Fei Yin and Cheng-Lin Liu. “Robust Classification with Convolutional Prototype Learning.” CVPR (2018)\n* Villmann, Thomas and Biehl, Michael and Villmann, Andrea and Saralajew, Sascha. “Fusion of deep learning architectures, multilayer feedforward networks and Learning Vector Quantizers for deep classification learning.” WSOM+ (2017)\n* to name just a few…\n\nNote that prototype-based classifiers and especially nearest prototype classifiers have been invented long before Tibshirani et al. (2002). For example, see Kohonen’s Learning Vector Quantization (LVQ) algorithms, including the original heuristic versions and the end-to-end trainable versions like Generalized LVQ (by Sato and Yamada).\nMoreover, the incorporation of a cost matrix into the learning processes for prototypes was also done before. For example, Kaden, Marika, Wieland Hermann, and Thomas Villmann. \"Attention Based Classification Learning in GLVQ and Asymmetric Misclassification Assessment.\" Advances in Self-Organizing Maps and Learning Vector Quantization. Springer, Cham, 2014. 77-87. Of course, in this work, the authors have not used the cost matrix to penalize the prototype positions, but it was included to account for different severity between class confusions.\n\n### Reproducibility: Are there enough details to reproduce the major results of this work?\n\nThe paper provides a lot of details about the experimental setup, especially in the appendix. However, some information is not provided, for example, the initialization strategy of the architecture.\n\n\n### Additional feedback, comments, suggestions for improvement, questions for the authors, and a **recommendation (accept or reject)**.\n\nThe authors should be precise about the assumptions of $D$. In the current version, the definition of $D$ is not correct to talk about a finite metric space (see page 3 Method and page 1). The missing property is the identity of indiscernibles.\n\nThe authors provide a good motivation for why the incorporation of a cost matrix is useful. However, could the authors explain why they then assume that the matrix $D$ is symmetric? Because, according to the motivation, the severity of misclassifying a *crossing pedestrian* as a *street lamp* is different from misclassifying a *street lamp* as a *crossing pedestrian*.\n\nThe first sentence in Section 3.1 is difficult to read due to “sample $n$”, consider writing $x_n$.\n\nWhat is the difference of Equation (3) to a cross-entropy loss with a one-hot encoding for the correct class?\n\nThe authors observed that the squared Euclidean distance does not perform as well as the Euclidean distance. I assume that this happens because the squared Euclidean distance does not fulfill the triangle inequality that is, on the other hand, assumed for $D$. Could the authors comment on this? \n\nI don’t see that the non-differentiability of the square root at zero causes any issues in the training. If the argument becomes zero, the distance is zero and, therefore, the prototype is equal to the feature vector of the sample. Given the given interpretation of the attraction and repulsion forces on the data and the prototypes (see page 3 below), in which direction should they be pushed or repelled if they have a perfect match? There exists no update direction that could improve the model. Therefore, it is feasible to just stop the gradient or to ignore such samples.\n\nHave the authors observed training instabilities due to large values of the distance $d$?\n\nIf I consider Figure 1, I wonder why the learned prototypes tend to be outside the class centers (e.g., see figure c). Can the authors elaborate on that?\n\nThe statement on page 1 bottom about the cross-entropy is incorrect. Whether the cross entropy “singles out the prediction” depends on the true probability vector.\n\nThe experimental results are not convincing as they don’t show a significant benefit compared with state-of-the-art methods. \n\nI have problems with the statement following Equation (4). Could the authors explain in more detail why a prototype arrangement regarding $D$ and low distortion is advantageous in terms of AC?\n\nThe google colab link provided in the supplementary was not accessible. \n\nConsidering all the above points, I vote for rejection, since the paper does not have the right maturity level to be published at ICLR and the scientific contribution is small.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OFHClMFdS-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths and weaknesses:\n+\nNice idea\nConsistent improvements over cross entropy for hierarchical class structures\nImprovements w.r.t other competitors (though not consistent) \nGood ablation study\n-\nThe improvements are small\nThe novelty is not very significant\n \n\nMore comments:\n\n\nFigure 1:\n-\tIt is not clear what distortion is at this stage\n-\tIt is not clear what perturbed MNist is, and respectively: why is the error of a 3-layer CNN so high (12-16% error are reported)? CNNs with 2-3 layers can solve MNist with accuracy higher than 99.5%?\n-\tThis figure cannot be presented on page 2 without proper definitions. It should be either presented on page 5, where the experiment is defined, or better explained  \nPage 4: It is said that s can be computed efficiently and this is shown in the appendix, but the version I have do not have an appendix\nPage 6: the XE+EMD method is not present in a comprehensible manner. 1) p_k symbols are used without definition (tough I think I these are the network predictions p(\\hat{y}=k|I)  2) the relation of the formula presented to the known EMD is not clear. The latter is a problem solved as linear programming or similar, and not a closed form formula  3) it is not clear what the role of \\mu is and why can be set to 3 irrespective of the scale of metric D\npage 7: \nThe experiments show small, but consistent improvements of the suggested method over standard cross entropy, and improvements versus most competitors in most cases\n\nI have read the reviews of others and the author's response. My main impression of the work remains as it was: that it is  nice idea with small but significant empirical success. However, my acquaintance with the previous literature in this subject is partial compared to the acquaintance of other reviewers, so It may well be possible that they are in a better position than me to see the incremental nature of the proposed work. I therefore reduce the rating a bit, to become closer to the consensus.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2wPNS7FbyRQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a prototype-based approach to learning with prior hierarchical knowledge of categories. The main idea behind their approach is to perform metric-guided penalization on top of a prototype-based cross-entropy loss. The penalization is scale-free, yet the loss can be computed in closed form, allowing for a hierarchical guidance to prototype locations.\n\nStrengths:\n\nThe starting motivation and direction of the paper are clear. If hierarchical knowledge is available, it is intuitive to exploit such information. Care has been taken to make the research open and reproducible. Along with the submission, an anonymized repo and notebook with details are provided. Both help with understanding the approach and its workings in practice.\n\nThe scale-free component of the proposed regularization is interesting, most notably the fact that the corresponding loss can be computed in closed form. By making the hierarchical loss scale free, a clash with the cross-entropy loss - which tries to push prototypes as far away as possible from each other - is avoided.\n\nWeaknesses:\n\nThe main weakness of the paper is that its innovation is limited, especially in light of missed recent works on prototype-based approaches using hierarchies in non-Euclidean spaces. The innovation of the paper is in Section 3.2, which means that the technical novelty of the paper is limited to a loss with hierarchical loss. More pressingly, relevant recent literature is missed on using hierarchical knowledge in deep networks. What the authors consider future work (\"Among other promising avenues for further research, we plan to investigate the use of non-Euclidean embedding spaces, which are known to be well-suited for embedding hierarchies [...]\", page 8), has readily been proposed, see e.g.:\n\n[1] Liu, Shaoteng, et al. \"Hyperbolic Visual Embedding Learning for Zero-Shot Recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[2] Long, Teng, et al. \"Searching for Actions on the Hyperbole.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[3] Khrulkov, Valentin, et al. \"Hyperbolic image embeddings.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. (note: this paper does not explicitly utilize hierarchical knowledge)\n\nAll three above papers, from the same conference as the often cited work by Bertinetto et al. (2020) in the submission, investigate prototype-based approaches in non-Euclidean spaces. [1,2] explicitly use hierarchical knowledge to steer the prototypes. Hyperbolic spaces are known to be a better fit for hierarchies, see e.g. [4].\n\n[4] Nickel, Maximillian, and Douwe Kiela. \"Poincaré embeddings for learning hierarchical representations.\" Advances in neural information processing systems. 2017.\n\nCompared to recent works on prototypes in hyperbolic spaces with hierarchical knowledge, the paper provides limited novelty. As the authors state that moving the prototypes to non-Euclidean spaces is future work for this submission, the other recent works make the paper come across as somewhat outdated.\n\nEmpirically, the paper provides a comparison to multiple hierarchical alternatives on four datasets. The use of four datasets is appreciated, as it provides a clear picture of the strengths and limitations of the approaches. Besides missing comparisons to e.g. the supervised setup of [2], Figures 2 and Table 2 show that the difference with alternatives is minimal. E.g. the guided prototypes seem to only outperform soft labels on iNaturalist-19 in terms of Average Cost. Table 2 paints a picture that fixing the scale or using a coarser ranking-based approach hardly affect the Average Cost on all datasets and only provide some improvements on one of the four datasets.\n\nConclusion:\n\nThe paper addresses an open problem in deep learning literature in a clear manner. The open science setup of the submission is appreciated and the scale-free regularizer is interesting. In light of recent advances in hyperbolic prototype-based approaches with hierarchical knowledge, the innovation of this work is minimal. The empirical benefits of the proposed approach are also not always convincing. For the rebuttal, it would be interesting to discuss and compare the submission to hyperbolic hierarchical prototype approaches.\n\nOpinion post rebuttals:\n\nAfter reading the rebuttal and the other reviews, I remain of the opinion that more work is needed before warranting acceptance. This paper indeed learns prototypes on the fly in contrast to e.g. Mettes et al. 2019. That method was however not designed for hierarchical knowledge, while several recent CVPR papers were. Since the novelty over these papers is limited and direct comparisons are lacking, more research is needed.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to ICLR.

This paper had somewhat dissenting reviews, but three of four reviewers felt negatively about the paper.  On the positive side, the reviewers noted good motivation for the problem, a good ablation study and, in some cases, good performance over standard cross entropy.  On the negative side, the reviewers noted limited novelty, missing discussion or comparison to prior work, and in some cases only marginal improvement over existing methods.

The positive reviewer still remained positive after discussion, but noted that their confidence was low on the paper and that they would defer to others' opinions.  The other reviewers still had several concerns after the discussion phase.  Ultimately, it seems that the paper could use some additional work before it is ready for publication.  I would strongly encourage the authors to keep the reviewer comments in mind when preparing a future version of the manuscript.
Decision: Reject