Connecting Sphere Manifolds Hierarchically for Regularization
Authors: Damien Scieur
Youngsung Kim
Venue:  (2021)
Keywords: Hierarchy
Manifold
Classification

Abstract: This paper considers classification problems with hierarchically organized classes. We force the classifier (hyperplane) of each class to belong to a sphere manifold, whose center is the classifier of its super-class. Then, individual sphere manifolds are connected based on their hierarchical relations. Our technique replaces the last layer of a neural network by combining a spherical fully-connected layer with a hierarchical layer. This regularization is shown to improve the performance of widely used deep neural network architectures (ResNet and DenseNet) on publicly available datasets (CIFAR100, CUB200, Stanford dogs, Stanford cars, and Tiny-ImageNet).

Under review as a conference paper at ICLR 2021
CONNECTING SPHERE MANIFOLDS HIERARCHICALLY FOR REGULARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper considers classification problems with hierarchically organized classes. We force the classifier (hyperplane) of each class to belong to a sphere manifold, whose center is the classifier of its super-class. Then, individual sphere manifolds are connected based on their hierarchical relations. Our technique replaces the last layer of neural network by combining spherical fully-connected layer with a hierarchical layer. This regularization is shown to improve the performance of widely used deep neural network architectures (ResNet and DenseNet) on publicly available datasets (CIFARI00, CUB200, Stanford dogs, Stanford cars, and ImageNe))
INTRODUCTION
Applying inductive biases or prior knowledge to inference models is a popular strategy improve their generalization performance (Battaglia et al., 2018). For example, a hierarchical structure is found based on the similarity or shared characteristics between samples and thus becomes a basic criterion to categorize particular objects. The known hierarchical structures provided by the datasets (e.g., ImageNet (Deng et al., 2009) classified based on the WordNet graph; CIFARTO0 (Krizhevsky, 2009) in ten different groups) can help the network identify the similarity between the given samples.
In classification tasks, the final layer of neural networks maps embedding vectors to a discrete target space. However, there is no mechanism forcing similar categories to be distributed close to each other in the embedding. Instead, we may observe classes to be uniformly distributed after training, as this simplifies the separation by the last fully-connected layer. This behavior is a consequence of seeing the label structure as 'flat' i.e., when we omit to consider the hierarchical relationships between classes (Bilal et al., 2017).
To alleviate this problem, in this study, we force similar classes to be closer in the embedding by forcing their hyperplanes to follow a given hierarchy. One way realize that by making children nodes dependent on parent nodes and constraining their distance through a regularization term. However, the norm itself does not give relevant information on the closeness between classifiers. Indeed, two classifiers are close if they classify two similar points in the same class. This means similar classifiers have to indicate similar direction. Therefore, we have focus on the angle between classifiers, which can be achieved through spherical constraints.
Contributions. In this paper, we propose simple strategy to incorporate hierarchical information in deep neural network architectures with minimal changes to the training procedure, by modifying only the last layer. Given hierarchical structure in the labels under the form of a tree, we explicitly force the classifiers of classes to belong to sphere, whose center is the classifier of their super-class, recursively until we reach the root (see Figure 2). We introduce the spherical fully- fully-commccted layer and the hierarchically connected layer, whose combination implements technique. Finally, investigate the impact of Riemannian optimization instcad of simple norm normalization.
By its nature, the proposed technique is quite versatile because the modifications only affect the structure of last fully-connected layer of the neural network. Thus, it can be combined with many other strategies (like spherical CNN from Xie et al. (2017), other deep neural network architectures).
Related works. Hierarchical structures are well-studied, and their propertics can be effectively learned using manifold embedding. The design of the optimal embedding to learn the latent hierarchy

---

Under review as a conference paper at ICLR 2021
is a complex task, and was extensively studied in the past decade. For example, Werd2 (Mikolov et al., 2013b;a) and Poincaré embedding (Nickel & Kiela, 2017) showed a remarkable performance in hierarchical representation learning. (Du et al., 2018) forced the representation of sub-classes to "orbit" around the representation of their super-class to find similarity based embedding. Recently, using elliptical manifold embedding (Batmangeelich et al., 2016), hyperbolic manifolds (Nickel & Kiela, 2017; De Sa et al., 2018; Tifrea et al., 2018), and combination of the two (Gu et al., 2019; Bachmann et al., 2019), shown that the latent structure of many data was -on-Euclidean (Zhu et al., 2016; Bronstein et al., 2017; Skopek et al., 2019). (Xie et al., 2017) showed that spheres (with angular constraints) in the hidden layers also induce diversity, thus reducing over-fitting in latent space models.
Mixing hierarchical information and structured prediction not new, especially text analysis (Koller & Sahami, 1997; McCallum et al., 1998; Weigend et al., 1999; Wang et al., 1999; Dumais & Chen, 2000). Partial order structure the visual-semantic hierarchy exploited using a simple order pair with max-margin loss function in (Vendrov et al., 2016). The results of previous studies indicate that exploiting hierarchical information during training gives better and more resilient classifiers, in particular when the number of classes is large (Cai & Hofmann, 2004). For given hierarchy, it possible to design structured models incorporating this information to improve the efficiency of the classifier. For instance, for support vector machines (SVMs), the techniques reported in (Cai & Hofmann, 2004; 2007; Gopal et al., 2012; Sela et al., 201 use hierarchical regularization, forcing the classifier of super-class to be close the classifiers of its sub-classes. However, the intuition very different in this case, because SVMs do not learn the embedding.
In this study, we consider that the hierarchy of the class labels is known. Moreover, we do not change prior layers of the deep neural network, and only work on the last layer that directly contributed to build hyperplanes for a classification purpose. Our work is thus orthogonal to those works on embedding learning, but not incompatible.
Comparison with hyperbolic/Poinearggraaa networks. Hyperbolic network is recent technique that shows impressive results for hierarchical representation learning. Poincaré networks (Nickel & Kiela, 2017) were originally designed learn the latent hierarchy of data using low-dimension embedding. To alleviate their drawbacks due to a transductive property which cannot be used for unseen graph inference, hyperbolic neural networks equipped set aggregation operations have been proposed (Chami et al., 2019; Liu et al., 2019). These methods have been mostly focused on learning embedding using a hyperbolic activation function for hierarchical representation. Our technique is orthogonal to these works: First, we assume that the hierarchical structure is not learnt but already known. Second, our model focuses on generating individual hyperplanes of embedding vectors given by the network architecture. While spherical geometry has positive curvature, moreover, that of hyperbolic space has a constant negative curvature. However, our technique and hyperbolic networks are not mutually exclusive. Meanwhile focusing on spheres embedded in IRd in this study, is straightforward to consider spheres embedded in hyperbolic spaces.
2 HIERARCHICAL REGULARIZATION
2.1 DEFINITION AND NOTATIONS
We assume we have samples with hierarchically ordered classes. For instance, apple, banana, and orange are classes that may belong to the super-class "fruits." This represents hierarchical relationships with trees, as depicted in Figure 1.
We identify nodes in the graph through the path taken in the tree. To represent the leaf (high- lighted in blue in Figure 1), we use the notation n ((1,3,2)- This means it is the second child of the super-class n (1.3). and recursively, until we reach the root.
Figure I:
To reference the node at the bottom,
we use the notation n, with = {1, 3, 2}. We use curly brackets (} to write path, and angle brackets (.) for the concatenation paths.
2

---

Under review as a conference paper at ICLR 2021
More formally, we identify nodes 7p, where p is the path to the node. A path uniquely defines node where only one possible path exists. Using the concatenation, between the path p and its child i, new path P can be defined as follows,
p= (p. x)
We denote the set of all paths in the tree starting from the root, with cardinality [P]. Notice that [P] is also the number of nodes in the tree (i.e., number of classes and super-classes). We distinguish the set P from the set L, the set of paths associated to nodes whose label appears in the dataset. Although L may equal to P. this is not the case in our experiments. We show an example in Appendix A.
2.2 SIMILARITY BETWEEN OBJECTS AND THEIR REPRESENTATION
Let X be the network input (e.g. an image), and Φo be its representation, i.e., the features of X extracted by deep neural network parameterized by 6. We start with the following observation:
Given representation, super-class separators should be similar to separators for their sub-classes.
This assumption implies the following direct consequence.
All objects whose labels belong to the same super-class have a similar representation.
That is a natural property that we may expect from a good representation. For instance, two dogs from different breeds should share more common features than that of a dog shares with an apple. Therefore, the parameter of the classifiers that identify dog's breed should also be similar. Their difference lies the parameters associated to some specific features that differentiate breeds of dogs.
Although this is not necessarily satisfied with arbitrary hierarchical classification, observe this in many existing datasets. For instance, Caltech-UCS Birds 200 and Stanford dogs are datasets that classify, respectively, birds and dogs in term of their breeds. A possible example where this assumption may not be satisfied is a dataset whose super-classes are "labels whose first letter is ao.."
2.3 HIERARCHICAL REGULARIZATION
Starting from a simple observation in the previous section, we propose a regularization technique that forces the network to have similar representation for classes along path p. which implies having similar representation between similar objects. More formally, if we have an optimal classifier up for the super-class p and a classifier W(pp)) for the class (p, i), we expect that
|| W(p.() ||
is small.
(2)
this is satisfied, separators for objects the same super-class are also similar because [[ w(p)) [[ wp) (W(r.) wp)
|| -
[t
(ps))
small
small
However, the optimal classifier for an arbitrary representation Φe may not satisfy equation 2. The naive and direct way to ensure equation 2 is through hierarchical regularization, which forces classifiers in the same path to be close to each other.
2.4 HIERARCHICAL LAYER AND HIERARCHICALLY CONNECTED LAYER
In the previous section, we described the hierarchical regularization technique given hierarchical structure in the classes. In this section, show how to conveniently parametrize equation 2. We first express the classifier as a sum of vectors 5 defined recursively as follows:
W(.s) up + 66.,)) d0 0,
(4)
where { is the root. possible to consider 61) A which shifts separating hyper-planes. We do not consider this case in this paper. Given equation 4, have that (| is small in equation 2. Finally, suffices to penalize the norm of 8 (p..) during the optimization. Notice that, by construction, the number of 8's is equal to the number of nodes in the hierarchical tree.
3

---

Under review as a conference paper at ICLR 2021
Next, consider the output of CNNs for classification,
(5)
where 0 denotes the parameters of the hidden layers, W (.r,.... w/ denotes the last fully- connected layer, and w, denotes the separator for the class For simplicity, omit potential additional nonlinear functions, such as a softmax, on top of the prediction.
We have parametrized w; following the recursive formula in equation 4. To define the matrix formulation equation 4, we first introduce the Hierarchical layer И which plays an important role. This hierarchical layer can be identified to the adjacency matrix of the hierarchical graph.
Definition 1. (Hierarchical layer). Consider ordering over the sets P and L, i.e., for i = 1,.... (P]
and 1, ..... [4),
P (p....., . PIPI) and L (p....., Pj. pj..... Plc)}- . In other words, we associate to all nodes an index. Then, the hierarchical layer H is defined as H EBXIIC H. = if np 3 mpp otherwise.
where np. np, means R2, is parent of np..
We illustrate example of H in Appendix A. The next proposition shows that equation 5 can be written using simple matrix-mann matrix multiplication, involving the hierarchical layer.
Proposition 1. Consider representation (-(-), where (- E Rdd Let W be the matrix of separators
W
Wpp Wpeil: Pi EL
where the separators are parametrized as equation 4. Let A be defined as
R [Pl A 18p1 ..... Sppzy]
(8)
where and are defined in Section 2.1. Consider the hierarchical layer defined in Definition 1. Then, the matrix of separators W can be expressed as
W = AH.
We can see W = AH as combination of an augmented fully-connecte layer, combined with the hierarchical layer that selects the right columns of A, hence the term hierarchically connected layer. The & regularization of the 8 can be conducted by the parameter weight decay, which is widely used in training of neural networks. The hicrarchical layer affect the complexity of the back-propagation significantly, as AH is simple linear form.
fixed, while is learnable. This does not
The size of the last layer slightly increases, from [C] x d to [P] x d, where d the dimension the representation (-)-). For instance, in the of ImageNe,, the number of parameters of the last layer only increases by roughly 36% nevertheless, the increased number of parameters of the last layer is still usually negligible in comparison with the total number of parameters for classical network architectures.
HIERARCHICAL SPHERES
The hierarchical (E2) regularization introduced the previous section induces separated hyper-planes along a path to be close to each other. However, this approach has significant drawback.
We rewind equation 2, which models the similarity of two separators wy and W 4). The similarity between separators (individual hyper-planes) should indicate that they point roughly the same direction, i.e..
Up
W
(p.s) s)
is small.
(10)
[wpl
However, this property is not necessarily captured by equation 2. For instance, assume that Wp = -W(p, i). i.e., the separators point in opposite directions (and thus completely different). Then, equation 2 can be arbitrarily small in the function of W,,l| but not in equation 10:
[wp "(p,. 1 = 2/wwpl1
Wp
Wip.
(11)
[w
[Wp 1)
4

---

Under review as a conference paper at ICLR 2021
(1.3
(1.2)
8t
Riemannian gradient
P
(11)
Projected"
gradient
(L220
Figure 2: (Left) Example of hyper-planes Wp. formed through the sum of 6p. The hyper-plane (1.3.221) associated to the class (13222211) is in green, the construction with the d's in blue, and all intermediate in red. (Right) Riemannian versus "projected" gradient descent. Riemannian optimization follows approximately geodesics, while projected gradient steps can jump very far from d
This can avoided, for example, by deploying the regularization parameter (or weight decay) independently for each |6pp However, it costly in terms of hyper-parameter estimation.
In order to enforce the closeness of embedding vectors whose paths are similar, we penalize large norms of 8. We also want to bound it away from to avoid the problem of separators that point in different direction may have a small norm. This naturally leads to a spherical constraint. Indeed, we transform the f2 regularization over 6p by fixing its norm in advance, i.e.,
|| R
(12)
In other words, we define 6p on a sphere of radius Rp. The fully-connected layer A is then constrained on spheres, hence it is named spherical fully- connected layer.
Hence, we have W(p.)) constrained on a sphere centered at wp. This constraint prevents the direction of ((p.) from being too different from that of Wp, while bounding the distance away from zero. This does not add hyperparameters. instead of weight decay, we have the radius Rp the sphere.
3.1 RADIUS DECAY W.R.T. PATH LENGTH
We allow the radius of the spheres, RRp to be defined function of the path. In this study, use simple strategy called radius decay, where Ry decreases w.r.t. the path length:
Rp = Ro7
(13)
where Ro is the initial radius, is the radius decay parameter, and [p] is the length of the path. The optimal radius decay can be easily found using voos-validation. The radius decay is applied prior to learning (as opposed to weight-decay); then, the radius remains fixed during the optimization. As opposed to weight-decay, whose weight are multiplied by some constant smaller than one after each iteration, the radius decay here depends only on the path length, and the radius remains fixed during optimization process.
The simplest way to apply the radius decay is by using the following predefined diagonal matrix D,
D. 1o:1 P. E P, 0 otherwise,
(14)
where p; follows the ordering from Definition 1. Finally, the last layer of the neural network reads,
do
ADH
(15)
Network
layer
The only learnable parameter in the last layer is A.
3.2 OPTIMIZATION
There are several ways to optimize the network in the presence of the spherical fully-comeectdd layer: by introducing the constraint in the model, "naively" by performing normalization after each step, or
5

---

Under review as a conference paper at ICLR 2021
by using Riemannian optimization algorithms. For simplicity, we consider the minimization problem,
min f(0,A), 0.3
(16)
where are the parameters of the hidden layers, A the spherical fully-connected layer from equation 8, and f the empirical expectation of the loss of the neural network. For clarity, we use noiseless gradients, but all results also apply to stochastic ones. The superscript :' denotes the f-th iteration.
3.2.1 INTEGRATION OF THE CONSTRAINT IN THE MODEL
We present the simplest way to force the column of A to lie on sphere, this does not require dedicated optimization algorithm. It is sufficient to normalize the column of A by their norm in the model. By introducing a dummy variable A, which the normalized version of A, the last layer of the neural network equation 15 reads
6 |
..... (
(17)
Then, any standard optimization algorithm can be used for the learning phase. Technically, A is not constrained on a sphere, but the model will act if A follows such constraint.
3.2.2 OPTIMIZATION OVER SPHERES: RIEMANNIAN (STOCHASTIC) GRADIENT DESCENT
The most direct way to optimize over a sphere is to normalize the columns of A by their norm after each iteration. However, this method has no convergence guarantee, and requires a modification in the optimization algorithm. Instead, we perform Riemannian gradient descent which we explain only briefly in this manuscript. We give the derivation of Riemannian gradient for spheres in Appendix B.
Riemannian gradient descent involves two steps: first, a projection to the tangent space, and then, a retraction the manifold. The projection step computes the gradient of the function on the manifold (as opposed to the ambient space Rd), such that its gradient is tangent to the sphere. Then, the retraction simply maps the new iterate to the sphere. With this two-step procedure, all directions pointing outside the manifold, (i.c., orthogonal to the manifold, thus irrelevant) are discarded by the projection. These two steps are summarized below,
8 +h's
s' = (5, 6 f(
50+1
(18)
1 +
where 8 is the projection of the descent direction to the tangent space, and 51+1 is the retraction of the gradient descent step with stepsize h. In our numerical experiments, we used the Geoopt optimizer (Kochurov et al., 2020), which implements Riemannian gradient descent on spheres.
4 NUMERICAL EXPERIMENTS
We experimented the proposed method using five publicly available datasets, namely CI- FAR 100 (Krizhevsky, 2009), Caltech-UCSD Birds 200 (CUB200) (Welinder et al., 2010), Stanford- Cars (Cars) (Krause et al., 2013), Stanford-dogs (Dogs) (Khosla et al., 2011), and Tiny-ImageNe (Tiny-ImNet) (Deng et al., 2009). CUB200, Cars, and Dogs datasets are used for fine-grained visual categorization (recognizing bird, dog bleeds, or models), while CIFAR 100 and Tmne-I datasets are used for the classification of objects and animals. Unlike the datasets for object classifica- tion, the fine-grained visual categorization datasets show low inter-class variances. See Appendix C.2 and C.3 for more details about the dataset and their hierarchy, respectively.
4.1 DEEP NEURAL NETWORK MODELS AND TRAINING SETTING
We used the deep neural networks (ResNet (He et al., 2016) and DenseNet (Huang et al., 2017)). The input size of the datasets CUB200, Cars, Dogs, and Tiny-ImNi for CIFAR100. Since the input-size of CIFAR 100 does not fit to the original ResNet and DenseNet, we used smaller kernel size (3 instead of 7) at the first convolutional layer and a smaller stride (1 instead of 2) the first block.
224 x 224, and 32 x 32 pixels

---

Under review as a conference paper at ICLR 2021
Remark: do not pretrained networks. All networks are trained from scratch, i.e., did not use pre-trained models. This is because most publicly available pre-trained models used ImageNet for training while Dogs and Tiny-ImNet are parts of ImageNet.
We used the stochastic gradient descent (SGD) over 300 epochs, with a mini-batch of and a momentum parameter of 0.9 for training. The learning rate schedule is the same for all experiments, starting at 0.1, then decaying by a factor 10 after 150, then 255 epochs. All tests are conducted using NVIDIA Tesla V100 GPU with the same random seed. Settings in in the supplementary material. We emphasize that we used the same parameters and learning rate schedule for all scenarios. Those parameters and schedule were optimized for SGD on plain networks, but are probably sub-optimal for our proposed methods.
detail are provided
4.2 RESULTS
Tables and 2 show comparison of the results obtained with several baseline methods and our methods. The first method, "Plain", ,, is plain network for subclass classification without hierarchical information. The second one, "Multitask" is simply the plain network with multitask (subclass and super-class classificationn setting using the hierarchical information. The third one, "Hierarchy", uses our parametrization W AH with the hierarchical layer H. but the columns of A are not constrained on spheres. Then, "+Manifold" means that A is restricted on sphere using the normalization technique from Section 3.2.1. Finally, "+Riemann" means we used Riemannian optimization from Section 3.2.2. We show the experimental results on fin-grained visual classification (Table I) and general object classification (Table 2).
Note that the multitask strategy in our experiment (and contrary to our regularization technique) does require an additional syper-parammeer that combines the two losses, because we train classifiers for super-classes and sub-classes simultaneously.
4.2.1 FINE-GRAINED CATEGORIZATION
As shown in Table I, our proposed parameterization significantly improves the test accuracy over the baseline networks (ResNet- 18/50, DenseNet-121/1600 Even the simple hierarchical setting which uses the hierarchical layer only (without spheres) shows superior performance compared to the baseline networks. Integrating the manifolds with Riemannian SGD further improves the generalization performance.
Surprisingly, the plain network with deeper layers shows degraded performance. This can be attributed to overfitting which does not occur with our regularization technique, where larger networks show better performance, indicating the high efficiency of our approach.
Table 1: Test accuracy (%) for fine-grained classification. Radius decay is fixed at 0.5.
Baseline
Proposed parametrization
Dataset
Architecture
Plain
Multitask
Hierarchy
+Manifold
+Riemann
ResNet-18
54.88
53.99
58.28
60.42
60.98
ResNet-50
54.09
52.17
57.59
59.00
60.01
CUB200
DenseNet-121
50.55
56.61
61.10
60.22
61.98
DenseNet-161
50.91
60.67
60.67
62.73
63.55
ResNet-18
59.17
59.88
60.30
61.83
61.36
ResNet-50
57.44
58.97
59.31
59.81
63.70
Dogs
DenseNet-121
56.00
64.39
62.19
64.95
65.89
DenseNet-161
55.49
64.23
65.28
65.68
65.90
ResNet-18
79.83
82,85
84.96
84.74
84. 16
ResNet-50
82.86
82,86
83.34
84.51
84.65
Cars
DenseNet-122 121
79.78
85.39
85.97
86.00
85.54
DenseNet-161
79.85
85.79
86.23
86.90
85.76

---

Under review as a conference paper at ICLR 2021
Table 2: Test accuracy (%) for object classification. Radius decay is fixed 0.5 and 0.9 for CIFARIO0 and Tiny-Imned respectively.
Baseline
Proposed parametrization
Dataset
Architecture
Plain
Multitask
Hierarchy
+Manifold
+Riemann
ResNet-18
69,47
69.37
70.89
70.06
71.89
ResNet-550
71.04
71.74
73.75
73.76
73.97
CIFARIO0
DenseNet-122
74.50
75.62
76.38
76.52
76.28
DenseNet-161
75.30
76.57
77.01
77.01
76.64
ResNet-18
64.70
64.81
64.33
64.74
65.13
ResNet-50
66.43
66.39
66.52
66.67
65.69
Tiny-ImNet
DenseNet-125
64.27
67.15
67.19
67.86
67.45
DenseNet- 161
67.22
67.62
67.63
68.95
67.82
4.2.2 OBJECT CLASSIFICATION
We show test accuracy (%) of our proposed methods with different network models using CIFAR-100 and Tiny-ImNet, in Table 2. From the table, it can be seen that the proposed method has better accuracy than the baseline methods. Compared to the fine-grained classification datasets, the general object classification datasets have less similar classes on the same super-class. In these datasets, our method achieved relatively small gains.
higher inter-class variance may explain the lower improvement compared to fine- fine-grained categoriza- tion. Nevertheless, for Tmny- e.g., ResNet- 18 (11 .28M parameters) with our parametrization achieves better classification performance than plain ResNet-50 (23.91M parameters). The same applies to DenseNet-121 and DenseNet-161. These results indicate that our regularization tech- nique, which does not introduce new parameters in the embedding layer, can achieve a classífication performance similar that of more complex models.
4.3 RIEMANNIAN VS. PROJECTED SGD
Overall, Riemannian SGD showed slightly superior performance compared to projected SGD for fine-grained datasets, although, in most cases, the performance was similar. For instance, with the Dogs dataset on Resner-50, Riemannian SGD shows a performance 4% higher than the projected SGD, For object classification, Riemannian SGD performs bit more poorly. We suspect that, owing to the different radius decay parameters (0.5 in Table and 0.9 in Table 2), the learning rate of Riemannian SGD should have been changed to larger value.
CONCLUSION AND FUTURE WORK
We presented simple regularization method for neural networks using given hierarchical structure of the classes. The method involves of the reformulation the fully connected layer of the neural network using the hierarchical layer. We further improved the technique using spherical constraints, transforming the last layer into a spherical fully-connected layer. Finally, we compared the optimiza- tion of the neural network using several strategies. The reformulation using the hierarchical layer AH and the spherical constraint had a considerable impact on the generalization accuracy of the network. The Riemannian optimization had lower overall impact, showing sometimes significant improvement and sometimes similar to its projected counterpart.
In this paper, we used the proposed regularization technique only on classical architectures. In the future, it would be interesting to use it other architectures, e.g. Inception and SqueezeNet, for embedding, e.g. Poincaré, and other applications, e.g. Natural Language Processing (NLP). Moreover, in this paper, we used given hierarchy mostly based on taxonomy designed by experts. This hierarchical structure, which convenient for humans, may not most convenient for classification algorithms. A self- self-supervised algorithm that learns classification and the hierarchy may be convenient because we do not need to access hierarchy and lead to better results (because the structure will be more adapted to the task).
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
P.A. Absil, R. Mahony, and R. Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ, USA, 2007. ISBN 0691132984 9780691 132983.
Gregor Bachmann, Gary Bécigneul, and Octavian-Eugen Ganea. Constant eurvature graph convolu- tional networks. arXiv preprint arXiv: 191100076, 2019.
Kayhan Batmanghelich, Ardavan Saeedi, Karthik Narasimhan, and Sam Gershman. Nonparametric spherical topic modeling with word embeddings. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2016, pp. 537. NIH Public Access, 2016.
Peter Battaglia, Jessica Blake Chandler Hamrick, Victor Bapst, Alvaro Sanchez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andy Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Jayne Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. arXiv, 2018.
Gary Bécigneul and Octavian- Eugen Ganea. Riemannian adaptive optimization methods. arXiv preprint
2018.
Alsallakh Bilal, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren. Do convolutional neural networks learn class hierarchy? IEEE transactions on visualization and computer graphics, 24(1): 152-162, 2017.
S. Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transuctions on Automatic Control, 589))22177222 Sep. 2013.
Nicolas Boumal. An introduction to optimization on smooth manifolds. Available online, May 2020.
URL http::tt/// /www. .iccololl.l.o aacooasbbummll net/book.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
Lijuan Cai and Thomas Hofmann. Hierarchical document categorization with support vector machines. In Proceedings of the thirteenth ACM international conference on Information and knowledge management, pp. 78 78-87,20 2004.
Lijuan Cai and Thomas Hofmann. Exploiting known taxonomies in learning overlapping concepts.
In IJCAI, volume 7, pp. 708-713,2000
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. In Advances in Neural Information Processing Systems 32, pp. 4868 4879, 2019.
Christopher De Sa, Albert Gu, Christopher Ré, and Frederic Sala. Representation tradeoff's for hyperbolic embeddings. Proceedings of machine learning research, 80:4460, 2018.
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Scale Hicrarchical Image Database. In CVPR09, 2009.
Lun Du, Zhicong Lu, Yun Wang, Guojie Song, Yiming Wang, and Wei Chen. Galaxy network embedding: A hierarchical community structure preserving approach. In IJCAI, pp. 2079-20002000 2085, 2018.
Susan Dumais and Hao Chen. Hierarchical classification of web content. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 256-263, 2000.
Siddarth Gopal, Yiming Yang, and Alexandru Niculescu-Mizil. Regularization framework for large scale hierarchical classification. Proceedings of European Conference on Machine Learning, 2012.
Albert Gu, Frederic Sala, Beliz Gunel, and Christopher Ré. Learning euxeaccuuvattue representations in product spaces. In International Conference on Learning Representations, 2019.

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA. June 27-30, 2016, pp. 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, pp. 2261-2269. IEEE Computer Society, 2017.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on TieeGrained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, June 2011.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch, 2020.
Daphne Koller and Mehran Sahami. Hierarchically classifying documents using very few words. Technical report, Stanford InfoLab, 1997.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fme-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. In Advances in Neural Information Processing Systems 32, pp. 8230-8241, 2019.
Andrew McCallum, Ronald Rosenfeld, Tom M Mitchell, and Andrew Ng. Improving text classification by shrinkage in hierarchy of classes. In ICML, volume 98, pp. 359-367, 1998.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. arXiv preprint arXiv:1301.3780 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 311 3111-3119, 2013b 3b.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. 2017.
In Advances neural information processing systems, pp.
Yehonatan Sela, Moti Freiman, Elia Dery, Yifat Edrei, Rifaat Safadi, Orit Pappo, Leo Joskowicz, and Rinat Abramovitch. firi--based hierarchical svm model for the classification and grading of liver fibrosis. IEEE transactions on biomedical engineering, 58(9))2574-2555 2571,-2581,,00 2011.
Ondrej Skopek, Octavian-Bugen Ganea, and Gary Bécigneul. Mixed-curvature variational autoen- coders. arXiv preprint 19Xvv:1911.084/1 2019.
Alexandru Tifrea, Gary Bécigneul, and Octavian-Eugee Eugen Ganea. Poincaré glove: Hyperbolic word embeddings. arXiv preprint arXiv: 1010.000000000 181.066666, 2018
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Ke Wang, Senqiang Zhou, and Shiang Chen Liew. Building hierarchical classifiers using class proximity. In VLDB, volume 99, pp. 363-374. Citeseer, 1999.
Andreas S Weigend, Erik D Wiener, and Jan O Pedersen. Exploiting hierarchy in text categorization. Information Retrieval, 1(3):193-116 1999.
Welinder. S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSS Birds 200. Technical Report CNS-TR-2010--0 California Institute of Technology, 2010.
Pengtao Xie, Yuntian Deng, Yi Zhou, Abhimanu Kumar, Yaoliang Yu, James Zou, and Eric P Xing. Learning latent space models with angular constraints. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 3799-3810. JMLR. org, 2017.
10

---

Under review as a conference paper at ICLR 2021
Jun Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on the natural image manifold. In European Conference on Computer Vision, pp. 597-613. Springer, 2016.
A EXAMPLE OF HIERARCHICAL STRUCTURE
Consider dataset composed by the following labels: cats, dogs, apple, orange. These labels can be organized trough hierarchical structure, with super-classes animal and fruit. In such case, the set P composed by
p= { (fruit), (animal); (fruit, apple), (fruit, orange), {animal, cat), {animal, dog) }
while the set is composed by
(fruit, apple), {fruit, orange), {animal, cat), {animal, dog} }
Then, its hierarchical layer reads (labels were added to ease the reading)
{ [fruit, apple}
[fruit, orange)
{animal, cat]
[animal, dog)
[friit]
[animal]
0
(fruit, apple]
(fruit, orange)
0
1
O
0
(animal, cat)
(animal, dog)
0
0
0
1
B OPTIMIZATION OVER SPHERES: RIEMANIAN (STOCHASTIC) GRADIENT DESCENT
We quickly recall some elements of optimization on manifolds, see e.g. Boumal (2020); Absil et al. (2007). For simplicity, we consider the optimization problem
min
f(x)
(19)
#E84
where 3d-1 is the sphere manifold with radius centered zero and embedded in Rd. The generic Riemannian gradient descent with stepsize h reads
8 frrdf((x))
(20)
Øk+1 = Rzg (hs²)
(21)
where gradf is the gradient of on the sphere, which is a vector that belongs to the tangent space Tay 8d (plane tangent to the sphere that contains I&), and Rgg is a second-order retraction, i.c., mapping from the tangent space Tæ Sa-1 to the sphere S that satisfies some smoothness properties. The vector (that belongs to the tangent sphere) represents the local descent direction. We illustrate those quantities in Figure 3. Stochastic Riemannian gradient descent directly follows from (Bonnabel, 2013), replacing the gradient by its stochastic version.
In the special case of the sphere, we have an explicit formula for the tangent space and its projection, for the Riemannian gradient, and for the retraction:
T,$a-1 = fy y² 0}
(a(x) = = (x"yyz; x y
(22)
grad (v)()))
(23)
|| + vll
The retraction is not necessarily unique, but this one satisfies all requirement to ensure good conver- gence properties. The gradient descent algorithm on sphere thus reads
(a V fzx)) I fx(x))
(24)
T& + h4SK
0k+1
(25)
I + h4-8k
In case, have matrix A, whose each column 8, belongs sphere. It suffices to apply the Riemannian gradient descent separately on each 6,. For practical reasons, we used the toolbox Geoopt (Kochurov al., 2020; Bécigneul & Ganea, 2018) for numerical optimization.
11

---

Tiny-ImNet Under review as a conference paper at ICLR 2021
-7 ((L)
H (xx))
M
TryM
2241
Figure 3: Illustration of a Riemanian gradient step manifold M. In blue, the projection operator from the ambient space (in our case, Rd) the tangent space T, This maps the standard gradient of the function, Vf, to its Riemannian gradient gradf. Then, in red we have the retraction that maps vector from the tangent space M to the manifold M. This converts the gradient step Th hgraa/ (xxx (that belongs to the tangent space) to F (that belongs to the manifold).
Table 3:
Size of the datasets used in our experiments. |L denote the number of classes in the dataset, and PP the total number of classes and super-classes.
Dataset
CIFAR100
CUB200
Stanford Cars
Stanford dogs
C
100
200
196
121
200
P
120
270
205
295 194
# Samples
50k
6k
16k
21k
100k
C NUMERICAL EXPERIMENTS, SUPPLEMENTARY MATERIALS
C.1 DEEP NEURAL NETWORK MODELS AND TRAINING DETAILS
We used ResNet which consists of the basic blocks or the bottleneck blocks with output channels [64, 128, 256, 512] in Conv. layers. A dimensionality of an input vector to the FC layer is 512. We used DenseNet which includes hyperparameters such as ("growth rate", "block configuration", and "initial feature dimension"] for 1De [32, (6, 12, 24, 16), 64] and 'DenseNet-161" [48, (6, 12, 36, 24), 96], respectively. A dimensionality an input vector for DenseNets the FC layer is 64 and 96.
Parameters in our proposed method using ResNet and DenseNet are optimized using the SGD with several settings: we fixed 1) the weight initialization with Random-Serd number "0" in pytorch, 2) learning rate schedule 10.1, 0.01, 0.001], 3) with momentum 0.9, 4) regularization: weight decay with 0.0001. A bias term in the FC layer is not used. The images (CUB200, Cars, Dogs, and Tiny-ImNo in training and test sets resized to 256 x 256 size. Then, the image is cropped with 224 x 224 size at random location in training and at center location test. Horizontal flipping is applied in training. The learning rate decay by 0.1 at [150, 225] epochs from initial value of 0.1. The experiments conduced using GPU "NVIDIA TESLA V100". We used one GPU for ResNet-18, 18, and two GPUs for ResNet-50, 1enseNet-121, and DenseNet-161.
C.2 DATASET
We summarize the important information of the previous datasets in Table 3. The next section describe how we build the hierarchical tree for each dataset.
C.3 HIERARCHY FOR DATASETS
In this section, we describe how we build the hierarchy tree for each dataset. We provide also the files containing the hierarchy used in the experiments in the folder Hi
12

---

Under review as a conference paper at ICLR 2021
Before explaining how generate the hierarchy, quickly describe the content of the files. Their name follow the pattern DATASETNAME caapTAAMME__iiidddl parent parent_pairs, txt. The first line the file corresponds to the number of entries. Then, the file is divided into two columns, representing pairs of (chi 1d, parent). This means if the pair (ni, n2) exists in the file, the node n2 is the direct parent of the node n1. All labels have been converted into indexes.
C.3.1 CIFAR100
The hierarchy of C0 is given by the authors.
C.3.2 CUB200
We classified the breed of birds into different groups, in function of the label name. For instance, the breeds Bl bacck_ootee_e footed booss, Laysan_Ale blat bat ross and cooty_Albar are classified in the same super-class Albatross.
C.3.3 STANFORD CARS
We manually classified the dataset into nine different super-classes: SUV, Sedan, Coupe, Hatchback, Convertible, Wagon, Pickup, Van and Mini- Van. most cases, the super-class name appears in the name of the label.
C.3.4 STANFORD DOGS
The hierarchy is recovered trough the breed presents at the end of the name of each dog specie. For instance, English Setter, Irish Setter, and Gordon Setter classified under the class Setter.
C.3.5 (TINY) IMAGENET
The labels of (mage-)t also Wordnet classes. We used the Wordnet hierarchy to build the ones (Tiny) Imagenet. There are also two post-processing steps:
tree, which means one node can have more than one ancestor. The choice was systematic: we arbitrarily chose unique ancestor the first one in the sorted list.
Wordnet hierarchy
2. In the case where a node has one and only one child, the node and its child are merged.
C.4 GENERALIZATION PERFORMANCE ALONG DIFFERENT RADIUS DECAY VALUES
In this section, show in Table 4 how the radius decay affects the test accuracy. In all experiments, we used the Resnet18 architecture with Riemannian gradient descent to optimize the spherical fully-connected layer.
Globally, we see that radius decay may influence the accuracy of the network. However, in most cases, the performance is not very sensitive to this parameter. The exception is for tiny-imagenet, where the hierarchy tree has many levels, and thus small values degrade a lot the accuracy.
13

---

Under review as a conference paper at ICLR 2021
Table Influence of radius decay on the test performance for ResNet18.
Radius Decay
CUB200
Dogs
Cars
CIFAR100
Imny-ImageNet
0.50
60.98
61.35
84.74
71.65
38.38
0.55
60.74
61.28
84.75
71.16
47.77
0.60
60.84
60.93
84.53
71.21
55.66
0.65
59.79
60.09
84.80
71.03
60.44
0,70
58.72
60.19
84.43
71.01
62.03
0.75
58.87
60.57
84.72
70.44
63.28
0,80
58.89
60.12
84.70
70.79
64.10
0.85
57.47
60.46
84.67
70.29
64.45
0.90
58.51
60.07
84.47
70.49
64.16
0.95
56.51
58.67
84.78
70.63
64.14
1.00
57.13
58.21
84.63
70.76
64.60
C.5 LEARNING RADIUS DECAY
Here, we replace the diagonal matrix D in equation 14 with a learnable parameter matrix which is trained using backpropagation without an additional constraint or loss function for simplicity, As shown in Table 5, this learnable radius is not effeective the in terms of an classification performance compared to that the predefined radius decay.
Table 5: Test accuracy (%) with a learnable radius. Clearly, using learnable parameter degrades considerably the performance using the predefined radius decay (shown in parentheses, selected from Table 1).
Proposed parametrization
Dataset
Architecture
Hierarchy
+Manifold
+Riemann
CUB200 Cars
ResNet-18
53,40 (58.28)
58.35 (60.42)
58.24 (60.98)
ResNet-18
81.51 (84.96)
82.54 (84.74)
82.40 (84.16)
C.6 LEARNING WITH RANDOM HIERARCHY
As shown in Table 6, the methods with randomly generated hierarchy showed degraded perfor- mance compared to that with a reasonable hierarchical information.
Table 6: Test accuracy (%) with random hierarchy tree. Radius decay is fixed at 0.5. Clearly, using random hierarchy degrades considerably the performance (shown parentheses, selected from Table I which are the results with the original hierarchical tree). This validates the importance of the proper hierarchy information.
Baseline
Proposed parametrization
Dataset
Architecture
Multitask
Hicrarchy
+Manifold
+Riemann
CUB200 ResNet-18
47.55 (53.99)
50.28 (58.28)
56.96 (60.42)
56.43 (60.98)
Cars
R8
79.98 (82.85)
81.07 (84.96)
82.02 (84.74)
81.84 (84.16)
C.7 SUPER-CLASS CLASSIFICATION EFFICIENCY
As shown in Table 7, our proposed methods (Hicrarchy, +Manifold, and +Riemann) outperformed multitask (multilabel) classification method terms of test accuracy performance. Note that, in the multitask classification, loss function for classification using superclasses is used additionally.
14

---

Under review as a conference paper at ICLR 2021
Table 7: Test accuracy (%) for super-class classification. Radius decay is fixed at 0.5. For the super-class classification, without any modification of the proposed layers, we trained the model on the dataset, then calculate classification accuracy using the 8's corresponding the parent classes.
Baseline
Proposed parametrization
Dataset
Architecture
Multitask
Hierarchy
+Manifold
+Riemann
CUB200
ResNet-18 ResNet-18 18
53.68 86.88
58.87
61.17 91.23
62.22
Cars
87.91
90.97
C.8 VISUALIZATION OF EMBEDDING VECTORS
In this section, we visualized an embedding vector which is an input of the last classification layer. As suggested by the reviewer, first, we show a distribution of two-dimensional vector (IE2) learned by the networks. To obtain these vectors, we added new layers (i.e. a mapping function R + IR?. m 512) prior to the last FC layer of BesNet-18. As two-dimensional vector is not enough to represent a discriminative feature for the fine-grained dataset which have large number of classes different from MNIST dataset with ten classes with gray level images, we observe multidimensional vectors used in the ResNet. Second, we use t-SNE, to visualize the high dimensional embedding vector (I2512) of the original ResNet, which is one of popular methods for exploring nigh-dimensional vector, introduced by van der Maaten and Hinton. Even though this method is known to have limitation which highly dependent hyperparameters such perplexity values, it is still useful observe distribution of those high dimensional vectors by using fixed hyperparameters. For a deterministic way of visualization on 2D plane regardless of hyperparameters, finally, we visualized the embedding vectors using the traditional dimension reduction technique, namely Principal Component Analysis (PCA). Note that t-SNE and PCA have complementary characteristies (e.g., stochastic vs. deterministic, non-linear vs. linear, capturing local vs. global structures).
1) Learned two-dimensional representation. We show a distribution of two-dimensional em- bedding vector which is used for classification. As shown in Figure 4, embedding vectors of our proposed methods are distributed more closely (clustered) with regard to their superclasses (Figure 4c,4d, and 4e). We used Cars dataset since the number of superclasses (nine super-classes) small enough to show their distribution clearly. We show the results where the classification performances of all compared methods are similar. We captured the distribution of embedding vectors from an early epoch due to their slow convergence rate. Two-dimensional vector seems too small to classify images of 196 classes which highly non-linearly distributed.
2) Observation of locality preserving structure via t-SNE. We visualize high high-dimension embed- ding vector extracted from R8 which mapped into two-dimensional space by preserving local pairwise relationship of those vectors, using t- t-Distributed Stochastic Neighboring Entities (t- SNE). As in Figure 5, embedding vectors of our proposed methods are clearly clustered (Figure 5c,5d, and 5e) with regard to their superclasses compared to that of the baseline methods (Figure 5a and 5b).
3) Observation of global structure via PCA. Using PCA, we capture a global structure of high dimensional embedding vectors by projection onto two-dimensional space. As shown in Figure 6, embedding vectors extracted from ResNet-18 of our proposed methods have Bess-Gaussian shape distribution (Figure 6d and 6e) with regard to their superclasses than that of the baseline methods (Figure 6a and 5b). We observe that individual distribution of each classes is similar to that of Figure 4.
15

---

Under review as a conference paper at ICLR 2021
@ Baseline
(b) Multitask
13
proatiige praayposre
Hierarchy
(d) +Manifold
(c) +Riemann
Figure 4: Visualization of 2-dimensional embedding vector. We added the fully connected layer (RR*x2, e.g., for ResNet18, 512) prior the last FC layer. (a) Baseline, (b) Baseline (Multitask), and proposed parameterization ((c) Hicrarchy, (d) +Manifold, and (c) +Riemann).
(a) Baseline
(b) Multitask
Hierarchy
(d) +Manifold
+Riemann
Figure 5: Visualization of high dimensional embedding vector using t-SNE on 2D plane. (a) Baseline. (b) Baseline (Multitask), and proposed parameterization ((c) Hierarchy, (d) +Manifold, and (e) +Rieman)).
16

---

Under review as a conference paper at ICLR 2021
(a) Baseline
(b) Multitask
(c) Hierarchy
(d) +Manifold
(c) +Riemann
Figure 6: Visualization of high dimensional embedding vector 2D plane using PCA. Baseline (b)
Baseline (Multitask), and proposed parameterization ((c) Hierarchy, (d) +Manifold, and (e) +Riemann).
17


Reviews:
[{"review_id": "lrfxt72LwEy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I generally like the neat idea of introducing hierarchical spheres to model the intra- and inter-class relationships among the hierachical labels. It is naturally motivated to combine the hierarchical structure in the label space as a prior knowledge to supervise the training of neural networks. The overall idea is simple and easy to understand. The method models the labels at different levels by adding a free vector that is constrained on a specific hypersphere, then uses a smart way of formulating this procedure with simple matrix multiplication, and finally considers an alternative manifold optimization method to train the neural network in an end-to-end fashion. As far as I'm concerned, the intuition has also been explored in [Deep neural decision forests, ICCV 2015], but differently, the ICCV 2015 paper considered the hierarchical label structure with a decision forest. I believe this direction is of sufficient significance to the ML community.\n\nThis paper has several aspects that I found most interesting:\n\n(1) The formulation is interesting and is novel from my perspective. Modeling the fine-grained classes by successively adding a \"perturbation\" vector makes sense to me. Then, the authors are able to formulate this in a matrix multiplication, which is basically a linear matrix factorization that over-parameterizes the classifiers. Although technically the linear matrix multiplication is still equivalent to a linear classifier, the fact that it can still improve the network generalization is interesting and is partially verified by a number of theory works. Besides, as a way to combine prior knowledge to supervise the neural networks, such a simple linear matrix factorization (with some constraints like sphericity, radius decay, etc.) provides a potentially useful way to incorporate some regularization priors.\n\n(2) The use of spherical constraints is interesting and empirically make senses to me. By constraining the learning on the spherical space can ease the training difficulties of the over-parameterized classification layers. This is, in fact, also observed and verified by [Neural Similarity Learning, Neurips 2019]. It will be potentially interesting to connect these two papers and have some discussions. The radius decay for the hierarchical spheres is also novel to me, because labels in finer-grained level should be modeled with less capacity.\n\nDesipte these interesting aspects, I also have a few concerns and suggestions to improve the paper:\n\n(1) The empirical evaluation is relatively weak and the evaluation metric seems not to well reflect the advantages of hierarchically modelling the label space. For example, I think it will be more informative to incorporate the classification accuracy of the super-classes. It will make this paper more interesting to have more experiments that analyzes the difference in feature distributions between normally trained neural networks and the hierarchically trained neural networks. For example, an intuitive visualization of the feature space will be of great interest. An easy way for the visualziaiton is to set the outpute feature dimension as 2 and directly plot them, similar to [A Discriminative Feature Learning Approach for Deep Face Recognition, ECCV 2016] and [Large-Margin Softmax Loss for Convolutional Neural Networks, ICML 2016].\n\n(2) Some important ablation studies to justify some heuristic designs are very important and necessary. For example, there is a hyperprameter in the radius decay, how it will affect the performance is crucial. Potentially, the authors can also evaluate what if no sphericity constraint is applied, or what if no radius decay is used, etc. Since this paper proposes a number of heuristic designs, it is very important to justify them (either from theoretical perspective, or from empirical evaluations).\n\n(3) Although I believe it is useful to model the hierachical label space in an explicit way, the empirical evaluation does not really convince me on that, especially experiments on CIFAR-100 and Tiny-ImageNet. The method uses additional prior knowledge on the label space, but only yields very limited performance gain. I think using some other SOTA regularization can easily improve more. What is the underlying reason? I think more discussions and insights will be useful.\n\n(4) The usefulness of the hierachical label structure should be evaluated and verified in the first place. A simple way to evaluate it is to use some random assignment or simple K-means assignments for the super-classes. If using the ground truth hierachical strucutre can consistently outperform the random or K-means super-class assignment, then one can believe that incorporating the ground truth hierachical label structure is indeed useful. Until then, it makes little sense to argue it is beneficial to generalization to combine the ground truth hierachical label structure. I highly suggest the authors conduct such an experiment.\n\nSome minor concerns and suggestions:\n\n(5) I cannot find the Spherical CNN from Xie et al. (2017) on page 1. I think the paper is more closely related to [Deep Hyperspherical Learning, Neurips 2017] in terms of the spherical regularization. The authors may discuss the connections and differences to this paper.\n\n(6) Since the authors consider regularizations for the intra-class hierachical label structure, it will be interesting to see whether the regularization on the inter-class regularization will be beneficial or not. For example, the authors can use some diversity regularization on sphere to push away classifiers from different super classes. A potential regularization for this is [\nLearning towards Minimum Hyperspherical Energy, Neurips 2018]. I want to note that it is a suggestion for the paper rather than a weakness.\n\nTo summarize, I think the paper proposes a very interesting and potentially widely useful method to incorporate the hierachical label structure to train neural networks. Currently, I feel postive to accept this paper, and I am sitting between 6 and 7 (I give a 6 for now). I will consider to increase my score if the authors well address the concerns.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Yyp3YRmxAY1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Section 1: The third paragraph is confusing. It is not clear why Euclidean distance is not sufficient for learning with such a hierarchical regularization. Can't the parent class mean just the mean of all its children?\n\nSection 1: From the second last paragraph, it reads that the paper does not train the neural network, but adopts a pre-trained off-the-shelf network and works on its last layer. While this makes the method simple and compatible to other network architectures, is there a reason not to train an end-to-end network with the proposed technique? However, in Section 4.1, the paper says all networks are trained from scratch. Does the training of the whole model follow end-to-end training or stage-wise training?\n\nSection 2.2: How to define \"separator\"? \"Separator\" does not seem like a formal word. Moreover, the sentence is confusing, \"the parameter of the classifiers that identify dog’s breed should also be similar\". If similar, how can they differentiate dog breeds?\n\nSection 3.1: Is radius decay a new method proposed in the paper? If so, what is the rationale behind this design? If not, are there any related work adopting this method? Why not use other linear decay methods, e.g., $R_p=R_0*|p|$?\n\nEq.(16) implies that the whole model (backbone and the proposed spherical fully-connected layer) is end-to-end optimized. What is the optimization method used for learning other layers?\n\nMoreover, the paper says \"the most direct way to optimize over a sphere is to normalize the columns of ∆ by their norm after each iteration. However, this method has no convergence guarantee, and requires a modification in the optimization algorithm\". To construct the spherical fully-connected layer, isn't it equivalent to learn a normal fully-connected layer followed up by L2-normalization and a scaling operation?\n\nEven though the authors provide C.2, C.3 and plain files on the prepared hierarchies in the datasets, it is disappointing that the paper does not present the hierarchical structure in a nice way. Perhaps a visualization on the class labels w.r.t the hierarchy serves the paper better?\n\nSection 4.1: Tiny-ImageNet dataset seems to have lower resolution of images (64x64). When authors say \"input size...224x224\", do the authors mean that Tiny-ImageNet images are resized from 64x64 to 224x224? \n\nSection 4.1: The paper trains all networks from scratch by explaining \"Dogs and Tiny-ImNet are parts of ImageNet\". Does it mean that images from datasets Dogs and Tiny-Imagenet are part of ImageNet? Or Does it mean that classes in the two datasets are included in the set of ImageNet classes?\n\nSection 4.1: How to define \"plain networks\"? The paper uses two networks ResNet and DenseNet, then what is \"plain networks\"? Once the authors state that the parameters are \"probably sub-optimal for our proposed methods\", it also implies that the parameters may be even more sub-optimal to the compared methods?\n\nSection 4.2.1: When the paper claims \"high efficiency of our approach\", it does not justify the \"efficiency part\". How to tell if the training or inference efficiency is higher than other methods? \n\nFigure 2 right depicts Riemannian gradient and \"projected gradient\", but the paper does not formally compare them. It is not clear which one may be better than the other? For learning, gradient guides the direction to update parameters, but the scale in the update also matters. Is there a discussion on which one may be more efficient (or compute time) during a training iteration? Section 4.2 explicitly notes the performance difference between the two methods. The paper should discuss this further for better understanding. Moreover, given that the two methods are so different, setting the same learning rate schedule (cf. Section 4.1) is not sensible, because they can perform quite differently with different learning rates.\n\nThe paper does not discuss how the proposed method may work if classes do not follow a tree hierarchy, although C.3 talks a bit in the context of Tiny-ImageNet. As the paper focuses on a generic regularization based on hierarchical information, it may also need to discuss how this can be applied in multi-label classification problems.\n\n\n---------------------------------\npost-rebuttal\n---------------------------------\nI appreciate that authors have provided rebuttal that addresses many of my questions, though I'd like to maintain my initial rating due to the following comments. I think this paper is at the borderline.\n\nIn terms of explaining why \"Euclidean distance is not sufficient for learning such a hierarchical regularization\", I don't find the illustration example in Section 2.3 intuitive or concrete. I don't think Eq3 adds much as the paper does not explain further. Perhaps the confusion is from that the paper does not explicitly explain what \"optimal classifier\" mean in terms of Eq3.\n\nThe authors only say \"those parameters and schedule were optimized for SGD on plain networks, probably sub-optimal for our proposed methods.” It is not clear whether other methods suffer severely from the choice of learning rate and scheduler. As far as I know, SGD is sensitive to the initial learning rate. So I am worried that setting the same learning rate is not fair to comparing different models that have different structures.\n\nFrom the updated paper, I find the blue line in Page-2 confusing. It is not clear about the logic: why diversity reduces over-fitting. (Xie et al. 2017) studies this point with a complete paper. But the way that authors simply put it is quite unclear how this statement is related in the context.\n\nVisualization is interesting to look at. But it should be better analyzed. For example, visually all methods produce similar tSNE visuals in Figure 5. But are there any essential difference?", "extra_scores": {}, "extra_text": {}}, {"review_id": "RETrq3vRt1I", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The idea of introducing class hierarchy as a regularization into deep networks seems to be novel. \n\nThe following comments could be relevant:\n\n1. The reviewer finds Section 2 is not easy to follow:\n- Authors may consider to give more specific definitions to terms such as, classifier, separators, etc. For example, in (2), Wp and Wpi are called classifiers. What are they, hyperplanes? \n- In Definition 1, authors may like to give some early examples about P and L. Otherwise, it is not easy to interpret the matrix H.\n- Authors may consider to use a different notation for Delta in (8), as Delta may remind an operator on H in (9).\n\n2. In (9), do we require or observe deltas in the same subtree roughly the same direction? \n\n3. In Section 3, it is claimed no hyperparameters are added. However, it seems that, initial radius R0, radius decay parameter, even how to organize classes may all be considered as additional hyperparameters.\n\n4. In reality, it can be non-trivial, or even impossible,  to define mutual exclusive class partitions to form the required class tree in Figure 1. Authors may discuss how different class hierarchy adopted affects the classification accuracy, e.g., in Table 2.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ITMCTrwF2s8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a novel reparameterization framework of the last network layer that takes semantic hierarchy into account. Specifically, the authors assume a predefined hierarchy graph, and model the classifier of child classes as a parent classifier plus offsets $\\delta$ recursively. The authors show that such hierarchy can be parameterized a matrix multiplication $\\Delta \\mathbf{H}$ where $\\mathbf{H}$ is predefined by the graph. In addition, the authors further propose to fix the norm of $\\delta$ in a decaying manner with respect to path length. The resulting spherical objective is optimized via Riemannian gradient descent.\n\nThe strengths and weaknesses are very obvious in this paper.\n\nOn the strength side:\n+ The paper itself is very well written. The notations are well defined and the methods are very clearly explained. The presentation is fluent.\n+ The proposed method seems novel and interesting. The derivations are technically correct.\n+ Experiments show performance improvement over baselines, especially on CUB200/Dogs/Cars.\n\nOn the weakness side:\n- I think experiment presents the most significant weakness of this paper: 1) The comparison is rather weak without any reference to existing prior arts such as [1]. A simple search with respect to the 5 experiment datasets also show significant performance gaps between the proposed method and latest methods. 2) I remain skeptical about the solidness of the baseline performance as they show considerable gaps to standard baseline training without bells and whistles (https://github.com/weiaicunzai/pytorch-cifar100). 3) The performance gain diminishes very quickly on bigger dataset such as Tiny ImageNet. What about the results on ImageNet?\n- The proposed method depends on a pre-defined semantic hierarchical graph rather than a learned one, which potentially limits the technical value of this work. In certain cases, semantic hierarchy may not always be a reasonable choice to guide the learning of visual embedding.\n- I have some concern about the selection of initial radius $R_0$ and its decay policy. I think this parameter should be dataset dependent due to different numbers of categories and the densities of class distributions. As a result, how such parameter and policy can be optimally determined becomes a question.\n- Finally, forcing a fixed radius does not sound as reasonable as allowing a learnable radius with soft regularization.\n\n[1] Chen et al., Fine-grained representation learning and recognition by exploiting hierarchical semantic embedding, ACM-MM 2018\n\n========================== Post Rebuttal ==============================\n\nThe authors did a good job in addressing some of my concerns in the rebuttal. Thus I am increasing the score in response to the clarifications. However, I feel there is still some improvement space for the experiment part of this section, and I encourage the authors to incorporate the changes, including ImageNet experiment and following stronger baselines to make the results more solid and convincing.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a method for hierarchical classification with deep networks. The idea is interesting, and as far as I know novel: namely, the authors add a regularizer to the last layer in order to enforce a hierarchical structure onto the classifiers. The idea of placing spheres (with a fixed radius) around each classifier and forcing the child-classifiers to lie on these spheres is quite clever. 
The reviewers have pointed out some concerns with this paper. Some had to do with terminology (which the authors should fix but which is no big deal), but the main weakness are the experimental results and the ablation study. The reviewers were not convinced that the optimization in the Euclidean space wouldn't be sufficient. A more thorough ablation study could help here. 

This is the kind of paper that I really want to see published eventually, but right now isn't quite ready yet. If you make one more iteration (in particular adding a stronger ablation study) it should be a strong submission to the next conference. Good luck!
Decision: Reject