Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors proposed a novel reparameterization framework of the last network layer that takes semantic hierarchy into account. Specifically, the authors assume a predefined hierarchy graph, and model the classifier of child classes as a parent classifier plus offsets $\delta$ recursively. The authors show that such hierarchy can be parameterized a matrix multiplication $\Delta \mathbf{H}$ where $\mathbf{H}$ is predefined by the graph. In addition, the authors further propose to fix the norm of $\delta$ in a decaying manner with respect to path length. The resulting spherical objective is optimized via Riemannian gradient descent.

The strengths and weaknesses are very obvious in this paper.

On the strength side:
+ The paper itself is very well written. The notations are well defined and the methods are very clearly explained. The presentation is fluent.
+ The proposed method seems novel and interesting. The derivations are technically correct.
+ Experiments show performance improvement over baselines, especially on CUB200/Dogs/Cars.

On the weakness side:
- I think experiment presents the most significant weakness of this paper: 1) The comparison is rather weak without any reference to existing prior arts such as [1]. A simple search with respect to the 5 experiment datasets also show significant performance gaps between the proposed method and latest methods. 2) I remain skeptical about the solidness of the baseline performance as they show considerable gaps to standard baseline training without bells and whistles (https://github.com/weiaicunzai/pytorch-cifar100). 3) The performance gain diminishes very quickly on bigger dataset such as Tiny ImageNet. What about the results on ImageNet?
- The proposed method depends on a pre-defined semantic hierarchical graph rather than a learned one, which potentially limits the technical value of this work. In certain cases, semantic hierarchy may not always be a reasonable choice to guide the learning of visual embedding.
- I have some concern about the selection of initial radius $R_0$ and its decay policy. I think this parameter should be dataset dependent due to different numbers of categories and the densities of class distributions. As a result, how such parameter and policy can be optimally determined becomes a question.
- Finally, forcing a fixed radius does not sound as reasonable as allowing a learnable radius with soft regularization.

[1] Chen et al., Fine-grained representation learning and recognition by exploiting hierarchical semantic embedding, ACM-MM 2018

========================== Post Rebuttal ==============================

The authors did a good job in addressing some of my concerns in the rebuttal. Thus I am increasing the score in response to the clarifications. However, I feel there is still some improvement space for the experiment part of this section, and I encourage the authors to incorporate the changes, including ImageNet experiment and following stronger baselines to make the results more solid and convincing.
review_title: Interesting method but rather weak experiment

final_score: rating: 6: Marginally above acceptance threshold
review_content: I generally like the neat idea of introducing hierarchical spheres to model the intra- and inter-class relationships among the hierachical labels. It is naturally motivated to combine the hierarchical structure in the label space as a prior knowledge to supervise the training of neural networks. The overall idea is simple and easy to understand. The method models the labels at different levels by adding a free vector that is constrained on a specific hypersphere, then uses a smart way of formulating this procedure with simple matrix multiplication, and finally considers an alternative manifold optimization method to train the neural network in an end-to-end fashion. As far as I'm concerned, the intuition has also been explored in [Deep neural decision forests, ICCV 2015], but differently, the ICCV 2015 paper considered the hierarchical label structure with a decision forest. I believe this direction is of sufficient significance to the ML community.

This paper has several aspects that I found most interesting:

(1) The formulation is interesting and is novel from my perspective. Modeling the fine-grained classes by successively adding a "perturbation" vector makes sense to me. Then, the authors are able to formulate this in a matrix multiplication, which is basically a linear matrix factorization that over-parameterizes the classifiers. Although technically the linear matrix multiplication is still equivalent to a linear classifier, the fact that it can still improve the network generalization is interesting and is partially verified by a number of theory works. Besides, as a way to combine prior knowledge to supervise the neural networks, such a simple linear matrix factorization (with some constraints like sphericity, radius decay, etc.) provides a potentially useful way to incorporate some regularization priors.

(2) The use of spherical constraints is interesting and empirically make senses to me. By constraining the learning on the spherical space can ease the training difficulties of the over-parameterized classification layers. This is, in fact, also observed and verified by [Neural Similarity Learning, Neurips 2019]. It will be potentially interesting to connect these two papers and have some discussions. The radius decay for the hierarchical spheres is also novel to me, because labels in finer-grained level should be modeled with less capacity.

Desipte these interesting aspects, I also have a few concerns and suggestions to improve the paper:

(1) The empirical evaluation is relatively weak and the evaluation metric seems not to well reflect the advantages of hierarchically modelling the label space. For example, I think it will be more informative to incorporate the classification accuracy of the super-classes. It will make this paper more interesting to have more experiments that analyzes the difference in feature distributions between normally trained neural networks and the hierarchically trained neural networks. For example, an intuitive visualization of the feature space will be of great interest. An easy way for the visualziaiton is to set the outpute feature dimension as 2 and directly plot them, similar to [A Discriminative Feature Learning Approach for Deep Face Recognition, ECCV 2016] and [Large-Margin Softmax Loss for Convolutional Neural Networks, ICML 2016].

(2) Some important ablation studies to justify some heuristic designs are very important and necessary. For example, there is a hyperprameter in the radius decay, how it will affect the performance is crucial. Potentially, the authors can also evaluate what if no sphericity constraint is applied, or what if no radius decay is used, etc. Since this paper proposes a number of heuristic designs, it is very important to justify them (either from theoretical perspective, or from empirical evaluations).

(3) Although I believe it is useful to model the hierachical label space in an explicit way, the empirical evaluation does not really convince me on that, especially experiments on CIFAR-100 and Tiny-ImageNet. The method uses additional prior knowledge on the label space, but only yields very limited performance gain. I think using some other SOTA regularization can easily improve more. What is the underlying reason? I think more discussions and insights will be useful.

(4) The usefulness of the hierachical label structure should be evaluated and verified in the first place. A simple way to evaluate it is to use some random assignment or simple K-means assignments for the super-classes. If using the ground truth hierachical strucutre can consistently outperform the random or K-means super-class assignment, then one can believe that incorporating the ground truth hierachical label structure is indeed useful. Until then, it makes little sense to argue it is beneficial to generalization to combine the ground truth hierachical label structure. I highly suggest the authors conduct such an experiment.

Some minor concerns and suggestions:

(5) I cannot find the Spherical CNN from Xie et al. (2017) on page 1. I think the paper is more closely related to [Deep Hyperspherical Learning, Neurips 2017] in terms of the spherical regularization. The authors may discuss the connections and differences to this paper.

(6) Since the authors consider regularizations for the intra-class hierachical label structure, it will be interesting to see whether the regularization on the inter-class regularization will be beneficial or not. For example, the authors can use some diversity regularization on sphere to push away classifiers from different super classes. A potential regularization for this is [
Learning towards Minimum Hyperspherical Energy, Neurips 2018]. I want to note that it is a suggestion for the paper rather than a weakness.

To summarize, I think the paper proposes a very interesting and potentially widely useful method to incorporate the hierachical label structure to train neural networks. Currently, I feel postive to accept this paper, and I am sitting between 6 and 7 (I give a 6 for now). I will consider to increase my score if the authors well address the concerns.
review_title: Interesting idea with some concerns

final_score: rating: 5: Marginally below acceptance threshold
review_content: The idea of introducing class hierarchy as a regularization into deep networks seems to be novel. 

The following comments could be relevant:

1. The reviewer finds Section 2 is not easy to follow:
- Authors may consider to give more specific definitions to terms such as, classifier, separators, etc. For example, in (2), Wp and Wpi are called classifiers. What are they, hyperplanes? 
- In Definition 1, authors may like to give some early examples about P and L. Otherwise, it is not easy to interpret the matrix H.
- Authors may consider to use a different notation for Delta in (8), as Delta may remind an operator on H in (9).

2. In (9), do we require or observe deltas in the same subtree roughly the same direction? 

3. In Section 3, it is claimed no hyperparameters are added. However, it seems that, initial radius R0, radius decay parameter, even how to organize classes may all be considered as additional hyperparameters.

4. In reality, it can be non-trivial, or even impossible,  to define mutual exclusive class partitions to form the required class tree in Figure 1. Authors may discuss how different class hierarchy adopted affects the classification accuracy, e.g., in Table 2.  
review_title: This paper proposes a technique to classify hierarchically organized classes.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Section 1: The third paragraph is confusing. It is not clear why Euclidean distance is not sufficient for learning with such a hierarchical regularization. Can't the parent class mean just the mean of all its children?

Section 1: From the second last paragraph, it reads that the paper does not train the neural network, but adopts a pre-trained off-the-shelf network and works on its last layer. While this makes the method simple and compatible to other network architectures, is there a reason not to train an end-to-end network with the proposed technique? However, in Section 4.1, the paper says all networks are trained from scratch. Does the training of the whole model follow end-to-end training or stage-wise training?

Section 2.2: How to define "separator"? "Separator" does not seem like a formal word. Moreover, the sentence is confusing, "the parameter of the classifiers that identify dog’s breed should also be similar". If similar, how can they differentiate dog breeds?

Section 3.1: Is radius decay a new method proposed in the paper? If so, what is the rationale behind this design? If not, are there any related work adopting this method? Why not use other linear decay methods, e.g., $R_p=R_0*|p|$?

Eq.(16) implies that the whole model (backbone and the proposed spherical fully-connected layer) is end-to-end optimized. What is the optimization method used for learning other layers?

Moreover, the paper says "the most direct way to optimize over a sphere is to normalize the columns of ∆ by their norm after each iteration. However, this method has no convergence guarantee, and requires a modification in the optimization algorithm". To construct the spherical fully-connected layer, isn't it equivalent to learn a normal fully-connected layer followed up by L2-normalization and a scaling operation?

Even though the authors provide C.2, C.3 and plain files on the prepared hierarchies in the datasets, it is disappointing that the paper does not present the hierarchical structure in a nice way. Perhaps a visualization on the class labels w.r.t the hierarchy serves the paper better?

Section 4.1: Tiny-ImageNet dataset seems to have lower resolution of images (64x64). When authors say "input size...224x224", do the authors mean that Tiny-ImageNet images are resized from 64x64 to 224x224? 

Section 4.1: The paper trains all networks from scratch by explaining "Dogs and Tiny-ImNet are parts of ImageNet". Does it mean that images from datasets Dogs and Tiny-Imagenet are part of ImageNet? Or Does it mean that classes in the two datasets are included in the set of ImageNet classes?

Section 4.1: How to define "plain networks"? The paper uses two networks ResNet and DenseNet, then what is "plain networks"? Once the authors state that the parameters are "probably sub-optimal for our proposed methods", it also implies that the parameters may be even more sub-optimal to the compared methods?

Section 4.2.1: When the paper claims "high efficiency of our approach", it does not justify the "efficiency part". How to tell if the training or inference efficiency is higher than other methods? 

Figure 2 right depicts Riemannian gradient and "projected gradient", but the paper does not formally compare them. It is not clear which one may be better than the other? For learning, gradient guides the direction to update parameters, but the scale in the update also matters. Is there a discussion on which one may be more efficient (or compute time) during a training iteration? Section 4.2 explicitly notes the performance difference between the two methods. The paper should discuss this further for better understanding. Moreover, given that the two methods are so different, setting the same learning rate schedule (cf. Section 4.1) is not sensible, because they can perform quite differently with different learning rates.

The paper does not discuss how the proposed method may work if classes do not follow a tree hierarchy, although C.3 talks a bit in the context of Tiny-ImageNet. As the paper focuses on a generic regularization based on hierarchical information, it may also need to discuss how this can be applied in multi-label classification problems.


---------------------------------
post-rebuttal
---------------------------------
I appreciate that authors have provided rebuttal that addresses many of my questions, though I'd like to maintain my initial rating due to the following comments. I think this paper is at the borderline.

In terms of explaining why "Euclidean distance is not sufficient for learning such a hierarchical regularization", I don't find the illustration example in Section 2.3 intuitive or concrete. I don't think Eq3 adds much as the paper does not explain further. Perhaps the confusion is from that the paper does not explicitly explain what "optimal classifier" mean in terms of Eq3.

The authors only say "those parameters and schedule were optimized for SGD on plain networks, probably sub-optimal for our proposed methods.” It is not clear whether other methods suffer severely from the choice of learning rate and scheduler. As far as I know, SGD is sensitive to the initial learning rate. So I am worried that setting the same learning rate is not fair to comparing different models that have different structures.

From the updated paper, I find the blue line in Page-2 confusing. It is not clear about the logic: why diversity reduces over-fitting. (Xie et al. 2017) studies this point with a complete paper. But the way that authors simply put it is quite unclear how this statement is related in the context.

Visualization is interesting to look at. But it should be better analyzed. For example, visually all methods produce similar tSNE visuals in Figure 5. But are there any essential difference?
review_title: misleading statements and justifications should be addressed

Meta-review: This paper introduces a method for hierarchical classification with deep networks. The idea is interesting, and as far as I know novel: namely, the authors add a regularizer to the last layer in order to enforce a hierarchical structure onto the classifiers. The idea of placing spheres (with a fixed radius) around each classifier and forcing the child-classifiers to lie on these spheres is quite clever. 
The reviewers have pointed out some concerns with this paper. Some had to do with terminology (which the authors should fix but which is no big deal), but the main weakness are the experimental results and the ablation study. The reviewers were not convinced that the optimization in the Euclidean space wouldn't be sufficient. A more thorough ablation study could help here. 

This is the kind of paper that I really want to see published eventually, but right now isn't quite ready yet. If you make one more iteration (in particular adding a stronger ablation study) it should be a strong submission to the next conference. Good luck!
Decision: Reject