Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: ## Summary
The paper tackles the problem of promoting diversity in the weights of deep neural networks. The problem is interesting and useful. The paper argues that hierarchical learning and hyper spherical learning are important in addressing this problem. The paper provides experiments on CIFAR-10 and CIFAR-100 where the improvements of using such regularization is visible but not sufficiently significant.

## Contribution of the paper
1. The paper proposed a regularization to training neural networks with discrete angular distance metric on the weights.
2. The paper shows improved performance on CIFAR-10 and CIFAR-100.

## Overall feedback
I found the paper is well motivated and the proposed approach to be interesting. But I found the experimental validation a bit confusing. The improvments of the proposed approach also seems quite marginal. The contribution of different regularization terms is not understood clearly as well. So I am leaning towards rejection.

## Detailed feedback and questions for rebuttal
1. The writing could be improved significantly. I had a hard time to find exactly what different regularization terms are, e.g., E, H, L2. The paper could be more clear by clearly stating these regularization equations.
2. Please capitalize "eq. (1)" to "Eq. 1".
3. It seems there are three regularization E, L2 and H. But different tables show different combinations. For example, Table 1 has E and E+l2 while Table 2 has E and E+H and Table 3 has only E+H. Can you provide full results on all datasets on E, E+l2, E+H? Without seeing the full results it is hard to draw any conclusions.
4. Please correct the text "resnet-100" to "resnet-110" assuming you are using resnet-110.
5. It seems E+H improves marginally over E. Can you elaborate the explanation about it?
6. Why E+l2 improves so much (+2%) on CIFAR10?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a regularization strategy motivated with principles of hierarchical, hyperspherical and discrete metric learning. Through regularization of as designed in level-wise, group-wise with the hierarchy of network, in their experiments with classification dataset, better performance are achieved with various distance. 

Pros:
1: I think the paper is well organized and motivated, the regularization of parameters in deep neural network is one of the center problem for effective learning. 
2: The proposed strategy is effective with their experiments, various datasets and objective metrics are adopted to validate the regularization.  Combination ablation study is sufficient.

Cons:
1: The paper is also related with several popular normalization strategies such as weight normalization/standardization, group/batch normalization. It would be more convincing that some comparison could be performed against these strategies.
2: There would be better to show its performance using larger dataset such as ImageNet or COCO detection.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes a hierarchical regularization framework based on hierarchical hyperspheres. In particular, the paper tackles the problem of diversity promoting learning. Following (Liu et al., 2018), pairwise distances between parameters on hyperspheres are used in the regularization framework.
The topology of the parameter space is reformulated with multiple hypersphere spaces which are each defined by two parameters: the centroid of a sphere and its surface vector. Multiple strategies involving hierarchical hyperspherical structures are proposed in Section 3 (continuous and discrete). 
The relevance of the proposed method is experimentally demonstrated on different computer vision datasets.


I vote for reject for the following reasons:
- The paper is hard to read in general. Although the method section is understandable, its readability could be improved because each method currently just looks like a succession of equations. The paper also does not really give an intuition of why (or what contexts) one of the proposed regularizers would be better than the others. 
- The reported (test accuracy) scores do not seem significantly better than the l2 baseline: none of the reported scores beats the l2 baseline by at least 1 percent, and it is unclear how that difference is measurable. How many splits/different initalizations were used? Why not give standard deviation over different test splits? etc... Given the fact that the improvements do not seem significant compared to a single baseline, a proper evaluation with standard deviation should be provided.
- Although the paper cites (Liu et al., 2018) as motivation for their framework, why does the proposed method does not compare to the other related work (i.e. works by Xie)?

As a side note, the paper seems to motivate the use of multiple spherical spaces to represent hierarchies. Recent work in machine learning has shown the advantage of using hyperbolic geometry [A,B] to represent trees, hence hierarchies. 


[A] Nickel and Kiela, Poincaré Embeddings for Learning Hierarchical Representations, NIPS 2017
[B] Ganea et al., Hyperbolic Neural Networks, NeurIPS 2018


======= after the rebuttal

I have read carefully the updated manuscript, other reviews and rebuttal.
My score does not change since the proposed method does not seem to improve much compared to a simple weight decay (baseline + l2 regularization). The motivation of using the method for a very small improvement is not convincing. The "well-defined hierarchical information which is categorized by an expert as mentioned in the manuscript" can also be exploited by hyperbolic representations and should then also be compared (as baseline).
review_title: Official Blind Review #1

Meta-review: The paper proposes a hierarchical diversity promoting regularizer for neural networks. Experiments are shown with this regularizer applied to the last fully-connected layer of the network, in addition to L2 and energy regularizers on other layers. Reviewers found the paper well-motivated but had concerns on writing/readability of the paper and that it provides only marginal improvements over existing simple regularizers such as L2. I would encourage the authors to look for scenarios where the proposed regularizer can show clear improvements and resubmit to a future venue. 
Decision: Reject