Deep Hierarchical-Hyperspherical Learning (DH^2L)
Authors: Youngsung Kim
Jae-Joon Han
Venue:  (2020)

Abstract: Regularization is known to be an inexpensive and reasonable solution to alleviate over-fitting problems of inference models, including deep neural networks. In this paper, we propose a hierarchical regularization which preserves the semantic structure of a sample distribution. At the same time, this regularization promotes diversity by imposing distance between parameter vectors enlarged within semantic structures. To generate evenly distributed parameters, we constrain them to lie on \emph{hierarchical hyperspheres}. Evenly distributed parameters are considered to be less redundant. To define hierarchical parameter space, we propose to reformulate the topology space with multiple hypersphere space. On each hypersphere space, the projection parameter is defined by two individual parameters. Since maximizing groupwise pairwise distance between points on hypersphere is nontrivial (generalized Thomson problem), we propose a new discrete metric integrated with continuous angle metric. Extensive experiments on publicly available datasets (CIFAR-10, CIFAR-100, CUB200-2011, and Stanford Cars), our proposed method shows improved generalization performance,  especially when the number of super-classes is larger.

Under review as a conference paper at ICLR 2020
HIRRARCHHCA--HYPERSPHRRCCC LEARNING (DH'L)
Anonymous authors Paper under double-blind review
ABSTRACT
Regularization is known to be an inexpensive and reasonable solution to alleviate the overfitting problem arising in inference models, including deep neural net- works. In this paper, we propose hierarchical regularization which preserves the semantic structure of a sample distribution. At the same time, this regularization promotes diversity by imposing a distance between parameter vectors enlarged within semantic structures. To generate evenly distributed parameters which is considered less redundant, we constrain them to lie on hierarchical hyperspheres. To define a hierarchical parameter space, we propose to reformulate the topology space with multiple hyperspheres. On each hypersphere, projection is parame- terized by two individual parameters. Since maximizing a groupwise and pairwise distance between points on hypersphere is nontrivial (generalized Thomson prob- lem), we propose a new discrete metric integrated with a continuous metric. The proposed method shows improved generalization performance on extensive exper- iments using publicly available datasets (CIFAR-10, CIFAR-IOM R-00, CUB200-201, and Stanford Cars), especially when the number of super-classes is large.
INTRODUCTION
Diversity promoting learning has been widely adopted via enlarging pairwise distances (Xie al.) 2018 2017a Liu et al., 2018). increasing orthogonality (Xie et al.) 2018), reducing covariance be- Iween parameters (Xie et al. 20176), reducing correlation on feature (Cogswell al., 2016) to improve generalization performance. Among them, diversity promoting regularization (Xie et al.) bb Liu al., 2018) by enforcing large diversity between projection parameters achieves reasonable performance without modifying the model structure. While optimizing the objective function with a covariance matrix in (Xie et al. 2017b 2018) is nontrivial, the diversity promoting regularization via minimizing energy of parameters of deep neural networks has been proposed (Liu) cct al., 2018) in a simpler way. By minimizing a pairwise distance between parameters on a hyper- sphere with the known metric, they achieved the improved generalization performance.
Following an efficient regularization on the hypersphere, we explore further this direction with three main concepts (hierarchical and hyperspherical learning with discrete metrics).
1) Why hierarchical learning? Hierarchical inference explains human intelligence. In (Kurzweil) 2013). it states that "the neocortex contains about 300 million very general pattern recognizers, arranged in a hierarchy". Applying the hierarchy of multiple classes based on semantic taxonomy natural choice to devise machine intelligence. Effectiveness of the hierarchical learning can be found in (Verma et al., 2012).
2) Why hyperspherical learning? Hypersphere is the set of points at the equidistance (radius) from given point (centroid) in a certain dimensional space. Due to the denominator in the unit-length nor- malization ( w || w E IR.+)). the angular distance defined on the hypersphere converges when the magnitude of w goes infinity while Euclidean distance goes infinity. Due to this bounded property, hierarchical structure with multiple separated hyperspheres can be defined.
3) Why discrete metric learning? If vector points form discontinuous series with discrete represen- tation (e.g. multi-dimensional binary temaary), they isolated from each other with certain margin. This property may fit with a discooneecedd/rruppviii manifold space problem. We note that making points to be equidistributed where pairwise distance is maximized is nontrivial task. As

---

Under review as a conference paper at ICLR 2020
West
(0)
WaayA Wai?
(d)
Weat
Weajt
Figure I: Multiple (hyper)spheres as quotient spaces of topology space on Euclidcan space might be found by gluing process with identifying points. Those separated hyperspheres assumed to be under the quotient space conditions (Tu 2010). Within individual (hyper)sphere, projection parameters in deep neural networks preserve hierarchical structure. The space can formed in series: (a), (b), (c), and (d)
points in a discrete metric space are finite and isolated, search efforts satisfy those constraints can be reduced.
In this paper, propose to apply hierarchical structure to parameter regularization the multiple groupwise hyperspheres. In order to find an appropriate metric in this space, we explore a discrete angular metric. We examine the proposed method extensive experimental setups in terms of datasets and deep network models.
2 MULTIPLE SEPARATED HYPERSPHERES
Samples observed from the real world may be on disconnected manifolds. In other words, the disjoint union of those manifolds can generate the global manifold (Lee 2000). In this section, we decompose the one space into multiple spaces (manifolds) and re-define the spaces in hierarchical point of view.
2.1 DISCONNECTED MANIFOLD VIA EQUIVALENT RELATIONS
Since it is not suitable measure pairwise distance between high dimensional vectors which have the hicrarchical structure in the same space, we construct another identification space where sub- spaces (sub-manifolds) are isolated (via equivalence relation (Tu 2010)). Denote d-sphere Sd to be the set of points that satisfies ş = fw E Rd+1 ||w|| II. We construct multiple separated hyperspheres using multiple identifying relations. In Figure [ we use a center parameter and surface parameter define projection parameter vector w which constitutes hypershpere space.
2.2 PRIOR DISTRIBUTION AND REGULARIZATION
To make parameter vectors uniformly distributed on the unit hypersphere, parameters are sampled from the Gaussian normal distribution (Muller 1959 Harman & Lacko 2010). This because the normal distribution is spherically symmetric (Muleer 1959). In Bayesian point of view, neural net- works with Gaussian priors are known induce norm regularization |Vldimirova al. 2019). From two evidences, we know that enforcing parameters to have the Gaussian prior is important for hyperspherical learning neural networks. Note that a parameter which calculated from the difference arithmetic operation with two parameters the normal Gaussian distribution is on the normal difference distribution.
3 METHOD
In deep neural networks, the objective function J with regularization R in addition to a loss L, JR(w) L(x, W) XR(W), is optimized find the optimal W having near minimum loss L, arg minw JR(n.W), where æ E de denotes an input vector, W -W, E Rd-1xx; W. = (w; Rd.-).j 1, 1,....c,, = ,,...,L} denotes a set of parameter matrices (i.e. neurons/kernels), L denotes the number of layers, and À 0 is to control the degree of the regularization. For a classification task, the cross entropy loss is used for the loss function We propose a new regularization formulation R preserving hierarchical parameter structure in Section
2

---

Under review as a conference paper at ICLR 2020
(d)
Figure 2: A series levelwise (!) spheres with a radius (ri) is shown: (a) radius an oveall area converges T-x (= Li rao"; the sum of radius series) as goes to infinity where I denotes level, is their initial radius, and 6 is the ratio between radiuses (71, r1-1), - < 1. (b) The radius of the overall area is bounded to the initial radius ro of spheres. This bears a resemblance to the process of repeat of Hypersphere packing which arranges non-overlapping spheres within containing space. (c) 2-sphere is defined following (b) which appropriate model hierarchical structure. This can be generalized with hypersphere (8", 3) in higher dimensional space.
3.1 REGULARIZATION FOR HIERARCHICAL AND HYPERSPHERICAL HYPOTHESES
Denote w a parameter vector (called a projection parameter, an element W at a certain layer) to transform given input into embedding space defined in a Euclidean metric space: æ E R + w² æ E IR. By the definition of unit-length projection Tel
an a new parameter vector v can be defined
on d-sphere: $d = {u E IR++1 || = 1} where || denotes (~-oorm and a center is zero. In other word, the parameter v can be defined by center parameter vector E Rd+1 and surface parameter vector w, E IR++1 using an arithmetic operation: v =- w. w.. We define the d-sphere with the center and surface parameters: S {w, w. E R l|w, w 1}. For notation simplicity, we use w instead of v hereafter. While we consider radius equals to for simplicity, the parameter vector can have radius 0.
3.1.1 HIERARCHICAL PARAMETERIZATION VIA LEVELWISE AND GROUPWISE PARAIISEEPAAMEEEE PARAMETERS
We assume that the hierarchical structure consists of a levelwise structure with notation (1) and groupwise structure with a notation g below. Through defining these structures serially, we explain hierarchical parameters.
Levelwise structure The projection parameter an E Rd+1 on sd can be defined with a levelwise notation (l) follows,
w w w
where gd ! denotes the 1-th level d-sphere centered (1) in higher dimensional space than that shown in Figure 2
In this paper, define the parameter
In levelwise setting, w! and ! at the current level (1) are additively represented based
ter parameter of the previous level (I - 1): w!! (!-1) + (1 + (!)) where w. ((-1)
Σ (
is the accumulated and (! denotes connecting parameter from w (l-1) to w!!) By denoting
( as (!,-11), the center vector at the /-th level is defined as, w == w ((1-1)
+w ((-1) and the
((-1)
surface vector as w. = w (I.--) + (I-1) Both the center parameter and the surface parameter at
the current level are based the center parameter at the previous leve["] Hence, Eq. | is equivalent
to
w (I,--) w (!.--)))
Note that center parameter at the (I - 1)-th level.
superscript (t,
to denote parameters the l-th level connected from the
With a group notation ge. the center parameter in Eq can be rephrased as (I1-1) on gd ()-11) which denotes d-sphere of gl group at the I-th level. Each group constitutes We.ge
Groupwise structure
a group set at the 1-th level ( = {gx |
1=1 where g(d) C GO) G()) denotes batch group set at
the I-th level, and denotes the cardinality. The group set (()) at the 1-th level is conditioned on
As not every sample has a child sample, it might be more reasonahle to branch from representative param- center parameter rather than from individual projection parameters.
3

---

Under review as a conference paper at ICLR 2020
a group set at the (Z - 1)-th level, g = (sw 199 199h-1 1
== where - _ G(-1). By their group-
wise relation over levels, an adjacency indication plu-i (IG z00) {0, 13 1elle---ppee (x|e is given. Hence, the i-th parameter vector on Sd
(12-11) is defined as follows,
(I,-1)
w (1,-1), cigh
(3)
where (w (1-111)
s.g Wagg wagge-))
(1-11)) Vi is calculated based center vector wang (1-1) at the (1 - 1)-th leve
i 1,... [ and [gkl denotes the number of surface parameter vectors in the group gk-
3.1.2 HIBRARCHICAL REGULARIZATION
In this section, we define a regularization term using the hierarchical parameters defined above. A
set of parameters (Wit Wi.gh (1,1-1)
(-11) E W Vgk, Vgu where W (1.--11
{ aigu. Ti=1
(1,1-11 1) Isve
s.g.
an optimizing target of hierarchical regularization follows:
R(W)= = Σ (W
(1.--1),, 1)) + Ci(wwit (1-11) 1-11); p(l.t())
(4)
s,9
c.gg
where R2 works on individual spheres sd (.11) A > O. and C aims to apply geometry-aware con-
straints across spheres. R₁ consists of two terms for regularization: 1) Ri,p for projection parameters
in the same group Is on Sd (1111) c.sk
and 2) R1,e for center parameters across groups in the same level
on gd w c.al
(-1))
Rí RI( (LI-1)))
wink
((ll-1))= = R.,((WWli
8i
((1-1))) + Ru.c(w((ll
(5)
where
R1 ((--1) w (oll-11):==
Σ
2
Σ
d(w (llllt-)) medd
w (1-1)))
(6)
cigl
G
Gz -
(
{itjEgn)
Ri.c(u((b!!)) 0.0k
d(w
(()-1)))
C(C 1)
where w/ (1,1-1 = w w We.gr gu.t GR l{2 $j jEE g.)l. C Hg: * 9j E g and d(...) denotes a distance metric between parameters. The distance metric is defined in Section 3.2 We explain regularization terms for- mulated using these parameters and pairwise distances further in Figure 4 of Appendix. When a minibatch of inputs (mx: a set of inputs {æ,} ) is given, the regularization term becomes: B(R(W)) m= L R(W; m.))
(11-1) (w (w 1I1-1) is similarly defined,, = Hgg E gdH
9ed
The constraint term help construct geometry-aware relational parameters between different spheres on the same level and on the across levels. Multiple constraints are defined as Ct == Σx A&Ci,K. where Cr,& is the k-th constraint between parameters at the (-th and the (1- 1) (1-1) level, and A is a Lagrange multiplier. We propose three constraints (defined in Appendix) in a geometric point of view.
3.2 DISCRETE AND CONTINUOUS ANGULAR DISTANCE METRIC
Discrete metric is good fit for the above groupwise definition. We expect that points projected by parameters in discrete metric space are isolated each other. In Figure g we show that discrete distances of parameter pairs have different values while their continuous distances are same. Hence, maxinization of discrete distances of parameter pairs could help parameters distributed isolatedly and diversely.
2 This can be replaced with a probability model.
The center vector (!) indicates representative vector the group ga at the I-th level and is equivalent
to a mean vector of w s. Vi: (( ..) = 1 laxl ( s. The center vector for the group Sl can be (f,-1)
lokl
determined by a parameter vector the (1 1) th level using adjust factor (c, lel < 1): wo
(l-1) where w (1-1) Ed (-1)) aard c.
4

---

Under review as a conference paper at ICLR 2020
Using parameter vectors wy and w in R++,, we define discrete dis- tance metric using a sign function as follows:
[0,0m,
(0) =0
Dn 1 d
Σ
sign(w; (k:) (ign(w;(k));
(8)
(M, =1
(ww =
-1 S Dr S 1, and w (wr(k) vk 1, ......d d + 1 E [ This is normalized version of Hamming distance. For a ternary discrete, sign(x) E f, 1] is used. In order to consider the discrete distance
where sign(x) = {
=20
-1, otherwise
While the pair- wise angular distances between a pair vectors {wi, w22 and {wz, wy} are the same, the pairwise discrete product distances D2 between vectors are dif- ferent. To diversify a pa- rameter space. the space with sign could be effec- to recognize their dif- ference.
Figure 3:
angular dis- tance within [0, 1), normalized one is defined as Dho1 == =Dyt1 where S Phot S 1. The angular distance based on the above product can be rephrased as pp. Daot where 0 S 0p, S 1.
As the discrete distance could underestimate to approximate the model distribution, we merge the above discrete distance metric with contin- uous angular distance metric (0 = # arccos w;: ),0 S 0 S 1)
||m0; |
into a single metric. We simply use the definition of Pythagorean means which consist of the arithmetic mean (AM), the geometric mean (GM),
and the harmonic mean (HM). Pythagorean means using a pair of angular distances are defined as follows:
DAM := PD. +o
DOM == 0D,0, DAM == 49p, PDy +9"
(9)
In the angular distance using a pair of angles {0D,, 0). reversed form 1 Piep.. .) is adopted to maximize an angle in the optimization formulation by minimization instead (-) where 8 =
., which is used in Thomson problem that utilizes s-energy (Brauchart & Grabner 2015). The
cosine similarity using the pair of angles can be defined as follows:
19 @ +0 r), Deas(GM) cos (0p,, Deos(HM) cos
Doos(AM) cos
0p, +9 T
(10)
then the cosine similarity functions are normalized with cos(-)+1 to have value within a range [0, 1]. Finally, Pythagorean means of cosine similarities can be calculated as follows:
DAM Dp 0-00 +-+
DGM
(cos 0p, x00) 0n+1) DHM
(oos fD x #+1) +x+
oos 9D, +oos 0+2 (11)
Metric functions defined in ( (1)), and 11 satisfy metric conditions: non-negativity, symmetry. and triangle inequality. The distance using the above metric function between any two parameter points is bounded, because the hypersphere is a compact manifold.
3.3 GRADIENTS AND BACKPROPAGATION
As the sign function is not differentiable the value O, adopt alternative backpropagation func- tion. We adopt straight-through estimator (STE) (Bengio et al. 2013)) in the backward path of the neural networks for the sign function in the discrete metric. The derivative of the sign function is substituted with 1 wis1 in the backward pass, known as the saturated STE. As the derivative of arccos( (-==z>) is undefined at the value = +1, we apply clamping to the cosine function to
have I E [-0.99, 0.99 where x = cos() and 0 S 0 < 1.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets We conduct experiments using four publicly available datasets including small size im- ages (CIFAR-10 and CIFAR-100) and large size images (CUB200-2011 (Wah et al., 201T) and
*On the other hand, the angle can be considered as cosine similarity directly, Dr == 0r, n. as to get the angular distance, needs an arccosine function 0D, 1 arecos Dr. summary, for the angle distance O either "Dogi" " "D'
arccos D,." " where 0 Dom S L, can be adopted.
$ In < 0 < 1, the angle and its cosine value show an inverse relationship:0 < 0 < + 2 cos 0r 2 -1. f {0.99. 1 1.200.9. T. 0.99 10.99
5

---

Under review as a conference paper at ICLR 2020
Stanford- -aaff (Krause et al., 2013b), shortly CUB200 and Cars hereafter respectively). CIFAR-10 dataset is used to validate effectiveness of the proposed metric without hierarchical information. Except CIFAR-10 10, regularization on three datasets. In 7 Appendix, statistics of datasets in detail is provided.
two-level hierarchy pairs {parent, child} for the proposed hierarchical
Deep neural network models and training setting We used the deep residual neural network (resner) (He et al.] (216)) with different sizes of parameters (weights) dependent on an image size of datasets. For small size input (32 x 32 pixels) of CIFAR-10 and CIFAR-100, we used resner with smaller number of parameters (light models, resnet-20 (0.29M) and resnet- 1 10 (1.73 M)) so as not to have redundant parameters leading to over-fitting. For a fine-grained input (224 224 pixels) of CUB200 and Cars, the original resnet with larger number of parameters (heavy models" Resnet-18 (11.28M) and Resnet-50 (23.91M)) is used.
In training of the deep neural network with the hierarchical regularization, assume that the global hierarchical structure is not given. Instead, stochastic or partial hierarchical structure is shown within each mini-batch by hierarchy pairs. bini-batches, 512 for light models and 256 for heavy models, are used in the SGD optimizer. We applied the hierarchical regularization in the FC layer of the resnet. Even though SGD is known an unbiased estimation, stochastic hierarchical structure could affect the overall approximation performance upon the class distribution within the mini- batch. Settings in more detail are provided in Appendix.
Baseline setting In terms of the regularization, we used three reference settings, denoted 'base- line', 'baseline+/2", and 'E' for a reference. First, in 'baseline', we examined neural networks without regularization. Then, in 'baseline+!"' we added 12-norm minimization based regulariza-
tion (which is equivalent to a weight decay in SGD setting (Zhang et al. 2019)) via X Ex w l|. where wg E W and \S 0 to "baseline". Lastly, in 'E', energy mimiizationn (Luu et al. 2018 diversity promoting regularization without hierarchical information, via de Eiey d(w; wi) where .(.,.) is pairwise distance and X > 0, is added to baseline+?2. When the proposed hierar- chical regularization (Eq. (5) applied to the FC layer, we denoted as 'H', Hence, the reg- ularization is incrementally adopted to the baseline: 1'+'E'+'H. In terms of distance metric, we compared the proposed metric with other continuous metrics such as Euclidean distance with the unit- unit-length projection ('N-euclidean2", Ex4y |:::
w; 2 angular distance ('angular2',
Σizj arccos Tie, 1|
where '2' is from Riesz s-energy shown higher accuracy, and cosine we w.
similarity ('cosine", Σijs T0 [w
4.2 RESULTS
Low resolution object classification In Table rics with energy minimization regularization without the hierarchical regularization along weight decay ((2-norm) using CIFAR-10 and resnet-20. The 'baseline' method is examined without (x in Tabll 1 with (c in Table TD the weight decay (12). The discrete angular metric based regulariza- tion (Dier (ternary code), Dea (binary code), Deos (IIM), and DiHM.... on the baseline can improve the generalization performance in term of test accuracy compared to the other continuous metrics such as N-euclidean2, angular2, and cosine. Due to the unit-length projection in N-euclidean2, their performance comparable to that other angular metrics. D' denotes an arccosine adopted discrete distance used as D (see footnote 4 in Section 3.2p. Regularization methods are applied over all lavers except Batch Normalization (BN) layers. As explained in (van Laarhoven, 2017 Zhang et al.) 2019), 1--norm minimization based regularization (weight decay) showed much improvement due to their effective learning rate in SGD.
we showed test accuracy (%) using different met-
As shown in Table 2 the regularization with proposed metrics shows better accuracy performance than that of the baseline on both resnet-20 and resnet-1 using CIFAR-100 dataset. Comparing to the baselinet!? with energy minimization 'E' regularization, metrics such Dter, Dens(AM). and D' (oos(HM) showed better performance than other metrics. If the proposed hierarchical "H' regular-
whhic is available at the pytorch library we
8 Eney (2- 2 TT ||
'This is because that hicrarchy information is not included in CIFAR-10.

---

Under review as a conference paper at ICLR 2020
Table 2: Test accuracy (%) using different met- rics on energy minimization ('E') (E')ane and hierarchical
Test accuracy (%) using different met- rics on energy minimization without hierarchy information along -o-norm minimization (weight decay) CIFAR-I0 10 and resnet-20.
Table 1:
regularization ('H') CIFAR-100: 'E' "E+H' respectively.
metric \ db
resnct-20
resnet-110
metric \ 1
x
baseline
63.86
62.02
baseline+/
68.03
72.90
baseline
90.34
92.21
N-euclidean2
90.93
92.35
N-cuclidean2
67.59 / 68.65
73.95 73.96
angular2
90.47
92.38
angular2
67.83 / 67.76
74,40/ / 73.89
cosine
90.53
92,40
cosine
68.11 / 68.45
73.37 73.37
Dr
90.67
92,48
68.4 68.68866 68
D
90.67
92.48
D
68.52 68.69
73.97 74.26
D DOoE(TMM)
90.84
92.93
DAM
68.58 / 68.86
73/43/73.50
Dooe(AM)
68.58 68
73.14 73.65
DeMoon
90.94
92.69
67.57 / 68.36
73.14/73.72
Table 3: Test accuracy (%) using different met-
Doca(HM)
68.62 68.65 73.07 73.65
rics on energy minimization ('E') and hierarchical regularization ("H") on CUB200: 'E' 'E+H' re- spectively.
Table 4: Test accuracy (%) using different met-
rics with energy minimization ('E') and hierarchi- cal regularization ("H") on Cars: 'E' 'E+H' re- spectively.
metric { db
Resnet-18
Resnet-50
baseline
72.17
74.21
b
72.29
74.05
metric \\ db
Resnet-18
Resnet-50
N-euclidean2
72.61775.99
13,49 76.14
baseline
85.10
87.99
angular2
72.43 76.11
73.55 76.66
baseline+/"
85.58
87.92
cosine
75.12
73.26 / 76.85
N-euclidean2
85,48 / 83.56
87.967
72.58 75.99
73.57 76.37
angular2
85.11 / 85.13
88.34 87.85
D
76.25
73.57 / 76.99
cosine
85.57 / 85.73
88.01 88.8868
D DAM
73.04/76.02
73.88 75.99
D
85.35 85.99
85.35 88.07
DAMgos
76..14
73.59 77.32
D
86.22 / 85,99
88.32 / 88.14
D'MooR
72.28/75..
72.42 74.12
DAM
85.66 85,66
88.397 / 88.
DGM
72.90 76.35
74.16 75.30
DMMon
85.52 86.05
87.92 88.57
DYMaoon
72.55 / 76.11
74.64 / 76.94
DAM
85.76 86.43
88.07 87.96
D (Hce(H
72.55 76.32
72.86 76.56
Doca(AM)
85.54 / 85.52
88.22 / 88.13
ization is applied, the generalization performance is further improved in most metric cases both resnet-20 and 110. As the binary metric shows hetter performance than that ternary, we adopt binary discretization for proposed angular metries (D., Deus(e), and in experiments.
Fine-grained visual categorization In this experiment, we used two large size image datasets with fine-grained category. One with birds (CUB200) and another with cars (Cars) that focus on single species of objects. A hierarchy of CUB200 is defined by the academic expert on birds whereas that of Cars is categorized manually based on a name of cars by a non-cxpert. Morcover, the rate between the number of superclass (parent) per subclass of CUB200 (0.35) is much larger than that of Cars (0.0459) (as shown in Table 7 at Appendix). That rate of Cars is even smaller than that of CIFAR-100.
As shown in Table 3 the proposed hierarchical regularization significantly improved the test accu- racy all metrics on both Resnet- 18 and Resnet-50. Compared to CUB200, shown in Table the improvement of the proposed method is not that significant for Cars dataset. This is because CUB200 dataset has not only better hierarchical categorization by the expert but also more diversi- tified pairs between the superclass and the subclass.
Ablation study In Table 5] we showed how metries affect the generalization performance using resnet-20 and CIFAR- 100 dataset. The proposed method with different proposed metrics showed significantly improved performance compared to the baseline+?2. Individual means (AM, GM, and HM) show different improvement patterns.
In Table 6 we showed how heterogeneous metrics on convolutional (conv.) layers (with the en- ergy minimization, 'E') and fully connected (FC) layers (with the hierarchical 'H' regularization, 'H') affect the generalization performance using resnet-20 and CIFAR- 100 dataset. The cases ap- plying hierarchical regularization showed better performance than that the baseline applying only energy minimization regularization. In this experiment, a combination GM and HM shows better improvement than that other combinations.
7

---

Under review as a conference paper at ICLR 2020
Table 5: Test accuracy (%) using different met- rics on energy minimization ('E') and hierarchical regularization ('H' on CIFAR-100 and resnet-20.
Test accuracy (%) using heterogeneous metrics at Conv, layers and FC layer respec- tively, on energy minimization ('E") and hier- archical regularization ('H) on CIFAR-100 and resnet-20.
Table 6:
metric
bascline+!
68.03
DAM
68.66
metries (in cony,, in FC)
D GM
68.70
baseline+!
68.03
Dam
68.80
(DeM, null)
68.22
D DAMoon
69.24
(Das, DAM)
68.58
D DGMoon
68.55
D DrIMoon
68.77
(Dou. DαM)
68.62
(DaM. DrN
69.04
D Aooo
68.96
(Dam, DAM)
68.62
D DeMoon
69.00
(Des. DeM)
68.65
D
68.83
(Dou. DHM)
68.70
5 RELATED WORKS
Promoting of diversity in an embedding space or model parameters is an widely adopted strategy in machine learning related area to improve the generalization performance (Cogswell et al. 2016
Yang et al. 2019 Li et al. 2012: Ratzlaff & Fuxin 2019: Xie et al.] 2017b] 2018: 201 7aj Liu et al, 2018). This diversity based strategy
be applied in variety of levels such in feature level
(Cogswell et all 2016 Xie et al.
2018), in a projection parameter level (Xie et al. 2017 Liu et al.,
2018), in model ensemble level (Zhou et al. 2018 Ratzlaff & Fuxin 2019). in latent space model
level Ratzlaff & Fuxin 2019 Liu et al. 2018), and in a generative model level (Yang et al. 2019
Ratzlaff & Fuxin 2019). Among these approaches, some regularization methods require additional
optimization effort. Enlarging pairwise distance between features (Xie et al., (2018) requires com- putational efforts due their covariance matrix. (Xie et al., 2017b), unit-eigenvalue
utilized via singular value decompasition. To optimize direction and magnitude of the parameter vector alternatively, an alternating direction method of multipliers (ADMM) (Xie et al. 2017a) used.
In terms of learning on hypersphere, (Liu et al.) 2017) proposed that hyperspherical convolution (SphereCov) replaces the traditional inner-product based convolution in order to conduct learning angular representation. On hypersphere, regularization via promoting diversity of parameters has been proposed (Liu et al. 2018) based on Minimum Hyperspherical Energy.
In terms of hierarchical learning, (Nickel & Kicla 2017: Ganea et al. 2018) proposed to apply hy- perbolic space to embed the data in deep neural networks. They showed that learning representations in the hyperbolic space is effective the data preserving a latent hierarchy compared to in the Eu- clidean space. Different from these works which are aimed for representation learning, our proposed method is focused on regularization learning using explicit (predefined) hierarchy information.
6 CONCLUSION
We proposed the regularization method, which maximizes a pairwise distance between parameters preserving a hierarchical structure. To define hierarchical parameter space, we reformulated the topology space with multiple hyperspheres. In each hypersphere, projection is parameterized by the surface parameter with the center parameter. By imposing maximum distance between hierarchical parameters, diversity of parameters preserving semantic structure was promoted. For the distance metric in multiple separated spaces, proposed a discrete metric integrated with continuous metric. Extensive experiments using publicly available datasets (CIFAR-10, CIFAR- 100, CUB200-2011, and Stanford Cars), the deep neural network with our proposed regularization method showed superior classification performance, especially when the number of super-classes large. For further exploration future, our proposed method can be combined with hierarchical representation learning such as hyperbolic (or Poincaré) embeddings (Nickel & Kiela 2017 Ganea
al. 2018).
REFERENCES
Y. Bengio, Nicholas Lonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv, Aug. 2013.
8

---

Under review as a conference paper at ICLR 2020
Johann S. Brauchart and Peter J. Grabner. Distributing many points spheres. J. Complex. 31(3): 293-326, June 2015.
Michael Cogswell, Faruk Ahmed, Ross B. Girshick, Larry Zitickk and Dhruv Batra. Reducing Overfitting in Deep Networks by Decorrelating Representations. In International Conference on Learning Representations, 2016.
Octavian Ganea, Gary Becigneul, and Thomas Hofmann. Hyperbolic neural networks, In Advances in Neural Information Processing Systems 31, pp. 5345-5355. 2018.
Radoslav Harman and Vladimr Lacko. On decompositional algorithms for uniform sampling from n-spheres and n n-balls. Journal of Multivariate Analysis, 101(10):2297 - 2304, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp- 770-778, 2016.
Jonathan Krause, Jun Deng, Michael Stark, and Li Fei-Fei. Collecting a large-scale dataset of fine- grained cars. In Second Workshop on Fine-Grained Visual Categorization (FGVC2), 2013a.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine- grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR- 13), Sydney, Australia, 2013b. Ddataset available at htttp::/immggnnt...... wwwraagnnssstaanaiin led./internal/car1se
Ray Kurzweil. How to Create a Mind: The Secret of Human Thought Revealed. Penguin Books, New York, NY, USA, 2013.
J.M. Lee. Introduction Topological Manifolds. Graduate texts in mathematics. Springer, 2000.
Nan Li, Yang Yu, and Zhi-Hua Zhou. Diversity regularized ensemble pruning. In Proceedings of the 2012th European Conference on Machine Learning and Knowledge Discovery Databases Volume Part I, ECMLPKDD 12, pp. 330-345, 2012.
Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep hyperspherical learning. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp- 3950-3960, 2017.
Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 6236, USA, 2018. Curran Associates Inc.
Mervin E. Muller. note on method for generating points uniformly on n-dimensional spheres. Commun. ACM, 2(4):19-20, April 1959.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representa- tions. In Advances in Neural Information Processing Systems 30, pp. 1338-6347. 2017.
Neale Ratzlaff and Li Fuxin. HyperGAN: A generative model for diverse, performant neural net- works. Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 5361-5369, 09-15 Jun 2019.
L.W. Tu. An Introduction to Manifolds. Universitext. Springer New York, 2010.
Twan van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint, 106.05070 2017.
N. Verma, D. Mahajan, S. Sellamanickam, and V. Nair. Learning hierarchical similarity metrics. In
2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 2280-2287, June 2012.

---

Under review as a conference paper at ICLR 2020
Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, and Julyan Arbel. Understanding priors in Bayesian neural networks at the unit level. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6458-6467, Long Beach, California, USA, 09- 15 Jun 2019. PMLR.
C. Wah, S. Branson, Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds- 200-2011 Dataset. Technical Report CNS-TR-2011-000 California Institute of Technology, 2011. Dataset available at hhtt::/ /wwp://wwwww//w.w/////// vision. caltech. edw.tteiiedddiiiispeeeedeeeeeeed data/ EUB-2200-0011
Pengtao Xie, Yuntian Deng, Yi Zhou, Abhimanu Kumar, Yaoliang Yu, James Zou, and Eric Xing- Learning latent space models with angular constraints. In Proceedings of the 34th International Conference on Machine Learning, ICML 20/7, Sydney, NSW, Australia, 6-11 August 2017, pp- 3799-3810, 2017a.
Pengtao Xie, Aarti Singh, and Eric Xing. Uncorrelation and evenness: A new diversity-promotiin regularizer. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 3811 1-200. JMLR.org. 2017b.
Pengtao Xie, Wei Wu, Yichen Zhu, and Eric P. Xing. bathogonality-promooti distance metric learning: Convex relaxation and theoretical analysis. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan Stockholm, Sweden, July 10- 15, 2018, pp. 5399-5408, 2018.
Dingdong Yang. Seunghoon Hong. Yunseok Jang, Tiangchen Zhao, and Honglak Lec. Diversity- sensitive conditional generative adversarial networks. In International Conference on Learning Representations, 2019.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. In International Conference on Learning Representations, 2019.
Tianyi Zhou, Shengjie Wang, and Jeff A Bilmes. Diverse ensemble evolution: Curriculum data- model marriage. In Advances in Neural Information Processing Systems 31, pp. 5905-5916. Curran Associates, Inc., 2018.
APPENDIX
Dataset acquisition details CIFAR-100 dataset provides a pair of superclass-subelas labels. In CUB200 dataset, the pairs be extracted from their filename. In Cars dataset, we parsed each subclass label to one of nine superclass vehicle types, such as "Sedan", "SUV", "Van" and etc., following (Krause et al. 2013a
Table 7: Statistics of benchmark datasets
#classes {pa,
#samples
#spper
Dataset
#train
#test
input size
/class
/subclass
CIFAR-I0
[1, 10]
50,000
10,000
32 x 32
5000.00
0. 1000
CIFAR-I00
(20, 100)
50,000
10,000
32
32
500.00
0.2000
CUB200
{70, 200)
5,994
5,794
224
x 224
29.97
0.3500
Cars
{9, 196]
8,144
8,041
224
224
41.55
0.0459
Deep neural network models and training details First, resnet-20 (0.29M) and resnet- 110 (1.73 M), which include a combination of Basic blocks with output channels [16, 30, 64] are adopted for light models. An input dimensionality of the fully connected (FC) layer (a classier) is 64 for both resnet-20 and res Second, heavy models are adopted such as Resnet-18 (1 (11.28M TO and
Dependennt the number of classes and the corresponding center parameters, the size could variate (c.g. 11.42M for CUB200, 11.31M for Cars).
10

---

Under review as a conference paper at ICLR 2020
Resnet-50 (23.91M) which consists of the basic blocks (Resnet- 18) the bottleneck blocks with output chamnels [64, 128, 256, 512] in Conv. layers. An input dimensionality of the FC layer is 512 for 18 and 2048 for Resnet-50.
Networks are optimized with SGD for both light and heavy models: we fixed 1) the weight initializa- tion with Kandom-Seet number '0' in pytorch, learning rate schedule [0.1, 0.01, 0.001]. 3) with momentum 0.9, 4) regularization: 22-norm minimization with Xy = 0.0005, 5) energy minimization 0.1 {1, 5). All the regularization is not applied to the parameters of Batch Normalization (BN) layers, since BN is also regularization function. A bias term in the FC layer is not used. The images in training and test, images are resized to 256 size. The images is cropped with 224 x 224 size at random location in training and at center location in test. Horizontal flipping applied in training. The light models are trained from scratch without the pretrained weights for 300 epochs. The heavy model is trained using pretrained model provided by pytorch library [1] with 100 epochs. The experiments are conduced using GPU "NVIDIA TESLA P40".
de- = {0.1, 13. and hierarchical minimization
W (1,L-)
C,
W 1
c,f (11-1)
W s.cck
dw !
w(())
w ((wel-1) waik
W Bikk
Bikk
gin)
d d(w (
W
a1
w
aum
gikk
(W-) wold-1)
W (LA-1)
w
Sua
d (w (1)
Blm
gm.m
S.HEM
(1111-2
w c.8t
Figure 4: Ri, Ct.pWWll1- (1,
(1.-11)
d(wwlit" wimm
(1.1-1) (1d-11) + d( d(ws..m ggit
(1d-1) (1d-1)) +
Wen Wa,9
d(w gick Wy..
(11-1) (1,1-1)
+ ... corresponds Eq (6) and R1,e (we.g (1 d( d(w.tt))) we.0.
(1d-1) (1.-11) +
d(wess ((ww()111) wo.0
(l.-1)
(1,-1)) )(l-)))... ).... corresponds Eq
Constraints in Eq. C=== Σx AnCi, can be given follows:
1. Constraint (C1): This constraint describes that radius of an inner sphere must be smaller
than that its outer sphere.
((-)) - (1) 2 0 = ((-) - (()) = ||wl (t-1) w w(-") (1-1) - |! - w (1) 2 0.
2. Constraint 2 (C2): This constraint describes that center of an inner sphere must be located
within outer sphere.
((-1) - (m/w (1,1-1) + ((()) 2 0 + (1-1)
(! (!
(1-1) w !1 + +0))
|| ( (!-1)
w (1-2)) (1/w. (!
(I-1) w!
(1)
||+ |/o w::h (1) (l) 2 0.
3. Constraint (Cz): This constraint describes that margin between spheres must be larger
than zero.
E mn! 00d 0.5
1 (1.1-1) [2 2 00 (r(()
|| (n
w
from http:://owwllod.pyyorrrtto Maccce pth, http:///www./..//////oo/ooooooo. 19c8c357. respectively
11

---

Under review as a conference paper at ICLR 2020
21les (1)
(1) 11. | (nn(h-1))(( ((-1--1) sin o² - (r(1-1)
we
where
|120 (1.1 1) 11(2 2cos 0.5
((--) cos
/
12


Reviews:
[{"review_id": "Hyg-cD3ycB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a regularization strategy motivated with principles of hierarchical, hyperspherical and discrete metric learning. Through regularization of as designed in level-wise, group-wise with the hierarchy of network, in their experiments with classification dataset, better performance are achieved with various distance. \n\nPros:\n1: I think the paper is well organized and motivated, the regularization of parameters in deep neural network is one of the center problem for effective learning. \n2: The proposed strategy is effective with their experiments, various datasets and objective metrics are adopted to validate the regularization.  Combination ablation study is sufficient.\n\nCons:\n1: The paper is also related with several popular normalization strategies such as weight normalization/standardization, group/batch normalization. It would be more convincing that some comparison could be performed against these strategies.\n2: There would be better to show its performance using larger dataset such as ImageNet or COCO detection.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hklvx0P-9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe paper tackles the problem of promoting diversity in the weights of deep neural networks. The problem is interesting and useful. The paper argues that hierarchical learning and hyper spherical learning are important in addressing this problem. The paper provides experiments on CIFAR-10 and CIFAR-100 where the improvements of using such regularization is visible but not sufficiently significant.\n\n## Contribution of the paper\n1. The paper proposed a regularization to training neural networks with discrete angular distance metric on the weights.\n2. The paper shows improved performance on CIFAR-10 and CIFAR-100.\n\n## Overall feedback\nI found the paper is well motivated and the proposed approach to be interesting. But I found the experimental validation a bit confusing. The improvments of the proposed approach also seems quite marginal. The contribution of different regularization terms is not understood clearly as well. So I am leaning towards rejection.\n\n## Detailed feedback and questions for rebuttal\n1. The writing could be improved significantly. I had a hard time to find exactly what different regularization terms are, e.g., E, H, L2. The paper could be more clear by clearly stating these regularization equations.\n2. Please capitalize \"eq. (1)\" to \"Eq. 1\".\n3. It seems there are three regularization E, L2 and H. But different tables show different combinations. For example, Table 1 has E and E+l2 while Table 2 has E and E+H and Table 3 has only E+H. Can you provide full results on all datasets on E, E+l2, E+H? Without seeing the full results it is hard to draw any conclusions.\n4. Please correct the text \"resnet-100\" to \"resnet-110\" assuming you are using resnet-110.\n5. It seems E+H improves marginally over E. Can you elaborate the explanation about it?\n6. Why E+l2 improves so much (+2%) on CIFAR10?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eHQGojFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a hierarchical regularization framework based on hierarchical hyperspheres. In particular, the paper tackles the problem of diversity promoting learning. Following (Liu et al., 2018), pairwise distances between parameters on hyperspheres are used in the regularization framework.\nThe topology of the parameter space is reformulated with multiple hypersphere spaces which are each defined by two parameters: the centroid of a sphere and its surface vector. Multiple strategies involving hierarchical hyperspherical structures are proposed in Section 3 (continuous and discrete). \nThe relevance of the proposed method is experimentally demonstrated on different computer vision datasets.\n\n\nI vote for reject for the following reasons:\n- The paper is hard to read in general. Although the method section is understandable, its readability could be improved because each method currently just looks like a succession of equations. The paper also does not really give an intuition of why (or what contexts) one of the proposed regularizers would be better than the others. \n- The reported (test accuracy) scores do not seem significantly better than the l2 baseline: none of the reported scores beats the l2 baseline by at least 1 percent, and it is unclear how that difference is measurable. How many splits/different initalizations were used? Why not give standard deviation over different test splits? etc... Given the fact that the improvements do not seem significant compared to a single baseline, a proper evaluation with standard deviation should be provided.\n- Although the paper cites (Liu et al., 2018) as motivation for their framework, why does the proposed method does not compare to the other related work (i.e. works by Xie)?\n\nAs a side note, the paper seems to motivate the use of multiple spherical spaces to represent hierarchies. Recent work in machine learning has shown the advantage of using hyperbolic geometry [A,B] to represent trees, hence hierarchies. \n\n\n[A] Nickel and Kiela, Poincaré Embeddings for Learning Hierarchical Representations, NIPS 2017\n[B] Ganea et al., Hyperbolic Neural Networks, NeurIPS 2018\n\n\n======= after the rebuttal\n\nI have read carefully the updated manuscript, other reviews and rebuttal.\nMy score does not change since the proposed method does not seem to improve much compared to a simple weight decay (baseline + l2 regularization). The motivation of using the method for a very small improvement is not convincing. The \"well-defined hierarchical information which is categorized by an expert as mentioned in the manuscript\" can also be exploited by hyperbolic representations and should then also be compared (as baseline).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a hierarchical diversity promoting regularizer for neural networks. Experiments are shown with this regularizer applied to the last fully-connected layer of the network, in addition to L2 and energy regularizers on other layers. Reviewers found the paper well-motivated but had concerns on writing/readability of the paper and that it provides only marginal improvements over existing simple regularizers such as L2. I would encourage the authors to look for scenarios where the proposed regularizer can show clear improvements and resubmit to a future venue. 
Decision: Reject