Year: 2019

Abstract:
Regularization is known to be an inexpensive and reasonable solution to alleviate the overfitting problem arising in inference models, including deep neural networks. In this paper, we propose hierarchical regularization which preserves the semantic structure of a sample distribution. At the same time, this regularization promotes diversity by imposing a distance between parameter vectors enlarged within semantic structures. To generate evenly distributed parameters which is considered less redundant, we constrain them to lie on hierarchical hyperspheres. To define a hierarchical parameter space, we propose to reformulate the topology space with multiple hyperspheres. On each hypersphere, a projection is parameterized by two individual parameters. Since maximizing a groupwise and pairwise distance between points on hypersphere is nontrivial (generalized Thomson problem), we propose a new discrete metric integrated with a continuous metric. The proposed method shows improved generalization performance on extensive experiments using publicly available datasets (CIFAR-10, CIFAR-100, CUB200-2011, and Stanford Cars), especially when the number of super-classes is large.

heading: 1 INTRODUCTION
text: Diversity promoting learning has been widely adopted via enlarging pairwise distances (Xie et al., 2018; 2017a; Liu et al., 2018), increasing orthogonality (Xie et al., 2018), reducing covariance between parameters (Xie et al., 2017b), or reducing correlation on feature (Cogswell et al., 2016) to improve generalization performance. Among them, diversity promoting regularization (Xie et al., 2017a;b; Liu et al., 2018) by enforcing large diversity between projection parameters achieves a reasonable performance without modifying the model structure. While optimizing the objective function with a covariance matrix in (Xie et al., 2017b; 2018) is nontrivial, the diversity promoting regularization via minimizing energy of parameters of deep neural networks has been proposed (Liu et al., 2018) in a simpler way. By minimizing a pairwise distance between parameters on a hypersphere with the known metric, they achieved the improved generalization performance.
Following an efficient regularization on the hypersphere, we explore further this direction with three main concepts (hierarchical and hyperspherical learning with discrete metrics). 1) Why hierarchical learning? Hierarchical inference explains human intelligence. In (Kurzweil, 2013), it states that “the neocortex contains about 300 million very general pattern recognizers, arranged in a hierarchy”. Applying the hierarchy of multiple classes based on semantic taxonomy is a natural choice to devise machine intelligence. Effectiveness of the hierarchical learning can be found in (Verma et al., 2012).
2) Why hyperspherical learning? Hypersphere is the set of points at the equidistance (radius) from a given point (centroid) in a certain dimensional space. Due to the denominator in the unit-length normalization ( w‖w‖ , w ∈ R
d+1), the angular distance defined on the hypersphere converges when the magnitude of w goes infinity while Euclidean distance goes infinity. Due to this bounded property, a hierarchical structure with multiple separated hyperspheres can be defined.
3) Why discrete metric learning? If vector points form discontinuous series with discrete representation (e.g. multi-dimensional binary or ternary), they are isolated from each other with a certain margin. This property may fit with a disconnected/groupwise manifold space problem. We note that making points to be equidistributed where a pairwise distance is maximized is a nontrivial task. As
points in a discrete metric space are finite and isolated, search efforts to satisfy those constraints can be reduced.
In this paper, we propose to apply a hierarchical structure to parameter regularization on the multiple groupwise hyperspheres. In order to find an appropriate metric in this space, we explore a discrete angular metric. We examine the proposed method on extensive experimental setups in terms of datasets and deep network models.

heading: 2 MULTIPLE SEPARATED HYPERSPHERES
text: Samples observed from the real world may be on disconnected manifolds. In other words, the disjoint union of those manifolds can generate the global manifold (Lee, 2000). In this section, we decompose the one space into multiple spaces (manifolds) and re-define the spaces in a hierarchical point of view.

heading: 2.1 DISCONNECTED MANIFOLD VIA EQUIVALENT RELATIONS
text: Since it is not suitable to measure a pairwise distance between high dimensional vectors which have the hierarchical structure in the same space, we construct another identification space where subspaces (sub-manifolds) are isolated (via equivalence relation (Tu, 2010)). Denote d-sphere Sd to be the set of points that satisfies Sd = {w ∈ Rd+1 : ‖w‖ = 1}. We construct multiple separated hyperspheres using multiple identifying relations. In Figure 1, we use a center parameter and a surface parameter to define a projection parameter vector w which constitutes a hypershpere space.

heading: 2.2 PRIOR DISTRIBUTION AND REGULARIZATION
text: To make parameter vectors uniformly distributed on the unit hypersphere, parameters are sampled from the Gaussian normal distribution (Muller, 1959; Harman & Lacko, 2010). This is because the normal distribution is spherically symmetric (Muller, 1959). In a Bayesian point of view, neural networks with Gaussian priors are known to induce l2-norm regularization (Vladimirova et al., 2019). From two evidences, we know that enforcing parameters to have the Gaussian prior is important for hyperspherical learning in neural networks. Note that a parameter which is calculated from the difference arithmetic operation with two parameters on the normal Gaussian distribution is on the normal difference distribution.

heading: 3 METHOD
text: In deep neural networks, the objective function J with regularization R in addition to a loss L, JR(W) = L(x,W) + λR(W), is optimized to find the optimal W having a near minimum loss L, arg minW JR(x,W), where x ∈ Rd0 denotes an input vector, W = {Wi ∈ Rdi−1×ci : Wi = {wj ∈ Rdi−1}, j = 1, . . . , ci, i = 1, . . . , L} denotes a set of parameter matrices (i.e. neurons/kernels), L denotes the number of layers, and λ > 0 is to control the degree of the regularization. For a classification task, the cross entropy loss is used for the loss function L. We propose a new regularization formulationR preserving a hierarchical parameter structure in Section 3.1.
Figure 2: A series of levelwise (l) spheres with a radius (rl) is shown: (a) A radius of an oveall area converges to r0 1−δ (=
∑∞ l r0δ
l: the sum of radius series) as l goes to infinity where l denotes a level, r0 is their initial
radius, and δ is the ratio between radiuses (rl, rl−1), rlrl−1 < 1. (b) The radius of the overall area is bounded to the initial radius r0 of spheres. This bears a resemblance to the process of repeat of Hypersphere packing which arranges non-overlapping spheres within a containing space. (c) 2-sphere is defined following (b) which is appropriate to model a hierarchical structure. This can be generalized with a hypersphere (Sd, d ≥ 3) in a higher dimensional space.

heading: 3.1 REGULARIZATION FOR HIERARCHICAL AND HYPERSPHERICAL HYPOTHESES
text: Denote w a parameter vector (called a projection parameter, an element of W at a certain layer) to transform a given input into an embedding space defined in a Euclidean metric space: x ∈ Rd+1 7→ wTx ∈ R. By the definition of unit-length projection w‖w‖ , a new parameter vector ŵ can be defined on d-sphere: Sd = {ŵ ∈ Rd+1 : ‖ŵ‖ = 1} where ‖ · ‖ denotes l2-norm and a center is zero. In other word, the parameter ŵ can be defined by a center parameter vector wc ∈ Rd+1 and a surface parameter vector ws ∈ Rd+1 using an arithmetic operation: ŵ := ws−wc. We define the d-sphere with the center and surface parameters: Sdwc = {ws − wc ∈ R
d+1 : ‖ws − wc‖ = 1}. For a notation simplicity, we use w instead of ŵ hereafter. While we consider a radius equals to 1 for simplicity, the parameter vector can have a radius r > 0.

heading: 3.1.1 HIERARCHICAL PARAMETERIZATION VIA LEVELWISE AND GROUPWISE PARAMETERS
text: We assume that the hierarchical structure consists of a levelwise structure with a notation (l) and a groupwise structure with a notation g below. Through defining these structures serially, we explain hierarchical parameters.
Levelwise structure The projection parameter w ∈ Rd+1 on Sdwc can be defined with a levelwise notation (l) as follows,
w(l) := w(l)s −w(l)c (1)
where Sd w (l) c denotes the l-th level d-sphere centered at w(l)c . In this paper, we define the parameter in a higher dimensional space than that shown in Figure 2.
In a levelwise setting, w(l)s and w (l) c at the current level (l) are additively represented based on a center parameter of the previous level (l− 1): w(l−1)c + −−→ ∆w(l) 7→ w(l)c , where w(l−1)c = ∑l−1 i −−→ ∆w(i) is the accumulated and −−→ ∆w(l) denotes a connecting parameter from w(l−1)c to w
(l) c . By denoting−−→ ∆w(l) as w(l,l−1), the center vector at the l-th level is defined as, w(l)c := w (l,l−1) c +w (l−1) c , and the surface vector as w(l)s := w (l,l−1) s +w (l−1) c . Both the center parameter and the surface parameter at the current level are based on the center parameter at the previous level1. Hence, Eq. 1 is equivalent to w(l) = w(l,l−1)s −w(l,l−1)c . (2) Note that we use a superscript (l, l − 1) to denote parameters at the l-th level connected from the center parameter at the (l − 1)-th level.
Groupwise structure With a group notation gk, the center parameter in Eq. 2 can be rephrased as w
(l,l−1) c,gk , on Sdw(l,l−1)c,gk which denotes d-sphere of gk group at the l-th level. Each group constitutes
a group set at the l-th level g(l) := {gk}|g (l)| k=1 where g (l) ⊆ G(l), G(l) denotes a batch group set at the l-th level, and | · | denotes the cardinality. The group set g(l) at the l-th level is conditioned on 1As not every sample has a child sample, it might be more reasonable to branch from representative parameter or center parameter rather than from individual projection parameters.
a group set at the (l − 1)-th level, g(l−1) := {gk′}|g (l−1)| k′=1 where g (l−1) ⊆ G(l−1). By their groupwise relation over levels, an adjacency indication2 P (l,l−1)({G(l−1),G(l)}) ∈ {0, 1}|G(l−1)|×|G(l)| is given. Hence, the i-th parameter vector on Sd
w (l,l−1) c,gk
is defined as follows,
w (l) gk,i := w (l,l−1) s,gk,i −w(l,l−1)c,gk , (3)
where {w(l,l−1)s,gk,i ,w (l,l−1) c,gk } ∀i is calculated based on a center vector w (l−1) c,gk′ at the (l − 1)-th level3, i = 1, . . . , |gk|, and |gk| denotes the number of surface parameter vectors in the group gk.

heading: 3.1.2 HIERARCHICAL REGULARIZATION
text: In this section, we define a regularization term using the hierarchical parameters defined above. A set of parameters {W (l,l−1)s,gk ,w (l,l−1) c,gk ,w
(l−1) c,g′k } ∈W ∀gk, ∀gk′ where W (l,l−1)s,gk := {w (l,l−1) s,gk,i }|gk|i=1, is an optimizing target of hierarchical regularization as follows:
R(W ) := ∑ l λlRl(W (l,l−1)s,gk ,w (l,l−1) c,gk ;P (l,l−1)) + ∑ l Cl(w(l,l−1)c,gk ,w (l−1) c,g′k ;P (l,l−1)) (4)
where Rl works on individual spheres Sd w
(l,l−1) c,gk , λl > 0, and Cl aims to apply geometry-aware constraints across spheres. Rl consists of two terms for regularization: 1)Rl,p for projection parameters in the same group gk on Sd
w (l,l−1) c,gk
and 2) Rl,c for center parameters across groups in the same level
on Sd w
(l−1) c,g′
k
,
Rl(W (l,l−1)s,gk ,w (l,l−1) c,gk ) := Rl,p(W (l,l−1)s,gk ,w (l,l−1) c,gk ) +Rl,c(w(l,l−1)c,gk ), (5) where
Rl,p(W (l,l−1)s,gk ,w (l,l−1) c,gk
) := 1
G ∑ {gk∈g(l)}
2 Gk(Gk − 1) ∑
{i 6=j∈gk}
d(w (l,l−1) gk,i ,w (l,l−1) gk,j ), (6)
Rl,c(w(l,l−1)c,gk ) := 2 C(C − 1) ∑
{gi 6=gj∈g(l)}
d(w(l,l−1)c,gi ,w (l,l−1) c,gj ), (7)
where w(l,l−1)gk,i := w (l,l−1) s,gk,i − w(l,l−1)c,gk , (w (l,l−1) gk,j
is similarly defined), G = |{gk ∈ g(l)}|, Gk = |{i 6= j ∈ gk}|, C = |{gi 6= gj ∈ g(l)}|, and d(·, ·) denotes a distance metric between parameters. The distance metric is defined in Section 3.2. We explain regularization terms formulated using these parameters and pairwise distances further in Figure 4 of Appendix. When a minibatch of inputs (mx: a set of inputs {xi} ) is given, the regularization term becomes: E(R(W )) = 1|mx| ∑ mx R(W ;mx).
The constraint term help construct geometry-aware relational parameters between different spheres on the same level and on the across levels. Multiple constraints are defined as Cl := ∑ k λkCl,k, where Cl,k is the k-th constraint between parameters at the l-th and the (l − 1)-th level, and λk > 0 is a Lagrange multiplier. We propose three constraints (defined in Appendix) in a geometric point of view.

heading: 3.2 DISCRETE AND CONTINUOUS ANGULAR DISTANCE METRIC
text: Discrete metric is a good fit for the above groupwise definition. We expect that points projected by parameters in a discrete metric space are isolated each other. In Figure 3, we show that discrete distances of parameter pairs have different values while their continuous distances are same. Hence, maximization of discrete distances of parameter pairs could help parameters distributed isolatedly and diversely.
2This can be replaced with a probability model. 3The center vector w(l)c,gk indicates a representative vector of the group gk at the l-th level and it is equivalent
to a mean vector of w(l)s,gk,i ∀i: µ(w (l) s,gk,i ) = 1|gk| ∑|gk| i w (l) s,gk,i . The center vector for the group gk can be determined by a parameter vector at the (l − 1) − th level using an adjust factor ( , | | < 1): w(l,l−1)c,gk = ·w(l−1)gk′ ,i , where w (l−1) gk′ ,i ∈ Sd w
(l−1) c,g′
k
.
Using parameter vectors wi and wj in Rd+1, we define a discrete distance metric using a sign function as follows:
Dh := 1
d d∑ k sign(wi(k)) · sign(wj(k)), (8)
where sign(x) := {
1, if x≥0 −1, otherwise , −1 ≤ Dh ≤ 1, and w =
{w(k) | ∀k = 1, . . . , d + 1} ∈ Rd+1}. This is a normalized version of Hamming distance. For a ternary discrete, sign(x) ∈ {−1, 0, 1} is used. In order to consider the discrete distance as an angular distance within [0, 1], normalized one is defined asDh01 := −Dh+1 2 , where 0 ≤ Dh01 ≤ 1. The angular distance based on the above product can be rephrased as θDh = Dh01
4 where 0 ≤ θDh ≤ 1. As the discrete distance could underestimate to approximate the model distribution, we merge the above discrete distance metric with continuous angular distance metric (θ = 1π arccos ( wi·wj ‖wi‖‖wj‖ ), 0 ≤ θ ≤ 1) into a single metric. We simply use the definition of Pythagorean means which consist of the arithmetic mean (AM), the geometric mean (GM),
and the harmonic mean (HM). Pythagorean means using a pair of angular distances are defined as follows:
DAM := θDh+θ 2 , DGM := θDhθ, DHM := 4θDhθ
θDh+θ . (9)
In the angular distance5 using a pair of angles {θDh , θ}, a reversed form 1 − D{θDh ,θ} is adopted to maximize an angle in the optimization formulation by minimization instead of (·)−s where s = 1, 2, . . . which is used in Thomson problem that utilizes s-energy (Brauchart & Grabner, 2015). The cosine similarity using the pair of angles can be defined as follows:
Dcos(AM) := cos ( θDh+θ 2 π), Dcos(GM) := cos (θDhθπ), Dcos(HM) := cos ( 4θDhθ
θDh+θ π), (10)
then the cosine similarity functions are normalized with cos(·)+12 to have a value within a range [0, 1]. Finally, Pythagorean means of cosine similarities can be calculated as follows:
DAMcos := cos θDhπ+cos θπ+2 4 , DGMcos := (cos θDhπ+1)(cos θπ+1) 4 , DHMcos := (cos θDhπ+1)(cos θπ+1)
cos θDh+cos θ+2 .
(11)Metric functions defined in (9), (10), and (11) satisfy metric conditions: non-negativity, symmetry, and triangle inequality. The distance using the above metric function between any two parameter points is bounded, because the hypersphere is a compact manifold.

heading: 3.3 GRADIENTS AND BACKPROPAGATION
text: As the sign function is not differentiable at the value 0, we adopt alternative backpropagation function. We adopt straight-through estimator (STE) (Bengio et al., 2013) in the backward path of the neural networks for the sign function in the discrete metric. The derivative of the sign function is substituted with 1|w|≤1 in the backward pass, known as the saturated STE. As the derivative of arccos(x) ( −1√
1−x2 ) is undefined at the value x = ±1, we apply clamping to the cosine function to have x ∈ [−0.99, 0.99]6 where x = cos(θπ), and 0 ≤ θ ≤ 1.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: Datasets We conduct experiments using four publicly available datasets including small size images (CIFAR-10 and CIFAR-100) and large size images (CUB200-2011 (Wah et al., 2011) and
4On the other hand, the angle can be considered as a cosine similarity directly, Dh := cos θDhπ. So as to get the angular distance, it needs an arccosine function θDh = 1 π arccosDh. In summary, for the angle distance θDh , either “Dh01” or “D ′ h01 = 1 π arccosDh” where 0 ≤ Dh01 ≤ 1, can be adopted.
5In 0 ≤ θ ≤ 1, the angle and its cosine value show an inverse relationship:0 ≤ θ ≤ 1→ 1 ≥ cos θπ ≥ −1. 6x = {0.99 · 1x>0.99, x,−0.99 · 1x<−0.99}
Stanford-Cars (Krause et al., 2013b), shortly CUB200 and Cars hereafter respectively). CIFAR-10 dataset is used to validate effectiveness of the proposed metric without hierarchical information. Except CIFAR-10, we use two-level hierarchy pairs {parent, child} for the proposed hierarchical regularization on three datasets. In Table 7 at Appendix, statistics of datasets in detail is provided.
Deep neural network models and training setting We used the deep residual neural network (resnet) (He et al., 2016) with different sizes of parameters (weights) dependent on an image size of datasets. For a small size input (32× 32 pixels) of CIFAR-10 and CIFAR-100, we used resnet with smaller number of parameters (light models, resnet-20 (0.29M) and resnet-110 (1.73 M)) so as not to have redundant parameters leading to over-fitting. For a fine-grained input (224× 224 pixels) of CUB200 and Cars, the original resnet with larger number of parameters (heavy models7, Resnet-18 (11.28M) and Resnet-50 (23.91M)) is used.
In training of the deep neural network with the hierarchical regularization, we assume that the global hierarchical structure is not given. Instead, stochastic or partial hierarchical structure is shown within each mini-batch by hierarchy pairs. Mini-batches, 512 for light models and 256 for heavy models, are used in the SGD optimizer. We applied the hierarchical regularization in the FC layer of the resnet. Even though SGD is known as an unbiased estimation, a stochastic hierarchical structure could affect the overall approximation performance upon the class distribution within the minibatch. Settings in more detail are provided in Appendix.
Baseline setting In terms of the regularization, we used three reference settings, denoted ‘baseline’, ‘baseline+l2’, and ‘E’ for a reference. First, in ‘baseline’, we examined neural networks without regularization. Then, in ‘baseline+l2’, we added l2-norm minimization based regularization (which is equivalent to a weight decay in SGD setting (Zhang et al., 2019)) via λf ∑ k ‖wk‖, where wk ∈ W and λf > 0 to ‘baseline’. Lastly, in ‘E’, energy minimization (Liu et al., 2018), diversity promoting regularization without hierarchical information, via λe ∑ i 6=j d(wi,wj), where d(·, ·) is a pairwise distance and λe > 0, is added to baseline+l2. When the proposed hierarchical regularization (Eq. 5) is applied to the FC layer, we denoted it as ‘H’. Hence, the regularization is incrementally adopted to the baseline: l2+‘E’+‘H’. In terms of a distance metric, we compared the proposed metric with other continuous metrics such as Euclidean distance with the unit-length projection (‘N-euclidean2’, ∑ i 6=j ‖ wi ‖wi‖ − wi ‖wi‖‖
−2)8, angular distance (‘angular2’,∑ i 6=j arccos ( wi·wj ‖wi‖‖wj‖ )
−2) where ‘2’ is from Riesz s-energy shown higher accuracy, and cosine similarity (‘cosine’, ∑ i6=j wi·wj ‖wi‖‖wj‖ ).

heading: 4.2 RESULTS
text: Low resolution object classification In Table 1, we showed test accuracy (%) using different metrics with energy minimization regularization without the hierarchical regularization9 along a weight decay (l2-norm) using CIFAR-10 and resnet-20. The ‘baseline’ method is examined without (× in Table 1) or with (◦ in Table 1) the weight decay (l2). The discrete angular metric based regularization (Dterh (ternary code), D bin h (binary code), Dcos (HM), and D ′ HMcos ) on the baseline can improve the generalization performance in term of test accuracy compared to the other continuous metrics such as N-euclidean2, angular2, and cosine. Due to the unit-length projection in N-euclidean2, their performance is comparable to that other angular metrics. D′ denotes an arccosine adopted discrete distance used as D′h (see footnote 4 in Section 3.2). Regularization methods are applied over all layers except Batch Normalization (BN) layers. As explained in (van Laarhoven, 2017; Zhang et al., 2019), l2-norm minimization based regularization (weight decay) showed much improvement due to their effective learning rate in SGD.
As shown in Table 2, the regularization with proposed metrics shows better accuracy performance than that of the baseline on both resnet-20 and resnet-110 using CIFAR-100 dataset. Comparing to the baseline+l2, with energy minimization ‘E’ regularization, metrics such as Dterh , Dcos(AM), and D′cos(HM) showed better performance than other metrics. If the proposed hierarchical ‘H’ regular-
7which is available at the pytorch library 8 = ∑ i6=j (2− 2 wi·wj ‖wi‖‖wj‖ ) −1 9This is because that hierarchy information is not included in CIFAR-10.
Table 1: Test accuracy (%) using different metrics on energy minimization without hierarchy information along l2-norm minimization (weight decay) on CIFAR-10 and resnet-20.
metric \ l2 × ◦ baseline 90.34 92.21 N-euclidean2 90.93 92.35 angular2 90.47 92.38 cosine 90.53 92.40 Dterh 90.67 92.48 Dbinh 90.67 92.48 Dcos(HM) 90.84 92.93 D′HMcos 90.94 92.69
Table 2: Test accuracy (%) using different metrics on energy minimization (‘E’) and hierarchical regularization (‘H’) on CIFAR-100: ‘E’ / ‘E+H’ respectively.
metric \ db resnet-20 resnet-110 baseline 63.86 62.02 baseline+l2 68.03 72.90 N-euclidean2 67.59 / 68.65 73.95 / 73.96 angular2 67.83 / 67.76 74.40 / 73.89 cosine 68.11 / 68.45 73.37 / 73.37 Dterh 68.44 / 68.68 73.73 / 73.97 Dbinh 68.52 / 68.69 73.97 / 74.26 DAM 68.58 / 68.86 73.43 / 73.50 Dcos(AM) 68.58 / 68.60 73.14 / 73.65 D′cos(AM) 67.57 / 68.36 73.14 / 73.72
D′cos(HM) 68.62 / 68.65 73.07 / 73.65Table 3: Test accuracy (%) using different met-
rics on energy minimization (‘E’) and hierarchical regularization (‘H’) on CUB200: ‘E’ / ‘E+H’ respectively.
metric \ db Resnet-18 Resnet-50
ization is applied, the generalization performance is further improved in most metric cases on both resnet-20 and resnet-110. As the binary metric shows a better performance than that ternary, we adopt binary discretization for proposed angular metrics (D•, Dcos(•), and D•cos ) in experiments.
Fine-grained visual categorization In this experiment, we used two large size image datasets with a fine-grained category. One is with birds (CUB200) and another is with cars (Cars) that focus on single species of objects. A hierarchy of CUB200 is defined by the academic expert on birds whereas that of Cars is categorized manually based on a name of cars by a non-expert. Moreover, the rate between the number of superclass (parent) per subclass of CUB200 (0.35) is much larger than that of Cars (0.0459) (as shown in Table 7 at Appendix). That rate of Cars is even smaller than that of CIFAR-100.
As shown in Table 3, the proposed hierarchical regularization significantly improved the test accuracy over all metrics on both Resnet-18 and Resnet-50. Compared to CUB200, as shown in Table 4, the improvement of the proposed method is not that significant for Cars dataset. This is because CUB200 dataset has not only better hierarchical categorization by the expert but also more diversitified pairs between the superclass and the subclass.
Ablation study In Table 5, we showed how metrics affect the generalization performance using resnet-20 and CIFAR-100 dataset. The proposed method with different proposed metrics showed significantly improved performance compared to the baseline+l2. Individual means (AM, GM, and HM) show different improvement patterns.
In Table 6, we showed how heterogeneous metrics on convolutional (conv.) layers (with the energy minimization, ‘E’) and fully connected (FC) layers (with the hierarchical ‘H’ regularization, ‘H’) affect the generalization performance using resnet-20 and CIFAR-100 dataset. The cases applying hierarchical regularization showed better performance than that the baseline applying only energy minimization regularization. In this experiment, a combination GM and HM shows a better improvement than that other combinations.

heading: 5 RELATED WORKS
text: Promoting of diversity in an embedding space or model parameters is an widely adopted strategy in machine learning related area to improve the generalization performance (Cogswell et al., 2016; Yang et al., 2019; Li et al., 2012; Ratzlaff & Fuxin, 2019; Xie et al., 2017b; 2018; 2017a; Liu et al., 2018). This diversity based strategy can be applied in a variety of levels such as in a feature level (Cogswell et al., 2016; Xie et al., 2018), in a projection parameter level (Xie et al., 2017a; Liu et al., 2018), in a model ensemble level (Zhou et al., 2018; Ratzlaff & Fuxin, 2019), in a latent space model level (Ratzlaff & Fuxin, 2019; Liu et al., 2018), and in a generative model level (Yang et al., 2019; Ratzlaff & Fuxin, 2019). Among these approaches, some regularization methods require additional optimization effort. Enlarging pairwise distance between features (Xie et al., 2018) requires computational efforts due to their covariance matrix. In (Xie et al., 2017b), unit-eigenvalue is utilized via singular value decomposition. To optimize a direction and magnitude of the parameter vector alternatively, an alternating direction method of multipliers (ADMM) (Xie et al., 2017a) is used.
In terms of learning on a hypersphere, (Liu et al., 2017) proposed that hyperspherical convolution (SphereCov) replaces the traditional inner-product based convolution in order to conduct learning angular representation. On a hypersphere, regularization via promoting diversity of parameters has been proposed (Liu et al., 2018) based on Minimum Hyperspherical Energy.
In terms of hierarchical learning, (Nickel & Kiela, 2017; Ganea et al., 2018) proposed to apply hyperbolic space to embed the data in deep neural networks. They showed that learning representations in the hyperbolic space is effective to the data preserving a latent hierarchy compared to in the Euclidean space. Different from these works which are aimed for representation learning, our proposed method is focused on regularization learning using explicit (predefined) hierarchy information.

heading: 6 CONCLUSION
text: We proposed the regularization method, which maximizes a pairwise distance between parameters preserving a hierarchical structure. To define a hierarchical parameter space, we reformulated the topology space with multiple hyperspheres. In each hypersphere, a projection is parameterized by the surface parameter with the center parameter. By imposing a maximum distance between hierarchical parameters, diversity of parameters preserving semantic structure was promoted. For the distance metric in multiple separated spaces, we proposed a discrete metric integrated with a continuous metric. Extensive experiments using publicly available datasets (CIFAR-10, CIFAR100, CUB200-2011, and Stanford Cars), the deep neural network with our proposed regularization method showed superior classification performance, especially when the number of super-classes is large. For further exploration in future, our proposed method can be combined with hierarchical representation learning such as hyperbolic (or Poincaré) embeddings (Nickel & Kiela, 2017; Ganea et al., 2018).

Review summaries:
The paper tackles the problem of promoting diversity in the weights of deep neural networks. The problem is interesting and useful. The paper argues that hierarchical learning and hyper spherical learning are important in addressing this problem. The paper provides experiments on CIFAR-10 and CIFAR-100 where the improvements of using such regularization is visible but not sufficiently significant.