Title: DEEP HIERARCHICAL-HYPERSPHERICAL LEARNING (DH 2 L)

Abstract: Regularization is known to be an inexpensive and reasonable solution to alleviate the overfitting problem arising in inference models, including deep neural networks. In this paper, we propose hierarchical regularization which preserves the semantic structure of a sample distribution. At the same time, this regularization promotes diversity by imposing a distance between parameter vectors enlarged within semantic structures. To generate evenly distributed parameters which is considered less redundant, we constrain them to lie on hierarchical hyperspheres. To define a hierarchical parameter space, we propose to reformulate the topology space with multiple hyperspheres. On each hypersphere, a projection is parameterized by two individual parameters. Since maximizing a groupwise and pairwise distance between points on hypersphere is nontrivial (generalized Thomson problem), we propose a new discrete metric integrated with a continuous metric. The proposed method shows improved generalization performance on extensive experiments using publicly available datasets (CIFAR-10, CIFAR-100, CUB200-2011, and Stanford Cars), especially when the number of super-classes is large.

1. INTRODUCTION:

Diversity promoting learning has been widely adopted via enlarging pairwise distances (Xie et al., 2018; 2017a; Liu et al., 2018) , increasing orthogonality (Xie et al., 2018) , reducing covariance between parameters (Xie et al., 2017b) , or reducing correlation on feature (Cogswell et al., 2016) to improve generalization performance. Among them, diversity promoting regularization (Xie et al., 2017a; b; Liu et al., 2018) by enforcing large diversity between projection parameters achieves a reasonable performance without modifying the model structure. While optimizing the objective function with a covariance matrix in (Xie et al., 2017b; 2018) is nontrivial, the diversity promoting regularization via minimizing energy of parameters of deep neural networks has been proposed (Liu et al., 2018) in a simpler way. By minimizing a pairwise distance between parameters on a hypersphere with the known metric, they achieved the improved generalization performance.

Following an efficient regularization on the hypersphere, we explore further this direction with three main concepts (hierarchical and hyperspherical learning with discrete metrics). 1) Why hierarchical learning? Hierarchical inference explains human intelligence. In (Kurzweil, 2013) , it states that "the neocortex contains about 300 million very general pattern recognizers, arranged in a hierarchy". Applying the hierarchy of multiple classes based on semantic taxonomy is a natural choice to devise machine intelligence. Effectiveness of the hierarchical learning can be found in (Verma et al., 2012) .

2) Why hyperspherical learning? Hypersphere is the set of points at the equidistance (radius) from a given point (centroid) in a certain dimensional space. Due to the denominator in the unit-length normalization ( w w , w ∈ R d+1 ), the angular distance defined on the hypersphere converges when the magnitude of w goes infinity while Euclidean distance goes infinity. Due to this bounded property, a hierarchical structure with multiple separated hyperspheres can be defined.

3) Why discrete metric learning? If vector points form discontinuous series with discrete representation (e.g. multi-dimensional binary or ternary), they are isolated from each other with a certain margin. This property may fit with a disconnected/groupwise manifold space problem. We note that making points to be equidistributed where a pairwise distance is maximized is a nontrivial task. As Figure 1 : Multiple (hyper)spheres as quotient spaces of a topology space on Euclidean space might be found by gluing process with identifying points. Those separated hyperspheres are assumed to be under the quotient space conditions (Tu, 2010) . Within an individual (hyper)sphere, projection parameters in deep neural networks preserve a hierarchical structure. The space can be formed in a series: (a), (b), (c), and (d) points in a discrete metric space are finite and isolated, search efforts to satisfy those constraints can be reduced.

In this paper, we propose to apply a hierarchical structure to parameter regularization on the multiple groupwise hyperspheres. In order to find an appropriate metric in this space, we explore a discrete angular metric. We examine the proposed method on extensive experimental setups in terms of datasets and deep network models.

2. MULTIPLE SEPARATED HYPERSPHERES:

Samples observed from the real world may be on disconnected manifolds. In other words, the disjoint union of those manifolds can generate the global manifold (Lee, 2000) . In this section, we decompose the one space into multiple spaces (manifolds) and re-define the spaces in a hierarchical point of view.

2.1. DISCONNECTED MANIFOLD VIA EQUIVALENT RELATIONS:

Since it is not suitable to measure a pairwise distance between high dimensional vectors which have the hierarchical structure in the same space, we construct another identification space where subspaces (sub-manifolds) are isolated (via equivalence relation (Tu, 2010) ). Denote d-sphere S d to be the set of points that satisfies S d = {w ∈ R d+1 : w = 1}. We construct multiple separated hyperspheres using multiple identifying relations. In Figure 1 , we use a center parameter and a surface parameter to define a projection parameter vector w which constitutes a hypershpere space.

2.2. PRIOR DISTRIBUTION AND REGULARIZATION:

To make parameter vectors uniformly distributed on the unit hypersphere, parameters are sampled from the Gaussian normal distribution (Muller, 1959; Harman & Lacko, 2010) . This is because the normal distribution is spherically symmetric (Muller, 1959) . In a Bayesian point of view, neural networks with Gaussian priors are known to induce l 2 -norm regularization (Vladimirova et al., 2019) . From two evidences, we know that enforcing parameters to have the Gaussian prior is important for hyperspherical learning in neural networks. Note that a parameter which is calculated from the difference arithmetic operation with two parameters on the normal Gaussian distribution is on the normal difference distribution.

3. METHOD:

In deep neural networks, the objective function J with regularization R in addition to a loss L, J R(W) = L(x, W) + λR(W), is optimized to find the optimal W having a near minimum loss L, arg min W J R(x,W) , where x ∈ R d0 denotes an input vector, W = {W i ∈ R di-1×ci : W i = {w j ∈ R di-1 }, j = 1, . . . , c i , i = 1, . . . , L} denotes a set of parameter matrices (i.e. neurons/kernels), L denotes the number of layers, and λ > 0 is to control the degree of the regularization. For a classification task, the cross entropy loss is used for the loss function L. We propose a new regularization formulation R preserving a hierarchical parameter structure in Section 3.1. 1-δ (= ∞ l r0δ l : the sum of radius series) as l goes to infinity where l denotes a level, r0 is their initial radius, and δ is the ratio between radiuses (r l , r l-1 ), r l r l-1 < 1. (b) The radius of the overall area is bounded to the initial radius r0 of spheres. This bears a resemblance to the process of repeat of Hypersphere packing which arranges non-overlapping spheres within a containing space. (c) 2-sphere is defined following (b) which is appropriate to model a hierarchical structure. This can be generalized with a hypersphere (S d , d ≥ 3) in a higher dimensional space.

3.1 REGULARIZATION FOR HIERARCHICAL AND HYPERSPHERICAL HYPOTHESES Denote w a parameter vector (called a projection parameter, an element of W at a certain layer) to transform a given input into an embedding space defined in a Euclidean metric space: x ∈ R d+1 → w T x ∈ R. By the definition of unit-length projection w w , a new parameter vector ŵ can be defined on d-sphere: S d = { ŵ ∈ R d+1 : ŵ = 1} where • denotes l 2 -norm and a center is zero. In other word, the parameter ŵ can be defined by a center parameter vector w c ∈ R d+1 and a surface parameter vector w s ∈ R d+1 using an arithmetic operation: ŵ := w s -w c . We define the d-sphere with the center and surface parameters:

S d wc = {w s -w c ∈ R d+1 : w s -w c = 1}.

For a notation simplicity, we use w instead of ŵ hereafter. While we consider a radius equals to 1 for simplicity, the parameter vector can have a radius r > 0.

3.1.1. HIERARCHICAL PARAMETERIZATION VIA LEVELWISE AND GROUPWISE PARAMETERS:

We assume that the hierarchical structure consists of a levelwise structure with a notation (l) and a groupwise structure with a notation g below. Through defining these structures serially, we explain hierarchical parameters.

Levelwise structure The projection parameter w ∈ R d+1 on S d wc can be defined with a levelwise notation (l) as follows,

EQUATION

)

where

S d w (l) c

denotes the l-th level d-sphere centered at w (l) c . In this paper, we define the parameter in a higher dimensional space than that shown in Figure 2 .

In a levelwise setting, w (l) s and w (l) c at the current level (l) are additively represented based on a center parameter of the previous level (l -1):

w (l-1) c + --→ ∆w (l) → w (l)

c , where w

(l-1) c = l-1 i --→ ∆w (i) is the accumulated and --→ ∆w (l) denotes a connecting parameter from w (l-1) c to w (l) c

. By denoting --→ ∆w (l) as w (l,l-1) , the center vector at the l-th level is defined as, w

(l) c := w (l,l-1) c +w (l-1) c

, and the surface vector as w (l)

s := w (l,l-1) s + w (l-1) c

. Both the center parameter and the surface parameter at the current level are based on the center parameter at the previous level 1 . Hence, Eq. 1 is equivalent to

EQUATION

Note that we use a superscript (l, l -1) to denote parameters at the l-th level connected from the center parameter at the (l -1)-th level.

Groupwise structure With a group notation g k , the center parameter in Eq. 2 can be rephrased as w

(l,l-1) c,g k , on S d w (l,l-1) c,g k

which denotes d-sphere of g k group at the l-th level. Each group constitutes a group set at the l-th level g l) , G (l) denotes a batch group set at the l-th level, and | • | denotes the cardinality. The group set g (l) at the l-th level is conditioned on a group set at the (l -1)-th level, g (l-1) :

(l) := {g k } |g (l) | k=1 where g (l) ⊆ G (

= {g k } |g (l-1) | k =1

where g (l-1) ⊆ G (l-1) . By their groupwise relation over levels, an adjacency indication 2 P (l,l-1) ({G (l-1) , G (l) }) ∈ {0, 1} |G (l-1) |×|G (l) | is given. Hence, the i-th parameter vector on S d w (l,l-1) c,g k is defined as follows,

EQUATION

where {w

(l,l-1) s,g k ,i , w (l,l-1) c,g k } ∀i is calculated based on a center vector w (l-1)

c,g k at the (l -1)-th level 3 , i = 1, . . . , |g k |, and |g k | denotes the number of surface parameter vectors in the group g k .

3.1.2. HIERARCHICAL REGULARIZATION:

In this section, we define a regularization term using the hierarchical parameters defined above. A set of parameters {W

(l,l-1) s,g k , w (l,l-1) c,g k , w (l-1) c,g k } ∈ W ∀g k , ∀g k where W (l,l-1) s,g k := {w (l,l-1) s,g k ,i } |g k | i=1 ,

is an optimizing target of hierarchical regularization as follows:

R(W ) := l λ l R l (W (l,l-1) s,g k , w (l,l-1) c,g k ; P (l,l-1) ) + l C l (w (l,l-1) c,g k , w (l-1) c,g k ; P (l,l-1) ) (4)

where R l works on individual spheres S d w (l,l-1) c,g k , λ l > 0, and C l aims to apply geometry-aware constraints across spheres. R l consists of two terms for regularization: 1) R l,p for projection parameters in the same group g k on S d w (l,l-1) c,g k and 2) R l,c for center parameters across groups in the same level

on S d w (l-1) c,g k , R l (W (l,l-1) s,g k , w (l,l-1) c,g k ) := R l,p (W (l,l-1) s,g k , w (l,l-1) c,g k ) + R l,c (w (l,l-1) c,g k ),

(5) where

EQUATION

EQUATION

where w  and d(•, • ) denotes a distance metric between parameters. The distance metric is defined in Section 3.2. We explain regularization terms formulated using these parameters and pairwise distances further in Figure 4 of Appendix. When a minibatch of inputs (m x : a set of inputs {x i } ) is given, the regularization term becomes:

(l,l-1) g k ,i := w (l,l-1) s,g k ,i -w (l,l-1) c,g k , (w (l,l-1) g k ,j is similarly defined), G = |{g k ∈ g (l) }|, G k = |{i = j ∈ g k }|, C = |{g i = g j ∈ g (l) }|,

E(R(W )) = 1 |mx| mx R(W ; m x ).

The constraint term help construct geometry-aware relational parameters between different spheres on the same level and on the across levels. Multiple constraints are defined as C l := k λ k C l,k , where C l,k is the k-th constraint between parameters at the l-th and the (l -1)-th level, and λ k > 0 is a Lagrange multiplier. We propose three constraints (defined in Appendix) in a geometric point of view.

3.2. DISCRETE AND CONTINUOUS ANGULAR DISTANCE METRIC:

Discrete metric is a good fit for the above groupwise definition. We expect that points projected by parameters in a discrete metric space are isolated each other. In Figure 3 , we show that discrete distances of parameter pairs have different values while their continuous distances are same. Hence, maximization of discrete distances of parameter pairs could help parameters distributed isolatedly and diversely.

2 This can be replaced with a probability model. 3 The center vector w (l) c,g k indicates a representative vector of the group g k at the l-th level and it is equivalent to a mean vector of w

(l) s,g k ,i ∀i: µ(w (l) s,g k ,i ) = 1 |g k | |g k | i w (l)

s,g k ,i . The center vector for the group g k can be determined by a parameter vector at the (l -1) -th level using an adjust factor ( , | | < 1): w

(l,l-1) c,g k = • w (l-1) g k ,i

, where w (l-1) Using parameter vectors w i and w j in R d+1 , we define a discrete distance metric using a sign function as follows:

g k ,i ∈ S d w (l-1) c,g k .

EQUATION

where sign(x) := As the discrete distance could underestimate to approximate the model distribution, we merge the above discrete distance metric with continuous angular distance metric (θ = 1 π arccos ( wi•wj wi wj ), 0 ≤ θ ≤ 1) into a single metric. We simply use the definition of Pythagorean means which consist of the arithmetic mean (AM), the geometric mean (GM), and the harmonic mean (HM). Pythagorean means using a pair of angular distances are defined as follows:

1, if x≥0 -1, otherwise , -1 ≤ D h ≤

EQUATION

In the angular distance5 using a pair of angles {θ D h , θ}, a reversed form 1 -D {θ D h ,θ} is adopted to maximize an angle in the optimization formulation by minimization instead of (•) -s where s = 1, 2, . . . which is used in Thomson problem that utilizes s-energy (Brauchart & Grabner, 2015) . The cosine similarity using the pair of angles can be defined as follows: 9), (10), and (11) satisfy metric conditions: non-negativity, symmetry, and triangle inequality. The distance using the above metric function between any two parameter points is bounded, because the hypersphere is a compact manifold.

EQUATION

3.3. GRADIENTS AND BACKPROPAGATION:

As the sign function is not differentiable at the value 0, we adopt alternative backpropagation function. We adopt straight-through estimator (STE) (Bengio et al., 2013) in the backward path of the neural networks for the sign function in the discrete metric. The derivative of the sign function is substituted with 1 |w|≤1 in the backward pass, known as the saturated STE. As the derivative of arccos(x) ( -1 √ 1-x 2 ) is undefined at the value x = ±1, we apply clamping to the cosine function to have x ∈ [-0.99, 0.99]6 where x = cos(θπ), and 0 ≤ θ ≤ 1.

4. EXPERIMENTS:



4.1. EXPERIMENTAL SETUP:



Datasets:

We conduct experiments using four publicly available datasets including small size images (CIFAR-10 and CIFAR-100) and large size images (CUB200-2011 (Wah et al., 2011) and Stanford-Cars (Krause et al., 2013b) , shortly CUB200 and Cars hereafter respectively). CIFAR-10 dataset is used to validate effectiveness of the proposed metric without hierarchical information. Except CIFAR-10, we use two-level hierarchy pairs {parent, child} for the proposed hierarchical regularization on three datasets. In Table 7 at Appendix, statistics of datasets in detail is provided.

Deep neural network models and training setting We used the deep residual neural network (resnet) (He et al., 2016) with different sizes of parameters (weights) dependent on an image size of datasets. For a small size input (32 × 32 pixels) of CIFAR-10 and CIFAR-100, we used resnet with smaller number of parameters (light models, resnet-20 (0.29M) and resnet-110 (1.73 M)) so as not to have redundant parameters leading to over-fitting. For a fine-grained input (224 × 224 pixels) of CUB200 and Cars, the original resnet with larger number of parameters (heavy models 7 , Resnet-18 (11.28M) and Resnet-50 (23.91M)) is used.

In training of the deep neural network with the hierarchical regularization, we assume that the global hierarchical structure is not given. Instead, stochastic or partial hierarchical structure is shown within each mini-batch by hierarchy pairs. Mini-batches, 512 for light models and 256 for heavy models, are used in the SGD optimizer. We applied the hierarchical regularization in the FC layer of the resnet. Even though SGD is known as an unbiased estimation, a stochastic hierarchical structure could affect the overall approximation performance upon the class distribution within the minibatch. Settings in more detail are provided in Appendix.

Baseline setting In terms of the regularization, we used three reference settings, denoted 'baseline', 'baseline+l 2 ', and 'E' for a reference. First, in 'baseline', we examined neural networks without regularization. Then, in 'baseline+l 2 ', we added l 2 -norm minimization based regularization (which is equivalent to a weight decay in SGD setting (Zhang et al., 2019) ) via λ f k w k , where w k ∈ W and λ f > 0 to 'baseline'. Lastly, in 'E', energy minimization (Liu et al., 2018) , diversity promoting regularization without hierarchical information, via λ e i =j d(w i , w j ), where d(•, •) is a pairwise distance and λ e > 0, is added to baseline+l 2 . When the proposed hierarchical regularization (Eq. 5) is applied to the FC layer, we denoted it as 'H'. Hence, the regularization is incrementally adopted to the baseline: l 2 +'E'+'H'. In terms of a distance metric, we compared the proposed metric with other continuous metrics such as Euclidean distance with the unit-length projection ('N-euclidean2', i =j wi wi -wi wi -2 ) 8 , angular distance ('angular2', i =j arccos ( wi•wj wi wj )

-2 ) where '2' is from Riesz s-energy shown higher accuracy, and cosine similarity ('cosine', i =j wi•wj wi wj ).

4.2. RESULTS:



Low resolution object classification:

In Table 1 , we showed test accuracy (%) using different metrics with energy minimization regularization without the hierarchical regularization9 along a weight decay (l 2 -norm) using CIFAR-10 and resnet-20. The 'baseline' method is examined without (× in Table 1 ) or with (• in Table 1 ) the weight decay (l 2 ). The discrete angular metric based regularization (D ter h (ternary code), D bin h (binary code), D cos (HM) , and D HMcos ) on the baseline can improve the generalization performance in term of test accuracy compared to the other continuous metrics such as N-euclidean2, angular2, and cosine. Due to the unit-length projection in N-euclidean2, their performance is comparable to that other angular metrics. D denotes an arccosine adopted discrete distance used as D h (see footnote 4 in Section 3.2). Regularization methods are applied over all layers except Batch Normalization (BN) layers. As explained in (van Laarhoven, 2017; Zhang et al., 2019) , l 2 -norm minimization based regularization (weight decay) showed much improvement due to their effective learning rate in SGD. 2 , the regularization with proposed metrics shows better accuracy performance than that of the baseline on both resnet-20 and resnet-110 using CIFAR-100 dataset. Comparing to the baseline+l 2 , with energy minimization 'E' regularization, metrics such as D ter h , D cos(AM) , and D cos(HM) showed better performance than other metrics. If the proposed hierarchical 'H' regular-7 which is available at the pytorch library 8 = i =j (2 -2 ization is applied, the generalization performance is further improved in most metric cases on both resnet-20 and resnet-110. As the binary metric shows a better performance than that ternary, we adopt binary discretization for proposed angular metrics (D • , D cos(•) , and D •cos ) in experiments.

As shown in Table:

w i •w j w i w j ) -1

Fine-grained visual categorization In this experiment, we used two large size image datasets with a fine-grained category. One is with birds (CUB200) and another is with cars (Cars) that focus on single species of objects. A hierarchy of CUB200 is defined by the academic expert on birds whereas that of Cars is categorized manually based on a name of cars by a non-expert. Moreover, the rate between the number of superclass (parent) per subclass of CUB200 (0.35) is much larger than that of Cars (0.0459) (as shown in Table 7 at Appendix). That rate of Cars is even smaller than that of CIFAR-100.

As shown in Table 3 , the proposed hierarchical regularization significantly improved the test accuracy over all metrics on both Resnet-18 and Resnet-50. Compared to CUB200, as shown in Table 4 , the improvement of the proposed method is not that significant for Cars dataset. This is because CUB200 dataset has not only better hierarchical categorization by the expert but also more diversitified pairs between the superclass and the subclass.

Ablation study In Table 5 , we showed how metrics affect the generalization performance using resnet-20 and CIFAR-100 dataset. The proposed method with different proposed metrics showed significantly improved performance compared to the baseline+l 2 . Individual means (AM, GM, and HM) show different improvement patterns.

In Table 6 , we showed how heterogeneous metrics on convolutional (conv.) layers (with the energy minimization, 'E') and fully connected (FC) layers (with the hierarchical 'H' regularization, 'H') affect the generalization performance using resnet-20 and CIFAR-100 dataset. The cases applying hierarchical regularization showed better performance than that the baseline applying only energy minimization regularization. In this experiment, a combination GM and HM shows a better improvement than that other combinations. 

5. RELATED WORKS:

Promoting of diversity in an embedding space or model parameters is an widely adopted strategy in machine learning related area to improve the generalization performance (Cogswell et al., 2016; Yang et al., 2019; Li et al., 2012; Ratzlaff & Fuxin, 2019; Xie et al., 2017b; 2018; 2017a; Liu et al., 2018) . This diversity based strategy can be applied in a variety of levels such as in a feature level (Cogswell et al., 2016; Xie et al., 2018) , in a projection parameter level (Xie et al., 2017a; Liu et al., 2018) , in a model ensemble level (Zhou et al., 2018; Ratzlaff & Fuxin, 2019) , in a latent space model level (Ratzlaff & Fuxin, 2019; Liu et al., 2018) , and in a generative model level (Yang et al., 2019; Ratzlaff & Fuxin, 2019) . Among these approaches, some regularization methods require additional optimization effort. Enlarging pairwise distance between features (Xie et al., 2018) requires computational efforts due to their covariance matrix. In (Xie et al., 2017b) , unit-eigenvalue is utilized via singular value decomposition. To optimize a direction and magnitude of the parameter vector alternatively, an alternating direction method of multipliers (ADMM) (Xie et al., 2017a) is used.

In terms of learning on a hypersphere, (Liu et al., 2017) proposed that hyperspherical convolution (SphereCov) replaces the traditional inner-product based convolution in order to conduct learning angular representation. On a hypersphere, regularization via promoting diversity of parameters has been proposed (Liu et al., 2018) based on Minimum Hyperspherical Energy.

In terms of hierarchical learning, (Nickel & Kiela, 2017; Ganea et al., 2018) proposed to apply hyperbolic space to embed the data in deep neural networks. They showed that learning representations in the hyperbolic space is effective to the data preserving a latent hierarchy compared to in the Euclidean space. Different from these works which are aimed for representation learning, our proposed method is focused on regularization learning using explicit (predefined) hierarchy information.

6. CONCLUSION:

We proposed the regularization method, which maximizes a pairwise distance between parameters preserving a hierarchical structure. To define a hierarchical parameter space, we reformulated the topology space with multiple hyperspheres. In each hypersphere, a projection is parameterized by the surface parameter with the center parameter. By imposing a maximum distance between hierarchical parameters, diversity of parameters preserving semantic structure was promoted. For the distance metric in multiple separated spaces, we proposed a discrete metric integrated with a continuous metric. Extensive experiments using publicly available datasets (CIFAR-10, CIFAR-100, CUB200-2011, and Stanford Cars), the deep neural network with our proposed regularization method showed superior classification performance, especially when the number of super-classes is large. For further exploration in future, our proposed method can be combined with hierarchical representation learning such as hyperbolic (or Poincaré) embeddings (Nickel & Kiela, 2017; Ganea et al., 2018) .

Resnet-50 (23.91M) which consists of the basic blocks (Resnet-18) or the bottleneck blocks with output channels [64, 128, 256, 512] in Conv. layers. An input dimensionality of the FC layer is 512 for Resnet-18 and 2048 for Resnet-50.

Networks are optimized with SGD for both light and heavy models: we fixed 1) the weight initialization with Random-Seed number '0' in pytorch, 2) learning rate schedule [0.1, 0.01, 0.001], 3) with momentum 0.9, 4) regularization: ) . . . corresponds to Eq (7)

Constraints in Eq. 4 C l := k λ k C l,k can be given as follows:

1. Constraint 1 (C 1 ): This constraint describes that a radius of an inner sphere must be smaller than that of its outer sphere. r (l-1) -r (l) ≥ 0 ⇒ w (l-1) -w (l) = w 

:

As not every sample has a child sample, it might be more reasonable to branch from representative parameter or center parameter rather than from individual projection parameters.

On the other hand, the angle can be considered as a cosine similarity directly, D h := cos θD h π. So as to get the angular distance, it needs an arccosine function θD h = 1 π arccos D h . In summary, for the angle distance θD h , either "D h01 " or "D h01 = 1 π arccos D h " where 0 ≤ D h01 ≤ 1, can be adopted.

In 0 ≤ θ ≤ 1, the angle and its cosine value show an inverse relationship:0 ≤ θ ≤ 1 → 1 ≥ cos θπ ≥ -1.

x = {0.99 • 1x>0.99, x, -0.99 • 1x<-0.99}

This is because that hierarchy information is not included in CIFAR-10.

Dependent on the number of classes and the corresponding center parameters, the size could variate (e.g.

11.42M for CUB200, 11.31M for Cars).

from https://download.pytorch.org/models/resnet18-5c106cde.pth, https://download.pytorch.org/models/resnet50-19c8e357.pth respectively

Figures and Tables:

Figure 2: A series of levelwise (l) spheres with a radius (r l ) is shown: (a) A radius of an oveall area converges to r 01-δ (= ∞ l r0δ l : the sum of radius series) as l goes to infinity where l denotes a level, r0 is their initial radius, and δ is the ratio between radiuses (r l , r l-1 ), r l r l-1 < 1. (b) The radius of the overall area is bounded to the initial radius r0 of spheres. This bears a resemblance to the process of repeat of Hypersphere packing which arranges non-overlapping spheres within a containing space. (c) 2-sphere is defined following (b) which is appropriate to model a hierarchical structure. This can be generalized with a hypersphere (S d , d ≥ 3) in a higher dimensional space.

Figure 3: While the pairwise angular distances Da between a pair of vectors{w1, w2} and {w2, w3}are the same, the pairwise discrete product distances D h between vectors are different. To diversify a parameter space, the space with sign could be effective to recognize their difference.

1, and w = {w(k) | ∀k = 1, . . . , d + 1} ∈ R d+1 }. This is a normalized version of Hamming distance. For a ternary discrete, sign(x) ∈ {-1, 0, 1} is used. In order to consider the discrete distance as an angular distance within [0, 1], normalized one is defined as D h01 := -D h +1 2 , where 0 ≤ D h01 ≤ 1. The angular distance based on the above product can be rephrased as θ D h = D h01 4 where 0 ≤ θ D h ≤ 1.

then the cosine similarity functions are normalized with cos(•)+1 2 to have a value within a range [0, 1]. Finally, Pythagorean means of cosine similarities can be calculated as follows: D AMcos := cos θ D h π+cos θπ+2 4 , D GMcos := (cos θ D h π+1)(cos θπ+1) 4 , D HMcos := (cos θ D h π+1)(cos θπ+1) cos θ D h +cos θ+2 . (11) Metric functions defined in (

l 2 -norm minimization with λ f = 0.0005, 5) energy minimization λ e = {0.1, 1}, and hierarchical minimization λ l = 0.1 × {1, 5}. All the regularization is not applied to the parameters of Batch Normalization (BN) layers, since BN is also a regularization function. A bias term in the FC layer is not used. The images in training and test, images are resized to 256 size. The images is cropped with 224 × 224 size at random location in training and at center location in test. Horizontal flipping is applied in training. The light models are trained from scratch without the pretrained weights for 300 epochs. The heavy model is trained using pretrained model provided by pytorch library 11 with 100 epochs. The experiments are conduced using GPU "NVIDIA TESLA P40".

Figure 4: R l,p (W (l,l-1) s,gi , w (l,l-1) c,gi ) = d(w (l,l-1) gi,n , w (l,l-1) gi,m ) + d(w (l,l-1) gi,m , w (l,l-1) gi,k) +

Constraint 2 (C 2 ): This constraint describes that a center of an inner sphere must be located within its outer sphere. r (l-1) -( w (l,l-1) c + r (l) ) ≥ 0 ⇒ r (l-1) -

Test accuracy (%) using different metrics on energy minimization without hierarchy information along l 2 -norm minimization (weight decay) on CIFAR-10 and resnet-20.

Table content: <table><tr><td/><td/><td/><td colspan="3">Table 2: Test accuracy (%) using different met-</td></tr><tr><td/><td/><td/><td colspan="3">rics on energy minimization ('E') and hierarchical</td></tr><tr><td/><td/><td/><td colspan="3">regularization ('H') on CIFAR-100: 'E' / 'E+H'</td></tr><tr><td/><td/><td/><td>respectively. metric \ db</td><td>resnet-20</td><td>resnet-110</td></tr><tr><td colspan="3">metric \ l 2 baseline N-euclidean2 90.93 92.35 × • 90.34 92.21 angular2 90.47 92.38 cosine 90.53 92.40 D ter h 90.67 92.48 D bin h 90.67 92.48 D cos(HM) 90.84 92.93 D HMcos 90.94 92.69</td><td colspan="3">baseline baseline+l 2 N-euclidean2 67.59 / 68.65 73.95 / 73.96 63.86 62.02 68.03 72.90 angular2 67.83 / 67.76 74.40 / 73.89 cosine 68.11 / 68.45 73.37 / 73.37 D ter h 68.44 / 68.68 73.73 / 73.97 D bin h 68.52 / 68.69 73.97 / 74.26 DAM 68.58 / 68.86 73.43 / 73.50 D cos(AM) 68.58 / 68.60 73.14 / 73.65 D cos(AM) 67.57 / 68.36 73.14 / 73.72</td></tr><tr><td colspan="3">Table 3: Test accuracy (%) using different met-</td><td>D cos(HM)</td><td colspan="2">68.62 / 68.65 73.07 / 73.65</td></tr><tr><td colspan="3">rics on energy minimization ('E') and hierarchical regularization ('H') on CUB200: 'E' / 'E+H' re-spectively. metric \ db Resnet-18 Resnet-50 baseline 72.17 74.21</td><td colspan="3">Table 4: Test accuracy (%) using different met-rics with energy minimization ('E') and hierarchi-cal regularization ('H') on Cars: 'E' / 'E+H' re-spectively.</td></tr><tr><td>baseline+l 2</td><td>72.29</td><td>74.05</td><td>metric \ db</td><td>Resnet-18</td><td>Resnet-50</td></tr><tr><td colspan="3">N-euclidean2 72.61 / 75.99 73.49 / 76.14 angular2 72.43 / 76.11 73.55 / 76.66</td><td>baseline baseline+l 2</td><td>85.10 85.58</td><td>87.99 87.92</td></tr><tr><td>cosine</td><td colspan="2">72.12 / 75.64 73.26 / 76.85</td><td colspan="3">N-euclidean2 85.48 / 85.56 87.96 / 87.97</td></tr><tr><td>D ter h</td><td colspan="2">72.58 / 75.99 73.57 / 76.37</td><td>angular2</td><td colspan="2">85.11 / 85.13 88.34 / 87.85</td></tr><tr><td>D bin h DAM DAM cos</td><td colspan="2">72.55 / 76.21 73.57 / 76.99 73.04 / 76.02 73.88 / 75.95 72.31 / 76.14 73.59 / 77.32</td><td>cosine D ter h D bin h</td><td colspan="2">85.57 / 85.73 88.01 / 87.86 85.35 / 85.99 85.35 / 88.07 86.22 / 85.99 88.32 / 88.14</td></tr><tr><td>D AMcos</td><td colspan="2">72.28 / 75.37 72.42 / 74.12</td><td>DAM</td><td colspan="2">85.66 / 85.66 88.39 / 88.11</td></tr><tr><td>DGM</td><td colspan="2">72.90 / 76.35 74.16 / 75.30</td><td>DAM cos</td><td colspan="2">85.52 / 86.05 87.92 / 88.57</td></tr><tr><td>D HMcos</td><td colspan="2">72.55 / 76.11 74.64 / 76.94</td><td>D AMcos</td><td colspan="2">85.76 / 86.43 88.07 / 87.96</td></tr><tr><td>D cos(HM)</td><td colspan="2">72.55 / 76.32 72.86 / 76.56</td><td>D cos(AM)</td><td colspan="2">85.54 / 85.52 88.22 / 88.13</td></tr></table>

Peer review (iclr2020):
## Summary The paper tackles the problem of promoting diversity in the weights of deep neural networks. The problem is interesting and useful. The paper argues that hierarchical learning and hyper spherical learning are important in addressing this problem. The paper provides experiments on CIFAR-10 and CIFAR-100 where the improvements of using such regularization is visible but not sufficiently significant. ## Contribution of the paper 1. The paper proposed a regularization to training neural networks with discrete angular distance metric on the weights. 2. The paper shows improved performance on CIFAR-10 and CIFAR-100. ## Overall feedback I found the paper is well motivated and the proposed approach to be interesting. But I found the experimental validation a bit confusing. The improvments of the proposed approach also seems quite marginal. The contribution of different regularization terms is not understood clearly as well. So I am leaning towards rejection. ## Detailed feedback and questions for rebuttal 1. The writing could be improved significantly. I had a hard time to find exactly what different regularization terms are, e.g., E, H, L2. The paper could be more clear by clearly stating these regularization equations. 2. Please capitalize *eq. (1)* to *Eq. 1*. 3. It seems there are three regularization E, L2 and H. But different tables show different combinations. For example, Table 1 has E and E+l2 while Table 2 has E and E+H and Table 3 has only E+H. Can you provide full results on all datasets on E, E+l2, E+H? Without seeing the full results it is hard to draw any conclusions. 4. Please correct the text *resnet-100* to *resnet-110* assuming you are using resnet-110. 5. It seems E+H improves marginally over E. Can you elaborate the explanation about it? 6. Why E+l2 improves so much (+2%) on CIFAR10?