Deep Hierarchical-Hyperspherical Learning (DH^2L)

1 INTRODUCTION . Diversity promoting learning has been widely adopted via enlarging pairwise distances ( Xie et al. , 2018 ; 2017a ; Liu et al. , 2018 ) , increasing orthogonality ( Xie et al. , 2018 ) , reducing covariance between parameters ( Xie et al. , 2017b ) , or reducing correlation on feature ( Cogswell et al. , 2016 ) to improve generalization performance . Among them , diversity promoting regularization ( Xie et al. , 2017a ; b ; Liu et al. , 2018 ) by enforcing large diversity between projection parameters achieves a reasonable performance without modifying the model structure . While optimizing the objective function with a covariance matrix in ( Xie et al. , 2017b ; 2018 ) is nontrivial , the diversity promoting regularization via minimizing energy of parameters of deep neural networks has been proposed ( Liu et al. , 2018 ) in a simpler way . By minimizing a pairwise distance between parameters on a hypersphere with the known metric , they achieved the improved generalization performance . Following an efficient regularization on the hypersphere , we explore further this direction with three main concepts ( hierarchical and hyperspherical learning with discrete metrics ) . 1 ) Why hierarchical learning ? Hierarchical inference explains human intelligence . In ( Kurzweil , 2013 ) , it states that “ the neocortex contains about 300 million very general pattern recognizers , arranged in a hierarchy ” . Applying the hierarchy of multiple classes based on semantic taxonomy is a natural choice to devise machine intelligence . Effectiveness of the hierarchical learning can be found in ( Verma et al. , 2012 ) . 2 ) Why hyperspherical learning ? Hypersphere is the set of points at the equidistance ( radius ) from a given point ( centroid ) in a certain dimensional space . Due to the denominator in the unit-length normalization ( w‖w‖ , w ∈ R d+1 ) , the angular distance defined on the hypersphere converges when the magnitude of w goes infinity while Euclidean distance goes infinity . Due to this bounded property , a hierarchical structure with multiple separated hyperspheres can be defined . 3 ) Why discrete metric learning ? If vector points form discontinuous series with discrete representation ( e.g . multi-dimensional binary or ternary ) , they are isolated from each other with a certain margin . This property may fit with a disconnected/groupwise manifold space problem . We note that making points to be equidistributed where a pairwise distance is maximized is a nontrivial task . As points in a discrete metric space are finite and isolated , search efforts to satisfy those constraints can be reduced . In this paper , we propose to apply a hierarchical structure to parameter regularization on the multiple groupwise hyperspheres . In order to find an appropriate metric in this space , we explore a discrete angular metric . We examine the proposed method on extensive experimental setups in terms of datasets and deep network models . 2 MULTIPLE SEPARATED HYPERSPHERES . Samples observed from the real world may be on disconnected manifolds . In other words , the disjoint union of those manifolds can generate the global manifold ( Lee , 2000 ) . In this section , we decompose the one space into multiple spaces ( manifolds ) and re-define the spaces in a hierarchical point of view . 2.1 DISCONNECTED MANIFOLD VIA EQUIVALENT RELATIONS . Since it is not suitable to measure a pairwise distance between high dimensional vectors which have the hierarchical structure in the same space , we construct another identification space where subspaces ( sub-manifolds ) are isolated ( via equivalence relation ( Tu , 2010 ) ) . Denote d-sphere Sd to be the set of points that satisfies Sd = { w ∈ Rd+1 : ‖w‖ = 1 } . We construct multiple separated hyperspheres using multiple identifying relations . In Figure 1 , we use a center parameter and a surface parameter to define a projection parameter vector w which constitutes a hypershpere space . 2.2 PRIOR DISTRIBUTION AND REGULARIZATION . To make parameter vectors uniformly distributed on the unit hypersphere , parameters are sampled from the Gaussian normal distribution ( Muller , 1959 ; Harman & Lacko , 2010 ) . This is because the normal distribution is spherically symmetric ( Muller , 1959 ) . In a Bayesian point of view , neural networks with Gaussian priors are known to induce l2-norm regularization ( Vladimirova et al. , 2019 ) . From two evidences , we know that enforcing parameters to have the Gaussian prior is important for hyperspherical learning in neural networks . Note that a parameter which is calculated from the difference arithmetic operation with two parameters on the normal Gaussian distribution is on the normal difference distribution . 3 METHOD . In deep neural networks , the objective function J with regularization R in addition to a loss L , JR ( W ) = L ( x , W ) + λR ( W ) , is optimized to find the optimal W having a near minimum loss L , arg minW JR ( x , W ) , where x ∈ Rd0 denotes an input vector , W = { Wi ∈ Rdi−1×ci : Wi = { wj ∈ Rdi−1 } , j = 1 , . . . , ci , i = 1 , . . . , L } denotes a set of parameter matrices ( i.e . neurons/kernels ) , L denotes the number of layers , and λ > 0 is to control the degree of the regularization . For a classification task , the cross entropy loss is used for the loss function L. We propose a new regularization formulationR preserving a hierarchical parameter structure in Section 3.1 . Figure 2 : A series of levelwise ( l ) spheres with a radius ( rl ) is shown : ( a ) A radius of an oveall area converges to r0 1−δ ( = ∑∞ l r0δ l : the sum of radius series ) as l goes to infinity where l denotes a level , r0 is their initial radius , and δ is the ratio between radiuses ( rl , rl−1 ) , rlrl−1 < 1 . ( b ) The radius of the overall area is bounded to the initial radius r0 of spheres . This bears a resemblance to the process of repeat of Hypersphere packing which arranges non-overlapping spheres within a containing space . ( c ) 2-sphere is defined following ( b ) which is appropriate to model a hierarchical structure . This can be generalized with a hypersphere ( Sd , d ≥ 3 ) in a higher dimensional space . 3.1 REGULARIZATION FOR HIERARCHICAL AND HYPERSPHERICAL HYPOTHESES . Denote w a parameter vector ( called a projection parameter , an element of W at a certain layer ) to transform a given input into an embedding space defined in a Euclidean metric space : x ∈ Rd+1 7→ wTx ∈ R. By the definition of unit-length projection w‖w‖ , a new parameter vector ŵ can be defined on d-sphere : Sd = { ŵ ∈ Rd+1 : ‖ŵ‖ = 1 } where ‖ · ‖ denotes l2-norm and a center is zero . In other word , the parameter ŵ can be defined by a center parameter vector wc ∈ Rd+1 and a surface parameter vector ws ∈ Rd+1 using an arithmetic operation : ŵ : = ws−wc . We define the d-sphere with the center and surface parameters : Sdwc = { ws − wc ∈ R d+1 : ‖ws − wc‖ = 1 } . For a notation simplicity , we use w instead of ŵ hereafter . While we consider a radius equals to 1 for simplicity , the parameter vector can have a radius r > 0 . 3.1.1 HIERARCHICAL PARAMETERIZATION VIA LEVELWISE AND GROUPWISE PARAMETERS . We assume that the hierarchical structure consists of a levelwise structure with a notation ( l ) and a groupwise structure with a notation g below . Through defining these structures serially , we explain hierarchical parameters . Levelwise structure The projection parameter w ∈ Rd+1 on Sdwc can be defined with a levelwise notation ( l ) as follows , w ( l ) : = w ( l ) s −w ( l ) c ( 1 ) where Sd w ( l ) c denotes the l-th level d-sphere centered at w ( l ) c . In this paper , we define the parameter in a higher dimensional space than that shown in Figure 2 . In a levelwise setting , w ( l ) s and w ( l ) c at the current level ( l ) are additively represented based on a center parameter of the previous level ( l− 1 ) : w ( l−1 ) c + −−→ ∆w ( l ) 7→ w ( l ) c , where w ( l−1 ) c = ∑l−1 i −−→ ∆w ( i ) is the accumulated and −−→ ∆w ( l ) denotes a connecting parameter from w ( l−1 ) c to w ( l ) c . By denoting−−→ ∆w ( l ) as w ( l , l−1 ) , the center vector at the l-th level is defined as , w ( l ) c : = w ( l , l−1 ) c +w ( l−1 ) c , and the surface vector as w ( l ) s : = w ( l , l−1 ) s +w ( l−1 ) c . Both the center parameter and the surface parameter at the current level are based on the center parameter at the previous level1 . Hence , Eq . 1 is equivalent to w ( l ) = w ( l , l−1 ) s −w ( l , l−1 ) c . ( 2 ) Note that we use a superscript ( l , l − 1 ) to denote parameters at the l-th level connected from the center parameter at the ( l − 1 ) -th level . Groupwise structure With a group notation gk , the center parameter in Eq . 2 can be rephrased as w ( l , l−1 ) c , gk , on Sdw ( l , l−1 ) c , gk which denotes d-sphere of gk group at the l-th level . Each group constitutes a group set at the l-th level g ( l ) : = { gk } |g ( l ) | k=1 where g ( l ) ⊆ G ( l ) , G ( l ) denotes a batch group set at the l-th level , and | · | denotes the cardinality . The group set g ( l ) at the l-th level is conditioned on 1As not every sample has a child sample , it might be more reasonable to branch from representative parameter or center parameter rather than from individual projection parameters . a group set at the ( l − 1 ) -th level , g ( l−1 ) : = { gk′ } |g ( l−1 ) | k′=1 where g ( l−1 ) ⊆ G ( l−1 ) . By their groupwise relation over levels , an adjacency indication2 P ( l , l−1 ) ( { G ( l−1 ) , G ( l ) } ) ∈ { 0 , 1 } |G ( l−1 ) |×|G ( l ) | is given . Hence , the i-th parameter vector on Sd w ( l , l−1 ) c , gk is defined as follows , w ( l ) gk , i : = w ( l , l−1 ) s , gk , i −w ( l , l−1 ) c , gk , ( 3 ) where { w ( l , l−1 ) s , gk , i , w ( l , l−1 ) c , gk } ∀i is calculated based on a center vector w ( l−1 ) c , gk′ at the ( l − 1 ) -th level3 , i = 1 , . . . , |gk| , and |gk| denotes the number of surface parameter vectors in the group gk .

Summary: The paper tackles the problem of promoting diversity in the weights of deep neural networks. The problem is interesting and useful. The paper argues that hierarchical learning and hyper spherical learning are important in addressing this problem. The paper provides experiments on CIFAR-10 and CIFAR-100 where the improvements of using such regularization is visible but not sufficiently significant.