Title: COMPRESSIVE HYPERSPHERICAL ENERGY MINIMIZATION

Abstract: Minimum hyperspherical energy (MHE) has demonstrated its potential in regularizing neural networks and improving the generalization. MHE was inspired by the Thomson problem in physics where the distribution of multiple propelling electrons on a unit sphere can be modeled via minimizing some potential energy. Despite its practical effectiveness, MHE suffers from some difficulties in optimization as the dimensionality of the space becomes higher, therefore limiting the potential to improve network generalization. To address these problems, we propose the compressive minimum hyperspherical energy (CoMHE) as a more effective regularization for neural networks. Specifically, CoMHE utilizes a projection mapping to reduce the dimensionality of neurons and minimizes their hyperspherical energy. According to different constructions for the projection mapping, we propose two major variants: random projection CoMHE and anglepreserving CoMHE. As a novel extension, We further consider adversarial projection CoMHE and group CoMHE. We also provide some theoretical insights to justify the effectiveness. Our comprehensive experiments show that CoMHE consistently outperforms MHE by a considerable margin, and can be easily applied to improve different tasks such as image recognition and point cloud recognition.

1. INTRODUCTION:

Recent years have witnessed the tremendous success of deep neural networks in a variety of tasks. With its over-parameterization nature and hierarchical structure, deep neural networks achieve unprecedented performance on many challenging problems [1] [2] [3] , but their strong approximation ability also makes it easy to overfit the training set, which greatly affects the generalization on unseen samples. Therefore, how to restrict the huge parameter space and properly regularize the deep networks becomes increasingly important. Regularization of neural networks can be roughly categorized into implicit and explicit. Implicit regularization usually does not directly impose explicit constraints on the neuron weights, and instead it regularizes the networks in an implicit manner in order to prevent overfitting and stabilize the training dynamics. A lot of classic methods fall into this category, such as batch normalization [4], dropout [5] , weight normalization [6], group normalization [7] , etc. Explicit regularization [8] [9] [10] [11] [12] [13] usually introduces some penalty terms for the neuron weights, and jointly optimizes it along with the other objective functions. Among many explicit regularizations, minimum hyperspherical energy (MHE) [13] is a simple yet effective regularization that encourages the hyperspherical diversity among neurons and significantly improves the network generalization. From geometric perspective, MHE regularizes the directions of neuron weights by minimizing a special potential energy on a unit hypersphere that characterizes the hyperspherical diversity (such energy is defined as hyperspherical energy [13] ). In contrast, standard weight decay only regularizes the norm of neuron weights, which can be viewed as regularizing one dimension of the neuron weight. In fact, MHE completes an important missing piece in the standard regularization in neural networks by introducing the regularization for the neuron directions (i.e., regularizing the rest dimensions of the neuron weight).

Despite its elegant interpretation and good performance, MHE still has a few critical problems which limit MHE to unleash its full potential. First, MHE suffers from huge number of local minima and stationary points due to its highly non-convex objective function. The problem can get even worse when the dimension gets higher and the number of neurons becomes larger [14, 15] . Second, the gradient w.r.t the neuron weight of the original MHE objective is deterministic. Unlike the weight decay whose objective is convex, MHE has a complex and non-convex regularization term. Therefore, deterministic gradients may quickly fall into one of the stationary points and get stuck there. Third, when the number of neurons is smaller than the dimension of the space (it is often the case in neural networks), MHE defines an ill-posed problem because the neurons can not even fully occupy the space. It will be less meaningful in this case to encourage the hyperspherical diversity. Last, in high-dimensional spaces, neurons are likely to be orthogonal to each other. As a result, in the original measure of diversity in MHE, these high-dimensional neurons can be trivially "diverse", leading to very small gradients that may result in optimization difficulties.

In order to address these problems, we propose the compressive minimum hyperspherical energy (CoMHE) as a more effective regularization for neural networks. The high-level intuition behind CoMHE is to project neurons to various spaces whose dimensions are significantly smaller than the original one such that the hyperspherical energy can get minimized more effectively. Specifically, CoMHE first maps the neuron weight from a high-dimensional space to a low-dimensional one and then applies standard MHE to regularize these neurons. As a result, how to map the neuron weights to a low-dimensional space while preserving some of the desirable information in highdimensional space is our major focus. Since MHE regualrizes the directions of neurons, the most important information we care in MHE is the angular similarity between different neuron weights. To this end, we explore multiple novel methods to perform the projection propose and propose two main approaches: random projection and angle-preserving projection, which can reduce the dimensionality of neurons while still partially preserving the angles.

Random projection (RP) [16] is a naturally motivated choice to achieve the dimensionality reduction in MHE due to its simplicity and nice theoretical properties. RP can provably preserve the angular information, and most importantly, introduce certain degree of randomness to the gradients, which can help MHE escape from some stationary points. The role that the randomness of RP serves in CoMHE is actually similar to the simulated annealing [17, 18] that is widely used in Physics to solve Thomson problem. Such randomness is also empirically shown to benefit the network generalization [19] . Furthermore, we also prove that using RP can well preserve the pairwise angles between neurons. Building upon RP, we propose the angle-preserving projection (AP) as an alternative way to project the neurons. AP is inspired by the goal that we aim to preserve the pairwise angles between neurons. Constructing an AP that can project neurons to a low-dimensional space and preserve the angles is usually very difficult even with powerful non-linear functions, which is suggested by the strong conditions required for conformal mapping in complex analysis [20] . Therefore, we adopt a different approach to construct AP by framing the AP construction as an optimization problem which can be solved jointly with the neural network. Furthermore, we propose the adversarial projection for CoMHE, which minimizes the maximal energy attained by learning the projection. We formulate it as a min-max optimization and solve it jointly with the neural network.

However, it is inevitable to lose some information in the high-dimensional space and the neurons may only get diverse in some low-dimensional spaces. To address this, we introduce multiple projections to approximate the learning objective of MHE in the original high-dimensional space more accurately. Specifically, we project the neurons to multiple subspaces, compute the MHE loss in each space separately and finally minimize the aggregation (i.e., average or max) of these MHE losses. Besides, we also reinitialize these projection matrix randomly every certain number of iterations to avoid trivial solutions. Our empirical study shows that CoMHE with RP or AP performs consistently better than the standard MHE in nearly all applications. Our contributions can be summarized as:

• To address the drawbacks of MHE, we propose CoMHE as a generic regularization to effectively minimize hyperspherical energy of neurons for better generalizability of neural networks. • We explore a number of ways to construct the projection for dimensionality reduction. Random projection and angle-preserving projection are proposed to achieve the dimensionality reduction of neurons while being able to partially preserve the angular information in MHE. Moreover, we consider several interesting variants such as adversarial projection CoMHE and group CoMHE. • We provide partial theoretical insights to show that some projections used in the paper can well preserve the angular similarity between different neurons. 

2. RELATED WORK:

Diversity-based regularization has been found useful in sparse coding [21, 22] , ensemble learning [23, 24] , self-paced learning [25], metric learning [26] , latent variable models [27] , person re-identification [28] , face recognition [13] , etc. There are a number of ways to characterize diversity in previous works. Early studies in sparse coding [21, 22] model the diversity with the empirical covariance matrix and show that the generalization capability of dictionary can be improved by regularizing its diversity. [29] promotes the uniformity among eigenvalues of the component matrix in a latent space model. [30] encourages the diversity of latent space models by imposing constraints to the pairwise angle among components. [9, 10, [31] [32] [33] 33] characterize diversity among neurons with orthogonality, and regularize the neural network using the orthogonality. Inspired by the Thomson problem in physics, MHE [13] defines the hyperspherical energy to characterize the diversity on a unit hypersphere and shows significant and consistent improvement in supervised learning tasks.

There are two MHE variants in [13] : full-space MHE and half-space MHE. Compared to full-space MHE, the half-space variant [13] further eliminates the collinear redundancy by constructing virtual neurons with the opposite direction to the original ones and then minimizing their hyperspherical energy together. The importance of regularizing angular information is also discussed in [34] [35] [36] [37] [38] [39] [40] [41] .

3 COMPRESSIVE MHE 3.1 REVISITING STANDARD MHE MHE characterizes the diversity of N neurons (W N = {w 1 , • • • , w N ∈ R d+1 }

) on a unit hypersphere using hyperspherical energy which is defined as

EQUATION

where • denotes 2 norm, f s (•) is a decreasing real-valued function (we use f s (z) = z -s , s > 0, i.e., Riesz s-kernels), and ŵi = wi wi is the i-th neuron weight projected onto the unit hypersphere

S d = {v ∈ R d+1 | v = 1}. For convenience, we denote ŴN = { ŵ1 , • • • , ŵN ∈ S d }, and E s = E s,d ( ŵi | N i=1

). Note that, each neuron is a convolution kernel in convolutional neural networks (CNNs). MHE minimizes the hyperspherical energy of neurons using gradient descent during backpropagation, and MHE is typically applied to the neural network layer-wise. For example, we write down the gradient of E 2 w.r.t ŵi and make the gradient to be zero:

EQUATION

where α j = ŵiŵj -4 . We use toy and informal examples to show that high dimensional space (i.e., d is large) leads to much more stationary points than low-dimensional one. Assume there are K = K 1 + K 2 stationary points in total for ŴN to satisfy Eq. 2, where K 1 denotes the number of stationary points in which every element in the solution is distinct and K 2 denotes the number of the rest stationary points. We give two examples: (i) For (d + 2)-dimensional space, we can extend the solutions in (d + 1)-dimensional space by introducing a new dimension with zero value. The new solutions satisfy Eq. 2. Because there are d + 2 ways to insert the zero, we have at least (d + 2)K stationary points in (d + 2)-dimensional space. (ii) We denote K 1 = K1 (d+1)! as the number of unordered sets that construct the stationary points. Then in (2d + 2)-dimensional space, we have

ŵE j = 1 √ 2 { ŵj ; ŵj } ∈ S 2d+1

, ∀j satisfying Eq. 2. Moreover, the order of the elements in ŵE j does not matter for Eq. 2.Thus, there are at least (2d+2)! 2 d+1 K 1 + K 2 stationary points for ŴN in (2d + 2)dimensional space, and besides this trivial construction, there are much more stationary points. Now one can know that there are increasingly more stationary points for MHE in higher dimensions.

3.2. GENERAL FRAMEWORK:

To overcome MHE's drawbacks in high dimension, we propose the compressive MHE that projects the neurons to a low-dimensional space and then minimizes the hyperspherical energy of the projected neurons. In general, CoMHE aims to minimize the following form of hyperspherical energy:

EQUATION

where g : S d → S k takes a normalized (d + 1)-dimensional input and outputs a normalized (k + 1)dimensional vector. g(•) can be either linear or nonlinear mapping. In this paper, we only focus on the linear mapping. For nonlinear mapping, the simplest case is to apply a multi-layer perceptron. Similar to MHE, CoMHE also serves as a regularization penalty in neural networks.

3.3. RANDOM PROJECTION FOR COMHE:

Random projection is in fact one of the most straightforward way to reduce dimensionality while partially preserving the angular information. More specifically, we use a random mapping g(v) = P v P v

where P ∈ R (k+1)×(d+1) is a Gaussian distributed random matrix (each entry follows i.i.d. normal distribution). In order to reduce the variance of the objective value, we use multiple random projection matrix to project the neurons and compute the MHE objective separately, shown as follows:

EQUATION

where P c , ∀c is a random matrix with each entry satisfying the normal distribution N (0, 1). According to the properties of normal distribution [42], every normalized row of the random matrix P is uniformly distributed on a hypersphere S d , which indicates that the projection matrix P is able to cover all the possible subspaces. Multiple projection matrices can also be interpreted as multi-view projection, because we are making use of information from multiple projection views. In fact, we do not necessarily need to average the energy for multiple projections, and instead we can use maximum operation (or some other meaningful aggregation operations). Then the objective becomes

max c N i=1 N j=1,j =i f s ( Pc ŵi Pc ŵi - Pc ŵj Pc ŵj

). Considering that we aim to minimize this objective, the problem is in fact a simple min-max optimization. Note that, we will typically re-initialize the random projection matrices every certain number of iterations. Most importantly, using RP can provably preserve the angular similarity between different neurons.

3.4. ANGLE-PRESERVING PROJECTION FOR COMHE:

Recall that our goal is to find a projection to project the neurons to a low-dimensional space while simultaneously preserving angular information. We transform the goal to an optimization problem:

EQUATION

where P ∈ R (k+1)×(d+1) is the projection matrix and θ (v1,v2) denotes the angle between v 1 and v 2 . For implementation convenience, we can replace the angle with the cosine value (e.g.., use cos(θ ( ŵi, ŵj ) ) to replace θ ( ŵi, ŵj ) ), so that we can directly use the inner product of normalized vectors to measure the angular similarity. With P obtained in Eq. 5, we use a nested loss function: 

E A s ( ŴN , P ) = N i=1 N j=1,

EQUATION

for which we propose two different ways to optimize the projection matrix P . We can approximate P using a few gradient descent updates. Specifically, we use two different ways to perform the optimization. Naively, we use a few gradient descent steps to update P in order to approximate P and then update W N , which proceeds alternatively. The number of iteration steps that we use to update P is a hyperparemter and needs to be determined by cross-validation. Besides the naive alternative one, we also use a different optimization of W N by unrolling the gradient update of P .

Alternative optimization. The alternative optimization is to optimize P alternatively with the network parameters W N . Specifically, in each iteration of updating the network parameters, we update P every number of inner iterations and use it as an approximation to P (the error depends on the number of gradient steps we take). Essentially, we are alternatively solving two separate optimization problems for P and W N with gradient descent.

Unrolled optimization. Instead of naively updating W N with approximate P in the alternative optimization, the unrolled optimization further unrolls the update rule of P and embed it within the optimization of network parameters W N . If we denote the CoMHE loss with a given projection matrix P as E A s (W N , P ) which takes W N and P as input, then the unrolled optimization is essentially optimizing E A s (W N , P -η • ∂L P ∂P ). It can also be viewed as minimizing the CoMHE loss after a single step of gradient descent w.r.t. the projection matrix. This optimization includes the computation of second-order partial derivatives. Note that, it is also possible to unroll multiple gradient descent steps. Similar unrolling is also applied in [43] [44] [45] .

3.5. NOTABLE COMHE VARIANTS:

We provide more interesting CoMHE variants as an extension and complement to the previous two major methods. We will have some preliminary empirical study on these variants, but our main focus is still on RP and AP.

Adversarial Projection for CoMHE. We consider a very novel CoMHE variant that adversarially learns the projection. The intuition behind is that we want to learn a projection basis that maximizes the hyperspherical energy while the final goal is to minimize this maximal energy. With such intuition, we can naturally construct a min-max optimization problem:

EQUATION

which can be solved by gradient descent similar to [46] . From a game-theoretical perspective, P and ŴN can be viewed as two players that are competing with each other. However, due to the instability of solving the min-max problem, the performance of adversarial projection is also unstable. We will empirically evaluate this approach in experiments.

Group CoMHE. Group CoMHE is a very special case in the CoMHE framework. The basic idea is to divide the weights of each neuron into several groups and then minimize the hyperspherical energy within each group. For example in CNNs, group MHE divides the channels into groups and minimizes within each group the MHE loss. Specifically, the objective function of group CoMHE is

EQUATION

where P c is a diagonal matrix with every diagonal entry being either 0 or 1, and c P c = I (in fact, this is optional). There are multiple ways to divide groups for the neurons, and typically we will divide groups according to the channels, similar to [7] . More interestingly, one can also divide the groups in a stochastic fashion.

Mixed CoMHE. Since all the proposed CoMHE variants and the original MHE share the same goal of diversifying the neurons on a unit hypersphere, we can in fact combine any of these MHE variants and use them together. It also has some flavor of multi-view learning [47] , since we optimize hyperspherical energy with different objectives. Experimenting all the possible combination is cumbersome and also out of the main scope of this paper, so we defer it to future investigation.

3.6. SHARED PROJECTION BASIS IN NEURAL NETWORKS:

In general, we usually need different projection bases for neurons in different layers of the neural network. However, we find it beneficial to share some projection bases across different layers. We only share the projection matrix for the neurons in different layers that have the same dimensionality. For example in a neural network, if the neurons in the first layer have the same dimensionality with the neurons in the second layer, we will share their projection matrix that reduces the dimensionality. Sharing the projection basis can effectively reduce the number of projection parameters and may also reduce the inconsistency within the hyperspherical energy minimization of projected neurons in different layers. Most importantly, it can empirically improve the network generalizability (as shown in our experiment) while using much fewer parameters and saving more computational overheads.

4. THEORETICAL INSIGHTS:

The section mainly discusses some theoretical insights towards understanding CoMHE in terms of angular approximation accuracy and statistical intuitions.

4.1. ANGLE PRESERVATION:

We start with some highly relevant properties of random projection and then delve into the analysis of angular preservation. Lemma 1 (Mean Preservation of Random Projection). For any w 1 , w 1 ∈ R d and any random Gaussian distributed matrix P ∈ R k×d where

P ij = 1 √ n r ij , if r ij , ∀i, j are i.i.d. random variables from N (0, 1), we have E( P w 1 , P w 2 ) = w 1 , w 2 .

This lemma indicates that the mean of randomly projected inner product is well preserved, partially justifying why using random projection actually makes senses. Lemma 2 (Johnson-Lindenstrauss Lemma [48, 49] ). Let w 1 , w 2 ∈ R d be vectors, and P ∈ R k×d , k < d be a random projection matrix with entries i.i.d. drawn from a 0-mean σ-subgaussian distribution. With P w 1 , P w 2 ∈ R k being the projected vectors of w 1 , w 2 , then, ∀ ∈ (0, 1),

(1 -) w1 -w2 2 kσ 2 < P w1 -P w2 2 < (1 + ) w1 -w2 2 kσ 2 (9)

holds with probability at least 1 -2 exp(-k 2 8 ).

Johnson-Lindenstrauss lemma (JLL) establishes a guarantee for the Euclidean distance between randomly projected vectors. Note that, N (0, 1) is a 1-subgaussian distribution, so Gaussian random projection satisfies this lemma. However, JLL does not show anything informative about the angle preservation, and it is also nontrivial to give a useful guarantee for angular similarity from JLL. Theorem 1 (Angle Preservation I). Given w 1 , w 2 ∈ R d , P ∈ R k×d is a random projection matrix that has i.i.d. 0-mean σ-subgaussian entries, and P w 1 , P w 2 ∈ R k are the randomly projected vectors of w 1 , w 2 under P . Then ∀ ∈ (0, 1), we have that

EQUATION

which holds with probability 1 -2 exp(-

k 2 8 ) 2 .

Theorem 2 (Angle Preservation II). Given w 1 , w 2 ∈ R d , P ∈ R k×d is a Gaussian random projection matrix where P ij = 1 √ n r ij (r ij , ∀i, j are i.i.d. random variables from N (0, 1)), and P w 1 , P w 2 ∈ R k are the randomly projected vectors of w 1 , w 2 under P . Then ∀ ∈ (0, 1) and w 1 w 2 > 0, we have that

EQUATION

which holds with probability 1 -6 exp(-k 2 (

2 2 - 3 3 )).

Theorem 1 is one of our main theoretical results and reveals that the angle between randomly projected vectors is well preserved. Note that, the parameter σ of the subgaussian distribution is not related to our bound for the angle, so any Gaussian distributed random matrix has the property of angle preservation. The projection dimension k is related to the probability that the angle preservation bound holds. Theorem 2 is a direct result from [50] . It again shows that the angle between randomly projected vectors is provably preserved. Both Theorem 1 and Theorem 2 give upper and lower bounds for the angle between randomly projected vectors. If θ (w1,w2) > arccos( +3 2 3 + 2 ), then the lower bound in Theorem 1 is tighter than the lower bound in Theorem 2. If θ (w1,w2) >

arccos( 1-3 2 -(1-) √ 1-2 3 -2

), the upper bound in Theorem 1 is tighter than the upper bound in Theorem 2. To conclude, Theorem 1 gives tighter bounds when the angle of original vectors is large.

Since AP is randomly initialized every certain number of iterations and learned to minimize the angular difference before and after the projection, AP usually performs better than RP in preserving angles. Without the angle-preserving optimization, AP reduces to RP.

4.2. STATISTICAL INSIGHTS:

In fact, we can also draw some theoretical intuitions from spherical uniform testing [51] in statistics. Spherical uniform testing is a nonparametric statistical hypothesis test that checks whether a set of observed data is generated from a uniform distribution on a hypersphere or not. Random projection is in fact an important tool [51] in statistics to test the uniformity on hyperspheres, while our goal is to promote the same type of hyperspherical uniformity (i.e., diversity). Specifically, we have N random samples w 1 , • • • , w N of S d -valued random variables, and the random projection p which is another random variable independent of w i , ∀i and uniformly distributed on S d . The projected points of w i , ∀i is y i = p w i , ∀i. The distribution of y i , ∀i uniquely determines the distribution of w 1 , which is specified by Theorem 3. Theorem 3 (Unique Distribution Determination of Random Projection). Let w be a S d -valued random variable and p be a random variable that is uniformly distributed on S d and independent of w. With probability one, the distribution of w is uniquely determined by the distribution of the projection of w on p. More specifically, if w 1 and w 2 are S d -valued random variables, independent of p and we have a positive probability for the event that p takes a value p 0 such that the two distributions satisfy p 0 w 1 ∼ p 0 w 2 , then w 1 and w 2 are identically distributed.

Theorem 3 shows that the distributional information is well preserved after random projection, providing the CoMHE framework a statistical intuition and foundation. We emphasize that the randomness here is in fact very crucial. For a fixed projection p 0 , Theorem 3 does not hold in general. As a result, random projection for CoMHE is well motivated from the statistical perspective.

5. DISCUSSIONS:

Comparison to existing works. One of the widely used regularizations is the orthonormal regularization [35, 52] that minimizes W W -I F where W denotes the weights of a group of neurons with each column being one neuron and I is an identity matrix. [10, 32] are also built upon orthogonality. In contrast, both MHE and CoMHE do not encourage orthogonality among neurons and instead promote hyperspherical uniformity and diversity.

The significance of CoMHE. CoMHE addresses some key problems in MHE, and further boosts the performance on various applications while enjoying elegant theoretical interpretations and properties. By projecting the neurons to a low-dimensional space, CoMHE is able to effectively reduce the number of stationary points and introduce more regularity to the neurons.

Randomness helps generalization? Both RP and AP introduce certain degree of randomness to CoMHE, and the empirical results show that such randomness can largely benefit the network generalization. It is well-known that randomness in SGD is one of the key ingredients that help deep models well generalize to unseen samples. Why CoMHE works well may also have some implicit relations to this observation. Beside this, [19] also theoretically shows that randomness can help generalization, which also partially justifies the effectiveness of CoMHE. 

6. EXPERIMENTS AND RESULTS:



6.1. IMAGE RECOGNITION:

We perform image recognition to show the improvement of regularizing CNNs with CoMHE. Our goal is to show the superiority of CoMHE rather than achieving stat-of-the-art accuracies on particular tasks. For all the experiments on CIFAR-10 and CIFAR-100 in the paper, we use the same data augmentation as [1, 37] . For ImageNet-2012, we use the same data augmentation in [35] . We train all the networks using SGD with momentum 0.9. All the networks use BN [4] and ReLU if not otherwise specified. By default, all the variants of CoMHE are built upon half-space MHE. More experimental details are given in each subsection and Appendix A.

6.1.1. ABLATION STUDY AND EXPLORATORY EXPERIMENTS:



Method:

Error (%) Baseline 28.03 HS-MHE [13] 25 ) and the number of projection to 5 (i.e., C = 5). For AP, the number of projection is 1 and the projection dimension is set to 30. For AP, we evaluate both alternative optimization and unrolled optimization. In alternative optimization, we update the projection matrix every 10 steps of network update. In unrolled optimization, we only unroll one-step gradient in the optimization. For G-CoMHE, we construct a group with every 8 consecutive channels. All these design choices are obtained using cross-validation. We will also study how these hyperparameters affect the performance in the following experiments.

The results in Table 1 show that all of our proposed CoMHE variants can outperform the original half-space MHE by a large margin. Network depth. We evaluate RP-CoMHE and AP-CoMHE with different network depth on CIFAR-100. We use three plain CNN with 6, 9 and 15 convolution layers, respectively. For all the networks, we set the filter number in Conv1.x, Conv2.x and Conv3.x to 64, 128 and 256, respectively. Detailed network architectures are given in Appendix A. For RP, we set the projection dimension to 30 and the number of projection to 5. For AP, the number of projection is 1 and the projection dimension is set to 30. The results in Table 5 show that both RP-CoMHE and AP-CoMHE can outperform half-space MHE by a considerable margin while regularizing a plain CNN with different depth.

Shared projection basis. We take RP-CoMHE as an example to empirically verify the advantages of shared projection basis across different layers. We set the projection dimension to 20 and the number of projections to 30. The plain CNN-9 is used as baseline. Specifically for shared projection basis, we share the random projection basis in Conv1.x, Conv2.x and Conv3.x separately. The shared projection yields 24.6% error rate. For independent projection basis, we use separated projection basis for different layers and only obtain 26.05% error rate. The results show that using shared random projection basis for neurons of the same dimensionality improves the network generalization while saving parameters. Note that, all the other experiments use shared projection basis by default. Effectiveness of optimization. In order to verify that our CoMHE can better minimize the hyperspherical energy, we compute the hyperspherical energy E 2 (Eq. 1) for basedline CNN and CNN regularized by orthogonal regularization, HS-MHE, RP-CoMHE and AP-CoMHE during training. Note that, we compute the original hyperspherical energy rather than the energy after projection. All the networks use exactly the same initialization (the initial hyperspherical energy is the same). The results are averaged over five independent runs. We show the hyperspherical energy after the 20000-th iteration, because at the begining of the training, hyperspherical energy fluctuates dramatically and is unstable. Interested readers can refer to Appendix C for the complete energy dynamics. From Fig. 1 , one can observe that both RP-CoMHE and AP-CoMHE can better minimize the hyperspherical energy. RP-CoMHE can achieve the lowest energy with smallest standard deviation. From the absolute scale, the optimization gain does not seem to be significant. However, this is not the case. In the high-dimensional space, the variance of hyperspherical energy is usually small (already close to the smallest energy value) and is already very difficult to minimize, so the improvement achieved by our CoMHE is in fact very significant.

Naively learning projection basis from training data. We study the case where we enable the back-propagation gradient to flow back to the projection basis. That is to say, the model learns the projection basis naively using training data. We find that naively learning the projection basis yields much worse performance (26.5%), compared to RP-CoMHE (24.6%). It is even worse than our baseline half-space MHE (25.96%). The results show that naively learning projection basis from training data leads to inferior performance. Allowing the projection basis to be updated according to the training data could undermine the strength of CoMHE regularization imposed on the neurons. All the experiments in ablation study are performed using a VGG-like plain CNN, so we use the more powerful ResNet [1] to show that CoMHE is architecture-agnostic. We use the same experimental setting in [53] for fair comparison. We use a standard ResNet-32 as our baseline and the network architecture is specified in Appendix A. From the results in We evaluate CoMHE for large-scale image recognition on ImageNet-2012 [54] . We perform the experiment using ResNet-18, ResNet-34 and ResNet-50, and then report the top-1 validation error (center crop) in Table 8 . Our results show consistent and significant performance gain of CoMHE in all ResNet variants. Compared to the baselines, CoMHE can reduce the top-1 error for more than 1%. Since the computational overhead of CoMHE is almost neglectable, the performance gain is obtained without many efforts. Most importantly, as a plug-in regularization, CoMHE is shown to be architecture-agnostic and produces considerable accuracy gain in most circumstances.

Besides the accuracy improvement, we also visualize in Fig. 2 

7. CONCLUDING REMARKS:

This paper first analyzes some critical problems that the original MHE [13] encounters in highdimensional space. To address these problems, we propose a compressive hyperspherical energy minimization framework which first projects the neurons to a low-dimensional space and minimizes the hyperspherical energy for the projected neurons. The CoMHE gradients will flow from the projected space to the original space via the projection mapping and then update the original neurons. Specifically, we propose two important CoMHE variants: random projection CoMHE and anglepreserving CoMHE. We also provide some theoretical insights for CoMHE. Experiments on image recognition and point cloud recognition show the effectiveness of CoMHE.

Point cloud recognition settings. For all the PointNet and PointNet++ experiments, we exactly follow the same setting in the original papers [55, 56] and their official repositories 1 2 . Specifically, we combine CoMHE regularization to neurons in all the 1 × 1 convolution layers before the max pooling layer and the multi-layer perceptron classifier after the max pooling layer. We evaluate on PointNet with T-Net and without T-Net in order to demonstrate that CoMHE is not sensitive to architecture modifications. We follow all the default hyperparameters used in the official released code, and the only difference is that we further combine an additional regularization loss for the neurons in each layer. One can observe that CoMHE consistently performs better than half-space MHE [13] .

Besides PointNet, we combine CoMHE to PointNet++ [56] and further show the improvement of generalization introduced by CoMHE is agnostic to the architecture. We evaluate PointNet++ with and without CoMHE on ModelNet-40. Note that, we exactly follow the released code in the official repository where PointNet++ uses the single scale grouping model. Because the original paper [56] uses the multi-resolution grouping model, the baseline performance reported in our paper is not as good as the accuracy reported in the original paper. However, our purpose is to validate the effectiveness of CoMHE, so we only focus on the performance gain. One can observe that CoMHE achieves about 0.5% accuracy gain, while half-space MHE [13] only has about 0.2% accuracy gain. All the networks use exactly the same initialized weights to ensure the hyperspherical energy is the same at the beginning. One can observe that the hyperpsherical energy is actually very low for the initialized weights. This is because the initialized weights follows Gaussian distribution and the hyperspherical energy is computed with normalized weights. The normalized weights (sampled from Gassuain distribution) follows the uniform distribution on the hypersphere (see the theorem below), which can obtain the lowest hyperspherical distribution in expectation. However, when the weights of the neural network start to fit the data and minimize the data approximation loss, the neuron weights no longer follow the hyperspherical uniform distribution. Therefore the hyperspherical energy will quickly get large. This is when MHE and CoMHE are useful. From Fig. 3 , one can see that without any regularization on hyperspherical energy, the hyperspherical energy of the baseline network gets extremely large at the beginning and then decreases as the training goes. However, the final hyperspherical energy of the baseline network is still way higher than the CNNs regularized by MHE and CoMHE. Notice that, the orthogonal regularized CNN also obtain high hyperspherical energy at the end (similar to the baseline network). In contrast to MHE, we can observe that CoMHE can effectively minimize the hyperspherical energy and RP-CoMHE achieves significantly lower hyperspherical energy in the end, which well verifies the superiority of the proposed CoMHE. Theorem 4. The normalized vector of Gaussian variables is uniformly distributed on the sphere. Formally, let x 1 , x 2 , • • • , x n ∼ N (0, 1) and be independent. Then the vector

EQUATION

)

follows the uniform distribution on S n-1 , where z = x 2 1 + x 2 2 + • • • + x 2 n is a normalization factor.

Proof. A random variable has distribution N (0, 1) if it has the density function

EQUATION

)

A n-dimensional random vector x has distribution N (0, 1) if the components are independent and have distribution N (0, 1) each. Then the density of x is given by

EQUATION

Then we introduce the following lemma about the orthogonal-invariance of the normal distribution.

Lemma 6. Let x be a n-dimensional random vector with distribution N (0, 1) and U ∈ R n×n be an orthogonal matrix (U U = U U = I). Then Y = U x also has distribution N (0, 1).

Proof. For any measurable set A ⊂ R n , we have that

EQUATION

because of orthogonality of U . Therefore the lemma holds.

Because any rotation is just a multiplication with some orthogonal matrix, we know that normally distributed random vectors are invariant to rotation. As a result, generating x ∈ R n with distribution N(0, 1) and then projecting it onto the hypersphere S n-1 produces random vectors U = x x that are uniformly distributed on the hypersphere. Therefore the theorem holds.

:

https://github.com/charlesq34/pointnet

https://github.com/charlesq34/pointnet2

Figures and Tables:

Computational overhead. One of the significant advantages of CoMHE is its savings of floatingpoint operations (FLOPs). Assume there are N neurons with dimension d and the projection matrix is of size k × d. Then the FLOPs of original MHE is N (N -1)(d + 2), while the FLOPs of CoMHE is 2kd + N (N -1)(k + 2). In a standard setting where N = 512, d = 3 × 3 × 512, k = 10, we have that MHE has nearly 1200M FLOPs and CoMHE has only 3.32M FLOPs (even with multiple projection matrix, it is still much less than MHE), showing that CoMHE is much more efficient to compute. Both MHE and CoMHE are only used in training and do not affect inference speed.

Figure 1: Hyperspherical energy during training. All networks are initialized with the same random weights, so the hyperpsherical energy is the same before the training starts.

Figure 3: 2D CNN features with or without MHE on both training set and testing set. The features are computed by setting the output feature dimension as 2, similar to [34]. Each point denotes the 2D feature of a data point, and each color denotes a class. The red arrows are the classifier neurons of the output layer.

Fig.3shows the entire training dynamics (from initialization to the end of training) of the hyperspherical energy of baseline CNN and CNN regularized by orthogonal regularization, HS-MHE, AP-CoMHE and RP-CoMHE. All the networks use exactly the same initialized weights to ensure the hyperspherical energy is the same at the beginning. One can observe that the hyperpsherical energy is actually very low for the initialized weights. This is because the initialized weights follows Gaussian distribution and the hyperspherical energy is computed with normalized weights. The normalized weights (sampled from Gassuain distribution) follows the uniform distribution on the hypersphere (see the theorem below), which can obtain the lowest hyperspherical distribution in expectation. However, when the weights of the neural network start to fit the data and minimize the data approximation loss, the neuron weights no longer follow the hyperspherical uniform distribution. Therefore the hyperspherical energy will quickly get large. This is when MHE and CoMHE are useful. From Fig.3, one can see that without any regularization on hyperspherical energy, the hyperspherical energy of the baseline network gets extremely large at the beginning and then decreases as the training goes. However, the final hyperspherical energy of the baseline network is still way higher than the CNNs regularized by MHE and CoMHE. Notice that, the orthogonal regularized CNN also obtain high hyperspherical energy at the end (similar to the baseline network). In contrast to MHE, we can observe that CoMHE can effectively minimize the hyperspherical energy and RP-CoMHE achieves significantly lower hyperspherical energy in the end, which well verifies the superiority of the proposed CoMHE. Theorem 4. The normalized vector of Gaussian variables is uniformly distributed on the sphere. Formally, let x 1 , x 2 , • • • , x n ∼ N (0, 1) and be independent. Then the vector

We apply CoMHE to image recognition and point cloud recognition, and demonstrate consistent

Table content: <table/>

Testing error (%) of different CoMHE variants on CIFAR-100.

Table content: <table><tr><td/><td>.96</td></tr><tr><td>G-CoMHE</td><td>25.08</td></tr><tr><td>Adv-CoMHE</td><td>25.09</td></tr><tr><td>RP-CoMHE</td><td>24.39</td></tr><tr><td>RP-CoMHE (max)</td><td>24.77</td></tr><tr><td>AP-CoMHE (alter.)</td><td>24.95</td></tr><tr><td>AP-CoMHE (unroll)</td><td>24.33</td></tr></table>

The unrolled optimization in AP-CoMHE shows the significant advantage over alternative one and achieves the best accuracy. Both Adv-CoMHE and G-CoMHE achieve decent performance gain over HS-MHE, but not as good as RP-CoMHE and AP-CoMHE. Therefore, we will mostly focus on RP-CoMHE and AP-CoMHE in the remaining experiments. Error (%) on CIFAR-100 under different dimension of projection. CoMHE to 20. Because AP-CoMHE does not need to use multiple projections to reduce variance, we only use one projection in AP-CoMHE. Results are given in Table2. In general, RP-CoMHE and AP-CoMHE with different projection dimensions can consistently and significantly outperform the half-space MHE, validating the effectiveness and superiority of the proposed CoMHE framework. Specifically, we find that both RP-CoMHE and AP-CoMHE usually achieve the best accuracy when the projection dimension is 20 or 30. Since the unrolled optimization in AP-CoMHE is consistently better than the alternative optimization, we will stick to the unrolled optimization for AP-CoMHE in the remaining experiments if not otherwise specified.

Table content: <table><tr><td>Dimension of projection. We evaluate</td><td colspan="2">Projection Dimension</td><td>10</td><td>20</td><td>30</td><td>40</td><td>80</td></tr><tr><td>how the dimension of projection (i.e., k)</td><td>RP-CoMHE</td><td/><td>25.48</td><td>25.32</td><td>24.60</td><td>24.75</td><td>25.46</td></tr><tr><td>affects the performance. We use the plain CNN-9 as the backbone network and test</td><td colspan="2">AP-CoMHE (alter.) AP-CoMHE (unroll.)</td><td>25.21 25.32</td><td>24.60 24.59</td><td>24.95 24.33</td><td>24.97 24.93</td><td>24.99 25.12</td></tr><tr><td>on CIFAR-100. We fix the number of pro-</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="2">jections in RP-# Projections</td><td>1</td><td>5</td><td>10</td><td>20</td><td>30</td><td>80</td></tr><tr><td/><td>RP-CoMHE</td><td>25.11</td><td>24.39</td><td>25.11</td><td>24.6</td><td>24.82</td><td>24.92</td></tr><tr><td/><td>AP-CoMHE</td><td>24.33</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></table>

Error (%) on CIFAR-100 under different numbers of projections.

Table content: <table><tr><td>Width</td><td>t = 1</td><td>t = 2</td><td>t = 4</td><td>t = 8</td><td>t = 16</td></tr><tr><td>Baseline</td><td>47.72</td><td>38.64</td><td>28.13</td><td>24.95</td><td>25.45</td></tr><tr><td>HS-MHE [13]</td><td>35.16</td><td>29.33</td><td>25.96</td><td>23.38</td><td>21.83</td></tr><tr><td>RP-CoMHE</td><td>34.73</td><td>28.92</td><td>24.39</td><td>22.44</td><td>20.81</td></tr><tr><td>AP-CoMHE</td><td>34.89</td><td>29.01</td><td>24.33</td><td>22.6</td><td>20.72</td></tr></table>

Error (%) on CIFAR-100 with different network width. 32 × t and 64 × t, respectively. Specifically, we test the cases where t = 1, 2, 4, 8, 16. Taking t = 2 as an example, then the filter numbers in Conv1.x, Conv2.x and Conv3.x are 32, 64 and 128, respectively. For RP, we set the projection dimension to 30 and the number of projection to 5. For AP, the number of projection is 1 and the projection dimension is set to 30. The results are shown in Table4. Note that, we use the unrolled optimization in AP-CoMHE. From Table4, one can observe that the performance gains of both RP-CoMHE and AP-CoMHE are very consistent and significant. With wider network, CoMHE also achieves better accuracy. Compared to the strong results of half-space MHE, CoMHE still obtains more than 1% accuracy boost under different network width.

Table content: <table><tr><td>Network width. We evaluate RP-CoMHE</td><td/><td/><td/></tr><tr><td>and AP-CoMHE with different network</td><td/><td/><td/></tr><tr><td>width on CIFAR-100. We use the plain</td><td/><td/><td/></tr><tr><td>CNN-9 as our backbone network architec-</td><td/><td/><td/></tr><tr><td>ture, and set its filter number in Conv1.x,</td><td/><td/><td/></tr><tr><td>Conv2.x and Conv3.x (see Appendix A) to</td><td/><td/><td/></tr><tr><td>16 × t, Depth</td><td>CNN-6</td><td>CNN-9</td><td>CNN-15</td></tr><tr><td>Baseline</td><td>32.08</td><td>28.13</td><td>Not Converged</td></tr><tr><td>HS-MHE [13]</td><td>27.56</td><td>25.96</td><td>25.84</td></tr><tr><td>RP-CoMHE</td><td>26.73</td><td>24.39</td><td>24.21</td></tr><tr><td>AP-CoMHE</td><td>26.55</td><td>24.33</td><td>24.55</td></tr></table>

Error on CIFAR-100 with different network depth.

Table content: <table/>

Err. of different # iterations for re-initialization.Frequency of re-initialization in RP-CoMHE. In RP-CoMHE, we need to re-initialize the random projections every certain number of iterations to avoid trivial solutions caused by bad initialization. Here, we test how the frequency of re-initialization will affect the accuracy on CIFAR-100, with the projection dimension being 30 and the number of projection being 20. The iteration number being ∞ in Table6represents that the random projection is fixed throughout the training once it is initialized. The results shows the performance is not very sensitive to the frequency of re-initialization, but we cannot fix the random projection during training as it may cause trivial solutions and hurt the performance.

Table content: <table><tr><td># Iterations</td><td>1</td><td>200</td><td>1000</td><td>∞</td></tr><tr><td>RP-CoMHE</td><td>24.6</td><td>24.84</td><td>24.62</td><td>26.09</td></tr></table>

Error (%) on CIFAR-10/100 using ResNets.

Table content: <table/>

one can observe that both RP-CoMHE and AP-CoMHE can consistently outperform half-space MHE, showing that CoMHE can boost the performance across different network architectures. More interestingly, ResNet-32 regularized by CoMHE achieves impressive accuracy and is able to outperform 1001-layer ResNet by a large margin. Additionally, we note that from

Table content: <table><tr><td/><td/><td>Method</td><td>ResNet-18</td><td>ResNet-34</td><td>ResNet-50</td></tr><tr><td/><td/><td>baseline</td><td>32.95</td><td>30.04</td><td>25.30</td></tr><tr><td/><td/><td>Orthogonal [32]</td><td>32.65</td><td>29.74</td><td>25.19</td></tr><tr><td/><td/><td>Orthnormal [35]</td><td>32.61</td><td>29.75</td><td>25.21</td></tr><tr><td/><td/><td>MHE [13]</td><td>32.50</td><td>29.60</td><td>25.02</td></tr><tr><td/><td/><td>HS-MHE [13]</td><td>32.45</td><td>29.50</td><td>24.98</td></tr><tr><td/><td/><td>RP-CoMHE</td><td>31.90</td><td>29.38</td><td>24.51</td></tr><tr><td>Baseline</td><td>CoMHE</td><td>AP-CoMHE</td><td>31.80</td><td>29.32</td><td>24.53</td></tr><tr><td colspan="2">Figure 2: Visualization of the first-layer filters in ResNet.</td><td/><td/><td/><td/></tr></table>

Top-1 center crop error (%) on ImageNet-2012.

Table content: <table/>

we can regularize a plain VGG-like 9-layer CNN with CoMHE and achieve 20.81% error rate, which is nearly 2% improvement over the 1001-layer ResNet.

Table content: <table><tr><td>6.1.3 IMAGENET-2012</td></tr></table>

the first-layer filters learned by the baseline ResNet and the CoMHE-regularized ResNet. The filters look quite different after we regularize the network using CoMHE. Each filter learned by baseline focuses on a particular local pattern (e.g., edge, color and shape) and each one has a clear local semantic meaning. In contrast, filters learned by CoMHE focuses more on edges, textures and global patterns which do not necessarily have a clear local semantic meaning. However, from a representation basis perspective, it seems that having such global patterns is beneficial to the recognition accuracy. We also observe that filters learned by CoMHE does not pay too much attention to the color information, which is reasonable since humans generally do not need to use color to identify an object. Testing accuracy (%) on ModelNet-40. We evaluate the performance on ModelNet-40[57]. Specifically, since Point-Net can be viewed as 1 × 1 convolutions before the max pooling layer, we can apply all these MHE variants similarly to CNN. After the max pooling layer, there is a standard fully connected network where we can still apply the MHE variants. We compare the performance of regularizing PointNet and PointNet++ with half-space MHE, RP-CoMHE or AP-CoMHE. Table9shows that all MHE variants consistently improve PointNet and PointNet++, while RP-CoMHE and AP-CoMHE again perform the best among all. We demonstrate that CoMHE is generally useful for different kinds of neural networks, not limited to CNNs.

Table content: <table><tr><td>6.2 POINT CLOUD RECOGNITION</td><td/><td/><td/><td/></tr><tr><td>Besides image recognition, we apply CoMHE to</td><td>Method</td><td>PointNet</td><td>PointNet (T)</td><td>PointNet++</td></tr><tr><td>improve point cloud recognition. Our goal is to</td><td>Original</td><td>87.1</td><td>89.20</td><td>90.07</td></tr><tr><td>validate the effectiveness of CoMHE on a differ-ent network architecture with a different form of</td><td>HS-MHE [13] RP-CoMHE AP-CoMHE</td><td>87.44 87.82 87.85</td><td>89.41 89.69 89.70</td><td>90.31 90.52 90.56</td></tr><tr><td>input data structure, rather than achieving state-of-</td><td/><td/><td/><td/></tr><tr><td>the-art performance on point cloud recognition. To</td><td/><td/><td/><td/></tr><tr><td colspan="5">this end, we conduct experiments on widely used neural networks that handles point clouds: Point-</td></tr><tr><td colspan="5">Net [55] and PointNet++ [56]. We combine half-space MHE, RP-CoMHE and AP-CoMHE into</td></tr><tr><td colspan="5">PointNet (without T-Net), PointNet (with T-Net) and PointNet++. More experimental details are</td></tr><tr><td>given in Appendix A.</td><td/><td/><td/><td/></tr></table>

All the regularization is added without changing any components in PointNet. For PointNet experiments, we use point number 1024, batch size 32 and Adam optimizer started with learning rate 0.001, the learning rate will decay by 0.7 every 200k iterations, and the training is terminated at 250 epochs. For PointNet++ experiments, since the MRG (multi-resolution grouping) model is not provided in the official repository, we use the SSG (single scale grouping) model as baseline. Specifically, we use point number 1024, batch size 16 and Adam optimizer started with learning rate 0.001, the learning rate will decay by 0.7 every 200k iterations, and the training is terminated at 251 epochs. For all experiments, we use s = 2 in both MHE and CoMHE.

Table content: <table/>

Peer review (iclr2020):
The paper extends previous work on using a hyperspherical energy function for regularizing neural networks, an apporach called minimum hyperspherical energy (MHE). The authors point out several problems with optimizing the original MHE objective in high dimensions and propose the compressive minium hyperspherical energy (CoMHE) to address these problems. The main idea is to project the network weights to a lower-dimensional space and then apply the original MHE approach to the projected weights. The paper proposes numerous different ways to perform these projections, where the two main variants are based on random projects and angle preserving projections. The paper provides a number of theoretical results, showing that these projections approximately preserve the angles between pairs of weight vectors. The paper has a comprehensive experimental section, where the authors empirically evaluate the different variants of CoMHE, and they compare CoMHE based models to state-of-the-art models on CIFAR-10/100 and ImageNet. Most considerable, with a plain 9-layer CNN they obtain close to a 2 percentage point improvement over ResNet-1001 on CIFAR-100. On the positive side, the paper is well written and easy to follow. The paper both investigates the theoretical aspects of the proposed methodology and has comprehensives empirical evaluation. In particular, many of the reported results are comparable to the state-of-the-art, showing that this is effective regularization approach that gives good generalization. The main shortcoming of the papers is that while it contains many new methodological contributions, the paper appears to be somewhat incremental work. Furthermore, while the experimental results are comprehensive, it appears that only for figure 1, multiple training-runs have been performed. When standard deviation over multiple runs is not reported, it makes it hard to draw conclusions from the reported results. For instance, are there any significant differences between the results reported in table 2, or does the table show that the methods are not sensitive to the dimension of the projection? Given these shortcomings, I recommend a weak reject of the paper. To clarify my understanding of the paper, I would like the authors to answer the following questions: (1) How can you conclude that your empirical results do not show any noticeable performance gain with using multiple projects for AP-CoMHE when you only report the result for one project in table 3? (2) In table 4, I do not understand why you did not increase t even further. When do you start to see overfitting? Could the baseline archive the same performance as the other models with a larger t? (3) On the top of page 2, you write *Third, when the number of neurons is smaller than the dimension of the space (it is often the case in neural networks), MHE defines an ill-posed problem because the neurons can not even fully occupy the space.* I do not understand which *space* you refer to here. Could you please clarify this?