Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap
Authors: Weiyang Liu
Longhui Yu
Adrian Weller
Bernhard Schölkopf
Venue: ICLR 2023 poster (2023)

Abstract: The neural collapse (NC) phenomenon describes an underlying geometric symmetry for deep neural networks, where both deeply learned features and classifiers converge to a simplex equiangular tight frame. It has been shown that both cross-entropy loss and mean square error can provably lead to NC. We remove NC's key assumption on the feature dimension and the number of classes, and then present a generalized neural collapse (GNC) hypothesis that effectively subsumes the original NC. Inspired by how NC characterizes the training target of neural networks, we decouple GNC into two objectives: minimal intra-class variability and maximal inter-class separability. We then use hyperspherical uniformity (which characterizes the degree of uniformity on the unit hypersphere) as a unified framework to quantify these two objectives. Finally, we propose a general objective -- hyperspherical uniformity gap (HUG), which is defined by the difference between inter-class and intra-class hyperspherical uniformity. HUG not only provably converges to GNC, but also decouples GNC into two separate objectives. Unlike cross-entropy loss that couples intra-class compactness and inter-class separability, HUG enjoys more flexibility and serves as a good alternative loss function. Empirical results show that HUG works well in terms of generalization and robustness.

Published as a conference paper at ICLR 2023
GENERALIZING AND DECOUPLING NEURAL COLLAPSE VIA HYPERSPHERICAL UNIFORMITY GAP
Weiyang Liu! Longhui Yu²,, Adrian Weller Bernhard Schölkopf'
Max Planck Institute for Intelligent Systems Tübiigen UUniversity of Cambridge *Peking University 'The Alan Turing Institute
ABSTRACT
The neural collapse (NC) phenomenon describes underlying geometric symme- try for deep neural networks, where both deeply learned features and classifiers converge to simplex equiangular tight frame. It has been shown that both cross- entropy loss and mean square error can provably lead to NC. We remove NC's key assumption the feature dimension and the number of classes, and then present generalized neural collapse (GNC) hypothesis that effectively subsumes the origi- nal NC. Inspired by how NC characterizes the training target of neural networks, we decouple GNC into two objectives: minimal intra-class variability and maximal inter-class separability. We then use hyperspherical uniformity (which character- izes the degree of uniformity on the unit hypersphere) as a unified framework to quantify these two objectives. Finally, we propose general objective heee-shpppprreee ical uniformity gap (HUG), which is defined by the difference between inter-class and intra-class hyperspherical uniformity. HUG not only provably converges GNC. but also decouples GNC into two separate objectives. Unlike cross-entropy loss that couples intra-class compactness and inter-class separability, HUG enjoys more flexibility and serves as good alternative loss function. Empirical results show that HUG works well in terms of generalization and robustness.
INTRODUCTION
Recent years have witnessed the great success of deep representation learning in a variety of applica- tions ranging from computer vision [37], natural language processing [16] to game playing [55, 64). Despite such success, how deep representations can generalize to unseen scenarios and when they might fail remain black box. Deep representations are typically learned by a multi-layer network with cross-entropy (CE) loss optimized by stochastic gradient descent. In this simple setup, [86] has shown that zero loss can be achieved even with arbitrary label assignment, After continuing train the neural network past zero loss with CE, [60] discovers an intriguing phenomenon called neural collapse (NC). NC can be summarized as the following characteristics:
Intra-class variability collapse: Intra-class variability of last-layer features collapses to zero, indicating that all the features of the same class concentrate to their intra-class feature mean.
Convergence to simplex ETF: After being centered at their global mean, the class-means are both linearly separable and maximally distant on a hypersphere. Formally, the class- meass form simplex equiangular tight frame (ETF) which is symmetric structure defined by a set of maximally distant and pair-wise equiangular points on a hypersphere.
Convergence to self-duality: The linear classifiers, which live in the dual vector space to that of the class mlass-means, converge to their corresponding class-mean and also form a simplex ETF.
Nearest decision rule: The linear classifiers behave like nearest class-mean classifiers.
The NC phenomenon suggests two general principles for deeply learned features and classifiers: minimal intra-class compactness of features (i.e., features of the same class collapse to single point), and maximal inter-class separability of classifiers feature mean (i.e., classifiers of different classes have maximal angular margins). While these two principles are largely independent, popular loss functions such as CE and square error (MSE) completely couple these two principles together. Since there is trivial way for CE and MSE to decouple these two principles, quantity - hyperspherical uniformity gap (HUG), which not only characterizes intra-class feature compactness and inter-class classifier separability whole, but also fully decouples these two
identify novel

---

Published as a conference paper at ICLR 2023
principles. The decoupling enables HUG to separately model intra-class compactness and inter-class separability, making it highly flexible. More importantly, HUG can be directly optimized and used to train neural networks, serving as alternative loss function in place of CE and MSE for classification. HUG is formulated as the difference between inter-class and intra-class hyperspherical uniformity. Hyperspherical uniformity [48] quantifies the uniformity of set of vectors on hypersphere and is used to capture how diverse these vectors are on hypersphere. Thanks to the flexibility of HUG, we are able to use many different formulations characterize hyperspherical uniformity, including (but not limited to) minimum hyperspherical energy (MHE) [45], maximum hyperspherical separation (MHS) [48] and maximum gram determinant (MGD) [48]. Different formulations yield different interpretation and optimization difficulty (e.g., HUG with MHE is easy to optimize, HUG with MGD has interesting connection to geometric volume), thus leading to different performance.
Similar to CE loss, HUG also provably leads to NC under the setting of unconstrained features [53]. Going beyond NC, we hypothesize a generalized NC (GNC) with hyperspherical uniformity, which extends the original NC to the scenario where there is no constraint for the number of classes and the feature dimension. NC requires the feature dimension no smaller than the number of classes while GNC no longer requires this. We further prove that HUG also leads to GNC at its objective minimum.
Another motivation behind HUG comes from the classic Fisher discriminant analysis (FDA) [19] where the basic idea is to find a projection matrix T that maximizes between-class variance and minimizes within-class variance. What we directly optimize the input data (without any projection) rather than optimizing the linear projection in FDA? We make a simple derivation below:
((T' T S.T)
Projection FDA:
max tr TERex
Data FDA:
max E
tr (S.) - tr (S.)
where the between-clase scatter matrix is Sw = =EE Ese,, - p.)
the within-class > n4(u, ( 1 n, is the number of samples in the i-th class,
scatter matrix is S. = 1 is the total number of samples, μ4 = ΣjeA, 2, is the i-th class- and Ø=n "EE= 2; is the global mean. By considering class-balanced data on the unit hypersphere, optimizing data FDA is equivalent to simultancously maximizing tr(S,) and minimizing tr( tr(Sw... Maximizing tr( tr(S,) encourages inter-class separability and is necessary condition for hyperspherical uniformity. Minimizing ) encourages intra-class feature collapse, reducing intra-class variability. Therefore, HUG can be viewed a generalized FDA criterion for learning maximally discriminative features.
However, one may ask the following questions: Why is HUG useful if we already have the FDA criterion? Could we simply optimize data FIA?? In fact, the FDA criterion has many degenerate solutions. For example, we consider a scenario of 10-class balanced data where all features from the first 5 classes collapse to the north pole on the unit hypersphere and features from the rest 5 classes collapse to the south pole on the unit hypersphere. In this case, tr(Sw.) ) is already minimized since it achieves the minimum zero. tr( (r(S)) also achieves its maximum n at the same time, In contrast, HUG naturally generalizes FDA without having these degenerate solutions and serves as more reliable criterion for training neural networks. We summarize our contributions below:
We decouple the NC phenomenon into two separate learning objectives: maximal inter-class sepa- rability (i.e., maximally distant class feature mean and classifiers on the hypersphere) and minimal intra-class variability (i.e., intra-class features collapse to a single point on the hypersphere).
Based on the two principled objectives induced by NC, we hypothesize the generalized NC which generalizes NC by dropping the constraint on the feature dimension and the number of classes.
We identify general quantity called hyperspherical uniformity gap, which well characterizes both inter-class separability and intra-class variability. Different from the widely used CE loss, HUG naturally decouples both principles and thus enjoys better modeling flexibility.
Under the HUG framework, we consider three different choices for characterizing hyperspherical uniformity: minimum hyperspherical energy, maximum hyperspherical separation and maximum Gram determinant. HUG provides unified framework for using different characterizations of hyperspherical uniformity to design new loss functions.
We first obtain the upper bound n of tr( (.)) from tr(S.)-- Because set of vectors {u:} ) achieving hyperspherical uniformity has En, _a. [|pا|]]] + 0 (as grows larger) [20]. Then we have that tr(5.)) attains n. Therefore, vectors achieving hyperspherical uniformity are one of its maximizers. tr( (w) can simultancously attain its minimum if intra-class features collapse a single point.
- n. | || S n.
2

---

Published as a conference paper at ICLR 2023
2 ON GENERALIZING AND DECOUPLING NEURAL COLLAPSE
NC describes an intriguing phenomenon for the distribution last-layer features and classifiers in overly-trainee neural networks, where both features and classifiers converge ETF. However, ETF can only exist when the feature dimension d and the number of classes C' satisfy d2C-1. This is not always true for deep neural networks. For example, neural networks for face recognition usually trained by classifying large number of classes (e.g., more than 85K classes in [23]), and the feature dimension (e.g., 512 in SphereFace [43]) is usually much smaller than the number of classes. In general, when the number of classes is already large, it is prohibitive to use a larger feature dimension. Thus question arises: what will happen in this case a nearal network fully trained?
Motivated by this question, we conduct a simple experiment to simulate the case of d C and the case of d' C- 1. Specifically, we train a con- volutional neural network (CNN) on MNIST with feature dimension 2. For the case of d2 C 1, we use only 3 classes (digit 0,1,2) as the training set. For the case of d<C' 1, we use all 10 classes the training set. We visualize the learned features of both cases in Figure 1. The results verify the case of d2 C - indeed approaches to NC, and ETF does not exist in the case of d Inter- estingly, one can observe that learned features in both cases approach to the configuration of equally spaced frames on the hypersphere. To accommodate the case of d d'C-LL 1. we extend NC to the generalized NC by hypothesizing that last-layer inter-class features and classifiers converge to equally spaced points on the hypersphere, which can be characterized by hyperspherical uniformity.
2D Figure I: 2D learned feature visanlization on MNIST. The features inherently 2-dimensional and are plotted without visualization tools. (a) Case 1: d= C = 3: (b) Case 2: d= C=
(6) fostur classce
Generalized Neural Collapse (GNC)
We define the feature global mean as JGG - aaa. where ai Rd is the last-layer feature of the i-th sample in the e-th class, the feature class-mean as f4. Ave for different classes eE (1, (1.... , C). the feature within-class covariance as = A (Wii, (at)
mce) and the feature between-class covariance as E = ( (G) - - GNC states that
(1) Intra-class variability collapse: Intra-class variability of last-layer features collapse to zero, indicating that all the features of the same class converge to their intra class feature mean. Formally, GNC has that Σ/ Ew 0 where denotes the Pboore-Penrose pseudoinverse.
(2) Convergence hyperspherical uniformity: After being centered at their global mean, the class-means are both linearly separable and maximally distant on a hypersphere. Formally, the class-means converge to equally spaced points on a hypersphere, i.e.,
Σ K K(jte (eee) ogte
min Σ cote
K K(ac Pe))
||p..
pGll l pall 0, Ve (1)
where action. Typically, we consider Riesz s-kernel K, (pc, (ae) = sign(s)) (s) A || or logarith- mie kernel (fpe f=e)=logg pip - م. ||| For example, the Riesz s-kernel with s ==d is a variational characterization of hyperspherical uniformity (e.g., hyperspherical energy [45]) using Newtonian potentials. the case of d= == 1, the Riesz kernel is called Coulomb potential and the problem of finding minimal coulomb energy is called Thomson problem [70].
Mall (me Tori--A HG) and (..,-) a kernel function that models pairwise inter-
(3) Convergence to self-duality: The linear classifiers, which live in the dual vector space to that of the class-ment meass, converge to their corresponding mlans, leading to hyperspherical uniformity. Formally, GNC has that ||w,ll w. a. - where w. ERd is the e-th classifier.
(4) Nearest decision rule: The learned linear classifiers behave like the nearest class-mean classifiers. Formally, GNC has that arg max, (w., 2)) + bb - arg min, || |a--Mell pell-
In contrast NC, GNC further considers the case of d< and hypothesizes that both feature class-means and classifiers converge to hyperspherically uniform point configuration that minimizes some form of pairwise potentials. Similar how NC connects tight frame theory [74] to deep learning, our GNC hypothesis connects potential theory [3] to deep learning, which may shed new light on understanding We show Theorem that GNC reduces to NC. in the case of d C
3

---

Published as a conference paper at ICLR 2023
Theorem (Regular Simplex Optimum for GNC) Let f (0,4) - B. be convex and decreasing function defined at v=0 by lim f(v). If d 1, then we have that the vertices of regular (C- simplices inscribed in St with centers at the origin (equivalent simplex ETF) minimize the hyperspherical energy Enje' K K(pes Pee)) on the unit hypersphere S (d 2 3) with the kernel I is stricily convex and strictly decreasing, then these are the only energy minimizing C-point configurations. Thus GNC reduces to NC when d2 C -1.
as K K(ūes (e)) f((ll (
We note that Theorem 2 guarantees the simplex ETF as the minimizer of general family of hyperspherical energies (as long as f is convex and decreasing). This suggests that there many possible kernel functions ((,,) -) in GNC that can effectively generalize NC. The case of d < C -1 is where GNC really gets interesting but complicated. Other than the regular simplex case, we also highlight a special uniformity case of Cd In this case, we can prove in Theorem that GNC(2) converges to the vertices of a cross-polytope as hyperspherical energy gets minimized. As the number of classes gets infinitely large, we show in Theorem 3 that GNC(2) leads to point configuration that is uniformly distributed on gd-1, Additionally, we show simple yet interesting result in Proposition 1 that the last-layer classifiers are already initialized to be uniformly distributed on the hypersphere in practice.
(4) Regular simplex
(b) Cross-polytope
Figure 2: Goometric illustration in R of (a) regu- lar simplex optimum (equivalent to simplex ETF NC) and (b) cross polytupe optimum in GNC.
Theorem 2 (Cross-polytope Optimum for GNC) =fC==d, then the verfices of the cross-polytope are the minimizer of the hyperspherical energy in GNC(2).
The cross-polytope optimum for GNC(2) is in fact quite intuitive, because it corresponds to the Cartesian coordinate system (up to a rotation). For example, the vertices of the unit cross-polytope in R are (=1,0, 0, 0). (0, +1, 0), (o, 0, =1). These vectors minimize the hyperspherical energy on S?. We illustrate both the regular simplex and cross pross-polytope cases in Figure 2. For the other cases of d<C-1, 1, there exists generally no simple and universal point structure that minimizes the hyperspherical energy, as heavily studied in [12, 27, 38, 63). For the point configurations that asymptotically minimize the hyperspherical energy as C' grows larger, Theorem 3 can guarantee that these configurations asymptotically converge a uniform distribution on the hypersphere.
Theorem (Asymptotic Convergence to Hyperspherical Uniformity) Consider a sequence of point configurations {uf ...' 16)2= that asymptotically minimizes the hyperspherical energy on Lo Sd-1 as C - Dc, then (pf, "PP -PCTC=2" is uniformly distributed on the hypersphere Sd-1
Proposition 1 (Minimum Energy Initialization) With zero-mean Gaussian initialization (e.g.. 122, 281), the C last-layer classifiers of neural networks are initialized as uniform distribution on the hy- persphere. The expected initial energy C(C (CC-11 1) Iga fg. p 1 )
With Proposition 1, one can expect that the hyperspherical energy of the last-layer classifiers will first increase and then decrease lower value than the initial energy. To validate the effectiveness of our GNC hypothesis, we conduct a few experiments to show how both class feature means and classifiers converge to hyperspherical uniformity (i.e., minimizing the hyperspherical energy), and how intra-class feature variability collapses to almost zero. We start with an intuitive understanding about GNC from Figure 1. The results are directly produced by the learned features without any visualization tool (such as I-SNE [73]), so the feature distribution can reflect the underlying one learned by neural networks. We observe that GNC is attained in both d dCC-1 1 and d C-1, while can no longer form a simplex ETF. To see whether the same conclusion holds for higher feature dimensions, we also train two CNNs on CIFAR-100 with feature dimension as 64 and 128, respectively. The results are given in Figure 3.
NC is violated in d<C- since the learned feature
Figure shows that GNC captures well the underlying convergence of the neural network training. Figure 3(a.c) shows that the hyperspherical energy of feature class-means and classifiers converge to small value, verifying the correctness of GNC(2) and GNC(3) which indicate both feature class-means and classifiers converge to hyperspherical uniformity. More interestingly, in the MNIST experiment, we can compute the exact minimal energy on S 2 in the case of d=2, C-=3 (1/3 for average energy) and = 82.5 in the case of d=2, C= 10 (= 0.917 for average energy). The final average energy in Figure 3(a) matches our theoretical minimum well. From Figure 3(c), we observe that the classifier energy stays close to its minimum the very beginning, which matches our Proposition 1
4

---

Published as a conference paper at ICLR 2023
Avergge teeagtuyppppee Uyporpiica tiang
Powtl (Hild)
2012 CH100
2000
dna Exrr
Dacatiin
bestee
Inter class separability on MNIST
(b) Intra clas vuriability on MNIST
(0) sepurability on CIFAR-100 (4) Intra variabilitt CIFAR-10 100
Figure Training dynamics of hyperspherical energy (which cuptures inter class separability) and hyperspherial reverse energy (which captures intra-class variability). (a.h) MNIST with d = 2, C = 10 and d = 2, C' = 3. (c.d) CIFAR- 100 with 64, C = 100 and d' 128, C 100.
that vectors initialized with zero-mean Gaussian are uniformly distributed over the hypersphere (this phenomenon becomes more obvious in higher dimensions). To evaluate the intra-class feature variability, we consider hyperspherical reverse-energy E, Eijjen. | ēū where Ø; at T and A. denotes the sample index set of the e-th class. The smaller this reverse-energy gets, the less intra-class variability it implies. Figure ((,d) shows that the intra-class feature variability approaches zero, as GNC(1) suggests. Details and more empirical results on GNC are in given Appendix A.
Now we discuss how to decouple the GNC hypothesis and how such a decoupling can enable us to design new objectives to train neural networks. GNC(I) and GNC(2) suggest to minimize intra-class feature variability and maximize inter-class feature separability, respectively. GNC(3) and GNC(4) natural consequences if GNC(1) and GNC(2) hold. It has long been discovered in [42, 68. 82] that last-layer classifiers serve as proxies to represent the corresponding class of features, and they GNC(3) indicates the classifiers converge to
also an approximation to the feature hyperspherical uniformity, which, together with GNC(1), implies GNC(4).
Until now, it has been clear that GNC really boils down to two decoupled objectives: maximize inter-class separability and minimize intra-class variability, which again echos the goal of FDA. The problem reduces to how to effectively characterize these two objectives while being decoupled for flexibility (unlike CE or MSE). In the next section, we propose to address this problem by characterizing both objectives with a unified quantity- hyperspherical uniformity.
3
HYPERSPHERICAL UNIFORMITY GAP
3.1 GENERAL FRAMEWORK
As GNC(2) suggests, the inter-class separability is well captured by hyperspherical uniformity so it is natural to directly use it as a learning target. On the other hand, GNC(I) does not suggest any easy-to-use quantity to characterize intra-class variability. We note that minimizing intra-class variability is actully equivalent to encouraging features of the same class to concentrate on a single point, which the opposite of hyperspherical uniformity. Therefore, we can unify both intra-class variability and inter-class separability with single characterization of hyperspherical uniformity. We propose to maximize the hyperspherical uniformity gap:
of feature
max LHUG := a (0,)
HU (1
-8. Σ
Ty: Inter-class Hyperspherical Uniformity
Tu: Intra-class Hyperspherical Uniformity
where ax B are hyperparameters, pa== | is the feature class-mean projected on the unit hyper- sphere, === Fr is the feature class-mean, n; the last-layer feature of the i-th sample and A. denotes the sample index set of the c-th class. Hll ((,))) denotes some measure of hyper- spherical uniformity for vectors {V1, of generality, we assume that the larger it gets, the stronger hyperspherical uniformity we have. We mostly focus on supervised learning with parameteric class proxies" where the CE loss is widely used as a de facto choice, although HUG can be used in much broader settings as discussed later. In the HUG framework, there is longer clear notion of classifiers (unlike the CE loss), but we still can utilize class proxies (i.e., a generalized concept of classifiers) to facilitate the optimization.
J. Eq. 2 is the general objective for HUG, Without loss
We observe that Eq. 2 directly optimizes the feature class-means for inter-class separability, but they are intractable to compute during training (we need to compute them in every iteration). Therefore it is nontrivial to optimize the original HUG for training neural networks. A naive solution is to
2 Parametric class proxies are set of parameters used to represent a group of samples in the same class. Therefore, these proxies store the information about a class. Last-layer classifiers are a typical example.
5

---

Published as a conference paper at ICLR 2023
approximate feature class-mean with few mini-batches such that the gradients of T3 can be still back- propagato to the last-layer features. However, it may take many mini-batchin in order to obtain sufficiently accurate class-mean, and the approximation gets much more difficult with large number of classes. To address this, we employ parametric class proxies to act as representatives of intra-class features and optimize them instead of feature class-means. We thus modify the HUG objective
max
HU. == a
Hld )om)
-9-
HM w.)
Inter-class Hyperspherical Uniformity
Intm-class Hyperspherical Unifoemity
where U. e is the parametric proxy for the c-th class. The intra-class hyperspherical uniformity term connects the class proxies with features by minimizing their joint hyperspherical uniformity, guiding features to move towards their corresponding class proxy. When training a neural network, the objective function in Eq. will optimize network weights and proxies together. There alternative ways to design the HUG loss from Eq. 2 for different learning scenarios, as discussed in Appendix C.
Learnable proxies. We can view the class proxy fá; learnable parameters and update them with stochastic gradients, similarly to the parameters of neural networks. In fact, learnable proxies play a role similar to the last-layer classifiers in the CE loss, improving the optimization by aggregating intra-class features. The major difference between learnable proxies and moving-averagee proxies is the way we update them. As GNC(3) implies, class proxies in HUG can also be used as classifiers.
Static proxies. Eq. 3 is decoupled into maximal inter-class separability and minimal intra-class variability. These two objects are independent and do not affect each other. We can thus optimize them independently. This suggests a even simpler way to assign class proxies -iiitializing class proxies with prespecified points that have attained hyperspherical uniformity, and fixing them the training. There are two simple ways to obtain these class proxies: (1) minimizing their hyperspherical energy beforehand; (2) using zero-mean Gaussian to initialize the class proxies (Proposition 1). After initialization, class proxies will stay fixed and the features are optimized towards their class proxies.
Partially learnable proxies. After the class proxies are initialized using the static way above, we can increase its flexibility by learning orthogonal matrix for the class proxies to find suitable orientation for them. Specifically, we can learn this orthogonal matrix using methods in [47].
3.2 VARIATIONAL CHARACTERIZATION OF HYPERSPHERICAL UNIFORMITY
While there exist many ways to measure hyperspherical uniformity, we seek variational characteriza- tion due to simplicity. As examples, we consider minimum hyperspherical energy [45] that is inspired by Thomson problem [66, 70] and minimizes the potential energy, maximum hyperspherical separa- tion [48] that is inspired by Tammes problem [69] and maximizes the smallest pairwise distance, and maximum gram determinant [48] that is defined by the volume the formed parallelotope.
Minimum hyperspherical energy. MHE seeks to find an equilibrium state with minimum potential energy that distributes n electrons on a unit hypersphere as evenly as possible. Hyperspherical unifor- mity is characterized by minimizing the hyperspherical energy for n vectors Vn, = [v1,- Un E ER]
min (6 Pre EBd
E. (V.))
K.( B.) K, 0,) = j=زاهر
Σ
|0: 65 6,|1= 8>0
1 6,1
8<0 0
(4)
v. where 0;:= = ||
is the i-th vector projected onto the unit hypersphere. With (V) = E, (V), we
apply MHE to HUG and formulate the new objective as follows (s)=2, sw 1):
min (a, ( (3) )8
LMHE-HUG == a Fes (( - B. Σ F.s.. (Lo))iEde m))
(5)
which can already be used as to train neural networks. The intra-class variability term in Eq. 5 can be relaxed to a upper bound such that we can instead minimize a simple upper bound of LMIEEHUG:
LAHE =
+ + 1'. ->> || w.ll Z LMHE-HUG
(6)
AE
which is much more efficient to compute in practice and thus can serve as relaxed HUG objective.
Moreover, LMUEAHUG and HUG share the same minimizer. Detailed derivation is in Appendix H.
Maximum hyperspherical separation. MHS uses a maximum geodesic separation principle by maximizing the separation distance 8 (V.) (Le., the smallest pairwise distance in V,, = =[01,-' " U,

---

Published as a conference paper at ICLR 2023
R): maxy (o(V)) minezz 0 +,]l). Because 2 (V)) is another variational definition,
cannot naively set H.)(.) = 0(). We define 0 0-1(V,)) v,, max $j 00 أزو and HUG becomes
LMHS-HUG a. (( ((=b))==)))) B Σ 0 (( wh),
max (6, 1-, 1 ( )=
which, by replacing intra-class variability with its surrogate, results in more efficient form:
LASS-HU == a- min
o#c || u - B Σ max
(8)
'Ede
which is max-min optimization with simple nearest neighbor problem inside. We note that HUG and LIs IUG share the same maximizer. Detailed derivation is given in Appendix H.
Maximum gram determinant. MGD characterizes the uniformity by computing proxy to the volume of the paralleiotope spanned by the vectors. MGD is defined with kernel gram determinant:
log det (G == (K(,, (K,,),))) (-)), K(;;, 05) exp = é5ll²)
( ( ES4 1)
where we use a Gaussian kernel with parameter and G(V,)) ) is the kernel gram matrix for V, {01, 8n}. With HV)(V,) = det G GVV,), minimizing intra-class uniformity cannot be achieved by minimizing det G(V,), since det G( =0 only leads linear dependence. Then we have
(a, 18 LMGD-HUG == a log det (G(( + +
max
ΣΣ
| dell
(10)
SEA.
where we directly use the surrogate loss from Eq. 6 as the intra-class variability term. With MGD, HUG has interesting geometric interpretation - encourages the volume spanned by class proxies to be as large as possible and the volume spanned by intra-class features to be as small possible.
3.3 THEORETICAL INSIGHTS AND DISCUSSIONS
There are many interesting theoretical questions concerning HUG, and this framework is highly related to a few topics in mathematics, such as tight frame theory [74], potential theory [39], sphere packing and covering [3, 18, 25]. The depth and breath of these topics are beyond imagination. In this section, we focus on discussing some highly related yet intuitive theoretical properties of HUG.
Theorem (Order of Minimum Hyperspherical Energy) If d 1> >>>00 O or 0> 8>> -2 and de N. we have that lim (n -2 -miny, E.(V.. (V.)) =c(s, d) where c(s, d) is constant involving s, d.
The result above shows that the leading term of the minimum energy grows of order On1²) as n co.
Theorem 4 generally holds with wide range of for the Riesz kernel in hyperspherical energy. Moreover, the following result shows that MHS is in fact a limiting case of MHE as s - 00.
Proposition 2 (MHS is Limiting Case of MHE) Let n E N, 2 2 be fixed and (g _12) be compact metric space. We have that sim (min C8 c.. E.
(VV)) (max CB 2 ((Vn))
Proposition 3 d ++ 1, converge to cross-polytope when C = 2d and asymptotically converge to GNC as C - xo.
The HUG objectives in both Eq. and Eq. converge to simplex ETF when S C
Proposition 3 shows that HUG not only decouples GNC but also provably converges to GNC. Since GNC indicates that the CE loss eventually approaches to the maximizer of HUG, we now look into how the CE loss implicitly maximizes the HUG objective in coupled way.
Proposition 4 The CE loss is LCE = Ei log(1+ + Eity, exp((ue, vz) (W,, xx))) where n is the number of samples, æs is the i-th sample with label ys and w, is the last-layer linear classifier for the j-th class. Bias is omitted for simplicity. LCE is bounded by (p=C-1)
(Wy, x2))
P Σ (wyr wi) Lox S log (1+)
(xp((w,,aa x.i)) +
(w,, 2.))).
Q:: Coupled and IV
Qa: Entee-class Variabliit
Qa: Coupled is and
Qs: Interclass Variability
We show in Proposition 4 that CE inherently optimizes two independent criterion: intra-class variability (IV) and inter-class separability (IS). With normalized classifiers and features, we can see that Q1 and Qa have similar minimum where a; P==uu ww, and w, Vi attain hyperspherical uniformity.
7

---

Published as a conference paper at ICLR 2023
We show that CE is lower bounded by the gap of inter-class and intra-class hyperspherical uniformity:
LCR
log
exp(p2
Σ (ee( 25)) j=l
ps
(E))))))
(1I)
i= JE
Inier-class flypershorical Uniformity
Intmeclass Hypersphorical Unifeemity
where P1s P2. P3 are constants and 4i is the softmax confidence of Φ; for the e-th class (Appendix L). This result [4] implies that minimizing CE effectively minimizes HUG. [50] proves that the minimizer of the normalized CE loss converges to hyperspherical uniformity. We rewrite their results below:
Theorem 5 (CE Asymptotically Converges to HUG's Maximizer) Considering unconstrained features of C classes (each class has the same number of samples), with features and classifiers normalized on some hypersphere, we have that, for the minimizer of the CE loss, classifiers converge weakly to the uniform measure on Sa-1 as C-> oo and features collapse to their corresponding classifiers. The minimizer of CE also asymptotically converges to the maximizer of HUG.
Theorem shows that the minimizer of the CE loss with unconstrained features [53] asymptotically converges to the maximizer of HUG (i.e., GNC). Till now, we show that HUG shares the same optimum with CE (with hyperspherical normalization), while being more flexible for decoupling inter-class feature separability and intra-class feature variability. Therefore, we argue that HUG can be an excellent alternative for the widely used CE loss in classification problems.
HUG maximizes mutual information. We can view HUG as a way to maximize mutual informa- tion Z(X; Y)= H(()) H(X Y). where X denotes the feature space and Y is the label space. Maximizing M( implies that the feature should be uniform over the space. Minimizing H(X YY) means that the feature from the same class should be concentrated. This is nicely connected to HUG.
The role of feature and class proxy norm. Both NC and GNC do not take the norm of feature and class proxy into consideration. HUG also assume both feature and class proxy norm are projected onto some hypersphere. Although dropping these norms usually improves generalizability [8, 9, 14, 43, 76), training neural networks with standard CE loss still yields different class proxy norms and feature norms. We hypothesize that this is due to the underlying difference among training data distribution of different classes. One empirical evidence to support this is that average feature norm of different classes is consistent across training under different random seeds (e.g.. average feature norm for digit I on MNIST stays the smallest in different run). [36, 46, 52] empirically show that feature norm corresponds to the quality of the sample, which can also viewed as a proxy to sample uncertainty. [56] theoretically shows that the norm of neuron weights (e.g., classifier) matters for its Rademacher complexity. As a trivial solution to minimize the CE loss, increasing the classifier norm (if the feature is correctly classified) can easily decrease the CE loss to zero for this sample, which is mostly caused by the softmax function. Taking both feature and class proxy norm into account greatly complicates the analysis (e.g.. it results in weighted hyperspherical energy where the potentials between vectors are weighted) and seem to yield little benefit for now. We defer this issue future investigation.
HUG as a general framework for designing loss functions. HUG can be viewed as an inherently decoupled way of designing new loss functions. As long as we design measure of hyperspherical uniformity, then HUG enables us to effortlessly turn it into a loss function for neural networks.
EXPERIMENTS AND RESULTS
Our experiments aims to demonstrate the empirical effectiveness of HUG, so we focus on the fair comparison to the popular CE loss under the same setting. Experimental details are in Appendix N.
4.1 EXPL.ORATORY EXPERIMENTS AND ABLATION STUDY
Different HUG variants. We compare different HUG variants and the CE loss on CIFAR- 10 and CIFAR- 100 with ResNet- 18 [29]. Specif- ically, we use Eq. 6, Eq. and Eq. 10 for MHE-HUG, MHS-HUG and MGD-HUG, respectively. The results are given in Table 1. We can observe that all HUG variants outperform the CE loss. Among all, MHE-HUG achieves the best testing accuracy with considerable improvement over the CE loss. We note that all HUG variants are used without the CE loss. The performance gain of HUG are actually quite significant, since the CE loss is currently a default choice for classification problems and serves as a very strong baseline.
Method
CIFAR 10
CIFAR: 100
CE Loss
5.45
24.90
HUG
23.50
MHS-HUG
5.09
24.38
MGD-HUG
5.38
24.59
Table I: Testing error (9) of HUG vari- ants on CIBAR-10 and CIFAR-100.
8

---

Published as a conference paper at ICLR 2023
Different methods to update proxies. We also evaluate how different proxy update methods will affect the classification per- formance. We use the same setting as Table 1. For all the proxy update methods, we apply them to MHE-HUG (Eq. 6) under the same setting. The results are given Table 2. We can observe that all the propose proxy update methods work reasonably well. More interestingly, static proxies work surprisingly well and out- perform the CE loss even when all the class proxies are randomly initialized and then fixed throughout the training. The reason the static proxies work for MHE-HUG is due to Proposition 1. This result is significant since we no longer have to train class proxies in HUG (unlike CE). When trained with large number of classes, is GPU-memory costly for learning class proxies, which is also known as one of the bottlenecks for face recognition (1]. HUG could be promising solution to this problem.
Method
CIFAR 10 CIFAR: 100
CE Loss
5.45
24.90
Fully learaalle
23.50
Static (random)
5.19
24.23
Static (optimized)
5.12
24.02
Partially learnable
5.08
23.89
Table Testing emur (5) of different peaxy
update methodk on CIFAR- a0 and CIFAR-100.
Loss landscape and convergence. We perturb neuron weights (refer to [40]) to visualize the loss landscape of HUG and CE in Figure 4. We use MHE in HUG here. The results show that HUG yields much flatter
CE Liss
do HUGI Rank loses
tO FUAG: Ierallas Versiilyy U8 HUG Teeralass Sepuckiity
Figure 4: Loss landscage visualization. (b.c.d) show SSEE HUG. Ta and Tur respectively.
local minima than the CE loss in general, implying that HUG has potentially stronger generaliza- tion [34, 57J. We show more visualizations and convergence dynamics in Appendix O.
Learning with different architectures. We evaluate HUG with different network architectures such as VGG-16 [65], ResNet- 18 [29] and DenseNet- 121 [31]. Results in Table 3 (Left number: CIFAR- 10. right number: CIFAR-100) show
Method ResNet-18
VGG-16
DemeNet-12
Loss 2.4.9 5.28 / 22.99
/2.47
HUG / 5.19/ 22,77
4.85 21.30
Table 3: Testing emor (5) with different architectures.
that HUG is agnostic to different network architectures and outperforms the CE loss in every case.
Although HUG works well on its own, any other methods that improve CE can also work with HUG.
4.2 GENERALIZATION AND ROBUSTNESS UNDER DIFFERENT LEARNING SCENARIOS
Long-tailed recognition. We consider the task of long-tailed recognition, where the data from differ- ent classes are imbalanced. The settings generally follow [6]. and the dataset gets more imbalanced if
CIFAR-100
CIFAR-10
IR
0.2
0.1
0.02
0.01
0.2
0.1 0.02
0.01
CF
66.74
62.31 48.79
43.82
90.29 79.17 74.11
HUG 67.83 63.33 50.48
45.63
90.41 88.20 79,88 75.14
Table 4: Testing accuracy (%) of long tailed recognition.
the imbalance ratio (IR) gets smaller. The potential
of HUG in imbalanced classification is evident, as the inter-class separability in the HUG is explicitly modeled and can be easily controlled. Experimental results in Table show that HUG can consistently outperform the CE loss in the challenging long-tailed setting under different imbalanced ratio.
Continual learning. We demonstrate the potential of HUG in the class-continual learning setting, where the training data is not sampled i.i.d but comes in class by class. Since training data is highly biased, hyperspher- ical uniformity among class proxies is crucial. Due to the decoupled nature of HUG, we can easily increase the importance of inter-class separability, unlike CE. We use a simple continual learning method - ER [62] where the CE loss with memory is used. We replace it with HUG. Table 5 shows HUG consistently improves ER under different memory size.
CIFAR- 100
CIFAR 10
Memory size
200 500 2000
2000
ER CR
22.14
31.02 43.54 49.07 61.58 76.89
ER + HUG
23.52
31.92 43.92 53.74 62.67 77.21
Table 5: Final testing accuracy (%) of continual learning.
Adversarial robustness. We further test HUG's ad- versarial robustness. In our experiments, we consider the classical white-box PGD attack [5 on ResNet- 18. The PGD attack iteration is set as 100 and the attack strength level is set as 2/255, 4/255, 8/255 in 1h norm. All networks are naturally training with either HUG or CE loss. Results in Table demonstrate that HUG yields consistently stronger adversarial robustness than the CE loss.
Method
Clean
Lx 2/255 L. 4/255 L 88255
Loss / 7
0.61 /0
0/0
HUG
5.03 23.50 15.24/5.24 5.26
3.45 1.24
1.76/ 0.44
Tabl 6: Testing accuracy (%) under adversarial attacks.
NLP tasks. As an exploration, we evaluate HUG on some simple NLP classification tasks. Our experiments follow the same settings as [32] and finetune the BERT model [15] in these tasks. Table 7 shows that HUG yields better generalizability than CE, demonstrating its potential for NLP.
Task
MRPC SST-2 WNLI
CE Loss
84.8
91.6
33.8
HUG
85.8
91.8
34.0
Table 7: NLP testing accuracy (%)

---

Published as a conference paper at ICLR 2023
5 RELATED WORK AND CONCLUDING REMARKS
We start by generalizing and decoupling the NC phenomenon, obtaining two basic principles for loss functions. Based on these principles, we identify a quantity hyperspherical uniformity gap, which not only decouples NC but also provides a general framework for designing loss functions. We demonstrate a few simple HUG variants that outperform the CE loss in terms of generalization and adversarial robustness. There a large body of excellent work in NC that is related HUG, such as [26, 33, 71, 89]. [88] extends the study of NC to more practical loss functions (e.g.. focal loss and losses with label smoothing). Different from existing work hyperspherical uniformity [41, 45, 48] and generic diversity (decorrelation) [2. 7, 11, 54, 77, 83], HUG works as a new learning target (used without CE) rather than acting as regularizer for the CE loss (used together with CE). Following the spirit of [32], we demonstrate the effectiveness and potential of HUG as valid substitute for CE.
Relevant theoretical results. [87] has discussed NC under the case of d < 1, and shown that the global solution in this case yields the best rand-d approximation of the simplex ETF, Along with [87], GNC gives a more profound characterization of the convergence of class-means. We show special case of 2=2, C. It is easy to see that hyperspherical uniformity in this case forms four vectors with adjacency ones being perpendicular. This also the case captured by the best rank-2 approximation (i.e., 2-dimensional hyperplane with simplex ETF projected onto it). Figure 5 gives geometric interpretation for the connection between [87] and GNC. [3] provides in-depth introduction and comprehensive theoretical analysis for the energy minimization problem, which significantly benefits this work.
2D sabspace
Simplex ETF (NC) d-3, C
space
Hopenphaiial Unifemilyy (GNC) d-2, C-2
Higure 5:
Geometric connection be- tween GNC and [87].
Connection to contrastive learning. The goal of contrastive learning [8, 10, 24, 30, 72, 78, 85] is to learn discriminative features through instance-wise discrimination and contrast. Despite the lack of class labels. [78] discovers that contrastive learning performs sample-wise alignment and sample-wise uniformity, sharing similar high-level spirit to intra-class variability and inter-class separability. [35] adapts contrastive learning to the supervised settings where labeled samples are available, which also shares conceptual similarity to our framework and settings.
Related work on (deep) metric learning. Metric learning also adopts similar idea where similar samples are pulled together and dissimilar ones are pushed away. HUG has intrinsic connections to a number of loss functions in metric learning [4, 17, 21, 24, 58, 59, 61, 67, 68, 79, 79-81, 84].
BROADER IMPACT AND FUTURE WORK
Our work reveals the underlying principle hyperspherical uniformity gap, for classification loss function, especially in the context of deep learning. We provide a simple yet effective framework for designing decoupled classification loss functions. Rather than previous objective functions that are coupled and treated as black-box, our loss function has clear physical interpretation and is fully decoupled for different functionalities. These characteristics may help neural networks to identify intrinsic structures hidden in data and true causes for classifying images. HUG may have broader applications in interpretable machine learning and fairnsss/ / bias problems.
Our work is by means perfect, and there are many aspects that require future investigation. For example, the implicit data mining in CE [49] is missing in the current HUG design, current HUG losses are more sensitive hyperparameters than CE (the flexibility of decoupling also comes at price), current HUG losses could be more unstable to train (more difficult to converge) than CE, and requires more large-scale experiments to fully validate the superiority of current HUG losses. We hope that our work can serve as good starting point to rethink classification losses in deep learning.
ACKNOWLEDGEMEN
The authors would like to sincerely thank the anonymous reviewers for all the detailed and valuable suggestions that have significantly improved the paper. This work is supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S18039A, 01IS 18099;; and by the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. AW acknowledges support from a Turing AI Fellowship under EPSRC grant EP/V025279/1, and the Leverhulme Trust via CFI.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Xiang An, Xuhan Zhu, Yuan Gao, Yang Xiao, Yongle Zhao, Ziyong Feng, Lan Wu, Bin Qin, Ming Zhang, Debing Zhang, et al. Partial fc: Training 10 million identities on a single machine. /CCV, 2021.
[2]
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can gain from orthogonality regularizations in training deep networks? In NeurlPS, 2018.
Sergiy V Borodachov, Douglas P Hardin, and Edward B Saff. Discrete energy on rectifiable sets. Springer, 2019.
Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. unifying mutual information view metric learning: cross-entropy vs, pairwise losses. In ECCV, 2020.
[5] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 101111999111999 2020.
[6] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label- distriiistiinttinn aware margin loss. In Neur/PS, volume 32, 2019.
[7] Tianlong Chen, Zhenyu Zhang. Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. In CVPR, 2022.
[8] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In ICML, 2020.
[9] Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv:/904.044 2019.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
[11] Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting deep networks by decorrelating representations. arXiv preprint arXiv:1511.0600 2015.
[12] Henry Cohn and Abhinav Kumar. Universally optimal distribution points on spheres. Journal of the American Mathematical Society, 20(1):99-148, 2007.
[13] Deng, Wei Dong. Richard Socher, Li-Jia Kai Li, Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
[14] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In CVPR, 2019.
[15] Jacob Devlin. Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 040004 2018.
[16] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In NAACL-HLT HLT, 2019.
[17] Ismail Elezi, Sebastiano Vascon, Alessandro Torcinovich, Marcello Pelillo, and Laura Leal-Taixé. The group loss for deep metric learning. ECCV, 2020.
[18] D Jack Elzinga and Donald W Hear. The minimum covering sphere problem. Management science, 19 (1):96-104, 1972.
[19] Ronald Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7(2): 179-188, 1936.
[20] Eduardo arXiv preprint arXiv: 1804.00286 2018.
and Thomas Verdehout, overview of uniformity on the hypersphere.
[21] Weifeng Ge. Deep metric learning with hierarchical triplet loss. In ECCV, 2018.
[22] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. AISTATS, 2010.
[23] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb- Iareelleemmm:: A datasct and benchmark for large-scale face recognition. In ECCV, 2016.
11

---

Published as a conference paper at ICLR 2023
[24] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In CVPR, 2006.
[25] Thomas C Hales. The sphere packing problem. Journal of Computational and Applied Mathematics, 44 (1):41-76, 1992.
[26] XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss; Proximity to and dynamics on the central path. arXiv preprint arXiv: PrXiv:2106.000 202222000022 2021.
[27] Doug P Hardin, Edward B Saff, et al. Discretizing manifolds via minimum energy points. Notices of the AMS, 51(10):1186-1199 2004.
[28] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenel classification. In ICCV, 2015.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[30] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[31] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[32] Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks. In ICLR, 2021.
[33] Like Hui, Mikhail Belkin, and Prectum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.08384, 2022.
[34] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint mnXiv 2016.
[35] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NewrlPS, 2020.
[36] Minchul Kim, Anil K Jain, and Xiaoming Liu. Adaface: Quality adaptive margin for face recognition. In CVPR, 2022.
[37] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[38] Arno Kuijlaars and E Saff. Asymptotics for minimal discrete energy on the sphere. Transactions of the American Mathematical Society, 350(2):523-538, 1998.
[39] NS Landkof. Foundations of modern potential theory, volume 180. Springer, 1972.
[40] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In NeurIPS, 2018
[41] Rongmei Lin, Weiyang Liu, Zhen Liu, Chen Feng, Zhiding Yu, James M Rehg, Li Xiong, and Le Song. Regularizing neural networks via minimizing hyperspherical energy. In CVPR, 2020.
[42] Weiyang Liu, Yandong Wen. Zhiding Yu, and Meng Yang. margennarginnan softmax loss for convolutional neural networks. JCML, 2016.
[43] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR, 2017.
[44] Weiyang Liu, Yan-Ming Zhang, Xingguo Li. Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep hyperspherical learning. In NIPS, 2017.
[45] Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. In NeurlPS, 2018
[46] Weiyang Liu, Zhen Liu, Zhiding Yu, Bo Dai, Rongmei Lin, Yisen Wang, James M Rehg, and Le Song. Decoupled networks. In CVPR, 2018.
12

---

Published as a conference paper at ICLR 2023
[47] Weiyang Liu, Rongmei Lin, Zhen Liu, James M Rehg. Liam Paull, Li Xiong, Le Song, and Adrian Weller. Orthogonal prrapaarmeteee training. In CVPR, 2021.
[48] Weiyang Liu, Rongmei Lin, Zhen Liu, Xiong, Bernhard Schölkopf, and Adrian Weller. Learning with hyperspherical uniformity. In AISTATS, 2021.
[49] Weiyang Liu, Yandong Wen, Bhiksha Raj, Rita Singh, and Adrian Weller. Sphereface revived: Unifying hyperspherical face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(02): 2458-2474, 2023.
[50] Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59:224 22122-241,222 2022.
[51] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:/706.06000 2017.
[52] Qiang Meng. Shichao Zhao, Zhida Huang, and Feng Zhou, Magface: A universal representation for face recognition and quality assessment. In CVPR, 2021.
[53] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):1-13 13, 2022.
[54] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint 180.008000.0555 2018.
[55] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540);529-555 2015.
[56] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In COLT, 2015.
[57] Behnam Neyshabur, Srinadh Bhojanapalli,, David McAllester, and Nati Srebro. Exploring generalization in deep lcarning. In NIPS, 2017.
[58] Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, 2016.
[59] Hyun Oh Song, Stefanie Jegelka, Vivek Rathod, and Kevin Murphy. Deep metric learning via facility location. In CVPR, 2017.
[60] Vardan Papyan, XY Han, and David I. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences. 117(40):24652 224663, 2020.
[61] Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metric learning without triplet sampling. In ICCV, 2019.
[62] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv: 1810. / 1910, 2018.
[63] Edward B Saff and Amo BJ Kuijlaars. Distributing many points sphere. The mathematical intlliiggeenn 19(1):--1,, 1997.
[64] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittweeser. Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot. et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):4844485 2016.
[65] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1449,1556,,, 2014.
[66] Steve Smale. Mathematical problems for the next century. The mathematical intelligencer, 20(2):7-15, 1998.
[67] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NIPS, 2016.
[68] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss; A unified perspective of pair similarity optimization. In CVPR, 2020.
13

---

Published as a conference paper at ICLR 2023
[69] Pieter Merkus Lambertus Tammes. On the origin of number and arrangement of the places of exit the surface of pollen-grains. Recueil des travaux botaniques néerlandais, 27(1):1-88 884, 1930.
[70] Joseph John Thomson. On the structure of the atom: an investigation of the stability and periods of oscillation a number of corpuscles arranged at equal intervals around the circumference of a circle; with application of the results to the theory of atomic structure. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 7(39):237-265, 1904.
[71] Christos Thrampoulidis, Ganesh R Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. arXiv preprint arXiv:2208.05512, 2022.
[72] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NeurlPS, 2020.
[73] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[74] Shayne FD Waldron. An introduction to finite tight frames. Springer, 2018.
[75] Feng Wang, Weiyang Liu, Haijun Liu, and Jian Cheng. Additive margin softmax for face verification. arXiv preprint arXiv:1801.05528 1000000559,201111 2018.
[76] Hao Wang. Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingehao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In CVPR, 2018.
[77] Jaayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In CVPR, 2020.
[78] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
[79] Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, Romain Garnier, and Neil Robertson. Ranked list loss for deep metric learning. In CVPR, 2019.
[80] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In CVPR, 2019.
[81] Kilian Q Weinherger and Lawrence Saul. Distance metric learning for large margin nearest neighbor classification. Journal of machine learning research, 10(2), 2009.
[82] Yandong Wen, Weiyang Liu, Adrian Weller, Bhiksha Raj, and Rita Singh. Sphereface2: Binary classifica- tion is all you need for deep face recognition. In ICLR, 2022.
[83] Pengtao Xie, Aarti Singh, and Eric P Xing. Uncorrelation and evenness: a new diversity-promoting regularizer. In ICMI,, 2017.
[84] Baosheng Yu and Dacheng Tao. Deep metric lcarning with tuplet margin loss. In ICCV, 2019.
[85] Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. ICML, 2021.
[86] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
[87] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In ICML, 2022.
[88] Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. arXiv preprint arXiv:2210.02199 2022.
[89] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. In NeurlPS, 2021.
14

---

Published as a conference paper at ICLR 2023
Appendix
Table of Contents
A Empirical Results on Generalized Neural Collapse
16
A.1 Detailed Metric Definition
16
A.2 Empirical Results of GNC on ImageNet
16
B 2D MNIST Feature Visualization
17
C Other Variants in the HUG Framework
20
C.I Proxy-free HUG
20
C.2 Coupled HUG
20
C.3 HUG without Hyperspherical Normalization
Proof of Theorem
22
E Proof of Theorem
23
F Proof of Theorem 3
24
G Proof of Proposition
25
H Derivation of HUG Surrogate for MHE and MHS
26
1 Proof of Proposition 2
27
J Proof of Proposition 3
28
K Proof of Proposition 4
29
L Derivation of CE's Lower Bound
30
M Proof of Theorem
31
N Experimental Details
32
Additional Experimental Results
33
15

---

Published as a conference paper at ICLR 2023
A EMPIRICAL RESULTS ON GENERALIZED NEURAL COLLAPSE
A.1 DETAILED METRIC DEFINITION
We consider four metries: average classifier energy (ACE), average class-mean energy (ACME), average feature revers--enery (AFRE) and average feature-mean reverse-energy (AFMRE) in the paper. Their definitions are given below:
1 PACE C(C
|00;
(12)
1)
(
EACMR
| 11111
(13)
CCC 1)
$
1
EAFRE
a - ا
(14)
C
Adl -1Ael - 1)
c=1
iøjed.
C
1
EAFMRE = C Acl
Σ
Σ - - ptell
(15)
tEAc
where |Ac| denotes the cardinality of the set Ac, Pae is the normalized feature mean of the c-th class and w. denotes the normalized class proxy of the e-th class.
A.2 EMPIRICAL RESULTS OF GNC ON IMAGENET
We find that the GNC hypothesis remains valid and informative even under the scenario of large number of classes (we use the 1000-class ImageNet-2012 dataset [13] here). Experimental results with ResNet-18 [29] (feature dimension as 512) are given in Figure 6. Experimental results with ResNet-50 [29] (feature dimension 2048) are given in Figure 7.
0.55
Avenge Hypensheiea Inergy
0.85
Energy Eucryy
0.54
Revene
0.8
0.53
0.75
0.7
0.52
0.65
0.51
0.6
Fostoremmzan Revena-Fnergy
Peonure Reverse Energy
0.5
0.55
Numbeer of Fpoccs
Number of Ppochs
Figure 6: Training dynamics of hyperspherical energy (which cuptures inter-class separability) and hyperspherical reverse-energy (which captures intra-clase class variahility). ImageNet 2012 [13] with ResNet- 18 [29] (d = 512, C 1000).
0.56
0.95
Arernge Hypersharical Enaggy
0.9
0.55
Enerryy Energy
Averaee Reversse Eneegyy
0.85
0.8
0.53
0.75
0.7
0.52
0.65
0.51
Reverse Energy
Energy
20
30
40
60
70
50
10
20
30
60
Epochs
Epochs
Figure 7: Training dynamics of hyperspherical energy (which captures claer-class sepurability) and hyperspherical reverse energy (which captures intra-class variability). MonageNet-201222 [13] with ResNet-50 [29] (d 2048, C= 1000).
16

---

Published as a conference paper at ICLR 2023
B 2D MNIST FEATURE VISUALIZATION
We also visualize the 2D MNIST feature in Figure 8, Figure 9 and Figure 10, which is done by directly setting the output feature dimension arrow denotes the class proxy. We compare the difference between the CE loss and the HUG-MHE loss (with either independently optimized proxies or fully learnable proxies). Specifically, for the HUG-MHE loss with independently optimized proxies, we use the following form:
Different color denotes different class and black
max
HP.HUG
(1
(u:(( SG
(16)
(ay) (m (1)
Inter-class Hyperspherical Uniformity
Intra-class Hyperspherical Uniformity
where we stop the gradient for the class proxies the intra-class hyperspherical uniformity term. Form the results, we observe that the our HUG losses generally learns better representations than the CE loss, and moreover, HUG learns more aligned class proxy and class feature- mean than CE.
30
10
20-
-10
10-
-30
-40
-20
-40
-20
20
30-
-10-
-10
-20-
-20
-30
-30
-30
-20
-10
10
20
30
0-
-20
Figure MNIST feature visualization CE loss 1,5,10.15.20 epochs (top left top right middle left middle right -bottom).
17

---

Published as a conference paper at ICLR 2023
-6
-5
-6
3-
2-
1-
0-
-3
Figure 9: 2D MNIST feature visualization for the HUG loss (randomly initialized and then optimized proxies) at 1,5, 1,5,00,1,200 opochs (top left top right middle middle right -boutom..
18

---

Published as a conference paper at ICLR 2023
2-
3-
2-
1-
Figure 2D MNIST feature visunlization HUG loss (fully learnable proxies) at 1,5.10,15,20 opochs (top left top right middle left
middle right botiom).
19

---

Published as a conference paper at ICLR 2023
C OTHER VARIANTS IN THE HUG FRAMEWORK
There are plenty of interesting and useful instantiations for the loss function under the HUG frame- work. In this section, we discuss few highly relevant and natural ones.
C. PROXY-FREE HUG
We have the following general HUG objective function:
max LUU == a (+1)
(
-B
HM (1
(17)
Ty: Inter-class Hyperspherical Uniformity
Tu:
Intra-class Hyperspherical Uniformity
where we can have many possible instantiations. Other than the proxy-based form proposed in the main paper, we can also have proxy-free version:
max LPF-HU =- a ( 1=
Σ
HU ((ssseiE
(18)
Inter-elass Hyperspherical Uniformity
Intra-class Hyperspherical Uniformity
where (eie J. denotes a set of vectors that consist of one random sample class. This is essentially to replace the class proxy with random sample from this class. The proxy-free HUG loss can be used in the scenario where extremely large amount of classes exist and storing class proxies can be very expensive, or in the scenario self-supervised contrastive learning where each instance and its augmentations viewed one class. A MHE-based instantiation of Eq. 18 is given by
min LMIE PF-HG- Esg (fiiEAd - B. Σ [ 17-1
E.s. ((0:)iEA.)
(19)
which can be similarly relaxed to
PH a. Σ
ILRIEA - $JEA, 1 B' Σ Σ
E
- 2/1
(20)
esko
iEAc.JEA...4
where A. denotes randomly selected sample from the e-th class. The first term in Eq. 20 can also be viewed scalable stochastic approximation to the first
the following loss function:
MHEE-PFIUU = =
Σ
|
- Σ Σ
1 2,
(21)
JEA.JEAA,
JEA.JEA,Nit
which is typically optimized by stochastic gradients (samples
mini batch) in practice.
C.2 COUPLED HUG
One advantage of HUG is that it decouples intra-class variability and inter-class separability. However, coupling may also bring some benefits (e.g., robustness on hyperparameters, stability training). To this end, we also propose coupled loss function using the HUG framework:
max LPF.HUG == a (0,))=1
8.
Hll (fat: (EE.))
Intra-class Hyperspherical Uniformity
Coupled Intra class and Inter class Ilyperspherical Uniforiity
(22)
which can be turned into MHE-based instantiation:
C CHHEE-CU
Σ -
-
w. -
Σ
12, - 2U
(23)
1=
where the first term itself couples intra-class and inter-class hyperspherical uniformity. Although the coupled HUG drops the flexibility that the original HUG framework brings, may introduce extra advantages (e.g., training stability).
20

---

Published as a conference paper at ICLR 2023
C.3 HUG WITHOUT HYPERSPHERICAL NORMALIZATION
While the CE loss does not necessarily require hyperspherical normalization for the proxies and features (but hyperspherical normalization does improve CE's generalizability [44, 75]), also consider the HUG framework without hyperspherical normalization here. We note that this issue remains an open challenge and we only aim to provide some simple yet natural designs.
The obvious problem to remove hyperspherical normalization is that HUG has trivial way to decrease its loss simply increasing the magnitude of features and proxies. A naive way address this is to introduce magnitude penalty terms for the features and proxies. This results in
LUN-P-HUG
Hu(l
-
Hu(( we
(my ( 1
Inter-class Hyperspherical Uniformity
Intra-class Hyperspherical Uniformity
A1
Σ We - s|l
A
| - $|1
Soft Magnitude Constraint Proxies
Soft Magnitude Constraint Features
where s denotes the magnitude hyperparameter.
21

---

Published as a conference paper at ICLR 2023
D PROOF OF THEOREM
We first Vc = {01," " ,0 be an arbitrary vector configuration in gd-1 Then we will have that
A(Vo)
i=1 1=1
C C
Σ
2 - 20; 0z)
(24)
j=l
=20 -
Σ 0
$20²
which holds if and only if if 0, 0. The vertices of regular (n 1) -ismplee at the origin well satisfy this condition. With the properties of the potential function f, we have that
Ey(Oc) = Σ Σ
1(1l;: - 0ú
1=1 jンjoi
A(oc)
(25)
20 1)f
C(C-1)
20
2CC 1)f C-11
which holds true if all pairwise distance || - أا are equal for * j and the center of mass the origin (i.e., Σ ú, 0). Therefore, for the vector configuration Va which contains the vertices of a regular (C' - 1-)--ppppee inscribed in gd and centered at the origin, we have that for 2 < C < d + 1
2C
= = C(C
C-1
(26)
Ev((e))
If is strictly convex and strictly decreasing, then (v(Vc) 2 C(C (()-)))( holds only when
V is regular (C 1)-simplex inscribed in gd-1 and centered at the origin.
22

---

Published as a conference paper at ICLR 2023
E PROOF OF THEOREM 2
This result comes as a natural conclusion from [12] where they prove that any sharp code is minimal
hyperspherical f-energy N-point configuration for any interaction potential f that is absolutely monotone on |-1, 1] including all Riesz s-potentials f(t) 2(t 2t)- for 0.
Before we move on, we need to introduce the definition of sharp code:
Definition Let VN "0i,,-.. , be N N-point configuration S
for every (d' + 1)-variate polynomial P of degree at most
Pdla
P(ó.)
N
then Vx is called spherical m-design.
If VN a configuration of N distinct points such that the set of inner products between
distinct points in Vx has cardinality k, then VN is called a spherical k-distance set.
The configuration VN is sharp code if is both a k-distance and spherical (2k 1)- design.
The Komn-Kumar Universal Optimality theorem [12] states that any sharp code is universally optimal. By universal optimality, we mean that
Definition 2 An N N-point configuration VN on gd is called universally optimal if
Eş(Vx)= = Σ
f(6] 62) = min Ey(VN)
VN
01, #02
holds for any absolutely monotone function f 1) + IR.
Then formally, Cohn-Kuma Universal Optimality Theorem states:
Theorem If VN is a sharp code on sd', then VN is universally optimal.
Because the vertices the cross-polytope are a sharp code, then this vertex set (2d' + points in total) is universally optimal, which implies that
E, (WWw) = min (W(w)
(27)
W. Cad
where Wy denote the vertex set of cross-polytope. Then we = for the f-energy and d' d - 1, and we prove our theorem.
23

---

Published as a conference paper at ICLR 2023
F PROOF OF THEOREM 3
This theorem is fact a well-known result (see 37, 63]). This general result is stated as Theorem If A C R is compact with dim A > and < < dim A, then
Es (A, N) (a(A),
lim N00
N
where Ex(A,n) n) == min CAA (WW,,) and W, (A) is Wiener constant. Moreover, the equilibrium measure AssAA on A is unique for the Riesz s-kernel when < s < dim A. Finally, any sequence (of.... N 100 of asympototically s-energy minimizing N N-point configuration on A satisfies
o(101 , week Ms,A, N + 00
From the theorem above, with 2, d 1 > N. C' and A = gd-1, we have that W. (Sd-1) is constant term, and most importantly, we have that these point sequences {ūf, ... (C) asymptoti- cally minimizes the hyperspherical energy on gd-1
Moreover, the same theorem also gives that the leading term of the minimum hyperspherical energy of order (n²) as n >> oo.
24

---

Published as a conference paper at ICLR 2023
G PROOF OF PROPOSITION
We show that zero-mean equal-variance Gaussian distributed vectors (after normalized to norm 1) are uniformly distributed over the unit hypersphere with Theorem 8.
Lemma Let x be n-dimmensional random vector with distribution (0, 1) and E R be an
orthogonal matrix (UU U = I). Then Y = =Ux also has the distribution of Nf.
Proof G.1 For any measurable set A C IR we have that
P(Y A) P(X EUTA A)
1
e (2,a))
UUA A (V2x)
1
e (Uv,Ux)
(28)
(V2T))
J (V2T))
1
e (m,a)
because of orthogonality of U. Therefore the lemma holds.
Theorem The normalized vector of Gaussian variables is uniformly distributed on the sphere.
Formally, let 21, D2 1 En ~ NO, 1) and be independent. Then the vector
I1
32
In
x
(29)
follows the uniform distribution on Sa-1, where /2 + zタ
x?? normalization
Proof G.2 random variable has distribution NO, 1) if has the density function
1 t=²
f(x)
(30)
V2T
A n-dimensional random vector has distribution (O0 1) if the components independent and have distribution (0,1) each. Then the density of x is given by
1 (s..))
(31)
(V2T))
Then we use Lemma about the orthogonal- -anaogoonl-invvranncee of the normal distribution.
Because any rotation is just multiplication with some orthogonal matrix, we know that normally distributed random vectors are invariant rotation. As a result, generating x B." with distribution N(O, 1) and then projecting it onto the hypersphere S produces random vectors U uniformly distributed on the hypersphere. Therefore the theorem holds.
that are
||]]||
The above results indicate that long as class proxies initialize with zero-mean Gaussian, they are uniformly distributed over the hypersphere in a probabilistic sense.
25

---

Published as a conference paper at ICLR 2023
H DERIVATION OF HUG SURROGATE FOR MHE AND MHS
The derivation of LAHE-HUG is as follows:
LMHE-HUG a. Ees ((we) -3. Σ
Esw ((i,)iEA,,
c=1
a
|10; - wer1 11
+ + 3
Σ
| +2 Σ
||ē; w
opc
iJEA
rEA.
Σ cyte
||0b uel| + Σ(
Σ
[ w + :. -tyll
+ Σ
| u.e (l)
(32)
ieAc
S a. Σ
||00 - welll=
-2 E(
+
Σ
(1 - Well + 11:02 - 2;,1l
LjEA
+ Σ
| w.
AE
Σ oftee
l 2..1112++ + B¹
ΣV E
| Dell MHEE-HU
tEAc
The derivation of L'MHs. is as follows:
C
LxMs HuG == (((w)))))) - Σ
0 (fdi)ies, we)
c=1
a min coke
u. - B. Σ 1.0
max u hie A. u. J.
|| u 2
(33)
S min ofde
wel B. Σ
max
| will L'MHS-HUG A
Most importantly, Ee max vE (1 Je Jufo ||u wl LMMIS and LAMSS-UG share the same minimizer (minimum is O, which happens when intra-class feature collapse to its class proxy).
Therefore, L MHHS--HU and LMSS-HUG share the same maximizer, and L HUG can be viewed surrogate loss for LUHS-HUG.
26

---

Published as a conference paper at ICLR 2023
1 PROOF OF PROPOSITION 2
For notational convenience, we first define Su(S5_1,)) .n) == miny, (V.).. and o (S4-1) =
max (V.). We then define that VI is a s-energy minimizing n-point configuration on $6 0 < < (i.e., MHE configuration) and Voe denotes best-packing configuration gd if 8 = oc (i.e. MHS configuration). Since we are considering > 0, we only need to discuss the case of (o, 0j) p(ù, ú₁)= -. Then we will have the following equation:
(S E. (VV))
(34)
(V
(84-1)
Moreover, have that
Ss (Sd-1 ,n)= S Es(iroo)
(0(2 a
Σ
(V
plop 800)) o800)
(35)
ISiAiSN
(n(n2 - (n(n-1))
1))
(V(2
Therefore, will end up with
lim sup 2 (S n)) (n)) S (v
(36)
8h (Su-1)
8-000
Then we take both Eq. 34 and Eq. 36 into consideration and have that
lim (S .n) 1
(37)
80000
which concludes the proof.
27

---

Published as a conference paper at ICLR 2023
J PROOF OF PROPOSITION 3
We write down the formulation of the HUG objectives (with MHE))
min [ 1- )
LMHE-HUG == a. Es, (f2cc - 3 Σ
E.n (f (eed 20c
c=1
a 1200 - w. 1 3
Σ
Σ
| - Ez11
(38)
cpc
+ Σ
|| - -ábell
1EAc
min
C MHE-HUG
= a
Σ 1 cpte
B'
| well
(39)
(#: 1.)
SEAc
For both objectives, we can see that the minimizer of the second term (i.e., the intra-class variability term) is all intra-class feature collapse their class proxy and therefore the second term achieves the global minimum 0.
For the first term of both objectives, the global minimizer can be obtain directly from Theorem Theorem 2 and Theorem 3. It is easy to see that the global minimizer of the inter-class separability term and the intra-class variability term does not contradict with each other and can be achieved simultaneously.
28

---

Published as a conference paper at ICLR 2023
K
PROOF OF PROPOSITION 4
log(1 + Σ
exp(( as) - (w0y, x;)))
log(1 + exp((ws (wyy x.i))
1=
(40)
((ur) (my: :n,))
i= 1=
W - -C - 1) 1=1
w y. x.)
Q1 Coupling and IV
2z: Inter-class Variability
Σ log(1
Σ
exp((w,, 21))
(Wg. xz)))
j=
S <log(1 +
exp((wr,,2:) (W a;:))
iml
(41)
<log(1 +
(exp((u),,x;)) + exp( (w0 x.))))
i=1 1=
llog (1+
Σ exp((w,,x,)) + (C 1) Σ exp(- (pp(- xz,)))
1=1 8733
Qai Coupling IS and IV
e Inter-class Variability
29

---

Published as a conference paper at ICLR 2023
L DERIVATION OF CE'S LOWER BOUND
The derivation is actually very simple and this result originally given by [4]. We find that naturally matches the intuition behind HUG. For our paper be self-contained, we briefly give the simple derivation below. For the details, please refer to Proposition in [4].
We start by rewriting the CE loss as
72
y Di) + An Σ We, we)
n
An Σ Wey we) (42)
LCE
Σ
log
eup mi))
(xw)
@2(w)
where A can be chosen such that both (iww) and Q2(w) become convex functions with respect to w. Taking advantage the convexity, we can separately set the gradient of Q1 (u)) and Q2 (1r) with respect to w and compute their minima. Specifically, end up with
Qı(w) 2 Q1(w),)) = 2An
(x) kz),
(43)
jEAy
n
1 n
Q2(w)= 2 Q2(w222)
log
exp
Σ
n
Li
(44)
An j=1
2X
where lic = (.p((w....)) Σ, exp((103:2..)
denotes the softmax confidence. Combining the two lower bounds above,
we can have that
Lcn 2 Qi(w@,) + Q2(w)22)
Σ log Σ cxp
"
1 An
n
Σ
n
( x.2)
2X
n
2An
i=1 JE
(45)
where the first two terms encourage larger inter-class hyperspherical uniformity, and the last term promotes smaller intra-class hyperspherical uniformity.
30

---

Published as a conference paper at ICLR 2023
M PROOF OF THEOREM 5
This theorem follows naturally from the main result in [50]. [50] has proved that the minimizer of simplified form of the cross-entropy loss the simplex ETF when C' d 1 and the minimizer also asymptotically converges to uniform measure on the hypersphere. More formally, we have
Theorem ([50) Consider the following variational problem
7t min (u) = Σ log j== exp((ug) u.))
exp( ui))
(46)
s.t. U; E RR || = 1, Vi Let pn, be the probability measure S generated by minimizer
Un
8a.
(47)
then for any > 0, Mn converges weakly the unform measure on St-1
From Theorem 3, know that HUG with specific potential energy also converges to the uniform measure on Sd-1 Combining the results above, we can conclude that HUG and CE share the same minimizer.
31

---

Published as a conference paper at ICLR 2023
N EXPERIMENTAL DETAILS
General settings. For MHE-HUG and MHS-HUG, a and are set as 0.15 and 0.015, respectively. For MGD-HUG, a and are set as 0.15 and 0.03, respectively. We train the model for 200 epochs with 512 batchsize for both the cross-entropy (CE) loss and HUG. We use the stochastic gradient descent with momentum 0.9 and weight decay x 10 The initial learning rate is set 0.1 for both CIFAR- 100 and CIFAR- 10 and is divided by 10 at 60, 120, 180 epoch. For the general classification experiments, we use multiple architectures, including ResNet-18, VGG16 and DenseNer121. we use the simple data augmentation: 4 pixels are padded each side, and image is randomly cropped.
Long-tailed recognition. We follow LDAM [6] to obtain imbalanced CIFAR-10 and CIFAR- 100 datasets with different imbalanced ratio. Following LDAM, we use ResNet-32 as our base network. The other setting is the same as our general setting.
Continual learning. We follow DER [5] to construct our continual learning experiments. We split both the CIFAR-I0 and CIFAR- training set into tasks. Each task has classes and 20 classes for CIFAR-IP 10 and CIFAR- -000, respectively. The training batchsize is set 64, where there are 32 incoming samples and 32 replayed samples. Different size of memory buffer is also studied.
Adversarial robustness. For the experiments of adversarial robustness, we first obtain the model trained with CE and HUG. With the information of the attacked model, PGD [51] generates some ad- versarial examples to mislead the attacked model. The test accuracy in the experiments of adversarial robustness shows the accuracy of the perturbed samples.
Visualizing loss landscape. We perturb neuron weights visualize the loss landscape, proposed in [40]. For details, we perturb the model weight with 400 interpolation points in two random vectors around the current model weight minima. The visualization method also the same as [47].
32

---

Published as a conference paper at ICLR 2023
O ADDITIONAL EXPERIMENTAL RESULTS
Training convergence. We observe the training convergence of HUG on CIFAR-10 and CIFAR-100 Both the evaluation accuracy and the training loss, including the overall losses, the intra-class loss and the inter-class loss, are shown in Figure 11. For both the CIFAR-I0 10 and CIFAR-100, the inter-class uniformity loss remains relatively small, which is consistent with the empirical finding in [41, 47]. Moreover, we find that the intra-class uniformity loss (i.e., intra-class variability) dominates the overall loss on CIFAR-100 dataset and is relatively difficult to optimize when the class number becomes large.
0.150
1p1
17-
1.0
3
Overll
Oseil
43
0.952
0.6
0.025
20
0.4
Baach
Figure 11: HUG's training loss and testing accuracy (%) on CIFAR-10 10 (left) and CIFAR-100 (right).
2D loss contour. We also utilize the method in [40] visualize the 2D loss landscape, which is more easy to visualize the flatness of the loss landscape. As shown in Figure 12, the 2D loss landscape of our HUG loss is flatter than the widely used CE loss, showing that HUG yields flat minima which may have better generalization ability.
Figure 12: The 2D Loss Contour of different loss objective. From left to the right: (1). CE loss. (2). HUG overall loss. (3). intra-class loss. (4).
mlter-llass
The ablation of a and 3. In our HUG framework, we introduce two scaling hyperparameters, a for the inter-class hyperspherical uniformity, for the intra-class hyperspherical uniformity. We investigate the effect of the two hyperparameters for the model performance. As shown in Table 8, HUG not sensitive to a, as the inter-class hyperspherical uniformity is always easy optimize. HUG is also not sensitive to 3 in a wide range. The ablations are conducted on CIFAR-100, a is set as 0.15 when we perform ablation on B as 0.015 when doing ablation on a.
0.0003
0.0015
0.015
0.05
0.15
0.5
1.5
5.0
Accuracy
76.31
75.99
76.28
76.16
76.48
76.32
76.1
76.03
0.005
0.015
0.05
0.15
0.3
0.5
1.5
5.0
Accuracy
74.15
76.48
76.12
75.87
75.59
75.24 74.81
74.00
Table Effect of hyperparameters a and
33


Reviews:
[{"review_id": "dmDRxo29PH", "reviewer": "Reviewer_k2o7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an interesting new Hyperspherical Uniformity Gap (HUG) objective that explicitly minimizes within-class variation while maximizing between-class variation. An especially interesting innovation is introducing---instead of canonical linear classifiers---a \"learnable proxy\" that is minimized directly with variation measures in the loss.", "summary_of_the_review": "The HUG objective is an interesting, well-motivated idea. The claims and experiments of this paper are promising. However, the paper, as is, suffers from some clarity issues. For example, there is no reproducible code and the exposition can be improved. Additionally, two very related prior works are missing.\n\nI feel like the contributions of this paper are novel and worthy of publication. But the above-described issues make it only marginally so. \n\n* I am willing to raise my score of this paper if the authors sufficiently address the above-described issues during the revision/discussion period.", "strength_and_weaknesses": "**Strengths**\n The authors show theoretical derivations and discussions as well as numerical experiments supporting the use of HUG. The theorems and experiments, for the most part, are concise and easy to understand---supporting the authors' arguments. The appendix provides additional thought-provoking experiments and ablation studies.\n\n**Weaknesses**\nThis paper has notable weaknesses in its clarity and discussion of prior works.\n\nI describe potential improvements below:\n\n**Major Comments**\n* The authors do not provide any reproducible code. Might is be possible for the authors to give short, self-contained code that (at least) shows the implementation of MHE, MHS, and MGD?\n* Lu and Steinerberger [39] also discusses the connection of hypersphere uniformity with neural collapse in their Section 1.4. Moreover, they discuss this relative to a \"frame potential\" along the same lines as to the energy minimization discussed by the authors of this paper's theory. I realize the authors already briefly acknowledge [39], but is done only in one passing sentence. Given the similarities in these two works, might the authors add a slightly longer discussion/comparison of these two works?\n* The following ICML paper also examines neural collapse when the number of classes is larger than the ambient dimension:\n\n  Zhou, J., Li, X., Ding, T., You, C., Qu, Q. &amp; Zhu, Z.. (2022). On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features. Proceedings of the 39th International Conference on Machine Learning in *Proceedings of Machine Learning Research* 162:27179-27202. https://proceedings.mlr.press/v162/zhou22c.html.\n\n  Might the authors include additional discussions on how their GNC definition compares to the generalization of NC in the $d < C-1$ case given by Zhou et al. 2022?\n\n* Page 6-7: The implementations of MHE, MHS, and MGD all require the computation of *all pairwise distance norms* for the vectors passed to the associated HU(...) function. Computationally, this seems significantly slower than MSE or CE which do not require a quadratically-sized outer loop. Might the authors comment on this?\n\n*  On the same note: Might the authors discuss or include an experiment comparing the compute/CPU time required to train using HUG vs. using CE or MSE?\n\n**Minor Comments**\n* The authors discuss HUG extensively in pages 1-4, but HUG is not introduced until Page 5. As a result, for four entire pages, the reader is left to purely imagine what HUG (hypothetically) might be. This significantly weakens the authors' discussion. I strongly recommend introducing the formal definition of HUG earlier. \n* Page 2: Are Projection FDA and Data FDA instances of HUG with a particular choice of HU(...)? This is especially unclear at this point of this paper since HUG has not been introduced yet.\n* Page 2: It seems there is an even bigger problem with Projection FDA and DataFDA as objectives: They require explicitly calculating the Sb and Sw of (at best) the minibatch and (at worse) the entire dataset. This is very memory and computationally expensive. The inverse will also be expensive. MSE and CE training do not require these matrix quantities and would be much more efficient. Might the authors comment on this?\n* Page 5: \"Nontrivial to optimize the original HUG...\" At this point, the reader has not seen a concrete example of the HU(...) function yet. Thus, it is unclear why the original HUG is difficult to optimize. Even after seeing concrete examples in later papers, I am not sure which particular difficulties the authors are referring to.\n* Page 8-9: I feel like the experimental results on these pages are the most compelling part of the paper. Might it be possible for the authors to move it to earlier in the paper? (After maybe also introducing HUG and its concrete realizations earlier as well?)\n\n**Typos**\n* Figure 3 Caption: \"verifyng\"\n* Appendix A General Settings: \"gradient descend\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VvLGKE0srM", "reviewer": "Reviewer_S3We", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Inspired by the recently observed neural collapse (NC) phenomena in deep learning classifiers where the feature dimension is often large than the number of classes, this paper considers a more general format of neural collapse (GNC) to include the case where feature dimension $d$ is smaller than the number of class $C$. Then the authors decouple NC into two objectives: minimizing intra-class variability and maximizing inter-class separability on the hypersphere via the proposed hyperspherical uniformity. After decoupling these two objectives, the authors then propose a new objective, called hyperspherical uniform gap (HUG),  with three choices of measurement (MHE, MHS, and MGD) as examples. The proposed HUG-based loss shows some improvements in long-tailed recognition, continual learning, and adversarial robustness. ", "summary_of_the_review": "This paper introduces a more general NC definition and introduces a new objective (HUG) by decoupling the objectives into inter-class variability and inter-class separability. Some theoretical analyses are provided for both GNC and HUG. The proposed new loss functions can achieve better performance in long-tailed recognition, continual learning, and adversarial robustness. Overall, the results could be of interest to researchers working in these areas. ", "strength_and_weaknesses": "## Strength:\n- Overall, the paper is well-written and the results are clearly presented.\n- The authors slightly relax the definition of neural collapse to include the case where the feature dimension $d$ is smaller than the number of class $C$.\n- The paper proposes a new class of objective functions based on the hyperspherical uniform gap (HUG), which captures the difference between inter-class and intra-class hyperspherical uniformity. \n- The proposed loss improves upon CE in long-tailed recognition, continual learning, and adversarial robustness.\n\n## Weakness:\n- For the case the feature dimension $d$ is smaller than the number of class $C$, it seems that characterizing the geometry of maximally distant features on the unit sphere is still a challenging problem.  What is the minimum hyperspherical energy could be in this case? In other words, how could one tell if the features converge to GNC since the smallest hyperspherical energy is not zero?\n- It was also observed in [A] that when the feature dimension $d$ is smaller than the number of class $C$, the features are still well separated via CE loss, but not for MSE loss. But there is no analysis for CE loss. Under the setting of unconstrained features, could the new losses or CE be proved to converge to a solution of the generalized NC for that case (i.e., feature dimension $d$ is much smaller than the number of classes (C))?\n\n\n- I am curious whether normalization is necessary for the new objective, hyperspherical uniform gap. In other words, could the objective be extended to the case without normalization of the features?\n- Typo: Caption of figure 3 c: is it about inter-class separability?\n- Typo: Should -2<s<0 in Theorem 4?\n\n[A] Zhou et al., On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features, ICML 2022. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VUEgf8nnljX", "reviewer": "Reviewer_bush", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyzes neural collapse (NC) phenomenon by decoupling it into two separate learning objectives: minimal intra-class variability and maximal inter-class separability, and proposes a Generalized NC applicable to broader applications. It then designs a quality as  hyperspherical uniformity gap (HUG) working as an alternative but meaningful loss as for CE and MSE. Theories and experiments on classification tasks show the effectiveness of the HUG loss.", "summary_of_the_review": "Based on the solidity of the work, as discussed above, I recommend for an acceptation.", "strength_and_weaknesses": "Strength:\n- The writing is good.\n\n- The decoupling and the generalization of NC is excited to me, especially the solid theoretical analysis along with the great numerical performance of the proposed HUG loss (though I didn't check the proofs).\n\nWeakness:\n- It mentions that HUG serves as an alternative loss function in place of CE and MSE, but HUG is only compared with CE loss in the experiments while MSE is missing. Moreover, the following two work closely related to MSE loss and the NC phenomenon are missing in the paper.\n\n    > Zhou, Jinxin, et al. \"On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features.\" arXiv preprint arXiv:2203.01238 (2022).\n\n    > Zhou, Jinxin, et al. \"Are All Losses Created Equal: A Neural Collapse Perspective.\" arXiv preprint arXiv:2210.02192 (2022).\n\n- The idea of GNC is great, but it is still under the assumption of a balanced dataset where each class has the same number of samples, which seems to be a bit conflict with the \"generalized\" claim. I think some comments and discussions are needed for this point.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "871jYdbVEL", "reviewer": "Reviewer_bWKx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the asymptotic training dynamics when classification loss approaches zero.\nIt generalizes the neural collapse phenomenon to settings to models trained on large numbers of classes.\nFurther, it proposes loss functions that decouple inter-classes separability and intra-class variability.\nTheoretical properties of such loss functions are derived and empirically the loss functions are shown to be preferable over the cross entropy loss.\n", "summary_of_the_review": "The main weakness is the limited novelty of the paper and I recommend rejection.", "strength_and_weaknesses": "**Strengths**\n- The paper addresses neural collapse in the setting when the number of classes is larger than the feature dimension, a much more difficult problem.\n- The empirical evaluation not only considers classification accuracy in a standard setting but considers additional aspects, such as long-tailed recognition or adversarial robustness.\n\n**Weaknesses**\n- Many of the presented theorems are essentially known.\n- Central to the paper is the design of loss functions that encourage inter-classes separability and intra-class variability. This is essentially the goal of contrastive learning. Relevant literature is neither mentioned nor discussed.\n- Actually, it is not proven that generalized neural collapse (GNC) occurs if the feature dimension is smaller than the number of classes.\n- The empirical evaluation misses relevant information.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster