Imbalance Trouble: Revisiting Neural-Collapse Geometry
Authors: Christos Thrampoulidis
Ganesh Ramachandra Kini
Vala Vakilian
Tina Behnia
Venue: NeurIPS 2022 Accept (2022)
Keywords: neural collapse
implicit bias
interpolation
max-margin
class imbalance
convex relaxations

Abstract: Neural Collapse refers to the remarkable structural properties characterizing the geometry of class embeddings and classifier weights, found by deep nets when trained beyond zero training error. However, this characterization only holds for balanced data. Here we thus ask whether it can be made invariant to class imbalances. Towards this end, we adopt the unconstrained feature model (UFM), a recent theoretical model for studying neural collapse, and introduce $\text{\emph{Simplex-Encoded-Labels Interpolation}}$ (SELI) as an invariant characterization of the neural collapse phenomenon. Specifically, we prove for the UFM with cross-entropy loss and vanishing regularization that, irrespective of class imbalances, the embeddings and classifiers always interpolate a simplex-encoded label matrix and that their individual geometries are determined by the SVD factors of this same label matrix. We then present extensive experiments on synthetic and real datasets that confirm convergence to the SELI geometry. However, we caution that convergence worsens with increasing imbalances. We theoretically support this finding by showing that unlike the balanced case, when minorities are present, ridge-regularization plays a critical role in tweaking the geometry. This defines new questions and motivates further investigations into the impact of class imbalances on the rates at which first-order methods converge to their asymptotically preferred solutions.

Imbalance Trouble: Revisiting Neural-Collapse Geometry
Christos Thrampoulidis', Ganesh R. Kini!, Vala Vakilian", Tina Behnia" "University of British Columbia, Uniiersity of California, Santa Barbara.
Abstract
Neural Collapse refers the remarkable structural properties characterizing the geometry class embeddings and classifier weights, found by deep nets when trained beyond zero training error. However, this characterization only holds for balanced data. Here we thus ask whether it can be made invariant class imbalances. Towards this end, adopt the ieooosttainedffatuure model (UFM), a recent theoretical model for studying neural collapse, Encoded- Labels Interpolation (SELI)
invariant characterization of the neural collapse phenomenon. We prove for the UFM with cross-entropy loss and vanishing regularization that, irrespective of class imbalances, the embeddings and classifiers always interpolate a simplex- encoded label matrix and that their individual geometries are determined by the SVD factors of this same label matrix. We then present extensive experiments on synthetic and real datasets that confirm convergence the SELI geometry. However, we caution that convergence worsens with increasing imbalances. We theoretically support this finding by showing that unlike the balanced case, when minorities are present, reggeregglarrizatinn plays a critical role tweaking the geomelry. This defines new questions and motivates further investigations into the impact of class inbalances on the rates at which first-order methods converge their preferred solutions.
and introduce Simplez-Enncueed-
Introduction
What are the unique structural properties of models learned by training deep neural networks to zero training error? Is there an implicit bias towards solutions of certain geometry? How does this vary across training instances, architcctures, and data? These questions are at the core of understanding the optimization landscape deep-nets. Also, they are naturally informative about the role of models since different paramoterizations might affect preferred geometries. Ultimately, such understanding makes progress towards explaining generalization of overparamederriee models.
Recently, remarkable new progress in answering these questions has been made by Papyan al. [26], who empirically discover and formalize the so-called Neural-collapse (NC) phenomenon. NC describes geometric properties of the learned embeddings (aka last-layer features) and of the classifier weights of deep-nets, trained with cross-entropy (CE) loss and balanced data far into the zero training-crror regime. The NC phenomenon produces a remarkably simple description of particularly symmetric geometry: (i) The embeddings of cach class collapse to their class mcan (sce (NC) property); and (ii) The class mcans align with the classifier weights and they form simplex equiangular light frame (see (ETF) property). Impportantly as noted by Papyan el al. [26], this simple geometry appears to be "cross-sit uational invariand" across different architectures and different. balonced datasets.
36th Conference on Neural Information Processing Systems (NeurIPS 2022).
"Supported by an NSERC Undergraduate Student Research Grant, an NSERC Discovery Grant, NSF CCF-2009030, CRGS-KAUST award, and by UBC Advanced Research Computing services.

---

Classifier Mrj
Eanbedding Maj
Classifie- min
Embeddiing
SEL matrix Z
0.75
625
(a) SELI geometry; (10, S/P imbalance
(b) ETF geometry; balanced
Figure 1: Visualizing and contrasting the SELI and ETF geometries for k 4 classes.
In this paper, study Neural collapse with imbalanced classes: Is there (ideally equally simple) description of the geometry that is invariant across laas-immalanced datasets?
Contributions. We propose a new description capturing the geometric structure of learned- embeddings and classifier-weightss on possibly class-imbalanced data, which we call the Simplez- Labels Interpolation (SELI) geometry. This new geometry is a generalization of the ETF geometry: It recovers the latter when data are balanced or when there are only two classes, and also, unlike ETF, it remains invariant across different imabalance levels. Importantly, it too, has a simple description: The matrix of learned logits interpolates simplex-encoded label (SEL) matrix Z, and, the individual geometries of the embeddings and classifiers are determined by the SVD factors of this same SEL matrix. Because the particular arrangement of columns of the SEL matrix changes with the imbalance level, this also impacts the geometric arrangement of the embedding and classifier vectors. Overall, the norms and angles of these voctors admit simple closed-form expressions in terms of the imbalance characteristies and the number of classes.
We use an example to illustrate this. Fig. la depicts the SEL matrix 2 R4x110 for a STEP-inbalanced k = 4-class dataset with two majority classes of 50 examples each, and, two minority classes of 5 examples each. Each column of includes the & learned logits for cach one of the 110 examples in the dataset. Each such column has exactly one entry equal to 1- 1/k = 0.75 and three entries equal to -1/k -025.. The corresponding geometry of the embeddings and classifiers, shown in the 3D plot, is found by an SVD of Ż: the left cigenvectors determine the classifiers and the right ones the embeddings. Note that ž is rank 3, hence the geometry is 3D. Since embeddings collapse to their class mcans (sce (NC) property), we only show the four class mean embeddings and the corresponding four classificrs. Two of each correspond to majoritics ("=" marker) and two to minoritics (*+" marker). The radii of the two concentric spheres are equal to the norms of the minority classifiers (red sphere) and of the minority embeddings (blue sphere), respectively. Note that the norms of minorities and majorities are different, and so the angles. Moreover, the classifiers are not aligned with the embeddings. Overall, the geometry is different compared to the ETF geometry seen in the balanced case, which is shown in Fig. 1b. What remains invariant across class-imbalances is that the logits (i.e. inner products between classifiers and embeddings) only take values either 1 1- 1/k or -1/k, so that the logit matrix equal the SEL matrix. Equivalently, the learned model interpolates the simplex-cncoding of the labels.
Below explain the conception of this geometry and our contributions in detail. The initial major challenge was: Assuming a tabas--imbalanc--nnerrr
geomelry exisls, how to find it?
To answer this question, we adopted the Unconstruined Feature Model (UFM) previously introduced as a two-layer proxy model to theoretically justify neural collapse [24, 3, 41]. Motivated by deep-learning practice and by studies on implicit bias of gradient descent (GD) for unregularized CE minimization, we analyze the geometry of solutions to an unconstrained- features Support Vector Machines (UF-SVM) problem. We prove, for immaaamala data, any solution of the UF-SVM follows the SELI gcometry. Thus, the learned end-to-end model always interpolates simplex labels encoding. We show that (ETF) However, (SELI) + (ETF) unless classes are balanced or there is just two of them (k: 2).
(SELI).
Next, we analyze training of the UFM with ridge-regularized CE. Unlike previous studies, we find in the presence of imbalances that regularization matters as changes the geometry of solutions. In fact, show that there is no finite regularization that leads to the SELI
2

---

CIFAR10
SELI/E1E
Thes Fonr
1.2
122
R=5
R=1
R-10
0.5
0.8
200
240
350
MNIST
0.3
SELI,
Trus
R-5
R-10
0.4
R=100
200
50
100
150
200
250
300
260
MaSSIon-MNIST
Distanne SELI/ETF
R=1
R=55
0.7
R=10
0.5
200
2.00
Epochs
Epochs (b) Embeddings
Epochs
(a) Classifiers
(c) Logits
Figure 2: Convergence of learned classifiers, embeddings and corresponding logits to the SELI (solid
lines) vs ETF (dashed lines) geometries, measured using ResNet-18 model, trained far beyond zero training
PVEEP-Imbalaccer CIFAR10, MNIST and Fashion-MNIST, for different imbalance ratios R;; see Sec. for metrics and discussion.
geometry. However, we also show that as regularization vanishes, the solutions do interpolate the SEL matrix (after appropriate normalization.) Finally, show that the SELI geometry differs from the minority-collapse phenomenon [3], since the latter does not correspond to solutions with zero training error. In fact, show minority collapse: (i) does not occur for small finite regularization and finite imbalance ratio, and (ii) occurs asymplotically for vanishing regularization, but only asymptotically as the imbalance ratio grows.
We numerically test convergence to the SELI geometry in both synthetic and real class- imbalanced datasets. For different imbalance levels, the learned geometries approach the SELI geometry significantly faster compared to the ETF geometry (Fig. 2). However, convergence worsens with increasing level of imbalance. A plausible theoretical justification is that we show regularization plays critical role under imbalanccs. We also consistently get better convergence rate for the classifiers. We believe our observations strongly motivate further investigations regarding potential frailties of "asymptotic" implicit bias characterizations and how these might vary in multiclass and possibly inbalanced settings.
1.1 Related works
The original contribution by Papyan et al. [66] has attracted lots of attention resulting in numerous followups within short time period, e.g., [41, 12, 9, 21, 24, 5, 40, 32]. (See also [9, Sec. E] for review of the recent literature.) Several works have proposed and/or used the UFM with CE training to analyze thcorctical abstractions of NC [41, 12, 3, 5). Other works analyze the UFM with square loss [24, 9, 40, 32) and recent extensions accounting for additional layers are st uddde in [32]. Here, we drove particular inspiration from Zhu al. [41], who presented particularly transparent and complete analysis of the optimization landscape of rnge-regularized CE minimization for the UFM under balanced data. the same spirit, also relied the UFM. However, work is, to the best of our knowledge, the first
3

---

Geometry of global minimizers for UFM UF-RidgeCE (Eqn. (1))
UF (Lqn. 12)
ETF
ETF
Balanced
[21, 41, 3,32
[1]]. [Cor. 1.3]
Vλ: NO SELI [Prop.
Imbalanced
A O: SELI [Prop. 2|
[Thm. 1]
SELI
d NO minority collapse [Sec. HI
Table 1: Summary of contributions and comparison to most-closely related work.
explicit goometry analysis for llas--imbllaced data. See also Table for a comparison. The only previous work on neural collapse with imbalances is [3], which was the first to note that collapse of the embeddings is preserved, but otherwise the geometry might skew away from ETF. Also, Fang et al. [3] first proposed studying the new geometry using the UFM and appropriate convex relaxations. With this setup, they presented an intriguing finding, which they termed minority collapse: for asymptotically large imbalance levels, the minorities' classifiers collapse to the same vector. Instead, we derive an explicit geometric characterization of both embeddings and classifiers for both majorities and minorities and for all imbalance levels. Specializing these findings to vanishing regularization and imbalance ratio growing to infinity recovers and gives new insights to minority collapse. Our results also draw from and relate to the literatures on implicit bias, matrix factorization, and imbalanced deep-learning. We defer detailed discussion these to Sec. I of the SM.
Notation. For matrix V E IRmx", V[ú,j] denotes its (i,j) entry, V, denotes the j-th column, the Frobenius,
VT its transpose and V its pseudoinverse. We denote | ||V],, spectral, muclear norms of V. tr(V) denotes the trace of V. and @ denote Hadammard and Kronecker products. V > O denotes V is positive semidefinite and V 2 that V has nonnegative entries. VvL E [Rmx is the gradient of scalar function L with respect to V. Im denotes an m-dimensional vector of all ones and Im the m-dimensional identity matrix. For prctors/matricss with all zero entries, we simply write O, with dimensions understood from context. Cj denotes a column with a single non-zero entry of in the j-th entry.
Problem setup
We adopt the unconstrained feature model (UFM) [24, 3] in a k-class classification setting. Let W dxk = [W1, W2."", be the matrix of classifier weights corresponding to the & classes. Here, d is the feature dimension. We assume throughout that d 2 k- Next, we let Haxn = [h,, [h,, h,, ..,] denote a matrix of n feature embeddings, each corresponding to different example in the training set. We assume each class é [A] has ne 1 examples (thus, ne embeddings) so that Eeels| ne n. Without loss of generality, we assume examples are ordered. Formally, we assume that examples i 1......,7 .., have labels yı = 1, examples i n1 + I,...,,, ni + 7i have labels 1/1 = 2, and so on. The UFM trains the features b [n] (jointly with the weights c e [k]) without any further constraints, i.e. by minimizing the ridge-regularized cross-entropy (CE) loss as follows [41]:
(Wx, H2) arg L(W" H) + A| A //22
(1)
where L(W"H) En log(1 + Eooy, (w,,,, -w.) h:) is the CE loss.
UFM as 2-layer linear net. The formulation above does not explicitly specify inputs. Alternatively, consider training a 2-layer linear net with hidden dimension d, first / second layers H W, over n examples with n-dim. inputs x; = , e [n] and labels as above.
wwwponstrainndfffttureeeee SVM (UF-SVM). Since neural-collapse is observed when train- ing with small vanishing regularization [26]. it is reasonable to consider unregularized version (1). In this special case, gradient descent (with sufficiently small step size) on (1) produces iterates that diverge in norm, but converge in direction [22, 12]. In fact, it has been recently shown that the GD solutions converge in direction to a KKT point of the following max-margin classifier [22, 12]:
(W, H) arg minw h | | sub. to ww. 2 1, ie [n], y1- (2) For convenience, refer to the optimization in (2) as umconstraiined--etuurree SVM (UF- SVM). This minimization (unlike 'standard' SVM) is non-convex. Hence, KKT points (thus, GD convergence directions) are not necessarily global minimizers; see discussion in Sec. 6.

---

Class-imbalance model. To streamline the presentation, we focus on a setting with STEP imbalances. This includes balanced data as special case by setting =
Definition 1 ((R, P)-STPP imbalance). In (R. (R. imbalance setting with label- imbalance ralio R 1 and minorily fraction p e (0,1), the following hold. All minorily (resp. majovily) clusses have the same sumple size mmmn (resp. manm))) There are (1 -p)k majorily and ph minority classes. Without loss of generality, closses (I....,,1
are majorities.
Global structure of the UF-SVM: SELI geometry
In this section, we show that the global minimizers of the non-convex program in (2) take particularly simple form that is best described in terms of a simplez- encoding of the labels. Definition 2 (SEL matrix). The
simplez-encoding
label (SEL) matrix Zixn is such that
Vee [A]].i [n] [c,
1- 1/k
c
(3)
-1/k
c
Onwards, let 2 UAV! be the compact SVD of 2² Specifically, A is positive (k - 1)- dimensional diagonal matrix and Unx(k-1), VEx(K-1) have orthonormal columns.
Each column 2; é R* of 2 represents class-membership encoding of datapoint ie [n]. This differs from the vanilla one-hot encoding y, ey, in that 2; 5, . 11.. Specifically, 2 has exactly & different and affinely independent columns, which together with the zero vector form a k-dimensional simplex, motivating the SEL name. Finally, note that 2²1,, = O; thus, rank(2 2) k- - 1. We gather useful properties about the cigenstructure of Ž in Sec. A. Theorem 1 (Structure of the UF-SVM minimizers). Suppose d k and ().P)-STEP imbalance setting. Let (W, H) any solution and p. the optimal cost of the UF-SVM in (2). Then, p. = |2]. 3 W Moreover, the following statements characterize the geometry of global minémiizess in terms of the the SEL matrix and its SVD.
(i) For the optimal logits holds that WTH 2.
(W) The Gram matrices satisfy f UAUT and WTW- VAVT.
(0i) For partial orthonormal matria R (((-1)xxd W = and = R. RT
We outline the theorem's proof in Sec. 3.3 and defer the details to Sec. C.1. The theorem provides an explicit characterization of the geometry of optimal embeddings and classifiers around the key finding that the optimal logit matrix is always equal to the SEL matrix.
Simplicity. The lack of symmetry in the imbalanced setting, makes a priori unclear whether simple geometry description is still possible, in the balanced case. But, the thcorem shows this to be the case. The key obsorvation is that the optimal logit matrix WTH equals 2 (cf. Statement (i)). Then, the Gram matrices embeddings and classifiers given simply in terms of the singular factors of the SEL matrix (cf. Statements (ii)).
Invariance to imbalances. Equality of the optimal logit matrix to the SET matrix is the key invariant characterization across changing imbalances. This also implies that at optimality all margins are equal irrespective
the imbalance type. The description of Gram matrices in terms of the SVD of 2 is also invariant. Of course, the particular arrangement of columns of 2 itself depends on the values of (R, p). In turn, the singular factors determining the geometry of embeddings and classifiers depend implicitly on the same parameters. Thus, the geometry differs for different imbalance levels; see Fig. for an example.
3.1 Invariant properties: NC and SELI
Here, we further discuss the geometry of embeddings and classifiers inducod by the SVD of the SEL matrix. The first realization is that the embeddings collapse under all settings.
Corollary 1.1. The UF-SVM solutions satisfy the following irrespective of imbalance:
(NC) The embeddings collopse to their class means h, fe. n. Ejjyre fij, Vee [k], i y
5

---

This can be inferred from Theorem 1 (specifically from Statement (ii) and that U has repeated columns.) A more straightforward argument by directly inspecting (2) is as follows. For any fixed (say optimal) W, the minimization over h, is: (i) separable and identical for all i y/ in same class c, and (ii) strongly convex. Hence, for all i y c, there is unique minimizer corresponding to the fixed W; this must be their class mean. Beyond (NC), Theorem 1 specifies the exact geometry of solutions.
Definition 3 (SELI geometry). The embedding and classifier matrices Haxn and Wask follow the simplez-enen ceweee-enoooddddddde labels interpolation geometry when for some scaling a > 0:
W
VAVT ZT
Ż
(SELI)
W
H
UAUT]: where 2 VAUT is the SEL malvia.
HT
Corollary 1.2. The UF-SVM solutions follow the SELI geometry, irrespective of imbalance.
The (SELI) geometry specifies (up to a global positive scaling) the Gram matrices Gw := WTW and GH := H'H, and the logit matrix Z := W The diagonals of the Gram matrices specily the norms ?, and logether with their off-diagonal entries, they further specify the angles between classifiers and between embeddings. Because of (NC), the norms and angles of the embeddings are uniquely determined in terms of the norms and angles of the mean-embeddiins, thus, for all i E [n]. GH[i.j] = (( ] if Vy = ye- Finally, the norms together with the entries of the logit matrix determine the angles between the sets of: (a) the & classifiers and (b) the k mean embeddings. Thus, they specify the degree of aligmment between the two sets of vectors. In the next section, we show that is in fact possible to obtain explicit closed-form formulas describing the norms, angles and alignment of classifier and embedding vectors, in terms of the imbalance characteristics and the number of classes.
Remark 3.1 (Why "SELI"?). For the UFM, Z - W is the learned end-to-end model. According to its definition, the SELI geometry implies W H aZ. Thus, the leorned model interpolates (a scaling of) the SEL matriz, motivates the naming in Definition 3.
3.1.1 Special case: Balanced or binary data
For the special cases of balanced or binary data, Theorem 1 recovers the ETF structure, i.e.
(ETF)) Let M = [A.,..., [ denote the matrix of mean embeddings.
Corollary 1.3 (R = 1 or k 2). Assume balanced data (R = 1) or binary classification 2k= Then, any UF-SVM solution (W,H) follows the ETF geometry defined in [26]:
(ETF) W-M and M WTW = I, I.1T
Thus, when data are balanced or binary: (i) the norms of the classifiers and of the embeddings are all equal; (ii) the angles between any two classifiers any two embeddings are all equal to -1/k- 1; and, (iii) the set of classifiers and the set of embeddings are aligned. See Fig. 1b.
3.2 How the SELI geometry changes with imbalances
For k 3, R > 1, (SELI) + (ETF) and geometry is determined in terms of the SVD factors of the SEL matrix. Sec. A in the SM explicitly characterizes these SVD factors and leads to explicit closed-form formulas for the norms, angles and aligument in terms of R,p and k. Mmin in Sec. B. For example, the following lemma gives formula for the ratio of majority and minority norms. For simplicity, assume equal majorities and minorities.
Lemma 3.1 (Norm ratios). Assume (12, S2)-STPP imbalance. Suppose (W,H) satisfies the (SELI) property. Let W Wmaj: hmaj (resp. W Wminor: )minor) denote majority (resp. minority) classifiers and embeddings, respectively. Then,
(1-2/k)VTC + (R-11)
JR (1-2/k) k/(R-1)/E
[mmalli
and
Wminno
(1-2/k)- V(R+11/2
[mminor
(1- kR(R-1122
Thus, [Wmaj 2
2
|miinor l2 and [lmma $ minoor l|,, with equalities iiʃ R= 1 or ==
2 We assume throughout that the regularization strength same for embeddings and classifiers. For completeness we treat the general case in Sec. C.3 in the SM, where it is shown that different regularization values do not change the SELI geometry as per Definition 3 apart from introducing (global) relative scaling factor between the norms of the embeddings and classifiers.

---

The fact that CE learns majority classifiers of larger norm has been empirically observed in the imbalanced deep-learning literature [17, 18). Lemma 3.1 provides a theoretical justification and precisely quantifies the ratio, not only for classifiers, but also for the learned embeddings.
As another example of closed-form formulas, the angles between any two majority (resp.
minority) classifiers Wmaj, W ma) (resp. W Wminor: W minnrr satisfy (see Sec. B.1.2):
Cos( Cos(Wmaj: Wmas) -2/R+ /R 1)/2 and Cos( Wiinnrr
2 VRR 1)/2
2 (R+ (4)
(k- v(R+ 1)/2
Both formulas evaluate to -1/(k- - 1) for R= 1. Also, the first is strictly decrcasing and the second strictly increasing R, i.c. with larger R majority classifiers go further away from each other, while minority classifiers come closer; see Fig. 1 and Fig. 6 in the SM.
Remark 3.2 (Asymptotics). While we focus on finite values of R, computing limits for our formulas gives asymptotic characterizations as R + co. As an enample, it is easy to see from (4) that the angle between the minority classifiers collapses
zero in that limit. This phenomenon is called "minority collapse" by Fang et al. [3]. Here, we recover it as special case of Theorem I and of SELI. Note also that the rate at which the minority angle collapses is rather slow (see also Fig. 6e in the SM). Additional details are included in Sec. B.
3.3 Proof sketch
We start from the following standard convex relaxation of the UF-SVM [31, 8, 41, 3]: min IZ|. subj. to Z[y. i] Z[e, c, i] 2 1, Vc# = yı [n].
(5)
Ze Roon
The relaxation follows by setting Z = W thus Z is the logit matrix (also, the end-to- end model) of the non-convex UF-SVM. Our key technical innovation is proving that ž is the unique minimizer of (5). There are three key ingredients in this. First, is a clever re- re-parameterivation of the dual program to (5), introducing the SEL matrix 2 in the dual:
max BeRax
tr (ZB) sub. to |B B1,=0, B 2T 20.
(6)
Second, we prove that B = UVT is the unique maximizer of the re- p--arameterized dual problem in (6). While it is not hard to check that B optimizes a relaxation of (6), it is far from obvious that B is unique, and, even more that it satisfies the third constraint. The key technical challenge here is that the third constraint acts entry-wise on B. In fact, to proceed with the proof we need that the constraint is not active, ie. B o 2T > O, or equivalently, that the sign pattern of the entries of B agrees with the sign pattern of the transpose SEL matrix 2". We prove this by an explicit construction of the singular factors U, V exploiting the structure of the SEL matrix. Once we have shown that B is the unique maximizer and is strictly feasible, we use the KKT conditions to prove that Ż is the unique minimizer of the muclear-norm relaxation in (5). To do this, we leverage that strict fcasibility of B implics by complementary slackness all constraints in the primal (5) must be active at the optimum. The proof of the theorem completes by arguing that the relaxation (6) is tight when d 2 kt --1 allowing us to connect the UF-SVM minimizers to the SEL matrix 2. Sec. C.1 for details.
Remark 3.3 (Comparison to literature). The common analysis strategy in all other related works is deriving tight bounds on the CE loss (or related quantities, such as the minimum margin), and, then identifying the structure. in the parameters that achieves those bounds. For example, [41, 5, 3] lower bound the CE loss and [12] upper bounds the minimum margin, all using similor elegant argument based on Schcary-Schwarzz and Jensen inequalities. The ETF geometry is then uncovered by recognizing that it uniquely achieves those bounds. It is not clear how to employ such exercises in the presence of imbolances, due to the absence of symmetry properties (e.g- aligrment of classifiers with embeddings). Our proof of Theorem 1 more direct and is in large enabled by identifying the key role played by the logit matrix.
4 The role of regularization
In this section, we study the geometry of solutions (W,, H)) of ridge-regularized CE. minimization in (1), as a function of both the imbalance and the regularization parameter X.

---

4.1 Global minimizers as solutions to a convex relaxation
The regularized CE minimization in (1) is non-convex. Yet, its landscape is benign and the global solution can be described in terms of the solution to convex relaxation program [41]. Theorem 2 (Reformulated from [41]). Let X> O, d k- and (R.P)-STEP imbolance. Let 2, e RRxx be the unique minömizer of the convex: nuctear-norm-rreulaae loss mindmization,
(7) and, denote LΛ = U its SVD. Any stationary point of (1) satisfies exactly one of the following two. Either it is a strict saddle, or, it is a global minimizer (W,,H) and satisfies
Zx mrg ((Z) A|lZll.
W
VA VA UA]. [W> H. = UU UUA H
This ensures that any first-order method escaping strict saddles finds a stationary point that is global minimizer [41]. Moreover, it describes the structure of the global minimizers of (1) in terms of Zy, the solution to the conver minimization in (7). Structurally, the characterization in (8) resembles the characterization in Theorem 1 regarding the UF-SVM. However, Theorem 1 goes a step further and gives an explicit form for the logit matrix, namely the SEL matrix 2. Instead, 2, in Theorem 2 is given implicitly as the solution to convex program. In the remaining of this section, different values of the regularizer? Also, how does the answer depend on the imbalance level?
ask: how does 2, compare to Ż for
Remark 4.1. Although not stated explicitly this form, Theorem 2 is essentially retrieved from the proof of [41, Theorem 3.2] with two small adjustments. First, Zhu et al. [41] only considers balanced data. Here, we realize their proof actually carries over to the imbalanced setting. Second, relox their assumption d > k to d k- thanks to simple observation: 1 (Z)(Z) = O, hence the CE gradient drops rank (see Sec. E. for details).
4.2 Regularization matters
For balanced data, the minimizers W.. H2 of (1) satisfy the (NC) and (ETF) properties (up to scaling by a constant) for every value of the regularization parameter A 0 [41] (see also (5, 3, 21].) In our language, for all X > 0, there exists scalar as such that a scaling (as W,) of any global solution of the regularized CE minimization in (1) satisfies the ETF geometry. Thus, for balanced data, up to a global scaling, the geometry is: (i) insensitive to X> 0 and (ti) the same as that of the UF-SVM minimizers.
Here, we show that the situation changes drastically with imbalances: the regularization now plays critical role and the solution is never the same as that of UF-SVM for finite X.
Proposition 1 (Imbalanced data: Regularization matters). Assume imbalanced data and k >>2 2. There does not earst finile X> O and corresponding scaling ax such that the scaled solution (ax W,,a,H, of (1) follows the (SELI) geometry. Equivalently, there does not enist X >>0 O and ax such that a scaling of the UF-SVM solution solves (1).
Proof. The proof relies on Theorem 1 as follows. For the sake of contradiction assume there exists X 0 and some ax 0 such that the scaled UF-SVM minimizer (ar, W, W, H) solves (1). Since then (ax W. a. H) is stationary point, it satisfies Vw WTH) (wC(o)WTH) + Xax W
- W. But, by Theorem 1: WT H 2 and W WT VAVT, Moreover, thanks to the special structure of Ž we can check that VzL(a} = 2a²2 for as == k/(ex((a)) + + 1); see Lemma Av. With these, and denoting a == aya', we arrive at the following about the singular values of a²A² XA
H (azC(a) W fD) -AW
ax W (
Z: "²Z²" W - V'A''V' AVAVT
A =>/aa)
Thus, all singular values of 2 must be the same. However, we show in Lemma A.3 that this is not the case unless data are balanced or k 2. Thus, arrive at contradiction. L
8

---

10
Distance to SELI
Distance to ETF
Min. margin
gigemery
10P
10"
7/A
10
n/X
10P
10
10P
A
10
(a) SELI Convergence.
(b) Minimum margin.
(c) ETF convergence.
Figure 3: Numerical study of global solutions (Wx, HA) of (1) across the regularization path.
4.3 Vanishing regularization
As À vanishes, it is not hard to check that minimizers diverge in norm and the relevant question becomes: where do they converge in direction? The following answers this. Proposition 2 (Regularization path leads to UF-SVM). Suppose d > k-1 and ()SPP imbalance. It then holds that lima-o W HX/(W/ 11..:///11//.////////1 2/121..
Put logether with the content of the previous section: For balanced data, the solution is always the same to global scaling. However, for imbalanced data, the solution changes with and only in the limit of x - does it align with that of the UF-SVM. Regarding the proof of the proposition: thanks to Theorems and 2, suffices that the solution ZA of (7) converges in direction to the SEL matrix 2; see Proposition 3 in Sec. D. To show this, we critically use from Theorem that Ž is unique minimizer of (5) (sce Sec. E.3). Closely related results are [28, 16), who studied the regularization path of p-norm regularized CE.
4.4 Imbalance emphasizes the impact of non-convexity
Recall interpreting the UFM as a two-layer linear net trained on the standard basis e; E R". Suppose instead that we train simple k-class linear classifier Exxn on the same data by minimizing ridge regularized CE: ming [(E)+ + E|
It is easy to check that (after scaling) Ż satisfies first-order optimality conditions. Thus, the optimal linear classifter is such that for all X > O, there exists or such that É, a, Contrasting this to Proposition 1, we find that the end-to-end models minimizing ridge-regularized CE for a linear versus a two-layer linear network are the same (in direction) when data are balanced, but differ under imbalances.
5 Experiments
In the experiments, we choose (R, ST) imbalances with varying and we measure convergence to either the (SELI) the (ETF), in terms of the three metrics below corresponding to classifiers, embeddings, and logits, respectively. Denote A = A/|AIr the
Euclidean normalization and GA ATA the Gram matrix of matrix A. Classifiers: G* and G VAVT (see Definition 3). Embeddings: Because of (NC), it suffices to work with the matrix M of mean embeddings (see Sec. 3.1.) Specifically, we measure || |M GMIIF, where G G* and ČELL is computed from the n-dimensional GSELI UAUT by only keeping the columns/root /rows corresponding we employ an additional centering of the class means with their (balanced) global mean; see Sec. G. Logits: We measure [W'M G' [.. Note that, when NC holds this metric is essentially analogous to measuring W H --22r-. 2(r
We measure Gw Gw F, where GTF lI - -
the first example of each class. In the deep-net experiments,
UFM: Global minimizers. Fig. 3 investigates the global minimizers of regularized CE (1) for the UFM and = 4 classes. Thanks to Theorem 2, we obtain such minimizers by solving (7) with CVX [6], and then, using (8) to infer the Gram matrices Gw, Gm and logits WT.. H. Fig. 3c shows that the distance to ETF is large and not approaching zero for any value of X. On the other hand, Fig. 3a numerically validates Propositions and 2: the distance to SELI for all three metrics is non-zero for any finite À > 0, but converges to zero
9

---

10P
10a
R=1
Distanon SELA, ETF
R-10
10
10-1
10
10'
10
10
10
10
10
10
10
10
10'
10'
10
10
10
Epochs
Epochs
Epochs
Logits Figure 4: Geometry of SGD solutions minimizing CE for the UFM; SEI.((Soid//ETFP((ahee)
(a) Classifiers
(b) Embeddings
(c)
as However, this convergence is slow and the rate becomes even worse as R increases. Finally, Fig. 3b depicts the minimum margins of solutions across A. For all sufficiently small X, the minimum margin is strictly positive (see Lemma D.4 in Sec. D.2). As byproduct, this shows that "minority collapse"
[3] can only possibly occur for large X; see also Sec. H.
UFM: SGD solutions. Fig. investigates whether the solutions found by SGD are consistent with the prediction of Theorem about global minimizers of the UF-SVM. We fix k and, for each R, we select Pmiin so that n ((R+ km & 400. The weights of the UFM are optimized using SGD with constant learning rate 0.4, batch size 4 and no weight decay. We train for 10 epochs, much beyond zero training error and plot the distance to SELI and ETF over time. We observe the following: (i) SGD iterates favor the SELI, instead of the ETF geometry. As a matter of fact, the distance SELI decreasing with epochs, suggesting implicit bias of SGD towards global minimizers of the UF-SVM. (ii) However, convergence is rather slow and rates get worse with increasing imbalance. (iii) Also, the embeddings convergence is more elusive compared to that of the classifiers. Interestingly, the last two observations are reminiscent connections between regularization path and (S)GD iterates, worth investigating further. Refer to Sec. for additional numerical results on the UFM.
the trends we observed in Fig. 3a, suggesting
Deep-lcarning experiments. We investigate convergence to SELI in deep-net training of p 1/2)-STEP imbalanced MNIST, Fashion-MNIIS and CIFAR10 datasets with ResNet-18 [10] and VGG-13 [29]: see Sec. G.1.1 for implementation details. The convergence to SEL. and ETF for the classifiers, (centered) mean-embeddings, and logits illustrated in Fig. 2 for ResNet and Fig. 11 in the SM for VGG. The vertical dashed lines mark the vero-training-ere epoch; see Sec. G.1.3. In all plots, the distance to SELI decreases as training evolves and convergence is consistently better compared to the ETF. However, convergence slows down for increasing imbalance (see R= 100). Also, convergence worse for the embeddings compared to classifiers. See Sec. G in the SM for additional results.
6 Outlook: Imbalance troubles and opportunities
We propose (SELI) as the chas-imbalancceinvariaaa geometry of classifiers and embeddings learnt by overparameteerieed models when trained beyond zero training error. We arrive at it after showing that the UF-SVM global minimizers follow this geometry. Subsequently, we conjecture and show experiments supporting that: (C1) GD the UFM leads to solutions approaching the SELI asymptotically in the number of epochs (Sec. T.2 for connection to implicit bias); (C2) training of deep-nets learns models that approach the SELI geometry (Sec. G for additional experiments). We hope our results motivate further theoretical and experimental investigations, especially since data imbalances appear frequently across applications. Beyond that, we believe that further similar studies on identifying geometric structures of learned embeddings and classifiers could offer new perspoctives on generalization. Our results could pave that way since they uncover different geometries (aka SELI for different R values), each leading different generalization (worse for increasing R [1])- Relatedly, envision that further such studies lead to algorithmic contributions in imbalanced deep- learning as they can facilitate studying the implicit-bias effect CE adjustments and post-hoc techniques tailored to imbalanced data [2, 23, 39, 18, 17, 19, 20).
10

---

References
[1] J. Byrd and Z. Lipton. What is the effect of importance weighting in deep learning? In International Conference on Machine Learning, pages 872 881. PMLR, 2019.
[2] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma. Learning imbalanced datasets with label-dissurbbution-aawar margin loss. In Advances in Neural Information Processing Systems, pages 1567 1578, 2019.
3] C. Fang, H. He, Q. Long, and W. J. Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43), 2021.
[4] T. Galanti, A. György, and M. Hutter. On the role of neural collapse in transfer learning. or Xiv preprint or orXic:2112.1111 15121, 2021.
[5] F. Graf, C. Hofer, M. Niethammer, and R. Kwitt. Dissecting supervised constrastive learning. In International Conference on Machine Learning, pages 3821 3830. PMLR, 2021.
6 M. Grant and S. Boyd. CVX: Matlab software for disciplined convex programming, version 2.1. http://cver htt////x.ccooo cwm/cvx, Mar. 2014.
[7] S. Gunasekar, J. Lee, D. Soudry, and N. Srebro. Characterizing implicit bias in terms of optimization geomelry. In International Conference Machine Learning, pages 1832-1844 PMLR, 2018.
[8] B. D. Haeffele and R. Vidal. Structured low-rank matrix factorization: Global optimality, algorithms, and applications. IEEE transactions on pallern analysis and machine intelligence, 42(6))1468 1482, 2019.
9 X. Han, V. Papyan, and D. Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. arXiv preprint arXi::2106002773 2021.
[10] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition." " computer vision and pattern recognition (2015). Google Scholor There is no corresponding record for this reference, pages 770-775 1000777,770 2015.
[1] L. Hui, M. Belkin, and P. Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint ar Xix::2202008384, 2022.
[12 W. Ji, Y. Lu, Y. Zhang, 7. Deng, and W. J. Su. An unconstrained layer-peeled perspective on neural collapse. arXiv preprint arXiv:2110.02796, 2021.
[13] Z. Ji and M. Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803, 07300, 2018
[14) 7. Ji and M. Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference on Learning Theory, pages 1772 1798. PMLR, 2019.
[15 Z. Ji and M. Telgarsky. Directional convergence and alignment in deep learning. Advances Newral Information Processing Systems, 33:17176 13:17176-17186, 2020.
[16] Z. Ji, M. Dudík, R. E. Schapire, and M. Telgarsky. Gradient descent follows the regularization path for general losses. Tn Conference on Learning Theory, pages 2109- 2136. PMLR, 2020.
[17] B. Kang. S. Xie, M. Rohrbach, Z. Yan, A. Gordo, Feng, and Y. Kalantidis. Decoupling representation and classifier for long- tong-tailed recognition, 2020.
[18] B. Kim and J. Kim. Adjusting decision boundary for class imbalanced learning. IEFE Access, 8:81674 281674-81685, 2020. doi: 10.1109/ 10111/9//ACSS.A.. ACCESS 201.SS...200.0000 29111911..
11

---

[19] G. R. Kini, O. Paraskevas, S. Oymak, and C. Thrampoulidis, Label-imbalanced and group-sensitive classification under overparameterization Advonces in Neural Information Processing Systems, 34:18970 19970-18983, 2021.
[20] M. Li, X. Zhang, C. Thrampoulidis,, J. Chen, and S. Oymak. Autobalance: Optimized loss functions for imbalanced data. Advances in Neural Information Processing Systems, 34:3163 3177, 2021.
[21] J. Lu and S. Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint ar Xii::2022.88465 2020.
[22] K. Lyu and J. Li. Gradient descent maximizes the margin of homogencous neural networks. arXiv preprint ar:: 05890, 2019.
[23] A. K. Menon, S. Jayasumana, A. S. Rawat, H. Jain, A. Veit, and S. Kumar. Long-tail learning via logit adjustment. Xiv preprint ar arxi2:007 07314, 2020.
[24] D. G. Mixon, H. Parshall, and J. Pi. Neural collapse with unconstrained features. arXiv preprint ar Xix:20011 11619, 2020.
[25] M. S. Nacson, J. Lee, S. Gunasekar, P. H. P. Savarese, N. Srebro, and D. Soudry. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Inlelligence and Slatislics, pages 3420 3428. PMLR, 2019.
[26] V. Papyan, X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117 (40):24652 24663, 2020.
[27] X. Peng, C. Lu, Z. Yi, and H. Tang. Connections between nuclear-norm and frobenius- norm- brrnd representations. IEEE transactions on neural networks and learning systems, 29(1):218 224, 2016.
[28] S. Rosset, Zhu, and T. Hastie, Margin maximizing loss functions. In NIPS, pages 1237 1244, 2003.
[29] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 400 1556, 2014.
[30] D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1): 2822 2878, 2018.
[31] N. Srebro, J. Rennic, and T. Jaakkola. Maximum-margin matrix factorization. Advances in neural information processing systems, 17, 2004.
[32] T. Tirer and J. Bruna. Extended unconstrained features model for exploring deep neural collapse. Xiv preprint arXic:2202.08087 2022.
[33] G. Vardi, O. Shamir, and N. Srebro. On margin maximization in linear and relu networks. arXio preprint arXiv:21 11222110.2202 02732, 2021.
[34] R. Vidal and P. Favaro. Low rank subspace clustering (Irsc). Pattern Recognition Letters, 43:47 61, 2014.
[35] K. Wang, V. Muthukumar, and C. Thrampoulidis. Benign overfitting in multiclass classification: All roads lead to interpolation. Advances in Neural Information Processing Systems, 34, 2021.
[36] J. Wright and Y. Ma. High- High-dimensional data analysis with low-dimensional models: Principles, computation, and applications. Cambridge University Press, 2022.
[37] T.. Xie, Y. Yang, D. Cai, D. Tao, and X. He. Neural collapse inspired Autraction-repulsionn balanced loss for imbalanced learning, arXiv preprint 2xXtt:2204.08733 2022.
12

---

[38] Y. Yang, L. Xie, S. Chen, X. Li, Z. Lin, and D. Tao. Do we really need learnable classifier the end of deep neural network? arXiv preprint arXio:2203. 09081, 2022.
[39] H.-J. Ye, H.-Y. Chen, D.-C. Zhan, and W.-L. Chao. Identifying and compensating for feature deviation in imbalanced deep learning, 2020.
[40] J. Zhou, X. Li, T. Ding, C. You, Q. Qu, and Z. Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. arXiv preprint ar Xix:2200. 01 01238, 2022.
[41] Z. Zhu, T. Ding, J. Zhou, X. Li, C. You, J. Sulam, and Q Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reffect the paper's contributions and scope? [Yes] We include concrete theoretical and experimental results substantiating our claims
(b) Did you describe the limitations of your work? [Yes] We discuss these throughout the paper. For example, we discuss that convergence to the proposed geometry is slower in certain settings. See also Section 6
(c) Did you discuss any potential negative socictal impacts of your work? [Yes] We do not foresee any potential negative socictal impact of this work
(d) Have you read the el hiic review guidelines and ensured that your paper conforms to them? [Yes] We have reviewed the guidelines and stand by the same
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We state our assumptions clearly
(b) Did you include complete proofs of all theoretical results? [Yes] While main body includes sketches of some of the proofs, more detailed proofs have been deferred to the supplementary material
3. Tf you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Our code scripts and instructions are included
(b) Did you specify all the training details (c.g., data splits, hyperparamoters, how they were chosen)? [Yes] We include these details with experimental results (c) Did you report error bars (e.g,, with respect to the random seed after running experiments multiple times)? [Yes] We show results over multiple runs to confirm invariance of the learned geometries across training instances.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We report these in Sec. 5 and in the SM.
4. Tf you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sec. 5
(b) Did you mention the license of the assets? [Yes] See Sec. 5
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le using/cune curatiig?? [N/A]
(e) Did you discuss whether the data you are using/ /ergg//uraiinggg contains personally identifiable information or offensive content? [N/A]
13

---

5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots,
if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "sKPgmh94GFl", "reviewer": "Reviewer_3oyr", "summary": "The paper studies the Neural-collapse (NC) phenomenon for data with class imbalances. The authors study two versions of the unconstrained feature model (UFM), which is a theoretical model for studying neural collapse, and introduce Simplex-Encoded-Labels Interpolation (SELI) geometry. The SELI geometry is a generalization of the equiangular tight frame (ETF) in the NC with balanced data. For the UF-SVM problem, they show that any solution follows the SELI geometry, and the major classes and embeddings are assigned with larger norms. For the UFM with ridge-regularized cross-entropy loss, they show that no finite regularization leads to the SELI geometry. Instead, as regularization vanishes, the solution converges in direction to the SEL matrix. The authors also provide experiments supporting that SELI is a better characterization than ETF for class-imbalanced data. The experiments also indicate that the convergence worsens with increasing level of imbalance. ", "questions": "1. In the experiments, it is reported that increasing levels of imbalance would worsen the convergence. This imbalance is characterized by the imbalance ratio $R$. There is another term $\\rho$ that describes the minority fraction, and it is not addressed in the theorem or the experiments. I am wondering how the $\\rho$ affects the SELI geometry or the speed of convergence. \n\n2. The NC phenomenon happens for very deep networks. The UFM can be viewed as an approximation for infinitely deep neural networks. Will the imbalance level affects the depth required to observe NC? In other words, will one need a deeper network to observe neural collapse for imbalanced data? It is fine if the authors don't have a good answer for this as the paper does not study the effect of the depth of the networks. ", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper studies data with class imbalances, which is an important extension to the existing NC theory. The paper is well organized and clearly written, making the contribution and methodology easy to understand. The proposed SELI geometry is justified as it serves as the optimal solution for the UF-SVM model. The authors also connect the SELI geometry to the UFM with ridge-regularized CE model, where the solution converges in direction to the SELI geometry when the regularization vanishes. As a result, it seems convincing to use SELI as a generalization for the ETF for class-imbalanced data. Lemma 1 showing that UF-SVM assigns major classes with larger classifiers and embeddings is also interesting. \n\nWeakness: \n\n1. The main weakness of the work is that it only studies the UFM. Even though the UFM is used in other analyses for the NC, it is an oversimplified model and is not equivalent to real neural networks. As a result, it is possible that the SELI geometry characterized in the paper is only true for the UFM instead of empirical neural networks. \n\n2. The main technical contribution of the work is theorem 1, where the authors use convex relaxation and the dual problem to show that $\\hat{Z}$ is the unique minimizer. The analysis for the UFM with ridge-regularized CE model, however, is mostly based on earlier analysis. Theorem 2 is a natural extension of [Zhu et al, 2021], and proposition 2 is an extension of [Rosset et al, 2003]. This may undermine the theoretical contribution of the work. \n\n3. The early works of the NC are mostly empirical observations, showing that one can observe the NC phenomenon in empirical classification tasks. This paper tries to argue that SELI serves as a generalization of the ETF for class-imbalanced data. So it would be nice if authors can provide empirical results where the SELI geometry can be observed over realistic datasets. Compared to Figure 2, a demonstration looks like Figure 1 would be more convincing. Still, the theoretical contribution of the paper can have independent interest to the field, so the additional experiment is not a hard requirement. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qQlMFsKlKi6", "reviewer": "Reviewer_NjgY", "summary": "The paper studies the effect of class-imbalanced data on neural collapse (NC) of DNNs through the unconstrained features model (UFM).\nIt does it by identifying the structure of the minima of the UFM in a max-margin optimization problem and connecting it to those of the minima of CE loss with vanishing regularization.\nIt shows that the new results are in some sense complementary to the \"minority collapse\" that has been shown before (they are two opposite extreme cases of the level of regularization) and shows correlation of the theory with practical NC of DNNs.\nIn my opinion, these are significant contributions.\n", "questions": "1. \nWhat can you say about how your results may be changed if W and H in Eq. 1 and in Eq. 2 have different regularization hyperparameters?\nThe reason I ask this is because only the dimension of H depends on the number of samples (unlike W), while it seems \"closer\" to the weight-decay that is used in practice if both regularization terms will be scaled similarly with respect to n.\nMany previous UFM works provide results that hold for separate hyperparameters for H and W and thus do not have this issue.\n\n2. \nI disagree with the claim that one can interpret the \"UFM as two-layer linear net\" (page 4, line 120), because it is limited to \"data\" $X=I_n$ which is extremely naïve. Also, in a two-layer linear net the dimensions of the weights do not depend on the number of samples n.\nThis problematic interpretation appears also in Section 4.4, which requires some rephrasing of this section in my opinion. \nAlso, you don't really need to repeat again this perspective for the \"UFM: SGD solutions\" that are presented in Figure 4 and are interesting on their own. \n\n3. \nIn page 5 line 188, what do you mean in: \"U has repeated columns\"? it has only k-1 columns. Perhaps you mean rows? Add more details how can you see it from Theorem 1.\nAlso, consider rewriting the \"more straightforward argument\" in a more mathematical/formal way (e.g., do you use Jensen's inequality per class?).\n\n4. \nIn page 6 line 205, note that [18] claims that W is aligned with the features' means after reducing their global mean. Mention why you do not do reduce the global mean in M. (Is it zero like in previous UFM works with CE loss?)\nAlso, don't the equalities hold (only) up to some scalar factors?\n\n5. \nYou should also refer in the paper to the following related work, which also analyzes the UFM (showing effect of bias regularization on the solution structure) and further extends it (to \"two-layer (non)linear net\" on-top of the features): \n\nTirer, T. and Bruna, J., 2022. Extended unconstrained features model for exploring deep neural collapse. arXiv preprint arXiv:2202.08087\n", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: Apart from the novel significant results that are mentioned above, I also find the paper to be nicely written and organized.\n\nWeaknesses: I have mostly minor comments that are listed below.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iIVzj0CTuQb", "reviewer": "Reviewer_6EtE", "summary": "The paper studies neural collapse for classification with class imbalanced data. The authors propose a novel characterization of the neural collapse phenomenon which is called Simplex-Encoded-Labels Interpolation (SELI). It has been shown both theoretically and experimentally that SELI is invariant across different imbalance levels.", "questions": "It would be helpful if there is a comparison between the proposed SELI and the ETF geometry in Figure 1.", "limitations": "The authors have adequately addressed the limitations and potential negative social impact of their work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is well-written and easy to follow.\n2. While the problem of neural collapse with class imbalance has been investigated in the previous work [1], this work derives an novel characterization that is invariant to class imbalances.\n3. The authors extensively studied the role of regularization in tweaking the geometry.\n\nWeaknesses: \n\n1. It is not clear how the proposed description of the geometric structure helps to explain the generalization problem in learning with imbalanced data, and how it helps to mitigate the problem of data imbalance.\n\n[1] Fang, Cong, et al. \"Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training.\" Proceedings of the National Academy of Sciences 118.43 (2021): e2103091118.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Vws-bVX3C3E", "reviewer": "Reviewer_2XxC", "summary": "The authors study the geometric of neural network's learned embeddings and classifier's weights. They investigates a simplified neural network model: unconstrained feature model. Unlike the existing works that focuses on balanced dataset, they focus on imbalanced dataset and establish a novel geometric characterization of NN's learned embeddings and classifier's weights. The results are insightful and can have strong impact towards better understanding of neural network's characteristics in practice.", "questions": "It would be interesting if author can discuss about the challenges to generalize the result to a more general imbalanced dataset. ", "limitations": "No negative impact.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "With existing work on neural collapse on balanced dataset, this paper moves an important step forward to the case of imbalanced dataset.\n1. The paper uses novel technical approaches and proposes SELI, a more generalized description of NN's geometric patterns instead of ETF. A connection between SELI and ETF is also established.\n2. A detailed analysis of how regularization influences the minimizer of UFM is provided, which I think might be insightful for NN as well.\n3. The paper extend the results of minority collapse from previous work and provides a more thorough classification of the UFM minimizer's geometry under imbalanced datasets.\n4. the paper is well written and clearly organized.\n\nWith above said, I believe that the practical impact of the results would be more significant if more experiments are conducted on various neural networks and different datasets, in addition to CIFAR-10 and MNIST.\n\nOverall, I believe that the analysis on imbalanced dataset is a significant step towards the understanding of NN, compared to the balanced dataset. With solid theoretical results, empirical experiment support and a clear presentation of the content, this is a very good paper and I anticipate it to have strong impact on the future study of NN's geometric patterns and dynamics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1e6drfmneqe", "reviewer": "Reviewer_DjjK", "summary": "This work studies the neural collapse for classification problems where the training data is imbalanced. Speciﬁcally, under the unconstrained feature model (UFM), the work theoretically studies the problem with cross-entropy loss and vanishing regularization. Irrespective of class imbalances, the work proved that embeddings and classiﬁers always interpolate a simplex-encoded label matrix and that their individual geometries are determined by the SVD factors of this same label matrix. The study is empirically verified by experiments on synthetic and real datasets that conﬁrm convergence to the SELI geometry and the convergence rate depends on the imbalance ratio.\n", "questions": "* can the authors provide more intuitions behind the SELI? It is a bit difficult to understand based on the current statement of geometry.\n\n* The authors study the neural collapse under the UFM-SVM formulation, arguing that neural collapse is happening with vanishing regularizations. Is there empirical support for this? With moderate regularization, neural collapse still happens. The UFM-SVM is not the commonly used formulation for training neural networks. Or is this for the ease of analysis?\n\n* can the authors explain more on the intuitions behind the SELI? It is a bit difficult to understand based upon the current statement.\n\n* what is the practical benefits of such a study for training deep networks with imbalanced data? It is a bit unclear to the reviewer.\n", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strengths: the work studies an interesting problem of neural collapse under imbalanced data and an unconstrained feature model. It shows a more general phenomenon of SELI geometry instead of Simplex ETF, when the training data is imbalanced. The study is well supported by experiments.\n\n* Weakness: The reviewer finds the paper is a little hard to follow, and the writing needs to be significantly polished and the paper needs to be reorganized to be more clear to the readers. \nAlso, the reviewer cannot find too much practical guidance from this study. For training on imbalanced data, some recent works on neural collapse are missing in the reference and have not been compared/discussed:\nhttps://arxiv.org/abs/2203.09081\nhttps://arxiv.org/abs/2204.08735\nhttps://arxiv.org/abs/2206.04041\nAlthough it is nice to show the convergence to SELI, what would be the benefits of understanding this? For example, one of the recent above works showed that fixing the linear classifier as an ETF can improve generalization performance. \n\nThe current analysis seems to be some trivial extension of [9] and [24], without very clear justifications/motivations behind them. For example, the paper studied two formulations (1) and (2). The KKT condition in Theorem 1, and Theorem 2 for (1) can be directly generalized from [9] and [24] for SELI.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept