Neural collapse vs. low-rank bias: Is deep neural collapse really optimal?
Authors: Peter Súkeník
Christoph H. Lampert
Marco Mondelli
Venue: NeurIPS 2024 poster (2024)
Keywords: neural collapse
deep neural collapse
unconstrained features model
deep unconstrained features model
low-rank bias
Primary area: optimization_for_deep_networks
TL;DR: We show theoretically and empirically that deep neural collapse is not an optimal solution in the general multi-class non-linear deep unconstrained features model due to a low-rank bias of weight regularization.

Abstract: Deep neural networks (DNNs) exhibit a surprising structure in their final layer known as neural collapse (NC), and a growing body of works is currently investigated the propagation of neural collapse to earlier layers of DNNs -- a phenomenon called deep neural collapse (DNC). However, existing theoretical results are restricted to either linear models, the last two layers or binary classification. In contrast, we focus on non-linear models of arbitrary depth in multi-class classification and reveal a surprising qualitative shift. As soon as we go beyond two layers or two classes, DNC stops being optimal for the deep unconstrained features model (DUFM) -- the standard theoretical framework for the analysis of collapse. The main culprit is the low-rank bias of multi-layer regularization schemes. This bias leads to optimal solutions of even lower rank than the neural collapse. We support our theoretical findings with experiments on both DUFM and real data, which show the emergence of the low-rank structure in the solution found by gradient descent.

Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?
Peter Súkeník
Christoph Lampert"
Institute of Science and Technology Austria
Institute of Science and Technology Austria 3400 Klosterneuburg Austria ch1 chl@ista.c ac.
3400 Klosterneuburg, Austria peter sukenik@ista.ac.
Marco Mondelli
Institute of Science and Technology Austria 3400 Klosterneuburg, Austria
marco www.mmlliiist...e.mmmmmmm ac at
Abstract
Deep neural networks (DNNs) exhibit surprising structure in their final layer known neural collapse (NC), and growing body of works has currently investi- gated the propagation neural collapse to earlier layers of DNNs a phenomenon called deep neural collapse (DNC). However, existing theoretical results stricted to special cases: linear models, only two layers or binary classification. In contrast, focus non-linear models of arbitrary depth in multi-class clas- sification and reveal a surprising qualitative shift. As soon as we go beyond two layers two classes, DNC stops being optimal for the deep unconstrained features model (DUFM)- the standard theoretical framework for the analysis of collapse. The main culprit is low-rank bias multi- layer regularization schemes: this bias leads to optimal solutions of even lower rank than the neural collapse. We support our thcoretical findings with experiments on both DUFM and real data, which show the emergence of the low-rank structure in the solution found by gradient descent.
Introduction
What is the geometric structure of layers and learned representations in deep neural networks (DNNs)? To address this question, Papyan et al. (39 focused on the very last layer of DNNs at convergence and experimentally measured what is now widely known Neural Collapse (NC). This phenomenon refers to four properties that simultaneously emerge during the terminal phase of training: feature vectors of training samples from the same class collapse to the common class-mean (NCI); the class means form a simplex equiangular tight frame or an orthogonal frame (NC2); the class means are aligned with the rows of the last layer's 's weight matrix (NC3): and, finally, the classifier in the last layer is nearest class center classifier (NC4). Since the influential paper [39]. a line of research has aimed at explaining the emergence of NC theoretically, mostly focusing on the unconstrained features model (UFM) [36]. In this model, motivated by the network's 's perfect expressivity, one treats the last layer's feature vectors as a free variable and explicitly optimizes them together with the last layer's weight matrix, "peeling off" the rest of the network 99 23 With UFM, the NC was demonstrated in a variety of settings, both as the global optimum and as the convergence point of gradient flow.
The emergence of the NC in the last layer led to a natural research question does some form of collapse propagate beyond the last layer earlier layers of DNNs? A number of empirical works
Eqqual contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

35| gave evidence that this is indeed the case, and we will refer to this phenomenon as Deep Neural Collapse (DNC). On the theoretical side, the optimality of the DNC was obtained (i) for the UFM with two layers connected by a non-linearity in [5]] (ii) for the UFM with several linear layers in 77, and (iii) for the deep UFM (DUFM) with non-linear activations in the context of binary classification 48] No existing work handles the general case in which there are multiple classes and the UFM is deep and non-linear.
20 17 43
40
In this work, we close the gap and reveal surprising behavior not occurring in the simpler settings above: for multiple classes and layers, the DNC as formulated in previous works not an optimal solution of DUFM. In particular, the class means at the optimum do not form an orthogonal frame (nor an equiangular tight frame), thus violating the second property of DNC.
Let J. and K denote the number of layers and classes, respectively. Then, if either I. 2 3 and K 2 10 or L, 2 4 and K 6, we provide an explicit combinatorial construction of class of solutions that outperforms DNC. Specifically, the loss achieved by our construction is a factor (443))(2L+22 lower than the loss of the DNC solution. Our result holds as long as all matrices are regularized.
We also identify the reason behind the sub- sub-optimality of DNC: a low-rank bias. Intuitively, this bias arises from the representation cost of DNN with 22 regularization, which equals the Schatten-p quasi norm [37 in the deep linear case. The quasi norm is well approximated by the rank, and this intuition carries over to the non-linear case as well. In fact, the rank of our construction is O K while the rank of the DNC solution is K. We note that after the application of the ReLU, the rank of the final layer is again equal to K, in order to fit the training data. We also show that the first property of neural collapse (convergence to class means) continues to be strictly optimal even in this general setting and its deep counterpart is approximately optimal with smoothed ReLU activations.
We support our theoretical results with empirical findings in three regimes: (i) DUFM training. (ii) training on standard datasets (MNIST BI,, CIFAR-10 (28) with DUFM-like regularization, and (ii)) training on standard datasets with standard regularization. In all cases, gradient descent retrieves solutions with very low rank, which can exhibit symmetric structures in agreement with our combinatorial construction, see e.g. the lower-right plot of Figure + We also investigate the effect of three common hyperparameters weight decay, learning rate and width- on the rank of the solution at convergence. On the one hand, high weight decay, high learning rate and small width lead to a strong low-rank bias. On the other hand, small (yet still non-zero!) weight decay, small learning rate or large width (and more complex datasets as well) lead to a higher-rank solution, even if that is not the global optimum, and this solution often coincides with DNC, which is in agreement with carlier experimental evidence. Altogether, our findings show that if a DNC solution is found, it is not because of its global optimality, but just because of an implicit bias of the optimization procedure.
The implications of our results go beyond deep neural collapse. In fact, our theory suggests that even the NC2 in the last layer is not optimal, and this is corroborated by our experiments, where the singular value structure of the last layer's class-mean matrices is imbalanced, ruling out orthogonality. This means that standard single-layer UFM, as well as its deep-linear or two-layer extensions, are not sufficient to describe the full picture, as they display a qualitatively different phenomenology.
2 Related work
Neural Collapse. Several papers (a non-exhaustive list includes 12 5 32 33 58
use neural collapse as a practical tool in applications, among which OOD detection and transfer learning are the most prevalent. On the theoretical side, the emergence of NC has been investigated, with the majority
of works considering some form of UFM [36 9.. (55 34 show global optimality of NC under the
cross-entropy (CE) loss, and [59] under the MSE loss. Similar results are obtained by [ 49 18, g)
for the lass-imbalanced setting. (61 23 59 refine the analysis by showing that the loss landscape of the UFM model is benign - all stationary points are either local minima or strict saddle points which can be escaped by conventional optimizers. A more loss- approach connecting CE and MSE loss is considered in [60] NC has also been analyzed for a large number of classes [25). in an NTK regime [45). in graph neural networks (2). We refer the reader to (26 for survey.
The emergence of NC has also been studied through the lens of the gradient flow dynamics. (36 considers MSE loss and small initialization, and [16] renormalized gradient flow of the last layer's features after fixing the last layer's s weights to be conditionally optimal. (23 studies the CE loss
dynamics and shows convergence in direction of the gradient flow to KKT point of the max- margin

---

problem of the UFM, extending similar analysis for the last layer's weights in 447]. The convergence
speed under both losses described in [53]. Going beyond UFM, [57 42 29 study the emergence of NC in homogeneous networks under gradient flow; 38 provides sufficient conditions for neural
collapse; and [52] perturbs the unconstrained features to account for the limitations of the model.
More recently, 20 mentions a possible propagation of the NC to earlier layers of DNNs, giving preliminary measurements. These are then significantly extended in [17 I1 40), which measure the emergence of some form of DNC in DNNs. On the theoretical front, an extension to two-layer non-linear model is provided in [S]] to a deep linear model in [4 and to deep non-lineat model
for binary classification in [48] Alternatively to DUFM, studies DNC in an end-to-end setting with a special layer-wise training procedure.
Low-rank bias. The low-rank bias is well-known phenomenon, especially in the context of matrix/tensor factorization and deep linear networks (see e.g. 12 6 24 54]). For non-linear
DNNs, [50] studies the gradient flow optimization of ReLU networks, giving lower and upper bounds on the average soft rank. [13] studies SGD training on deep ReLU networks, showing upper bounds on the rank of the weight matrices as a function of batch size, weight decay and learning rate. (30] proves several training invariances that may lead to low-rank, but the results require the norm of at least one weight matrix diverge and the architecture to end with a couple of linear layers. [ presents bounds on the singular values of non non-linear layers in a rather generic setting, not necessarily convergence. More closely related to our work is [37], which considers deep linear network followed by single non- non-linearity and then by a single layer. Their arguments to study the low-rank bias are similar the intuitive explanation of Section lower effective rank of the penultimate layer's Gram matrix both at initialization and at convergence. The true rank also measured, but on rather shallow networks and it is far above the DNC rank. M shows a strong low-rank bias of sharpness- serppess--wwree minimization, although only in layers where DNC does not yet occur and the rank is high. [10 22 of DNNs, providing asymptotic results and empirical measurements. These results are refined in 21 56), which show bottleneck structure of the rank both experimentally and theoretically. The measurements of the singular values at convergence in (56) are in agreement with those of Section We highlight that none of the results above allows to reason about DNC optimality, as they focus on infinite width/depth, effective or functional ranks, orthogonal settings, not quantitative enough.
shows that increasing the depth results in
study special functional ranks (Jacobi and bottleneck)
Preliminaries
We study the class balanced setting with N Kn samples from K classes, n per class. Let f(x) = ( )) be a DNN with backbone B(-). The backbone represents the majority of the deep network before the last L layers, e.g. the convolutional part of a ResNet20. Let
H2 E Rddx xN its feature vector representations in the last T. layers, with H, denoting their counterparts before applying the ReLU a. We refer h2 and h/2 to the i-th sample of c-th class of H; and Hi, respectively. Let n 1 Ei h/ and p. 1/2 Σm= ii be the class means layer/ after and before applying o, and Mr, Mi the matrices of the respective class means stacked into columns. We organize the training samples so that the labels Y E R equal Ix & I,, where I is a K x K identity matrix, @ is the Kronecker product and 1, the all-one vector of size n.
X E IR N be the training data, and H₃ = B(X) E Rd1 XN .H, = o (W,, H1) E Rd2xN
Deep neural collapse (DNC). As there are no biases in our network model, the second property of DNC requires the class mean matrices to be orthogonal (instead of forming ETF) 43 48).
Definition 1. We say that layer I exhibits DNC 1, 2 or 3 if the corresponding conditions are satisfied (the properties can be stated for both after and before the application of ReLU);
DNC1: The within-class variability of either H, or H. is O. Formally, It,, = Wo Rh = W for all
i,j E [m] or, in matrix notation, H1 M1 I H, M, @ 1T.
DNC2: The class-mean matrices Mi, Mr are orthogonal, i.e., Mf Mr x IK, Mf Mr x Ik.
DNC3: The rows of the weight matrix Wi are either 0 or collinear with one of the columns of the matrix Mi.
Deep unconstrained features model. arbitrary number of classes K.
To define DUFM, we generalize the model in 48 to an
3

---

Definition 2. The L-layer deep unconstrained features model (L-DUFM) denotes the following optimization problem:
L
Aw
AH1
min Hr.Wi....W WE 2N [WLo(WL- (2 -Y
YI
W
+
HH
P
2
(1)
where |I-1| denotes the Frobenius norm and XHy Awi.... Aw2 0 are regularization parameters.
4 Low-rank solutions outperform deep neural collapse
Intuitive explanation of the low-rank bias. Consider simplified version of L-DUFM:
H.WW Ws. 2N [Wro(W..1.... ... W.W.H.......... - Y Aw W + AH1
min
|
(2)
2
Compared to T. removes all non-linearities except in the last layer, making the remaining part of the network deep linear model, a construction similar the one in [37]. Now, we leverage the variational form of the Schatten-p muasi-norm [46), which gives
<
F
2/L
min
Σ
L-1
Aw
+ An H Iiil
S2/2
WL.W We 1 WT... W2
2
where be computed explicitly. Thus, after solving for H1, W Wi...., WL-1, the simplified L-DUEM problem (2) can be reduced to
(LL( - v Awz
Aū 2
2/6
min HLWE
W 2
Fr.
2N
2
S2/2
2/L
For large values of L, H
is well approximated by the rank of Hr. Hence, the objective
S2/2
value is low when the output HH fits Y closely, while keeping Hr low-rank, which justifies the low-rank bias. Crucially, the presence of additional non lon-liicasssies in the L-DUFM model (I) does not change this effect much, as long one is able to define solutions for which most of the negative (so that ReLU does not have effect).
intermediate feature matrices H1
Low-rank solution outperforming DNC. We define the combinatorial solution that outperforms DNC, starting from the graph structure on which the construction based.
Definition 3. A triangular gruph Tn of order is a line graph of a complete graph Kr, of order To has (2) vertices, each representing an edge of the complete graph, and there is an edge between a pair of vertices if and only the corresponding edges in the complete graph share vertex. Moreover. let Tn be the normalized incidence matrix of Kn, i.e., (Tr)iss -カー and 0 otherwise. Let Gn, denote the adjacency matrix of Ta
1 if vertex belongs to edge
We recall that Tn is strongly regular graph with parameters (n(n - 1)/2, 2), 2, 4) and spectrum 2(n 2) with multiplicity 1, - 4 with multiplicity n and -2 with multiplicity nnn ((n-3)/2 32. Next, we construct an explicit solution (H1, WI...., WL) based on the triangular graph. For ease of exposition,
focus the case where the number of classes K equals (g) for some r 2 4, deferring the general definition to Appendix A.
Definition 4. Let K (2) for 2 4. Then, a strongly regular graph (SRG) solution of the L-DUFM problem obtained by setting the matrices (H1, Wi,..., WL) as follows:
For all I, the feature matrices Hi, Hr are DNCI collapsed, iee. Hi Mi & 1 H,
Mi @ 1?.
For 2 3. and the sum of squared norms of the rows of Mr corresponding to a row of T, is the same for each row of Tr. Since Mr is entry-wise non-negative, Mj = Mi.
Mr. each row of M, is a non-negative multiple of a row of Tr (as
in Definition
For 1, W,, M1 are any pair of matrices minimizing the objective conditionally on M2 defined above.

---

30
0.175
0.150
10
15
0.20
15
00
0.000
20
0075
030
0005
0.00
Figure I: Strongly regular graph (SRG) solution with L = 4, K 10 and r = 5. Left: Class-mean matrix of the third layer Ma. The non-zero entries of each row have the same value and their number r - 1, which corresponds to the degree of the complete graph K,,. Middle: Class-mean matrix of the fourth layer before ReLU Mj (middle left), and its Gram matrix M, M, (middle right). The SRG construction has very low rank before ReLU: rank (M)) = r and rank (Ma)) = K. Right: Mf M4 for DNC. The DNC solution has rank K' all layers before and after ReI.U.
For 2 W, minimizes the objective conditional to input and output to that layer.
As for the last layer L, let AL be a K x matrix where the set of rows equals the set of vectors with two (-1) entries and r (+1) entries. Then, Mr. = o (MM). ) the rows of M are non-negative multiple of A1,T,, and the sum of their squared norms corresponding to either row of Tr is equal.
Finally, the Frobenius norms (i.e. scales) of M1, WI...., D.....,,, Wt. are chosen so as to mini- mize ① while satisfying the construction above.
In this construction, columns and rows of class- mean matrices are associated edges and vertices of the complete graph Ic,. Each row (corresponding to vertex) has non-zero entries at columns that correspond to edges containing the vertex. In the final layer, each row of Mr corresponds to a weighting of vertices in K,, s.t. exactly two vertices get -1 weight and the rest +1, and the value at column is the sum of the values of the vertices of the edge. The class-mean matrices of the SRG solution are illustrated Figure for L = and K = 10 (which gives r = 5): we display M3, M4, MT M4 and, for comparison, also Mf M4 of DNC solution. Very similar solutions to SRG are shown for K = 6 and K = 15 in and of Appendix B.
Let us highlight the properties of the SRG solution, which are crucial outperform DNC. First, the rank of the intermediate feature and weight matrices is very low, only of order ((K)/)), since by construction there only solution that has rank K in all intermediate feature and weight matrices. The low rank of the SRG solution is due to the specific structure of the triangular graph, which has many eigenvalues equal to -2 that become 0 after adding twice diagonal matrix. Second, the definition of M,, ensures that Mr. or( M2) has full rank K. This allows the output W,M, Mr also have full rank and, therefore, fit the identity matrix Jx, thus reducing the first term in the loss (T). Finally, the highly symmetric nature of the SRG solution balances the feature and weight matrices as minimize large entries and, therefore, the Frobenius norms, thus reducing the other terms in the loss
linearly independent rows. This is contrasted with the DNC
Main result. For any L-DUFM problem (specified by K, and all the regularization parameters), let LSRG, LDNG the losses incurred by the SRG and DNC solutions, see Definitions 4 and respectively. At this point we are ready to state our key result.
Theorem 5. If > 6, > or K > 10, L 3 and di > K for all I, then LSRG < LpNc- Moreover, consider any sequence of L-DUFM problems for which K - oo so that 0.499 LNNG for each problem. In that case.
LsnG O(K X(TTT.
LDNC
In words, as long as the number of classes and layers is not too small, the SRG solution always outperforms the collapsed one and the gap grows with the number of classes K.
5

---

The proof first computes the conditionally optimal values of ||Wi for both the SRG and DNC solutions. The specific structure of these solutions enables calculate pseudoinverses of the intermediate features, thus enabling the explicit computation of the weight norms. All these values depend only on the singular values of the feature matrices, which are explicitly given by their scale. As a result, both RS and DNC are expressed via an optimization problem in single scalar variable and, by comparing these problems, the statement follows. The details are deferred to Appendix A.
Although the argument requires L = 3, K 2 10 or L 2 4, K 2 6, the experiments in Appendix B. show that the DNC solution not optimal when L 2 4, K
L 3, K Furthermore, for L = 3 and large K, there is a large gap between LSRG and LNN (even if (3) trivializes). For either K- L = 2, the DNC is optimal, as shown in [51 48
5 Within-class variability collapse is still optimal
While the DNC2 property conflicts with the low-rank bias, the same is not true for DNCI, as the within-class variability collapse supports low rank. We show below that the last-layer NC1 property remains optimal for any L-DUFM problem. A proof sketch follows, with the complete argument deferred to Appendix[AA. (A.2
Theorem 6. The optimal solutions of the L-DUFM ①I) exhibit DNCI at layer L, i.e.,
Hi = Mī @ 1 holds for any optimal solution (H1, WT.
:-... :-... WL) of the L-DDFM problem.
Proof sketch: Assume by contradiction that there exists an optimal solution of (I) with regularization parameters (XH, AW..... Awz). denoted as (Hf, ....... Wi). which does not exhibit neural collapse at layer L. Then, we can construct two different optimal solutions of the L-DUFM problem with n = 1 and regularization parameters (nAAH, .......... Aw,) of the form (H1)), W W...... WT) and (H12). WT ..... Wi)) These two solutions share the weight matrices, and Hi (and, therefore, H2) only differs single column (w.l.o.g.. the first column). The optimality of these solutions can be proved using separability and symmetry of the loss function w.r.t. the columns of H1.
and y, respectively. By exploiting the linearity of the loss function on ray [th] t 2 0} for any h direct computation gives that r and y are not aligned. Let L the loss ( By optimality of both solutions, we get
Denote the first (differing) columns of (1(,)) and (1,
ac
OC
aWL WL).....W.)=(R")) WT.....Wt))
awL (H, (t.......-- (2)= WT.....W
An application of the chain rule gives
Oc OLF Oll++1 Aw, W, OWL
OLF HE, + Aw, Wr. 0HL+1
awp
011++1
where Hr+1 1 is the model output and LF the first term of C, corresponding the label fit. Plugging this back into (4) and using that Wi, is the same in both expressions, we get ((H = -B(H(2) where we have denoted by and B the partial derivatives OCr evaluated (H Wi...., Wi)
aif
and (H7(2) Wi,..., WI), respectively. As Lr is separable with respect the columns of Hi, H, for all 1, the matrices A, B can only differ in their first columns (denoted by a, b), and they are identical otherwise. This implies that ar by After some simple considerations and using that and are not aligned, we reach a contradiction, as we conclude that x f is impossible.
The difficulty in extending Theorem 6 to result on the unique optimality of DNCI for all layers stems from the special role W,, as the loss is differentiable w.r.t. it. By considering a differentiable relaxation of ReLU, we show below approximate result for relaxed L-DUFM model.
Definition 7. We denote by ReLU, (or a.) a function satisfying the following conditions: (i) a (z) = (x), for x E (-o0, 0] Ue, o0), (ii) 0 (z(x) < (xx) for (0, e), and (ii) a continuously differentiable with derivative bounded by universal constant and strictly positive on (0,

---

Theorem 8. Denote by L-DUFM, the equivalent of ①. with a replaced by Let D max d2, dg..... ".....,,d,, and x= AB, >w, ... ..,. with the regularization parameters upper bounded by 1)) Then, for any globally optimal solution of the I--DUFMe problem, the distance between any two feature vectors of the same class in any layer is at most
6E D(L+( (L 17-
(5)
In words, as the activation function approaches ReLU (i.e., c + 0), the within-class variability tends O. The proof starts with similar strategy as the argument of Theorem and then explicitly tracks the error due to replacing a with through the layers. The full argument is in Appendix A.22 AA.
6 Numerical results
We employ the standard DNC1 metric tr(Ew) /tr(E)) where Ew ΣB are the within and between class variabilities. This is widely used in the literature [52 43 and considered more stable than other metrics [43. We measure the DNC2 metric as the condition number of Mj for 1 Z 1 (48) We do not measure DNC3 here, it is not well-defined for solutions that do not satisfy DNC2. For end-to-end DNN experiments, we employ a model from 48 where an MLP with few layers is attached ResNet20 backbone. The output of the backbone is then treated unconstrained features, and DNC metrics are measured for the MLP layers.
6.1 DUFM training
We start with the L-DUFM model (I). training both features and weights. In the top row of Figure 2 we consider 4-DUFM, with K = 10 and n = 50, presenting the training progression of the losses (left plot), the DNC1 metrics (center plot) and the singular values convergence (right plot).
The results are in excellent agreement with our theory. First, the training loss outperforms that of the DNC solution, and it is rather close to that of the SRG solution. Second, DNC1 holds in clear way in all layers, especially in the last ones. Third, the solution at convergence exhibits a strong low rank bias: the ranks of intermediate layers range from 8, and they are always the same in all intermediate layers within one run. For comparison, we recall that the intermediate layers of the DNC solution have full rank K- 10. Third, for few runs, the Gram matrices of the intermediate class means resulting from gradient descent training coincide with those of an SRG solution. Finally we highlight that, similarly our theory, the solutions found in all our experiments in the entire Section 6 have non-negative pre-activations in all intermediate layers of the MLP head except the last one.
Impact of number of classes and depth. For K= 2 or L 2, we recover the results of 48 irrespective of other hyperparameters. The higher the number of classes, the more prevalent are low- rank solutions, while finding DNC solutions becomes challenging. The same holds for increasing the number of layers. For I. = 3 and low number of classes (K < 6), we weren't 't able to experimentally find solutions that would outperform DNC, which aligns nicely with the fact that SRG outperforms DNC only from K = 10 for I. = 3. For large number of classes, the difference between the loss of low-rank solutions and the DNC loss is considerable already for = and becomes even larger for higher L. This is illustrated in the left plot of Figure 3
For L < 5 and moderate number of classes (K S 30), gradient descent solutions are as follows: until layer L 1, feature matrices share the same rank and have similar Gram matrices; intermediate activations are typically non- nen-negative and the ReLU has no effect; then, the rank jumps K after the final ReLU, as pre-activations are also negative. For large L or large K, the rank of the first few layers is low, growing gradually in the last couple of layers (see Figure66lin Appendix (B.nndxxBBB])) [. the ReLU is active only in the final layers. This means that not only very low-rank solutions outperform DNC (as shown by our theory), but such solutions are routinely reached by gradient descent.
Impact weight decay and width. While neither weight decay nor width influence Theorem 5 which shows that DNC is not optimal - both quantities influence the nature of the solutions found by gradient descent. particular, the stronger the weight decay, the lower the rank, see the middle plot in Figure 3 For very small weight decay, DNC is sometimes recovered; for very high weight decay,
7

---

109
SRG loss
tranigg Joss
DNC Joun
10
10a
DNC1
layer
layer
1ayer_
I
2,0
1.5
Singular values
leyer
leyer.
layer_1
10
value
1.0
Inyer_
10
10-3
0.0
20
0.0
0.2
0.4
0.6
0.0
1.0
0.0
0.2
0.4
0.6
0.8
1.0
8
epoch
125
opech
1e5
Index
DNC1
Singuler values
10a
layer_1
2.0
lsyor_
16a
layer_3
layer_2
layer_2
kayer
10-3.
10
layer 4
105
10-2
value
1.0-
Iaver_4_rebb
10
10-15
10-1
0
2 4
apoch
163
opoch
le3
Index
Figure 2: Training loss compared against DNC and SRG losses (left), DNC1 metric training progres- sion (middle) and singular value distribution at convergence (right). Top row: 4-DUFM training with K = 10, À= = 0.004 for all regularization parameters, learning rate of 0.5 and width 30. Results averaged over 10 runs, and we show the confidence intervals at standard deviation. Bottom row: Training of a ResNet20 with a 4-layer MLP head on CIFARIO, using a DUFM- -ike regularization. We use weight decay 0.005 except λи, 0.000005 (to compensate for 5000, which significantly influences the total regularization strength), learning rate 0.05 and width 64 for all the MLP layers. Results averaged over 5 runs, and we show the confidence intervals at standard deviation.
1.0
14-
0.9-
average rank
12:
probability
0.8
1035 ratios
0.7
0.4
6-
0.2
0.5
10
-12
-10
20
40
60
80
log2 weight decay
width
Figure 3: All experiments refer to the training of an L-DUFM model. Results are averaged over runs, and we show the confidence intervals at standard deviation. Left: Ratio between SRG and DNC loss (CsRG/ C DNC), as a function of r, where the number of classes is K = (*) Different curves correspond to different values of L E {3, 4, 4,53. Middle: Average rank convergence, as function of the weight decay in lsgaes when L = 4 and K = 15. Right: Empirical probability of finding DNC solution as a function the width, when
and K 10.
it is never recovered. The width has an opposite effect, see the right plot of Figure For small width, low-rank solutions are much more likely to be found; large width has a strong implicit bias towards DNC and, thus, rank K solutions. This means that, surprisingly. a larger width leads to larger loss, since low-rank solutions exhibit smaller loss than DNC. Thus, at least DUFM, the infinite-widnt limit prevents gradient descent from finding a globally optimal solution, and sub-optimal solutions are reached with increasingly high probability.
6.2 End-to-end experiments with DUFM-like regularization
Next, we train a DNN backbone with an MLP head, regularizing only the output of the backbone and the layers of the MLP head (and not the layers of the backbone). This regularization is closer to our theory than the standard one, since we explicitly regularize the Frobenius norm of the unconstrained features. We also note that training with such regularization scheme is easier than training with the standard regularization scheme. In the bottom row of Figure 2 with 4-layer MLP head trained on CIFARI0.
we consider ResNer20 backbone
8

---

DNC1
Singuler values
101
_
lnyee]
10a
2
3
2.0
layer_3
layar_2
977
10
teyr_
1.5
leyer
layer
4
56
Syyer__rell
1.0
10 5
0.5
10
0.0
0.02
spoch
index
DNC1
Singular values
16.
layer 1
layor
10a
leyer_ 2
2.5
20
10:
2.0
layor_4
layer 5
layer_5
15
10
layer mebu
10
10
1.0
05
10
0.5
10-1
0.00
0.01
0.07
0.03
0.04
0.05
2
6
8
opoch
le5
Indox
Figure 4: Training of a ResNer20 with a 5-layer MLP head on CIFAR-IO (top row) and MNIST (bottom row), using the standard regularization. We pick a large weight decay (0.08 for CIFAR- 10 and 0.04 for MNIST) and a large learning rate (0.005 for CIFAR-I0 and 0.01 for MNIST), Results are averaged over runs, and we show the confidence intervals at standard deviation. Left: DNC1 metric training progression. Middle: Singular value distributions at convergence for all the layers. Right: Gram matrices of M3 (CIFAR-10) and M; (MNIST).
The results agree well with our theory, and they are qualitatively similar to those of Section 6.1 for DUFM training. The DNNs consistently outperform the DNC loss, but still achieve DNC1. The ranks of class-mean matrices range from 5 to 6, and they are always the same in all intermediate layers within one run. Remarkably, the SRG solution found by gradient descent also in this setting.
Both weight decay and learning rate affect the average rank of the solutions found by gradient descent. Varying the width can lead to unexpected results, as it changes the ratio between the number of parameters in the MLP and that in the backbone, so the effect of the width is harder to interpret. Similar results can be seen MNIST.
6.3 End-to-end experiments
Finally, we perform experiments with standard regularization and the same architecture (i.e., DNN backbone plus MLP head) in Section 6.2 In particular, in Figure 4 we consider a ResNet20 back- bone with a 5-layer MLP head trained on CIFARIO and MNIST with standard weight regularization.
Overall, the results remain qualitatively similar to those discussed above. This demonstrates that, in spite of different loss landscape compared to previous settings, the low-rank bias still responsible for DNC2 not being attained. Specifically, for CIFAR10, the rank in the third layer ranges between 8 and 9, and for MNIST ranges between 5 and 7: in contrast, the DNC solution has rank K 10. All DNNs display DNC1 across all layers. Remarkably, for the MNIST experiment the solution displayed in Figure found by gradient descent is the SRG solution (compare the gram matrices in bottom right plot of Figure with the right-most plot of Figure D.
The difficulty of the learning task plays significant role in this setting: when training on MNIST, is rather easy to reach low-rank solutions and rather difficult to reach DNC solutions and the rank depends heavily on the regularization strength as shown in Figure Appendix B.3] when training on CIFAR-10 10, the weight decay needs to be high for the class mean matrices to be rank deficient. Moreover, the learning rate no longer exhibits clear relation with the rank, since gradient descent diverges when the learning is too large. We also observe that the rank deficiency is the strongest in the mid-layer of the ML. head, creating a "rank bottleneck".. This can be seen by a closer look at the tails of the singular values, which better match zero at intermediate layers (the green and red curves corresponding to layers 3 and 4 have tails slightly lower than the other curves). In a more precise manner, further measured effective ranks of all the layers in Figure For instance, the effective

---

ranks of CIFARIO experiment layers are 8.96, 7.46, 6.88, 7.04. which shows the middle layer
closest low hard rank matrix. The rank bottleneck also mentioned in [22] 2T [6]. In fact, these works also measure extremely low ranks, but [22 21 do it on synthetic data with very low inner dimension, while [56] focuses on fully convolutional architectures trained with CE loss and including biases.
In summary, Figure shows that both the low-rank bias and the optimality of DNCI carry over the standard training regime. This means that there are hyperparameter settings for which deep neural collapse, including in the very last layer, is not reached (and likely not even optimal). Although the sub-optimality of DNC in the last layer is not proved formally, this phenomenon is supported by evidence across all experimental settings and further corroborated by our theory where our SRG construction is far from being DNC2-collapsed in the last layer.
7 Conclusion
In this work, we reveal that the deep neural collapse is nor an optimal solution of the deep uncon- strained features model - the extension of the widely used unconstrained features model. This finding considerably changes our overall understanding of DNC, as all the previous models in simplified settings showed the global optimality of neural collapse and of its deep counterpart. The main culprit- the low-rank bias - makes the orthogonal frame property of DNC, and thus DNC as a whole, too high rank to be optimal. We demonstrate this low-rank bias across variety of experimental settings, from DUFM training to end-to-end training with the standard weight regularization. While the structure of the Gram matrices of class means is not captured by orthogonal matrices (or by the ETF), the within-class variability collapse remains optimal. Our theoretical analysis proves this for the DUFM problem, and our numerical results showcase the phenomenon across various settings.
Our analysis focuses on the MSE loss, but we expect similar results to hold for the cross-entropy loss and, in particular, that the same SRG construction proposed here would still refute the optimality of DNC. We leave as an open question whether DNC1 is strictly optimal across all layers. While proving this would likely require new ideas, note that none of experiments converged to solution that would not be DNCI-collapsed.
Acknowledgments and Disclosure of Funding
Marco Mondelli is partially supported by the 2019 Lopez-Loreta prize. This research was supported by the Scientific Service Units (SSU) of ISTA through resources provided by Scientific Computing (SciComp).
References
[1] Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness- aware minimization leads to low-rank features. In Conference on Neural Information Processing Systems (NeurIPS), volume 36, 2023.
[2] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Conference Neural Information Processing Systems (NeurIPS), 2019.
[3] Bradley T Baker, Barak A Pearlmutter, Robyn Miller, Vince D Calhoun, and Sergey M Plis. Low-rank learning by design: the role of network architecture and activation linearity in gradient rank collapse. arXiv preprint arXiv:2402.06655 06700. 2024.
[4] Daniel Beaglchole, Peter Súkeník, Marco Mondelli, and Mikhail Belkin. Average gradient outer product mechanism for deep neural collapse. arXiv preprint wrXiv:2402.13728, 2024.
[5] Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. In Topological, Algebraic and Geometric Learning Workshops, 2022.
[6] Hung-Hsu Chou, Carsten Gieshoff, Johannes Maly, and Holger Rauhut. Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank. Applied and Computational Harmonic Analysis, 68, 2024.
10

---

[7] Hien Dang, Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho. Neural collapse in deep linear network: From balanced imbalanced data. In International Conference on Machine Learning (ICML., 2023.
[8] Hien Dang, Tho Tran, Tan Nguyen, and Nhat Ho. Neural collapse for cross-entropy class- imbalanced learning with unconstrained ReLU feature model. arXiv preprint arXiv:2401.02055: 2024.
[9] Cong Fang, Hangfeng He, Qi Long, and Weijie Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. In Proceedings of the National Academy of Sciences (PNAS), volume 1 118, 2021.
[10] Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. Rank diminishing deep neural networks. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[11] Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the implicit bias towards minimal depth of deep neural networks. arXiv preprint arXiv:2202.09028, 2022.
[12] Tomer Galanti, András György, and Marcus Hutter. Improved generalization bounds for transfer learning via neural collapse. In First Workshop on Pre-truining: Perspectives, Pitfalls, and Paths Forward at ICML., 2022.
[13] Tomer Galanti, Zachary S Siegel, Aparna Gupte, and Tomaso Poggio. SGD and weight decay provably induce low-rank bias in neural networks. arXiv preprint arXiv:2206.05794 2022.
[14] Connall Garrod and Jonathan P Keating. Unifying low dimensional observations in deep learning through the deep linear unconstrained feature model. arXiv preprint arXiv:2404.06106, 2024.
[15] Jarrod Haas, William Yolland, and Bernhard T Rabus. Linking neural collapse and 12 normal- ization with improved cut-of-distribution detection in deep neural networks. Transactions on Machine Learning Research (TMLR), 2022.
[16] X. Y. Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations (ICLR), 2022.
[17] Hangfeng He and Weijie Su. A law of data separation in deep learning. Proceedings of the National Academy of Sciences, 120(36), 2023.
[18] Wanli Hong and Shuyang Ling. Neural collapse for unconstrained feature model under cross- entropy loss with imbalanced data. arXiv preprint arXiv:2309.09725. 2023.
[19] Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research, 2022.
[20] Like Hui, Mikhail Belkin, and Prectum Nakkiran. Limitations of neural collapse for understand- ing generalization in deep learning. arXiv preprint arXiv:2202.08384 2022.
[21] Arthur Jacot. Bottleneck structure learned features: Low-dimension vs regularity tradeoff. In Conference on Neural Information Processing Systems (NeurIPS). 2023.
[22] Arthur Jacot, Implicit bias of large depth networks: notion of rank for nonlinear functions. In International Conference on Learning Representations (ICLR), 2023.
[23] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer- peeled perspective on neural collapse. In International Conference on Learning Representations (ICLR), 2022.
[24] Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference Learning Representations (ICLR), 2018.
11

---

[25] Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu. Dustin G Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for a large number of classes. Conference on Parsimony and Learning (Recent Spotlight Track), 2023.
[26] Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. In Transactions on Machine Learning Research (TMLR), 2023.
[27] Vignesh Kothapalli, Tom Tirer, and Joan Bruna. A neural collapse perspective feature evolution in graph neural networks. In Conference on Neural Information Processing Systems (NeurIPS), volume 36, 2023.
[28] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[29] Daniel Kunin, Atsushi Yamamura, Chao Ma, and Surya Ganguli. The asymmetric maximum margin bias of quasi-homogeneous neural networks. In International Conference on Learning Representations (ICLR), 2022.
[30] Thien Le and Stefanie Jegelka. Training invariances and the low-rank phenomenon: beyond linear networks. In International Conference on Learning Representations (ICLR), 2022.
[31] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998.
[32] Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez- Granda, Zhihui Zhu, and Qing Qu Principled and efficient transfer learning of deep models via neural collapse. In Conference on Parsimony and Learning (Recent Spotlight Truck), 2023.
[33] Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[34] Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59, 2022.
[35] Wojeiech Masarezyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Miloš, and Tomasz Trzcinski. The tunnel effect: Building data representations in deep neural networks. Conference on Neural Information Processing Systems (NeurIPS), 36, 2023.
[36] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11619, 2020.
[37] Greg Ongie and Rebecca Willett. The role of linear layers in nonlinear interpolating networks. arXiv preprint arXiv:2202.00856, 2022.
[38] Leyan Pan and Xinyuan Cao. Towards understanding neural collapse: The effects of batch normalization and weight decay. arXiv preprint S.ti::2309046444 2023.
[39] Vardan Papyan, Y. Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. In Proceedings of the National Academy of Sciences (PNAS), volume 117, 2020.
[40] Liam Parker, Emre Onal, Anton Stengel, and Jake Intrater. Neural collapse in the intermediate hidden layers of classification neural networks. arXiv preprint wwXiv:2308.02266. 2023.
[41] Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101.00072, 2020.
[42] Tomaso Poggio and Qianli Liao. Implicit dynamic regularization in deep networks. Technical report, Center for Brains, Minds and Machines (CBMM), 2020.
[43] Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso Poggio. Feature learning in deep classifiers through intermediate neural collapse. Technical Report, 2023.
12

---

[44] Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in tensor factorization. In International Conference on Machine Learning (ICML), 2021.
[45] Maríia Seleznova, Dana Weitzner, Raja Giryes, Gitta Kutyniok, and Hung-Hsu Chou. Neural (tangent kernel) collapse. In Conference on Neural Information Processing Systems (NeurIPS), volume 36, 2023.
[46] Fanhua Shang, Yuanyuan Liu, Fanjie Shang. Hongying Liu, Lin Kong, and Licheng Jiao. A unified scalable equivalent formulation for Schatten quasi-norms. Mathematics, 8(8), 2020.
[47] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. In Journal of Machine Learning Research, volume 19, 2018.
[48] Peter Súkeník, Mareo Mondelli, and Christoph Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[49] Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. In Conference Neural Information Processing Systems (NeurIPS), 2022.
[50] Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu networks. In Algorithmic Learning Theory (ALT), 2023.
[51] Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning (ICML), 2022.
[52] Tom Tirer, Haoxiang Huang, and Jonathan Niles-Weed. Perturbation analysis of neural collapse. In International Conference on Machine Learning (ICML), 2023.
[53] Peng Wang, Huikang Liu, Can Yaras, Laura Balzano, and Qing Qu. Linear convergence analysis of neural collapse with unconstrained features. In NeurIPS Workshop Optimization for Machine Learning (OPT), 2022.
[54] Zihan Wang and Arthur Jacot. Implicit bias of SGD in Læ-regularized linear DNNs: One-way jumps from high low rank. In International Conference on Learning Representations (ICLR), 2024.
[55] E Weinan and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Mathematical and Scientific Machine Learning, 2022.
[56] Yuxiao Wen and Arthur Jacot. Which frequencies do CNNs need? Emergent bottleneck structure in feature learning. arXiv preprint arXiv:2402.0800 2024.
[57] Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds. In Research, volume 6, 2023.
[58] Jiawei Zhang, Yufan Chen, Cheng Jin, Lei Zhu, and Yuantao Gu. EPA: neural collapse inspired robust -itt----iittibttionn detector. arXiv preprint 24X.v22401.01710, 2024.
[59] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features. In International Conference on Machine Learning (ICML), 2022.
[60] Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. In Conference Neural Information Processing Systems (NeurIPS), 2022.
[61] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
13

---

A Proofs
A.1 Low-rank solutions outperform deep neural collapse
We start by providing more detailed definitions of SRG and DNC solutions.
Definition 9. Let K K= x for r 2 4. Then, a strongly regular gruph (SRG) solution of the L-DDFFM problem ① is obtained by setting the matrices (H1, Wi..... WL) as follows. For all I, the feature matrices Hi, Hi are DNCI collapsed, i.e., Hi M, @ I Hi Mi @ 17 For S 1 L-
Mt = M, and M1- = Ar where each row of A is multiple of the standard basis vector of dimension and the sum of squared multiples corresponding to one basis vector equals the same parameter aq In other words, each row of Mi is a multiple of a row of y and the sum of squared norms of the rows of Mi corresponding
row T, is the same for each row Tr. For 1, W1, MI are any pair of matrices that minimize the objective conditionally on M2 being defined as above. For 1 2 2, Wi minimizes the objective conditional to the input and output to that layer. Let A be a K x r matrix where each row has exactly two (-1) entries and exactly r (+1) entries. Then, normalize the matrix T,, so that each row is unit norm and multiply from the left with AL of dimension dr, x K, where each of the rows of A is a multiple of a standard basis vector of dimension K and the lotal sum of squared multiplex corresponding to each basis vector is or.. Then, ML is obtained via this procedure, and ML- a (ML). Finally, the parameters {ai)l=2 satisfy
VnAN,,NN AwL (V2+ v(r - 1)(r -2))
2))
1-2
8' 2
01
nd21 1 Aw,
(6)
(nndH, Aw, (V2+ V(r - (/(r - 2)) pb- 1)3 1) Aw. 2
1-2
4((r ((r-2)(() - -3) +2)
and the parameter q 2 0 is chosen to minimize the objective function in
Definition 10. A deep neural collapse (DNC) solution for any number of classes K of the L-DUFM problem T obtained by setting the matrices (H1, W1, WI...., WL) as follows. For all 1, the feature
matrices H1, Hi are DNCI collapsed, i.e., Hi = M, @ 1 H. = Mj @ 1T For 2 L, M, = Mi and Mf Mi = cylk. For 1= 1, Wi, Mi are any puir of mutrices that minimize the objective conditionally on M2 being defined as above. For 1 2 2, Wt minimizes the objective conditional to the input and output that layen. Finally, the parameters {ai)=z satisfy
1-1
W
04
2 L
AHS 22
1= Aw:
L-1 Aw...
01.
AH n
Aw:
and the parameter q 2 0 is chosen to minimize the objective function in 1I
Next, we define the SRG solution when K + (2) for any r, and provide two constructions, each useful for different parts of the proof of Theorem 5
Definition 11. strongly regular graph (SRG) solution for K 2 6 of the L-DUFM problem ( is obtained in one of the two following ways.
1. First, we take the largest r s.t. K 2 (2) and construct the SRG solution (H1, WI...., Wr) as in Definition 9 setting the number of classes to (2), Next, we construct a DNC solution (H₂, WT.... Wn as in Definition 10 setting the number of classes K K -G)). Then, to construct H, of the SRG solution, we create it as diagonal block matrix with number of columns equal to K and number of rows equal to d2 + da, where these are the numbers of rows of the respective H matrices; the first block is Hy the second block is H and
"The order is not important, both the rows and the columns can afterward be permuted if we accordingly permute also the weight matrices.
14

---

the off off-diagonal blocks are zero matrices. Similarly, we extend the weight matrices such that, for any I, Wi is a block diagonal matrix where the number of rows is d + di+1 and the number of columns is di + d;; the first block is Wt, the second block is W,, and the off-diagonal blocks are zero matrices.
2. First, we take the smallest r s.t. K S (2) and construct the SRG solution as in Definition setting the number of classes to (2). Then, we just remove the (2) - columns of H that achieve the highest individual fil losses (in case of tie choose arbitrarity), and define the SRG solution as the original solution without these columns.
We recall our main result and give the proof.
Theorem 5. K > 6, 2 4 or K > 10, = 3 and di 2 K for all I, then CSRG < LDNC. Moreover, consider any sequence of L-DUFM problems for which K - oo so that 0.499 LDNG for each problem. In that case,
CSRG O(K 2((+1)) LDNC
(3)
Proof. We start by considering the case K (2) for some Without loss of generality we can assume n = 1, because all comparisons are between solutions that are by definition DNCI collapsed, and the ratio LsRC/ DNN in the theorem statement does not depend
We first compute the loss of the SRG solution as in Definition 9 up to only one degree of freedom.
Let us go term-by-term. The simplest to evaluate is Aw; ||i 12 for 2 S L 2. Using Lemma 13 (which relies on Lemma T2). we get
Awa
2 rdw, a++1
Will
2 oi
Similarly, for layer I. - 1, we use Lemma 14 (again relying on Lemma 12 to compute:
Awz
,2(r 1)² az
2
8((r - 2) 3) + 2) a--1
Combining Lemma 15 with Lemma 18 we get:
Awi
An
|[
+
|
l
AWw,AnI1 (V2+ v(r - (()rr 2) az.
2
2
Finally, combining Lemma 17 with Lemma 16 we get:
Awz
[Wrl = dwz
W - J 1- +
2K
2
2
(r 2-2 - 3) aL +
(r- 2)( br 19)
r(r-1)
2 AWL
Aw
r r-1
2
2(r-3)2
3)+2 aL + r(r-1)
Aw
(r-
r(r33)
Aw
2
() a1 + -(r-1) Aw,
2
The total loss of the SRG solution just the sum of all these terms, which is expressed terms of a22 o.,...,, We now verify that the choice in (6) minimizes the loss, having set = 02 so, we compute the partial derivatives of w.r.t. the aj and set them 0:
1/2
To do
dc
a3 a
A/w,A - (V2 + (r 1)(r 2) a2
0 = Dan
03 AW.AHH- (V2+ + Vrr 1)(r -2)
02
rdwa
For < 5 L - 2, we have
0L
rdw; 0141
TdW--1 1
0
dai
2
oj
2
+
2
04
a1+1
Aw
aD
01
Awi
15

---

And finally, for layer L 1, we have
0c XW (r 1)2 1)2 a1
TXWn--2 1
daL-1
2 4((r - 2) (3) 3) 2)
oz ai
2 0L-2
00 Xw
4((r - 2)(r - + +2)
O1.11
AWL-1 1
r(r- - 11)2
1k1.-2
Denoting by p = VWW,HH
(V2+ v(r- - 1)(rr - 2)) we can express these fractions as
a1+1
rdw, 9:
2 2515 T. 2
04
4((r - 2r 3) + 2)p
0L-1 ,2(r 1)2>wz
9,
which gives the expressions in (6. Finally, plugging this back into the loss function get univariate q-dependent function of the following form:
AWL
LSRG(
2 4(r 2) 19)
(V Aw, As, V2+ 1)(r
ab r(r-1) Aw
nb 1(r- 1 Aw,
2
AWz
2
Aww An, V2+ (r-1)(r-21) 8(r-3)2
9b + r(r-1) AWL
pli 1)a I1 Aw;
2
r(r-3]
Awz
2 2W, V2+ (r-1)(r-2))
ql + r(--)) AW 2
(r-1)" 11:-2" Aw,
V AW,AH, (V2+ v(r 1)(r 1)( q.
2
The loss of the DNC solution can be computed by simple extension of the expression (17) from 48
(DNC(C)= Awa
r(r-))
I Ir(r 1)
2 2 Aw.-.9.
L
qh + r(r 1) Awz
An 11 Aw.
2
At this point, we split our analysis for L = 3 and for L > 3. We start with L > 3, which is simpler.
Analysis for I, 3. As K 2 6, the following upper bound holds:
Lsno(g) s (g)a(g) = Awa
-(r-1)
2
8( ((VAn, An (V2+ (r- 1)(r 2)
-- r(r-1) Aw..
rt ( I Aw;
L
v Aw,dH (V2+ Vrr 1)(r 2) 9.
2
Now we reparametrize (DNC(4) and the upper bound on Esno(q), so that they look as similar as possible. By replacing Aw,_,9 with g, we get
Awz
(rr1)) 1)
L (r-1)
min LpNc(g) 220
min 920
- q.
2
n r(y-1) 1) Awz
2
Au
Awa
Awn Amn ( (22+ (r- 1)(r
Similarly, by replacing
9 with q, we get:
Awz
( 1)
min (SRO(G)
min 920
220
2
9¹ r(r-1) Aw
V2+ (r ()
(r- 1)= IT Aw.
2
+
rq.
2
16

---

Next, by replacing
9 with q. we get:
() (r-1)2
AWr
r(r-1)
min (SRG( 420
-min Q20
(8)
2
94 r(r-1) 1)
AWz
Am I1 Aw;
2
2
(22+ v(r 1)(r (2-2 2) (r 1)2
L
+
2
Sr
After the reparameterization, (DNc(() and LsRa(q) have almost the same form except for the multiplier 1 in the DNC case and
b
V2+ Vrr ()(r 2) (r 11)2
8r
in the SRG case. Therefore, the inequality between LNNG and LSRG is fully determined by the inequality between these two terms. We can write:
(V2+ V(r 1)(r 22) (r 1)2
2
8r
1)(r- r(r - 1) > 26-3 (v2+ - 12) 2))²
We first solve it for L = 4 and any 2 4 (which is guaranteed by K 2 6). We get the inequality r(r - 1)2 > 2(r - 1)(r - 2) + + 4 2(r 1)(r - 2). This inequality is equivalent to (r 1) 2 3r + 4) > 4 + 4, 2(r (r - 2). which holds for all r 4.
Compared to the case L = 4, for general L the LHS gets multiplied by (r 1)2-4 and the RHS gets multiplied by 2/-4, which is smaller for 2 Hence, the inequality holds as well.
Analysis for L= 3. Here, we need tighter upper bound than Esno(g). Thus, we write
Awa
LSRO(g) <
8(r-3) /Aw, An, (V2+ (r)
g3 + r(--)) Awa
r2 111ww
r(r-3)
Awa
2 AvA ANY V2+ (1+ ()-1)(r-2) 93 + r(r-1) Aws
ph (1--1)-Aw
2
3 Aw, (V2+ (rr 1)(r -2)
-)) q.
2
We equivalently re-write this by extending both of the ratios by 2 and then moving 7 to denominator. Thus,
r(r-1) 1)
Awg
CsRalg) s LsRo(q) =
2
2 8(r-3)2 /w,, ds, (VIt (r- ()
92 + s(r-1) Awa
( (nrr1)(222u
(r-3))(r1)
Awa
2(r-2)
+
2 8 A0 (V2t (r- 1)
93 + r(r-1) Awa
((r--) ( 2)Aw2
+ Aw, (2+ /(r - 1)(r -0)-2)) 2) q.
17

---

Now, we perform the same reparametrizations as for L > 4, with the only exception of treating
(r-1)(r 2) in the denominator of the denominator the current ratios as (r- (r=1)² in the previous case. Then, we have
r(r-1)
Awa
min LSRO(1) 920
min 920
r-
2
(r-3) 3)
r(r-1)
AWa
AH1 Aw, Awz
((r-3)(r-1)
AWa
2(r 2]
+
9ºh
r(r-1) 1)
AW
AN, Aw; Away
(V2+ v(r - 1)(r 2)) (r - 1)(r 2)
3 + 2"
q.
Assume that the following inequality holds:
1
(9)
(r-3)2 AN. Aw, Aw,
@3 +
rir
Awa
2
An, Aw, Aw, 92 + r(r-1) AWs
2
Then,
Awa
r(r-1)
min Lsno(1) S min LSRG(1) = min 920
920
920
2
Au Aw, Awa q² r(r-1) Awa
$
(22+ / 1)(r ()(r-2) (r- 1)(r 2)
(r-1)(r-
3 + 2"
q.
Sr
The only difference between the expression considered here and the one considered in the L case is that here we have (r - 1)(r 2) instead of (r 1)² within the expression. By comparing against
Love again, we get that LSRG LDNG if and only if (f)= (2 + v () 2) (r-
2). This equivalent to (r - 1) 4) > 2(r - 2) + 2(r 2) 2( 1)(r - 2), which holds for all 2 4.
remains to show that ( holds, and it suffices to do so for the minimizer q" of LDNC, as minz20 Lsna (q) < (sna(g") S Lsna(a") LDNC(I") = mingzu LDNc(q). Note that this is equivalent to
r(r 1) Aw3 Xw, >w,
(r- 3)- (q")
Note that the minimum of the function in ? (having the same reparametrizatioon as ESRG) - if not 9- O, in which case the statement of the theorem is trivial must come after the unique inflection point of the function. A direct computation yields that this inflection point satisfies
r(r
Awa
Xw, AW, An
Therefore, the minimum of 7 is attained at q" for which
r(r 1)
2 Awa <
Xw, Aw, An. (q*)3
For r 2 5, this implies that such satisfies which concludes the argument for K
-()
For a general K, the extension is rather simple. Note that the first type of SRG solution in Definition 11 is constructed in way so that the losses attained by the SRG and DNC parts sum up. Therefore, we can split the analysis for the SRG and DNC parts. The DNC part obviously attains equal loss to the DNC solution. For the SRG part, the analysis done above applies, and the argument is complete.
18

---

It remains to show the statement on the asymptotic relationship between LSRG and LDNG for K - oo when with an extra index corresponding to the order within the sequence. However, with an abuse of notation, we drop this indexing and switch to the O, notations whenever convenient.
Formally, we should consider sequences of the problems and label everything
(2 for some r. Let A AHH, I Aw, and As before, we start by considering K of the form
V (K) = 2 V2+ (1+v(r-1)(--2) t
where r corresponds to the value s.t. (2) = K. We note
1(r-1)=-2
that (KK O(K Since
interested in the ratio Loxe: we do few changes and reparametrizatioos to the expressions in 7 and (8): multiply both by 2, divide all terms in the left summands by Aw, rewrite -(r=1) as K, plug in the defined quantities, divide the terms in the
left summands by K and finally replace A t K tg with to obtain the following expression
1
LK
LK + Atq
(10)
9t +
for the DNC loss, and the following expression
1
LK A AZ (K)g
(11)
+1+1
for the SRG loss. Using a similar trick as in the previous analysis for I, = 3, we have that the minimum of the function in (IOp is achieved when > 1. Hence, we can lower bound (10) by
1
LK T Atg.
39
For this convex expression, we can find the optimal solution by setting to zero the derivative, which
gives that the optimal solution is (1 + L)3 cii KAZH Similarly,
upper bound [11 by
-- I. + LK (L)(K)g
and after finding the optimal solution we get that it equals (1 + ())) This allows us to conclude that
3-L LSRG = O(K 2(++1)).
LDNG
To get the same formula when the number of classes is not of the form (2) we only need simple adjustments. For this part, we will employ the upper-index notation to denote the number of classes K to which the solution corresponds. First, note that the optimal value of (10) is continuous in the coefficient in front of the linear term Therefore, if CK DNN < 0.499, then, choosing the smallest r for which K < (2) = K, we see that CB < 0.5 for the same set of regularization parameters,
as R K0000 1. Since the argument above does not need LDNC < 0.499 but only LDNG 0.5. 3-L ボ[ト+
). Finally, choosing the second construction in Definition T1 we construct the SRG solution for K classes from the SRG solution for K classes with the same regularization parameters (thus also the same regularization as for the DNC solution with classes). To conclude, it just suffices to see that C > Rg because we removed columns from H1, decreasing its norm and the fit loss is at most as big because the columns with the worst loss were removed and the loss an average over the columns. This concludes the proof also for general K.
that _ with the same regularization parameters is still O(K
We conclude the section by stating and proving few auxiliary lemmas that were used in the proof of Theorem 5 Lemma 12. Consider the following optimization problem:
min ||w|
(12)
w
s.t. T = n0 A,Tr.
(13)
Then, the value of the optimal solution is (r 1)2
3r
TT.
4(r - 1)2 1, T,z.
an(r - 2)²
19

---

Proof. Multiplying the constraint with T, (T,I,)) from the right we get 2 T (T,T,- w" Ar. Now, we can use that the minimum l norm solution of such system can be computed by multiplying with the right pseudoinverse of A,. Thus, we get w A1 (T) (T, (T,, TT Then the squared norm of this is simply:
1 (T,T)) AT A(T,,T))) 'T,= _ T TT ) TT
T w
Now, we know that
T.T -
11 r-2
1+ 1+,-2 r-2 11 (+--211²)
1
This can be seen by looking at the structure of K, where two vertices have exactly one edge between them. Now we can compute the the inverse of this matrix using the Sherman-Morrison formula
T+
1
11
=1
1 11
2(r 1)
and the square is:
-2
3r 4 11
I 2
11
= I
4(r 1)2
Putting this all together, the proof is complete.
Lemma 13. Let 2 S S - Consider Mi41, Mr as Definition of the SRG solution. Then, the following optimization problem:
W [|
(14)
8.1. M++1 = WM
(15)
achieves optimal value of ra141 oa
Proof. This is a direct consequence of Lemma 12
Denote y20 a row from M4+1 which corresponds
to the first row of T, and such that ||=o|| = Then, Ty-220 = 7(1, 1/(r - 1)...., 1/(r 1))".
Directly evaluating the expression in Lemma 12 will yield ?²aj for single row and thus for all rows we get the value from the lemma statement.
Lemma 14. Consider Mr, ML-1 as in Definition of the SRG solution. Then, the following optimization problem:
min ||
(16)
s.t. Mr = WM
(17)
achieves optimal value of
(2 1)2
01
aL-1 4((r - 2T - 3) + 2)
Proof. We first need to characterize the rows of Mr.- Note that M, comes from the multiplication of T, and ()) see Definition This operation can be seen weighting vertices of K,- (rows of T,) and then looking at what the sum of the weights of adjacent vertices of each edge (column of T,)) is.
We can easily that the resulting vector has exactly one "-1" entry (for the edge corresponding to the two vertices given negative weight) and exactly
("=2) entries with "+1". Therefore, it must be scaled with the inverse of s (r (rr-))r-3) 1 to be unit norm. Now, let ø be one of these vectors with the negative edge between first two vertices. Then,
2
T, = (-
3 T T-3 T 8Vr
SVT - .....
sVr
20

---

which can be easily derived if we imagine doing edge-wise dot dot-product between two vertex weightings, one with two "-1s" for 2 and the other type with one "+1" representing the rows of T, For the other vectors in Hr, the resulting vectors would be similar, except they would have the negative entries for different pairs of vertices of K,. Note that
(2 - 1) + (r 3) - 2)
T TT,TT,2= 2 TT.
(r - 1)
प.. 1, ((r 3) - 2) - 2) (2 (1--) 1)
Therefore, if we are optimizing for y=1, then an application of Lemma 12 gives
(r ( 1)²
2
(r 3)
37 - 4 ((r 3) - 2) 2)
w - (r 2)
(r
(r 4(r 1)²
(r
where w denotes a row of W. Simplifying this expression, we get
r(r - w² w = - 2((r - 2) - -)
To conclude, it suffices to sum up r(r-1) such rows having total l2 norm squared ag (and, hence, y² = aL), which gives
(r - 1) 2) 2) 3) 2 aL-1 4
WIF = at
and concludes the proof.
Lemma 15. For 2 < SISL-1 1, consider Mi, M, as in Desinition 9 of the SRG solution. Then,
Mi Mj and the eigenvalues of Mf Mi are:
μ1 2ag with multiplicity
H2 = r - 2
with multiplicity r- 1,
rrr-3) #3 0 with multiplicity
2
Proof. From the definition, it readily follows that Mj = Mr, so let us compute M² M₁. Looking at any row of Mr, we see that it has non-negative equal entries of value /any/ /r I, where E, 0ij = an on all the r r- 1 edges of K,, that contain the vertex corresponding to that row. Therefore, by definition of αq, the sum of squares of all entries corresponding to one row type within any column is az/ (r (r-1r) 1). Each edge (and, thus, column) contains exactly two vertices, thus the diagonal elements of M2 M, which are the la norms squared of the columns of Mr, are simply equal to 201. There are two possible off-diagonal values. One is for the pairs of columns that correspond to edges that share a vertex and one for those pairs that do not share vertex. The pairs of columns whose edges do not share a vertex do not have any entries which would both be jointly positive, because the other. Therefore, the value of off-diagonal entries corresponding to such pairs is simply 0. On the other hand, there is exactly one vertex that has the
r-
either vertex does not belong to one edge
non-zero values for both edges corresponding to columns whose edges do share a vertex shared vertex. Therefore the value of off- off-diagonal entries of this type is _ Crucially, the structure of the off-diagonal entries is fully determined by the graph T,. because two edges in Kr+ share a vertex if and only if they are connected in the graph T,. Therefore, M/" Mt can be written as a weighted sum of I and the adjacency matrix G, of T,, where the weight of [x simply corresponds to the size of the diagonal term and the weight of G, the positive off diagonal term. In conclusion, we get
201
Mf M:
7K +
r- Gr-
As T, is strongly regular graph with parameters (r(r - 1)/2, 2(r - 2). r 2,.). G, has a single eigenvalue equal to 2(r 2). r eigenvalues equal to 4 and r(r - 3)/2 eigenvalues equal to -2, which concludes the proof.
D
21

---

Lemma 16. Consider ML as in Definition 9 of the SRG solution. Then, the eigenvalues of M ML are:
(r - -5r 19)az
with multiplicity 1,
(r - 2) r- - 3) +
2(r -3)
P22 (r 2) 3) +
with multiplicity 1,
201
r(r - 3) 2
A3 (r - -2)((r- -3) -3)+2 +2
with multiplicity
Proof. Let us compute M Mr. Looking at any row of M,, we see that it has non-negane non-negative equal (r-2)(r-3) 1 and E, Oij ar all edges in subgraph of Kr of size - 2. Therefore, by definition of az, the of squares of all entries corresponding to one row type within any column is as/s². However, not all row types have non-zero value on any particular column. Namely, a row type will only have non-zero value on a column, if the row-type corresponds to such subgraph of Kr, which is disjoint with the edge corresponding to the column. This is because all edges outside the complete subgraph of r vertices corresponding to the row type are assigned O in ML. Therefore, the number of row types that assign non-zero value in a particular column is equal to the number of 2 vertex sets. This corresponds the number of edges in KG,, which is equal to (" s² Thus, the diagonal elements of MI Mr. which are
entries of value varj/s, where
the l2 norms squared of the columns of Mr. are simply equal to (s2-1)=, There are two possible off-diagonal values. One is for the pairs of columns that correspond to edges that share a vertex, and one is for those pairs that don't share vertex. Let us compute the number of row types assigning positive value to both of these columns jointly. Using the same interpretation, the columns correspond to edges and only row types that correspond to r 2 vertex subsets disjoint with them assign positive value to the column of that edge. If we want this to be satisfied for both rows jointly, we need to take the intersection of those r vertex subsets, which in this case will result an r - 3 vertex subset. Thus, exactly ("²") row types will jointly assign positive value. Therefore, we have value (r-3)(-4)ar on these off- off-diagonal entries. For the pairs of columns that correspond to edges with 23t disjoint vertices, the same intersection will now yield set of vertices of size only 4. Therefore, the value of this off-diagonal entry is (r-4)(r5)aa, Crucially, the structure of the off-diagonal entries 2n² is fully determined by the graph Tr. because two edges in Krp share vertex if and only if they are connected in the graph T,. Therefore, M.. Mr, can be written as a weighted sum of 1 f I and the adjacency matrix G, of T,. The weights can be determined as follows: we first subtract multiple of 1 x 1/ to make the smaller off fif-diagonal entry of M My zero, then we subtract what is left of the diagonal and we take the rest to be multiple of G,.. In conclusion, we get
M (r () 5)ar, 1
2s²
+ (s² -1)a
(r-4)(r-
(0(r-5)Cn. Tx
5²
25²
(r 3)(r -4)a 4 -5)oz) Gr-
+
2s²
2s²
As Tr is a strongly regular graph with parameters (r(r- - 1)/2, 2(r - 2), r- 2, 2),, G, has a single eigenvalue equal to 2(r 2). eigenvalues equal to and r(r 3)/2 eigenvalues equal to
-2. The summation with T only shifts all the eigenvalues. The term 1 1/ has only one non-zero eigenvalue, and the eigenvector is identical that of the eigenvector corresponding the dominant cigenvalue of G,. This concludes the proof.
Lemma 17. Assuming DNCI, let Mt be the mean matrix of the last layer. Let M2 = UEVT be the full SVD of Mr and let o;, i E [K], be the singular values of ML. Then, the following optimization problem:
1 Wr. 2K
Awz W| 12
min
1x 1x
2
22

---

that achieves smaller loss for this problem. Let us duplicate all the samples of f7(0) for n times, thus constructing मे H s 17, The solution (fi; Wi,..., Wi) has the same loss under the L-DUFM problem with regulariza- tion parameters (AH,, \w, ., Awz) as the solution (1( Wi..... Wi: for the L-DUFM with n = and parameters (nAH, Awar..., ..v....,AW... Awz). This is easy to see from the separability of both attains the minimum of
K
AW
KAWz
Proof. The proof consists in a direct computation. Let Wi, denote the minimizer. Computing the gradient and setting it to 0 gives that
W (M + Aw, KTd,)
VEETET" Xw, KTa,) '
which readily implies that
|
Σ
of
Aw, K)2
Ix12
VE
(ΣET
+ +AW, KTdi)
(lai)-IEV' - -VV
K
?
2
K
K2 KW>W
+ Aw K -1)
0,
(, + Aw
i=1
thus concluding the argument.
Lemma 18. The optimization problem
AA
+ AB |B
min A A,B,C=AB
|lll
(18)
2
attains the minimum of / \A |C and the minimizers are of the form A = /2 B BB RTA/2VT. Here, the constants "IA, "B only depend on XA, AB; UEVT is the SVD of C; and R an orthogonal matrix.
Proof. See Lemma C.1 of 51
A.2 No within-class variability is still optimal
Theorem 6. The optimal solutions of the L-DUFM ( exhibit DNCI at layer L, i.e.,
Hi Mi s 1
holds for any optimal solution (H1, W ..... Wi) of the L-DUFM problem.
Proof. Step 1: Reduction to = 1. In the first step. assume by contradiction that there ex- ists an optimal solution of (T) with regularization parameters (AH, AW,..... Awz) denoted as (Hi, Wi...., Wi) which does not exhibit deep neural collapse at layer L. This means that there exist indices i, s.t. No + h2. Let us construct two solutions of the = 1 L-DUFM. They will share the weight matrices which will equal (Wi..... W2) the weight matrices of the original solution. To construct the features, for every class except the e-th, pick any sample and share between both solutions. For the class take the samples h hé and put one in one solution and
the other one in the other solution. Denote ( (2 the two = sample matrices. It is not hard see that both (H Wi..... W2) and (Hz Wi...., Wi) are optimal solutions of (I) with regularization parameters (nAH, Aw..... ..... ). To prove it, assume by contradiction that, without loss of generality, (H W Wi) is not an optimal solution of the corresponding problem. Then, there exists an alternative (H))0)
Hj 2 and the fit part of the loss in w.r.t. the columns of H1. For the same reasons, the loss
23

---

functions for (Hi, Wi....,W2 in the original problem equals the loss function of the solutions (HJ") w....., W2) or (H .......... W in the reduced problem. In fact, this was not the case, there would need to be an inequality between the losses exhibited by two different columns of Hi belonging to the same class, from which we could arrive at a contradiction by taking the better column and multiplying it to all columns within that class, thereby obtaining better solution. This means that the loss of (Hi Wf.... ..... Wi) in the original problem is smaller than the loss of (H₁, .,..., W?),, which is a contradiction.
Step 2: Excluding an aligned case. By assumption we know that not only H(1) and H differ in the e-th column (from now on we assume without loss of generality that it the first column) but also ( and H do. Denote for simplicity the first (differing) columns of H and H[" as r, y respectively. We now show that is not possible that y az. First, a has be non non-negative since I, y are entry-wise non-negative given that they come after the application of a. Assume w.l.o.g. > (otherwise, we can just exchange the roles of x and y). Consider a reduced problem where we only optimize for the size of the first column of either H HL or ((2), focusing on that part of the problem (I) which is relevant for this column, being:
min 1 W 120
L(0 ei || ndH,
1(8) 11
2N
2
This problem is strongly convex, quadratic and simple enough to give the following: a > 1, then 11 1(2) and simultaneously | - 112 |Wy c1l12. This means that H(() is
a strictly better solution than H( contradiction.
Step 3: Contradiction by zero gradient condition. By optimality of both solutions we get
Oc
OC
= 0 = dW, (B..W......))((ff" (2) . WW. W......WEE)
aWr L (H..W....W..):(4f) "i............ WW)
An application of the chain rule gives
dc dCF Oll++1 Aw2 W.
OLF
HE + Aw,
aw,
OH1+1
OWL
0HL+1
where H1+1 is the output of our model. Plugging this back to the previous equation and using that W7, is the same in both expressions, get
0CF
(H())=
OLF
(H
0H (Hi, W: (2)=(H(( Wi.....WW
0HL+1
(Hs, WT.... (2)=(H(²) w W
Let us denote
A 0LF
OCF
B
HH (H, (W.:,,...W)):(HH" (W",,... WT).
H "..........)((FF). (W; .......
Due to the separability of Cp with respect to the columns of H₁ (and, thus, also of H1, H,, for all I < + 1), we get that the matrices A, B can only differ in their first columns and are identical otherwise. We denote these columns b for A, B, respectively. This implies that ar by"
Now we exclude a few cases. First, neither nor can be zero, because by the exact formula that exists for them, this would mean that exact fit was achieved for either of the columns. This is impossible with non-zero weight-decay, because decreasing the norm of the column of (1()) or 11 that achieves the exact fit by a sufficiently small value would necessarily lead to an improvement on the objective value. Moreover, if x y 0, then this is contradiction with the assumption that x # y. Finally, the case x + = or = x # is excluded already in the step 2.
Thus we get y, a. are all non-zero. Looking at any fixed row (column) of az? and by" we see that necessarily T. y (a, b) are aligned. However, this case already solved in step 2 and leads to y, which is the contradiction. This concludes the proof.
D
24

---

Theorem 8. Denote by L-DUFM, the equivalent of ① with a replaced by Let D max d2, dy..... ".......d,, and x= AB, Xw, ... AW with the regularization parameters upper bounded by 1)) Then, for any globally optimal solution of the I--DUFMe problem, the distance between any two feature vectors of the same class in any layer is at most
Ge D(L+) (L 1)4-1A4
(5)
Proof. In order not to mix approximation technical details with the gist of the proof, we split the argument into two parts: in the first part, provide a heuristic for differentiable at O; in the second part, we discuss what changes in the proof if we use the relaxation and then execute technical computation that bounds the error based on the strength of the approximation (the size of c). useful to split the loss function & into the two terms LE and La- The former represents the fit part of the loss (which penalizes for deviation of the predictions from Y), and the latter represents the regularization part of the loss.
by assuming that ReLU is
As in Step / of that argument, if we have globally optimal solution that does not exhibit DNCI in the first layer, we can construct two different solutions of the in the first column of the H1 matrix. Let us denote two such constructions H * HHY. where we denote * y, be the mentioned first columns, respectively. We emphasize that H H both form optimal solutions of the n = L-DUFM with the same tuple of weight matrices (WT,- WT). In particular, this means that
Part 1: Heuristic for O. We start identically to the proof of Theorem 6
L-DUFM problem where the two solutions only differ
ac
= Oc
aw
OW
(H,,W (H)
WHI.W....... (W;
An application of the chain rule gives
ac dCr 0H2 0H, ow1
Hł + Aw; W1. OLr
Aw; Wi
aW1
aHl
Plugging this back to the previous equation and using that the WT is the same in both expressions, we get
OLF
(HT)²= OLF
(HT)T.
0Hl (M. W2) (H)=(HFWW;; (.........W.
Oll,
(H1.Wi.... W2 W2) .W),,)))
Let us denote
= 0LF
OCF
B
HH,
AH₂ (H: (..,W.......)((HH. (H......W...
Due to the separability of L with respect to the columns of H1 (and, thus, also Hi for all | S L), we get that the matrices A, B can only differ in their first columns and are identical otherwise. We denote these columns a, b for A, B, respectively. This implies that az" = by"".
Now, we treat few cases. First, assume b 0. Since holds that (Hf Wi..... Wt) is the optimal solution, then necessarily
ac 0 = the
(WW))a a XH + Z.
(H. W2) (M; ..W.....WW. w.....))
Similarly, we get y = O, but that is a contradiction with æ * y. Therefore, at least one of a, is non-zero. Next assume the other hand, b + O. then the row of by that corresponds to a non-zero entry of must be zero we can get 0.
0. Then ax 0. If = 0 then y 0 and we have contradiction. On
and thus y = 0. Similarly, if
Let us therefore assume x, y are both non-zero, which also implies a, b are both non-zero. Looking at any fixed row (column) of ax? and by" we see that necessarily ær, (a, b) are aligned. Let write r ay and on = b for some a * O. We will first show that if > 0 then necessarily a = 1 and y. For this, Ict us fix any ray r E Rd, The ray represents set of possible first columns in
25

---

H1. Let us fix any optimization over 0 minimize L on (3r, Wi.... Wr.). However, CF is convex by assumption and the mapping h + hi is ray-linear, therefore Lr is convex in B. Moreover LR is strongly convex in h and therefore is strongly convex in B3. This means has a unique optimal solution 3".
WL). Since L is separable in the columns of H1, we can consider an
We have just showed that, if > O, then since æ, y are aligned and thus lie on the same ray and are both optimal together with the same tuple of weight matrices, they must necessarily be identical and so a 1, y. This a contradiction and thus we are left with the case a < O. Denote s = Wi a. t = w. From linearity, we have = at. Note that if s = t = O, then necessarily x y 0 because they don't have any effect on LF and they minimize L at O. Thus, s, are non-zero and negative entries of are positive entries of t and vere-versa. This means that (ss) and o have different sets of positive entries. However, from the optimality of the both solutions we know:
Oc
ac
= =
2 (Hi
(H) W: Ww)=(HH.W .W....W) OW2
WT........ ....... WL)=(HH WT ..... Wi)
Again, using the chain rule we get:
ac OCF OH,
+ Awa W₂ = OCF HT. + Aw;
OW2 HH, oW,
0H₃
Plugging this back to the previous equation and using that W2 is the same in both expressions, we get 0Cp
dCp
(H²)
(H²)T.
aHs
HH, (H1 W (H)= W2)
WE)
Let us denote
C OLF
D 0LF
dHa (H,, H...W..... (H)....)) W.. Due to the separability of DF with respect to the columns of H, (and, thus, also H, for all I S L), we get that the matrices C, D can only differ in their first columns and are identical otherwise. We denote these columns e, for C, D respectively. This implies that car(s) = (I)"".
0il3 (H,
As above, if either or d is zero, using the chain rule we would get that r or y (respectively) are zero too, which cannot happen. Therefore, both e and are non-zero and (s), ot) must be aligned. Since they are non-zero, non-negative and with different supports, we have reached a contradiction. This proves that a < 0 also impossible and the only possible case is a 1, but that forces y, which is also contradiction.
Part 2: Relaxation to ReLU. The difficulty in making the previous heuristic rigorous is that ReLU is not differentiable at O and, thus, we do not have the desired analytical statement that global solutions must necessarily admit zero derivative (because the loss might not be differentiable at them at all). If we tried to use the same proof as in part I with the differentiable relaxed version of ReLU - ReLU., an issue would occur when showing that, x, y are aligned and a 2 0. then a must be 1. The reason is that the mapping h1 + hi is no longer ray-linear and the corresponding optimization problem on any fixed ray is no longer quadratic and strongly convex. However, we can prove that the optimization problem admits a solution that is close to the solution of the corresponding ReLU optimization problem. For this, let us fix any direction r such that ||||| = and an optimization parameter t 2 0, and define the following two losses:
nAH 12
[o(l,r)= 2K [[ W20 11
(19)
(t(rr)) 2K ||Wi,o0 (W7 (W ...)) 11122
ndH 2
(20)
2
We now bound max [Lo(t, r) - L. L. r.). For this, we fix any 2 0. and bound the accumulated l2 120
error of using o. instead of a throughout the layers. For this, a useful statement that we will need is the following:
Awa |I = AWz
W
2
Aw
ndH |
| < 2(L+1)
2
2
2
2
26

---

This is true because, by simple computation, we get that the terms must be balanced and the inequality comes from the fact that the solution (0, 0, .., 0) achieves full loss 1/2 in the L-DUFM as well as L-DUFM, problems and, thus, Ln is trivially upper-bounded by this. This implies that, for each Wi (and H1): [Wll S [Will S
V(L+I)AW
For case of exposition, we set = (the same argument would work for all t). We see that Wir does not introduce any l2 error. Then, the le error that is introduced in the application of the ReLU is trivially upper-bounded by ev After applying W2, we can use the bound on the operator norm ||W]|| obtained above to upper bound the propagated l2 error by
d2.
VL+ + 1)>W,
After that, we obtain an additive la error of Ev dz by applying o. which gives a total l2 error of
1
V(L+ 11++1)AM cv d2 + t
Then, again, we multiply this whole expression with W3. Inductively, the upper bound on the l2 error in the output space is
((++)AAW to account for the multiplication by
dd
L
Vww,
j=l
This can be further upper bounded as
EV D V AH, AH,AA,.
11+1) VX
Using the triangle inequality, the upper bound on (l. (t, ] is this expression squared. On the other hand, the second derivative of Lo(t, r) with respect is lower bounded by nAH,-
Given two functions fi and f2, with f1 strongly convex with second derivative at least c and f2 everywhere most d distant from fi, then the distance between their global minimizers is at most //d/c. Applying this to our case, we get that the distance between the minimizers to and t, of r)(,,)) and Le(1, (1, r) is at most
2EVD(LII (T. + 1) VAn
Since the ray r is unit-norm, this is also the upper bound on the distance between two feature vectors of any globally optimal solution in the first layer. To obtain the upper bound the distance between two vectors in any layer, we proceed as follows.
Assume we have two input vectors of the first class (now we know they need to be aligned): x',y' = a'z', where a > As before, at, x', ...y' are the 1-th layer representations of these vectors before and after ae- If we would compute JW, with respect any layer > 2 and used the same arguments as in part 1 (that still carry the analysis for σz), we would find out that in each layer, we need to have y' = a'x'.. Moreover, we can assume that 8 ,²,x", ,',0", y/ are non-zero at all layers because otherwise the same argument in part would trivialize the rest of the proof. By a simple inductive argument, since ai > 1, we know that 0:2 > 1 as well because a, is strictly increasing on [0, oo). Similarly, on 1 for all I. Note that, if there exists at least one index on which 2 is bigger equal than e, then necessarily a' = a because or is the identity on inputs of least c, and the aj can be uniquely determined from v//=! 1/-1//a /at- Having a' al-1 makes us have more control over the distance between x' and y¹ If this fails to hold at some layer, we need separate analysis.
8C
For this, let lo denote the first layer (assuming it exists), where z does not have any entries that are bigger or equal than c. This means that ||a.5 || S < dig-- We now compute the maximal possible 01--1 for all 2 < I << lo and for each index i, either x i < or < 7/ j/ Otherwise, since (x( Vr e (0,e), w /// ag, contradiction. Thus, get
norm of y/ Since 01 02
27

---

Js' - 3'Il 2 nz 1l Therefore, ||اl - s' can only grow by applying W, Thus, after applying W1. W...... 1 and using the operator norm bound with all of the matrices, we get
||a = 2E D)
(I+ & Van (L 1) ' Aw, Aw, XW
Combining with | S dio and using the triangle inequality, we get
| - 2e 1)
/ddl diu
(1+1) Vnn (I. in Vww, Aw, .... -Wi
Applying the operator norm bound with Wio Wiot1...., WL-1, we get:
|| 2 D(L+())
(I+1) Vxn (L (I+1) Vxw, Aws ... AW
+ tv
(L+ = k-1a
v Awio AWig ... AWL-1
3 D(L+1)
(L+ 1)++))+
Since ||~4|| S ||
we can use the triangle incquality again to obtain the final:
1e"-~"ll 6c [DD(++1)
(I+ 11+++t/V/
Note that the same bound is upper bound for any layer, since in each layer we increasing the upper bound by least a factor (L + 1)-1/2 x//////
In case such a layer lo does not exist, the upper bound above is still valid, since we can simply use the error propagation through all the layers, as did for 1 < in the previous case. We would arrive at the same bound as above but with 4 instead of 6 as a multiplicative factor (because we would not introduce the additive error cV d.g))
B Further experimental evidence
B.1 L-DUFM experiments
Experiments on small number of classes and layers. For L 3, we weren't able to find any solutions that would outperform DNC for K < 6. For K = 7, the difference in the losses is extremely small. For K = 6, we did find some low rank solutions, but these were either slightly worse than DNC the losses were the same up four decimal places. Already for L 4, we find low-rank solutions for all K 2 3. We present some of them for E {3, 4, 5} in Figure S Note, that all these solutions were found automatically by gradient descent through the optimization process.
We specifically highlight the solution for = 4, as it represents another graph structure different from SRG. Namely, this solution is based on the square graph. We see that each column (edge) has positive scalar product with exactly two other columns (the corresponding edges share a vertex), and it is orthogonal to one other column (the non-touching sides of square). Each row (vertex) has only two positive values, and these correspond to the two sides (edges) to which they belong.
Experiments on large number of classes or layers. For large L or K, the behavior of the low-rank solutions is slightly different from the behavior when both and K are moderate. When is close to J., it no longer holds that M, Mr, and the rank of M, is larger as the singular values do not decay sharply to O. However, we note that the singular values are rather small and, thus, it is not clear whether they would disappear with a longer training (we trained for 200000 steps of full gradient descent with learning rate 0.5). In Figure we take L 7, K 15, and weight decay 0.0025 in all layers. We highlight that Me + Mg. and the singular values of Mg are small but non-zero after the dominant values. We also highlight that the rank of previous layers is 3, which is remarkably small (and also smaller than the rank of the SRG solution).
28

---

80
bal Iuss
0.0
25
25
00
DMC
5.0
00g00 0050
-0.3
7.
0.
-1.0
100
0.15
125
15
-1.3
15.0
010
-2.0
0.05
17.5
-44
-03
2.0
0.00
2505
losses
bral Itos
0.0
0.35
0.0
0.90
025
5.0
05
100 1950
7.
7.5
0.15
10.0
-15-
12.5
0.10
12.5
25
0.10
-2.0
17.5
-03
30
0.b
Log losses
0.
0.4
0.0
total
0.4
DAC
5.0
5.0
04
02
75
7.
1101
02
15.0
15.0
0.6
2.6
0.0
-066
200
400
4tep
Figure 5: 4-DUFM training for K (top), K = (middle), and (bottom). Left: Loss progression, also decomposed into the fit and regularization terms. Middle left: Visualization of the matrix My. Middle right: Visualization of the matrix M4. Right: Visualization of the matrix Mf M3.
SRG solutions. In Figure presented Pigure[preseetee in the main body, we recover the SRG solution for K = 10. In Figures 7 and we show that solutions very similar SRG are recovered for K and K 15. respectively. The only difference with SRG is in the construction of Mr. We note that the losses of these solutions are slightly lower than the loss of our construction, which proves that the SRG solution itself is not necessarily globally optimal.
B.2 End-to-end experiments with DUFM-like regularization
We complement the experiments of Figure 3 with two extra ablation studies for ResNet20 trained CIFAR-10 with layer MIP head. We focus the dependence of the average rank the weight decay and the learning rate, and present the results in Figure The weight decay has clear effect on the rank of the solutions found by gradient descent, similarly the results in Figure forr the L-DUFM model. The effect of the learning rate is slightly less clear, but we still see a general downward trend.
B.3 End-to-end experiments with standard regularization
Finally, in Figure 10 we include the analysis the average rank as a function weight decay in the standard regularization setting for training the MNIST dataset. The results confirm the trend from the previous experimental settings, showing that the weight decay strength is a crucial predictor of the final rank even in standard regularization setting, which has different loss landscape compared to the L-DUFM.
29

---

Singular values of
0.12
1.5
0.7
10
0.10
0.6
0.08
20
0.5
20
0.8
30
0.6
0.04
0.-
-0.5
0.4
0.1
40
0.02
40
02
-1.0
0.0
0.00
0.0
10
14
10
Index
Singular valuees of H_B
Singular values of sigme H_6
6-
4-
2-
0-
2
10
12
14
8
10
14
index
Figure 6: Class-mean matrices and singular values at convergence for a DUFM model with K = 15
and L 7. Top row: Singular values of M2, and visualization of the matrices M2. Mc. Mc and Bottom row: Singular values of Ms and Ms.
0.30
0-
0.30
0.25
5
0.25
0.2
2-
15
4
0.10
-0.2
0.05
0.05
5-
0
Singulet valuues of
Log losses
1.0
0.0
total less
0.8
0.5
0.6
1og10 less
0.4
-2.0
indlex
step
Figure 7: 4-DUFM training for K 6. Top row: Visualization of the matrices M3 M3, Ma, and M, Bottom row: Singular values of Hg, and loss progression including its decomposition into fit and regularization terms.
30

---

0.4
0.35
03
10
0.30
10
0.2
20
0.1
0.20
0.0
30
-0.1
0.10
0.2
40
0.05
-0.3
14
0.4
0.0
2.5
5.0
7.5
10.0
12.5
10
10
Log losses
total_lose loss
fit loss
-0.5
theoretical _pptamummmmmmmmmmmmmm
-1.0
10910
-2.0
step
Figure 8: 4-DUFM training for 15. Top row: Visualization of the matrices Mg Ma, Ma and M4. Bottom row: Loss progression including its decomposition into and regularization terms.
20
9
overage rank
average rank
15
10
5
-26
-24
-22
-20
-18
-10
-6
log2
weight deeayy
log2 learning rate
Figure 9: Average ranks as function of log, weight decay (left) and log, learning rate (right). We trained ResNet20 with 4-layer MLP head on CIFAR-10. The experiments are averaged over three and two independent runs, respectively.
31

---

10
average rank
8
7
5-
-11
-10
-9
-8
log2 weight decay
Figure 10: Average rank as function of the log2 weight decay. We trained ResNet20 with 5-yer MLP head on MNIST. The experiments averaged over 4 independent runs.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

The paper provides the full set of assumptions and complete proofs in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The paper includes enough details so as to perform the same kind experi- ments as the ones presented in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
34

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The contribution of our paper is mostly theoretical. The experiments are for illustrative purposes. The source code will be provided on request.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic/guudees/Cdeeeuuii.. www.eeeeuuuuisssoooooooo.ooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppplcccuui.aeaeeecccuea
Stteee///ooSSSeeissSioica www.ssssiicggcoooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [No]
Justification: We do provide most of the experimental details in the manuscript. Additional (minor) parameter choices can be derived from the source code, which is available on request.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The paper reports error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
35

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification:
The experiments do not require any specific hardware setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttlp::///euuiiii www./puur..../. ctcppp..ccpppppe cccppbllic/thhicccuueeeee....e.e
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: no socictal impact of the work performed
Guidelines:
The answer NA means that there is no societal impact of the work performed.
36

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We provide references for the datasets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
37

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
38

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "bgm6PG3WgP", "reviewer": "Reviewer_kT8c", "summary": "Papyan et. al that showed that at the terminal phase of training there are four phenomena called neural collapse on the last layer of a deep neural network architecture. The authors extended Papyan's work by considering the earlier layers of nonlinear deep neural networks. They proved the first neural collapse condition (convergence to class means) is optimal for DNNs (Theorem 6 in manuscript). They also proved the second neural collapse condition (within-class variability) does not hold if number of layers and classes are high enough by showing there exists a scenario with better performance (Theorem 5). The claims are also supported by the experiments.", "questions": "-", "limitations": "The authors adequately addressed the limitations.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Authors extended our understanding of how deep neural networks work by analyzing the neural collapse for deep neural networks.", "weaknesses": "The results only hold under certain assumptions such as using multi-layer regularization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. If any questions come up during the discussion phase, we’ll be happy to address them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zpZzgH0QSL", "reviewer": "Reviewer_dZSf", "summary": "This paper theoretically explores the deep neural collapse (DNC) phenomenon within non-linear deep models for multi-class classification. Neural collapse (NC) is a phenomenon in deep overparameterized networks where, the last layer's feature vectors align with class means and their corresponding classifier vectors, while also maximizing separation between classes. Previous works have studied the propagation of this symmetric geometry to earlier layers and proved its optimality which is termed as DNC. These works focus on DNC's optimality in multi-class linear networks or binary non-linear networks. This paper, however, extends the analysis to multi-class non-linear deep networks with ReLU activation, using the deep unconstrained features model (DUFM), where input features are treated as unconstrained parameters in the training objective.\n\nWhile in the binary or linear case, DNC was shown to be optimal across all layers of DUFM, the authors show that in the multi-class case and in the presence of non-linearity, DNC is not always optimal. They prove this by explicitly constructing a solution, called SRG, that achieves a lower objective value than DNC. They show SRG is rank-deficit and the rank/loss gap between SRG and DNC increases as the network becomes deeper or the number of classes increases. They further numerically show on DUFM that, although DNC is not optimal, wider networks have an implicit bias toward finding the DNC solution. To further support their findings, they conduct additional experiments on benchmark datasets MNIST and CIFAR10 and ResNet architecture.", "questions": "1. As discussed in the paper, the last-layer feature vectors before the activation ($\\tilde{M}_L$), have a lower rank, and after activation ($M_L$) recover the full rank structure of NC. This is also clear from Fig. 2 (right). However, what I find missing is whether after activation $M_L$ follows the NC geometry or not, i.e., $M_L \\propto I_K$? Or does it follow a different rank $K$ structure?\n\n2. Line 263 mentions that only for a few runs the training solutions match SRG. In the rest of the runs, do the solutions achieve a lower objective value than SRG (thus proving that SRG is a local optimal at best)?\n\n3. What about $W_\\ell$'s? Do they also exhibit a similar low-rank structure in the SRG solution as well as the numerical experiments? \n\n    I also don't see why DNC3 is not well-defined in this setup (line 250). Still, at optimality, we might have the mean feature vectors $M_\\ell$ and classifiers $W_\\ell$ align in a low-rank structure. This alignment property doesn't hold in either the SRG or SGD solutions in the experiments?\n\n4. Is there a small typo in Fig. 4 caption? The bottom right heatmap cannot be $M_5$ since the features after ReLU are non-negative and the heatmap cannot have negative values.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper closes a gap in the theoretical analysis of the NC literature. The result is interesting as it challenges the optimality of DNC established in previous works and the accuracy of the abstract UFM for modeling the dynamics of deep overparameterized models.", "weaknesses": "The proof of non-optimality of DNC in the paper relies on constructing a solution (SRG) that has a lower objective value. However, SRG might not be optimal. So, showing that SRG has a lower rank does not necessarily mean that the optimal solution also follows this low-rank structure. Still, the results and observations presented in this work remain interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and the positive evaluation of our paper. We address your concerns and questions below:\n\n**While the SRG solution has low-rank, it is not clear whether a globally optimal solution also has one.**\n\nThis is a great point. We agree that we do not rigorously show this in our paper. However, we provide an intuitive explanation in lines 140-150. Our view is also aligned with a line of work on low rank bias [1, 2, 3]. In fact, [2] even proves the global optimality of low-rank solutions for sufficiently deep networks. Moreover, we highlight that the reason why SRG achieves a better loss than DNC is primarily its low-rank structure, as can be seen from our Lemma 13 which shows that the cost of intermediate layers linearly depends on their rank. Nevertheless, proving that global optima have low rank in our setting remains an interesting open problem.\n\n**Do the last layer’s post activation features follow the NC geometry? Or a different structure?**\n\nIn our experiments, we do not recover the exact NC2 geometry, see Figure 2 (right) or Figure 4 (middle). In fact, the singular values are not all the same, and the tail is around 2-3x smaller than the largest singular value, which implies that the matrix cannot be orthogonal. This effect could be due to the large weight decay considered in our experimental setup. We also do not observe any specific type of different structure in the last layer.\n\n**Do the solutions that don’t achieve SRG structure outperform the SRG solution?**\n\nWe did not find a solution that outperforms the SRG construction when the number of classes $K$ and number of layers $L$ are moderate. However, when $K$ and $L$ are very large, there are indeed solutions that outperform the SRG construction. This indeed shows that the SRG solution is not optimal in general, and we suspect that even lower rank structures are optimal in those settings. \n\n**Do the weight matrices also exhibit low-rank structure? What about the DNC3?**\n\nYes, the weight matrices exhibit the same low-rank structure as features. This can be seen from the formula $W_l=H_{l+1}H_l^\\dagger$ for the min-norm interpolator. This shows that the rank of $W_l$ is at most the minimum between the rank of its input and the rank of the output layer features. If those two are the same, $W_l$ must match the rank of the features exactly. Regarding DNC3, in our numerical experiments we do not observe exact alignment of rows of weight matrices with columns of feature matrices, and we do not expect this to happen theoretically. In fact, the rows of the weight matrices are not aligned with the columns of the feature matrices in our SRG solution.\n\n**Typo in Figure 4?**\n\n You are right, thank you for spotting this. The layer in question is in fact $\\tilde{M}_5$ instead of $M_5.$\n\n----\n\n[1] G. Ongie and R. Willett. \"The role of linear layers in nonlinear interpolating networks.\" arXiv preprint arXiv:2202.00856 (2022).\n\n[2] A. Jacot. \"Implicit bias of large depth networks: a notion of rank for nonlinear functions.\" ICLR, 2023.\n\n[3] A. Jacot. \"Bottleneck structure in learned features: Low-dimension vs regularity tradeoff.\" NeurIPS, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8cv3UZMuoy", "reviewer": "Reviewer_jQsy", "summary": "Deep Neural Collapse (DNC) refers the the neural collapse phenomenon that has been observed on intermediate layers of a deep network (nb. whereas neural collapse (NC) focuses only on the penultimate layer). Similar to Unconstrained Feature Model (UFM) for analyzing NC, Deep UFM (DUFM) is the comparable framework to study DNC. However, as opposed to UFM for NC, it's been observed that once you go beyond two layers or two classes, DNC is no longer optimal for DUFM. \n\nIn this paper, the authors suggest that the reason for this is a low-rank bias of commonly used multi-layer regularization schemes, meaning the low-rank bias leads to optimal solutions that are lower rank than the neural collapse. \n\nThis paper extends the current research by looking at DUFM when there are more than two classes and the model is non-linear. Previous work has focused only on DUFM for binary classification with non-linear models or deep linear models or shallow models with multiple classes.\n\nMost notably, they show that in this more complicated setting, DNC is not optimal wrt DUFM. Namely, the class means fail one of the original conditions of NC; i.e., they fail to form an orthogonal frame. The authors suggest that the reason for this is the low-rank bias under L2 regularization. It is worth noting that the condition DFC1 (which states within class variability goes to zero) is still optimal and it's only the DNC2 property (which states that class means are orthogonal) that conflicts with the low-rank bias.\n\nTheir findings are supported with strong theoretical proofs and empirically on benchmark datasets such as MNIST and CIFAR10.\n\nOne of the main takeaways from this paper is that…if a DNC solution is found, it is not because of DUFM or global optimality. Instead, it is because of an implicit bias of the optimization procedure.", "questions": "line 209 (clarification): Does the statement regarding K=2 or L=2 indicate that the argument of your theorem also be used to show DNC is optimal in this setting as shown in [51, 48] or are you just saying that as comparison when K=2 or L=2 by [51, 48] we know DNC is in fact optimal?\n\nline 343: why the MSE loss? I didn't catch that part.\n\n(typos/nits)\nline 206 (typo): according to Theorem 5, should L \\geq 3 be replaced with L = 3?", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The literature review is excellent and they do a great job motivating the problem they aim to address and how it compares with the existing literature. \nThe mathematical notation is clear and precise. The theory itself is very well developed with careful and detailed proofs. \nThe experiments they cover are solid and in agreement with their theory.\n\nThe authors pose some interesting open questions for future work related to the DNC1 condition.", "weaknesses": "Based on my understanding of the current work, I don't see any major weaknesses. I have some minor points of clarification or questions below but no major flaws.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and the positive evaluation of our paper. We address all questions below.\n\n**line 209 (clarification) about K=2, L=2**\n\nOur statement was only meant to say that the cases $K=2$ and $L=2$ were already treated in prior work [51, 48]. We will clarify this in the manuscript.\n\n**Why the MSE loss?**\n\nWe focus on the MSE loss primarily for theoretical reasons,  because the MSE loss allows for a much more explicit analysis of the loss function. However, we expect that this is without much loss of generality, as both the MSE loss and the CE loss have similar behavior for well-fitting solutions. In addition, the low-rank bias does not come from the fit part of the loss, but rather from the regularization.\n\n**Typo:**\n\n Thank you for pointing this out, we will correct it in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qal3NdiBiX", "reviewer": "Reviewer_hQr6", "summary": "The paper studies the optimality of neural collapse in intermediate layers of a deep neural network on classification tasks with more than 2 classes.\n\n1. The paper shows that the NC2 property (formation of a simplex ETF/orthogonal frame) in the intermediate layer features is not optimal, and constructs a strongly regular graph (SRG) based lower-rank solution leading to lower loss than previously proposed deep neural collapse (DNC) optimal solutions.\n\n2. The NC1 property (variability collapse of features) is still shown to be optimal in these settings.", "questions": "1. Do numerical experiments with the 5-layer MLP head include a ReLU activation for L-DUFM and end-end training?\n\n2. The authors mention in section 6.3 that \"We also observe that the rank deficiency is the strongest in the mid-layer of the MLP head\". Can further justification be provided for this? Especially which aspect of the spectrum of Layer 3 or 4 in Figure 5 justifies this observation? \n\n3. Since the SRG solution relies on feature class means being non-negative, how valid is this assumption (based on your practical experiments)?\n\nnit: In the notation for $H_L = \\sigma(W_{L-1}X_{L-1})$ in section 3, shouldn't $X_{L-1}$ be replaced with $H_{L-1}$ ? The same applies to $\\widetilde{X}_l$.\n\nnit: Please fix the formula for NC1 in the first line of section 6. The pseudo-inverse is not required when dividing the traces.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The key contribution of this paper is the construction of a strongly regular graph (SRG) based lower-rank solution for a generic $L > 2$ layer neural network trained on classification tasks with $K > 2$ classes. The authors show that this construction leads to lower training loss than the deep neural collapse-based solution.", "weaknesses": "1. There seems to be an error in the Sherman-Morrison formula-based inverse in the final result of Lemma 12, i.e, $(I + \\frac{1}{r-1}11^\\top)^{-1} = I - \\frac{1}{2(r-1)}11^\\top$. (The authors can correct me on this). Since this result is used in Lemma 13 and affects the results in Theorem 5, verifying this result is important.\n\n2. Can Definition 4 be split into multiple parts for better readability? For instance, split it into parts w.r.t:\n\n- collapsed class means are unaffected by non-linearity.\n- The class means are scaled versions of $T_r$ where each row is scaled individually.\n- Choice of $A_L$ for the final layer.\n\nThis breakdown highlights a major assumption that the non-linearity is not in effect till the last layer. Additionally, the complete set of trainable and non-trainable parameters should be made clear. The full definition in Appendix A.1 mentions extra trainable terms corresponding to $q$ terms.\n\n3. In section 6.1 (DUFM training) Figure 2 (first row, last column), the plots for the singular values of class means seem to have a rank of around 7 or 8 for intermediate layers as well. The ideal rank $K=10$ might not exactly hold for intermediate layers but these observations should be discussed since it is one of the main claims in the paper. \n\n4. The experimental results in section 6.3 for MNIST datasets claim that the network trained using gradient descent is the SRG solution. A justification for this observation is missing. The heatmap for MNIST in Figure 4 corresponds to $M_5$ which as per notation indicates the class-means of the penultimate layer post-activations. However, shouldn't the SRG solution correspond to similar heat maps across intermediate layers? Also, a statement such as low-rank implying an SRG solution is not justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for a detailed review. The typos pointed out by the reviewer will be corrected in the revision, and we thank the reviewer for spotting those. We address all the other concerns below (including the technical issue about the Sherman-Morrison formula), and we would like to kindly ask the reviewer to raise the score (together with the soundness score) in case no additional issue remains. \n\n**An issue with Sherman-Morrison formula.** \n\nWe carefully checked our computation and believe there is no error. The matrix we have to invert is indeed $I+\\frac{1}{r-2}\\mathbb{1}\\mathbb{1}^T$ (not $I+\\frac{1}{r-1}\\mathbb{1}\\mathbb{1}^T$, as in the review), as this is the main factor of the previous computation of $T_r T_r^T$, where we exchange the multiplier of $\\mathbb{1}\\mathbb{1}^T$ from $\\frac{1}{r-1}$ to $\\frac{1}{r-2}$ by pulling $\\frac{r-2}{r-1}$ in front of the parenthesis. Then, we use the Sherman-Morrison formula $(A+uv^T)^{-1}=A^{-1}-\\frac{A^{-1}uv^TA^{-1}}{1+u^TA^{-1}v}$, where $A$ is the identity matrix and both $u$ and $v$ are $\\frac{1}{\\sqrt{r-2}}\\mathbb{1}$. Thus, we get $I$ as the left summand, $\\frac{1}{r-2}\\mathbb{1}\\mathbb{1}^T$ as the numerator in the right summand, and $1+\\frac{r}{r-2}=\\frac{2(r-1)}{r-2}$ as the denominator. The term $r-2$ cancels in numerator and denominator, and we are left with the right hand side we write in our computation. \n\n**The definition 4 should be split and clarified.**\n\n Thank you for pointing this out, we agree that this definition is rather complicated. In the revision, we will split the definition into different bullet points where we distinguish the construction of the intermediate layers and the final layer, highlight the non-negativity of intermediate activations and more clearly define the trainable parameters which are the scales of the involved matrices. \n\n**In Figure 2 the rank is 7-8, which is not K=10. This should be discussed more.**\n\n The rank 5-8 we found in our experiments is indeed smaller than the rank of DNC solution, which in this case is 10. This is exactly in agreement with Theorem 5, which shows that NC2 solutions are not globally optimal. We will make this explicit clarification in the revision, adding to the discussion in lines 259-263. \n\n**The sentence “solution found by gradient descent is the SRG solution” from line 313 should be more justified.**\n\nThank you for pointing this out, we will clarify the sentence in the revision. What we meant is that the particular solution (i.e. a single run) whose gram matrix is displayed in lower row right of Figure 4 is an SRG solution, not that all solutions that SGD finds are SRG. Regarding the justification of this claim, as the reviewer dZSf correctly pointed out, there is a typo in the label, the plotted matrix is in fact $\\tilde{M}_5^T\\tilde{M}_5.$ However, we note that the gram matrix looks exactly the same (up to scalar scaling) for all other layers (please see our global response where we upload a PDF showing all 5 layers). This can be also seen theoretically since all the involved matrices have by the construction of the SRG solution the same gram matrix. The reason why the gram matrices look exactly like this can be seen from the fact that the $H_l$ matrices are defined to be incidence matrices of the complete graph and their gram matrix is therefore naturally the adjacency matrix of the triangular graph. For a more detailed explanation, we refer you to Lemma 15 and Lemma 16, which are basically entirely devoted to compute the gram matrices and their entries and singular values explicitly. These calculations exactly agree with the plot in Figure 4. We will explain this in more detail in the revision. \n\n**Do numerical experiments include the ReLU activation?**\n\n Yes, we include the ReLU activation in all our experiments in all layers. \n\n**What do we mean by “the rank deficiency is strongest in the middle layers of the MLP head” and how can one see this?**\n\nThank you for this question, we will clarify the sentence. We do not mean that the rank in these middle layers is smaller than in the other layers, but rather that the tail singular values are closer to zero than the singular values of the other layers (i.e., 0.001 instead of 0.01). In fact, by looking closely at the green and red curves corresponding to layers 3 and 4, one can notice that their right tails are slightly below those of the other curves, meaning that the corresponding singular values are closer to 0 than those in the other layers.\n\n**Is the assumption on feature class-means being non-negative valid in experiments?**\n\nThis is a great question. Indeed, in our experiments we observe that the intermediate features of the MLP head are non-negative before the application of the  ReLU, regardless of whether we obtain the SRG solution or not. We will clarify this point in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their reviews. Here, as part of an answer to reviewer hQr6's question, we upload a PDF representing the gram matrices of class-means in all 5 layers of the SRG solution corresponding to the lower row (right) of Figure 4. As can be seen, all gram matrices (up to small imperfections) match the same SRG structure of the triangular graph.

Meta-review: This paper studies the Neural Collapse phenomenon (NC) in deep nonlinear networks. They prove that a collapsed solution is not optimal under the widely-used “deep unconstrained feature model.” This highlights a surprising aspect of neural collapse in deep nonlinear networks, which was not present in prior analyses. Further, the authors explore the low-rank bias of regularization as a potential mechanisms for this suboptimal behavior.

Reviewers agree that this is a strong, well-motivated technical contribution which yields insight on a timely topic. I recommend acceptance.
Decision: Accept (poster)