Title: Neural collapse vs. low-rank bias: Is deep neural collapse really optimal?
Venue: NeurIPS 2024
Accepted: yes
Keywords: neural collapse

deep neural collapse

unconstrained features model

deep unconstrained features model

low-rank bias
Abstract: Deep neural networks (DNNs) exhibit a surprising structure in their final layer known as neural collapse (NC), and a growing body of works is currently investigated the propagation of neural collapse to earlier layers of DNNs -- a phenomenon called deep neural collapse (DNC). However, existing theoretical results are restricted to either linear models, the last two layers or binary classification. In contrast, we focus on non-linear models of arbitrary depth in multi-class classification and reveal a surprising qualitative shift. As soon as we go beyond two layers or two classes, DNC stops being optimal for the deep unconstrained features model (DUFM) -- the standard theoretical framework for the analysis of collapse. The main culprit is the low-rank bias of multi-layer regularization schemes. This bias leads to optimal solutions of even lower rank than the neural collapse. We support our theoretical findings with experiments on both DUFM and real data, which show the emergence of the low-rank structure in the solution found by gradient descent.

Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?

Abstract

Deep neural networks (DNNs) exhibit a surprising structure in their final layer known as neural collapse (NC), and a growing body of works has currently investigated the propagation of neural collapse to earlier layers of DNNs - a phenomenon called deep neural collapse (DNC). However, existing theoretical results are restricted to special cases: linear models, only two layers or binary classification. In contrast, we focus on non-linear models of arbitrary depth in multi-class classification and reveal a surprising qualitative shift. As soon as we go beyond two layers or two classes, DNC stops being optimal for the deep unconstrained features model (DUFM) - the standard theoretical framework for the analysis of collapse. The main culprit is a low-rank bias of multi-layer regularization schemes: this bias leads to optimal solutions of even lower rank than the neural collapse. We support our theoretical findings with experiments on both DUFM and real data, which show the emergence of the low-rank structure in the solution found by gradient descent.

1 Introduction

What is the geometric structure of layers and learned representations in deep neural networks (DNNs)? To address this question, Papyan et al. [39] focused on the very last layer of DNNs at convergence and experimentally measured what is now widely known as Neural Collapse (NC). This phenomenon refers to four properties that simultaneously emerge during the terminal phase of training: feature vectors of training samples from the same class collapse to the common class-mean (NC1); the class means form a simplex equiangular tight frame or an orthogonal frame (NC2); the class means are aligned with the rows of the last layer's weight matrix (NC3); and, finally, the classifier in the last layer is a nearest class center classifier (NC4). Since the influential paper [39], a line of research has aimed at explaining the emergence of NC theoretically, mostly focusing on the unconstrained features model (UFM) [36]. In this model, motivated by the network's perfect expressivity, one treats the last layer's feature vectors as a free variable and explicitly optimizes them together with the last layer's weight matrix, 'peeling off' the rest of the network [9, 23]. With UFM, the NC was demonstrated in a variety of settings, both as the global optimum and as the convergence point of gradient flow.

The emergence of the NC in the last layer led to a natural research question - does some form of collapse propagate beyond the last layer to earlier layers of DNNs? A number of empirical works

[20, 17, 43, 40, 35] gave evidence that this is indeed the case, and we will refer to this phenomenon as Deep Neural Collapse (DNC). On the theoretical side, the optimality of the DNC was obtained (i) for the UFM with two layers connected by a non-linearity in [51], (ii) for the UFM with several linear layers in [7], and (iii) for the deep UFM (DUFM) with non-linear activations in the context of binary classification [48]. No existing work handles the general case in which there are multiple classes and the UFM is deep and non-linear .

In this work, we close the gap and reveal a surprising behavior not occurring in the simpler settings above: for multiple classes and layers, the DNC as formulated in previous works is not an optimal solution of DUFM. In particular, the class means at the optimum do not form an orthogonal frame (nor an equiangular tight frame), thus violating the second property of DNC.

Let L and K denote the number of layers and classes, respectively. Then, if either L ≥ 3 and K ≥ 10 or L ≥ 4 and K ≥ 6 , we provide an explicit combinatorial construction of a class of solutions that outperforms DNC. Specifically, the loss achieved by our construction is a factor K ( L -3) / (2 L +2) lower than the loss of the DNC solution. Our result holds as long as all matrices are regularized.

We also identify the reason behind the sub-optimality of DNC: a low-rank bias . Intuitively, this bias arises from the representation cost of a DNN with l 2 regularization, which equals the Schattenp quasi norm [37] in the deep linear case. The quasi norm is well approximated by the rank, and this intuition carries over to the non-linear case as well. In fact, the rank of our construction is Θ( √ K ) , while the rank of the DNC solution is K . We note that after the application of the ReLU, the rank of the final layer is again equal to K , in order to fit the training data. We also show that the first property of neural collapse (convergence to class means) continues to be strictly optimal even in this general setting and its deep counterpart is approximately optimal with smoothed ReLU activations.

We support our theoretical results with empirical findings in three regimes: (i) DUFM training, (ii) training on standard datasets (MNIST [31], CIFAR-10 [28]) with DUFM-like regularization, and (iii) training on standard datasets with standard regularization. In all cases, gradient descent retrieves solutions with very low rank, which can exhibit symmetric structures in agreement with our combinatorial construction, see e.g. the lower-right plot of Figure 4. We also investigate the effect of three common hyperparameters - weight decay, learning rate and width - on the rank of the solution at convergence. On the one hand, high weight decay, high learning rate and small width lead to a strong low-rank bias. On the other hand, small (yet still non-zero!) weight decay, small learning rate or large width (and more complex datasets as well) lead to a higher-rank solution, even if that is not the global optimum, and this solution often coincides with DNC, which is in agreement with earlier experimental evidence. Altogether, our findings show that if a DNC solution is found, it is not because of its global optimality, but just because of an implicit bias of the optimization procedure.

The implications of our results go beyond deep neural collapse. In fact, our theory suggests that even the NC2 in the last layer is not optimal, and this is corroborated by our experiments, where the singular value structure of the last layer's class-mean matrices is imbalanced, ruling out orthogonality. This means that standard single-layer UFM, as well as its deep-linear or two-layer extensions, are not sufficient to describe the full picture, as they display a qualitatively different phenomenology.

2 Related work

Neural Collapse. Several papers (a non-exhaustive list includes [12, 15, 5, 32, 33, 58]) use neural collapse as a practical tool in applications, among which OOD detection and transfer learning are the most prevalent. On the theoretical side, the emergence of NC has been investigated, with the majority of works considering some form of UFM [36, 9]. [55, 34] show global optimality of NC under the cross-entropy (CE) loss, and [59] under the MSE loss. Similar results are obtained by [9, 49, 18, 8] for the class-imbalanced setting. [61, 23, 59] refine the analysis by showing that the loss landscape of the UFM model is benign - all stationary points are either local minima or strict saddle points which can be escaped by conventional optimizers. A more loss-agnostic approach connecting CE and MSE loss is considered in [60]. NC has also been analyzed for a large number of classes [25], in an NTK regime [45], or in graph neural networks [27]. We refer the reader to [26] for a survey.

The emergence of NC has also been studied through the lens of the gradient flow dynamics. [36] considers MSE loss and small initialization, and [16] a renormalized gradient flow of the last layer's features after fixing the last layer's weights to be conditionally optimal. [23] studies the CE loss dynamics and shows convergence in direction of the gradient flow to a KKT point of the max-margin

problem of the UFM, extending a similar analysis for the last layer's weights in [47]. The convergence speed under both losses is described in [53]. Going beyond UFM, [57, 41, 42, 29] study the emergence of NC in homogeneous networks under gradient flow; [38] provides sufficient conditions for neural collapse; and [52] perturbs the unconstrained features to account for the limitations of the model.

More recently, [20] mentions a possible propagation of the NC to earlier layers of DNNs, giving preliminary measurements. These are then significantly extended in [17, 43, 11, 40], which measure the emergence of some form of DNC in DNNs. On the theoretical front, an extension to a two-layer non-linear model is provided in [51], to a deep linear model in [7, 14] and to a deep non-linear model for binary classification in [48]. Alternatively to DUFM, [4] studies DNC in an end-to-end setting with a special layer-wise training procedure.

Low-rank bias. The low-rank bias is a well-known phenomenon, especially in the context of matrix/tensor factorization and deep linear networks (see e.g. [2, 6, 44, 24, 54]). For non-linear DNNs, [50] studies the gradient flow optimization of ReLU networks, giving lower and upper bounds on the average soft rank. [13] studies SGD training on deep ReLU networks, showing upper bounds on the rank of the weight matrices as a function of batch size, weight decay and learning rate. [30] proves several training invariances that may lead to low-rank, but the results require the norm of at least one weight matrix to diverge and the architecture to end with a couple of linear layers. [3] presents bounds on the singular values of non-linear layers in a rather generic setting, not necessarily at convergence. More closely related to our work is [37], which considers a deep linear network followed by a single non-linearity and then by a single layer. Their arguments to study the low-rank bias are similar to the intuitive explanation of Section 4. [19] shows that increasing the depth results in lower effective rank of the penultimate layer's Gram matrix both at initialization and at convergence. The true rank is also measured, but on rather shallow networks and it is far above the DNC rank. [1] shows a strong low-rank bias of sharpness-aware minimization, although only in layers where DNC does not yet occur and the rank is high. [10, 22] study special functional ranks (Jacobi and bottleneck) of DNNs, providing asymptotic results and empirical measurements. These results are refined in [21, 56], which show a bottleneck structure of the rank both experimentally and theoretically. The measurements of the singular values at convergence in [56] are in agreement with those of Section 6.3. We highlight that none of the results above allows to reason about DNC optimality, as they focus on infinite width/depth, effective or functional ranks, orthogonal settings, or are not quantitative enough.

3 Preliminaries

We study the class balanced setting with N = Kn samples from K classes, n per class. Let f ( x ) = W L σ ( W L -1 σ ( . . . W 1 B ( x ) . . . )) be a DNN with backbone B ( · ) . The backbone represents the majority of the deep network before the last L layers, e.g. the convolutional part of a ResNet20. Let X ∈ R d × N be the training data, and H 1 = B ( X ) ∈ R d 1 × N , H 2 = σ ( W 1 H 1 ) ∈ R d 2 × N , . . . , H L = σ ( W L -1 H L -1 ) ∈ R d L × N its feature vector representations in the last L layers, with ˜ H l denoting their counterparts before applying the ReLU σ . We refer to h l ci and ˜ h l ci as to the i -th sample of c -th class of H l and ˜ H l , respectively. Let µ l c = 1 n ∑ n i =1 h l ci and ˜ µ l c = 1 n ∑ n i =1 ˜ h l ci be the class means at layer l after and before applying σ , and M l , ˜ M l the matrices of the respective class means stacked into columns. We organize the training samples so that the labels Y ∈ R K × N equal I K ⊗ 1 T n , where I K is a K × K identity matrix, ⊗ is the Kronecker product and 1 n the all-one vector of size n.

Deep neural collapse (DNC). As there are no biases in our network model, the second property of DNC requires the class mean matrices to be orthogonal (instead of forming an ETF) [43, 48].

Definition 1. We say that layer l exhibits DNC 1, 2 or 3 if the corresponding conditions are satisfied (the properties can be stated for both after and before the application of ReLU):

DNC1: The within-class variability of either H l or ˜ H l is 0 . Formally, h l ci = h l cj , ˜ h l ci = ˜ h l cj for all i, j ∈ [ n ] or, in matrix notation, H l = M l ⊗ 1 T n , ˜ H l = ˜ M l ⊗ 1 T n .

DNC2: The class-mean matrices M l , ˜ M l are orthogonal, i.e., M T l M l ∝ I K , ˜ M T l ˜ M l ∝ I K .

DNC3: The rows of the weight matrix W l are either 0 or collinear with one of the columns of the class-means matrix M l .

Deep unconstrained features model. To define DUFM, we generalize the model in [48] to an arbitrary number of classes K.

Definition 2. The L -layer deep unconstrained features model ( L -DUFM) denotes the following optimization problem:

where ∥·∥ F denotes the Frobenius norm and λ H 1 , λ W 1 , . . . , λ W L > 0 are regularization parameters.

4 Low-rank solutions outperform deep neural collapse

Intuitive explanation of the low-rank bias. Consider a simplified version of L -DUFM:

Compared to (1), (2) removes all non-linearities except in the last layer, making the remaining part of the network a deep linear model, a construction similar to the one in [37]. Now, we leverage the variational form of the Schattenp quasi-norm [46], which gives

where c can be computed explicitly. Thus, after solving for H 1 , W 1 , . . . , W L -1 , the simplified L -DUFM problem (2) can be reduced to

For large values of L , ∥ ∥ ∥ ˜ H L ∥ ∥ ∥ 2 /L S 2 /L is well approximated by the rank of ˜ H L . Hence, the objective value is low when the output W L H L fits Y closely, while keeping ˜ H L low-rank, which justifies the low-rank bias. Crucially, the presence of additional non-linearities in the L -DUFM model (1) does not change this effect much, as long as one is able to define solutions for which most of the intermediate feature matrices ˜ H l are non-negative (so that ReLU does not have an effect).

Low-rank solution outperforming DNC. We define the combinatorial solution that outperforms DNC, starting from the graph structure on which the construction is based.

Definition 3. A triangular graph T n of order n is a line graph of a complete graph K n of order n . T n has ( n 2 ) vertices, each representing an edge of the complete graph, and there is an edge between a pair of vertices if and only if the corresponding edges in the complete graph share a vertex. Moreover, let T n be the normalized incidence matrix of K n , i.e., ( T n ) i,j = 1 √ n -1 if vertex i belongs to edge j and 0 otherwise. Let G n denote the adjacency matrix of T n .

We recall that T n is a strongly regular graph with parameters ( n ( n -1) / 2 , 2( n -2) , n -2 , 4) and spectrum 2( n -2) with multiplicity 1, n -4 with multiplicity n -1 and -2 with multiplicity n ( n -3) / 2 . Next, we construct an explicit solution ( H 1 , W 1 , . . . , W L ) based on the triangular graph. For ease of exposition, we focus on the case where the number of classes K equals ( r 2 ) for some r ≥ 4 , deferring the general definition to Appendix A.1.

Definition 4. Let K = ( r 2 ) for r ≥ 4 . Then, a strongly regular graph (SRG) solution of the L -DUFM problem (1) is obtained by setting the matrices ( H 1 , W 1 , . . . , W L ) as follows:

For all l, the feature matrices H l , ˜ H l are DNC1 collapsed, i.e., H l = M l ⊗ 1 T n , ˜ H l = ˜ M l ⊗ 1 T n .

For 2 ≤ l ≤ L -1 , M l = ˜ M l , each row of ˜ M l is a non-negative multiple of a row of T r (as in Definition 3), and the sum of squared norms of the rows of ˜ M l corresponding to a row of T r is the same for each row of T r . Since ˜ M l is entry-wise non-negative, M l = ˜ M l .

For l = 1 , W 1 , M 1 are any pair of matrices minimizing the objective conditionally on M 2 defined above.

0.35

D

0.4

1.0

0.175

10

5

0.30

0.25

10

5

0.2

2

0.8

0.6

0.125

0.150

15

0.20

15

0.0

4

0.4

0.100

20

0.15

20

0.2

6

0.0

0.2

6

0.075

0.10

0.050

25

25

0.2

0.05

0.4

8

0.4

8

0.025

30

0.00

30

0

5

0

5

0

2

6

8

0

2

4

5

8

For l ≥ 2 , W l minimizes the objective conditional to input and output to that layer.

As for the last layer L, let A L be a K × r matrix where the set of rows equals the set of vectors with two ( -1) entries and r -2 (+1) entries. Then, M L = σ ( ˜ M L ) , the rows of ˜ M L are a non-negative multiple of A L T r , and the sum of their squared norms corresponding to either row of A L T r is equal.

Finally, the Frobenius norms (i.e. scales) of M 1 , W 1 , . . . , W L are chosen so as to minimize (1) while satisfying the construction above.

In this construction, columns and rows of class-mean matrices are associated to edges and vertices of the complete graph K r . Each row (corresponding to a vertex) has non-zero entries at columns that correspond to edges containing the vertex. In the final layer, each row of ˜ M L corresponds to a weighting of vertices in K r s.t. exactly two vertices get -1 weight and the rest +1 , and the value at a column is the sum of the values of the vertices of the edge. The class-mean matrices of the SRG solution are illustrated in Figure 1 for L = 4 and K = 10 (which gives r = 5 ): we display M 3 , ˜ M 4 , ˜ M T 4 ˜ M 4 and, for comparison, also ˜ M T 4 ˜ M 4 of a DNC solution. Very similar solutions to SRG are shown for K = 6 and K = 15 in Figures 7 and 8 of Appendix B.1.

Let us highlight the properties of the SRG solution, which are crucial to outperform DNC. First, the rank of the intermediate feature and weight matrices is very low, only of order Θ( K 1 / 2 ) , since by construction there are only r = Θ( K 1 / 2 ) linearly independent rows. This is contrasted with the DNC solution that has rank K in all intermediate feature and weight matrices. The low rank of the SRG solution is due to the specific structure of the triangular graph, which has many eigenvalues equal to -2 that become 0 after adding twice a diagonal matrix. Second, the definition of ˜ M L ensures that M L = σ ( ˜ M L ) has full rank K . This allows the output W L M L to also have full rank and, therefore, fit the identity matrix I K , thus reducing the first term in the loss (1). Finally, the highly symmetric nature of the SRG solution balances the feature and weight matrices so as to minimize large entries and, therefore, the Frobenius norms, thus reducing the other terms in the loss (1).

Main result. For any L -DUFM problem (specified by K,n and all the regularization parameters), let L SRG , L DNC be the losses incurred by the SRG and DNC solutions, see Definitions 4 and 1, respectively. At this point we are ready to state our key result.

Theorem 5. If K ≥ 6 , L ≥ 4 or K ≥ 10 , L = 3 and d l ≥ K for all l , then L SRG < L DNC . Moreover, consider any sequence of L -DUFM problems for which K - →∞ so that 0 . 499 > L DNC for each problem. In that case,

In words, as long as the number of classes and layers is not too small, the SRG solution always outperforms the collapsed one and the gap grows with the number of classes K .

The proof first computes the conditionally optimal values of ∥ W l ∥ 2 F for both the SRG and DNC solutions. The specific structure of these solutions enables to calculate pseudoinverses of the intermediate features, thus enabling the explicit computation of the weight norms. All these values depend only on the singular values of the feature matrices, which are explicitly given by their scale. As a result, both L SRG and L DNC are expressed via an optimization problem in a single scalar variable and, by comparing these problems, the statement follows. The details are deferred to Appendix A.1.

Although the argument requires L = 3 , K ≥ 10 or L ≥ 4 , K ≥ 6 , the experiments in Appendix B.1 show that the DNC solution is not optimal when L ≥ 4 , K ≥ 3 or L = 3 , K ≥ 7 . Furthermore, for L = 3 and large K , there is a large gap between L SRG and L DNC (even if (3) trivializes). For either K = 2 or L = 2 , the DNC is optimal, as shown in [51, 48].

5 Within-class variability collapse is still optimal

While the DNC2 property conflicts with the low-rank bias, the same is not true for DNC1, as the within-class variability collapse supports a low rank. We show below that the last-layer NC1 property remains optimal for any L -DUFM problem. A proof sketch follows, with the complete argument deferred to Appendix A.2.

Theorem 6. The optimal solutions of the L -DUFM (1) exhibit DNC1 at layer L , i.e.,

holds for any optimal solution ( H ∗ 1 , W ∗ 1 , . . . , W ∗ L ) of the L -DUFM problem.

Proof sketch: Assume by contradiction that there exists an optimal solution of (1) with regularization parameters ( λ H 1 , λ W 1 , . . . , λ W L ) , denoted as ( H ∗ 1 , W ∗ 1 , . . . , W ∗ L ) , which does not exhibit neural collapse at layer L . Then, we can construct two different optimal solutions of the L -DUFM problem with n = 1 and regularization parameters ( nλ H 1 , λ W 1 , . . . , λ W L ) of the form ( H (1) 1 , W ∗ 1 , . . . , W ∗ L ) and ( H (2) 1 , W ∗ 1 , . . . , W ∗ L ) . These two solutions share the weight matrices, and H 1 (and, therefore, H L ) only differs in a single column (w.l.o.g., the first column). The optimality of these solutions can be proved using separability and symmetry of the loss function w.r.t. the columns of H 1 .

Denote the first (differing) columns of H (1) L and H (2) L as x and y , respectively. By exploiting the linearity of the loss function on a ray { th 1 11 , t ≥ 0 } for any h 1 11 , a direct computation gives that x and y are not aligned. Let L be the loss in (1). By optimality of both solutions, we get

An application of the chain rule gives

where ˜ H L +1 is the model output and L F the first term of L , corresponding to the label fit. Plugging this back into (4) and using that W ∗ L is the same in both expressions, we get A ( H (1) L ) T = B ( H (2) L ) T , where we have denoted by A and B the partial derivatives ∂ L F ∂ ˜ H L +1 evaluated at ( H (1) 1 , W ∗ 1 , . . . , W ∗ L )

̸

and ( H (2) 1 , W ∗ 1 , . . . , W ∗ L ) , respectively. As L F is separable with respect to the columns of H l , ˜ H l for all l , the matrices A,B can only differ in their first columns (denoted by a, b ), and they are identical otherwise. This implies that ax T = by T . After some simple considerations and using that x and y are not aligned, we reach a contradiction, as we conclude that x = y is impossible.

The difficulty in extending Theorem 6 to a result on the unique optimality of DNC1 for all layers stems from the special role of W L as the loss is differentiable w.r.t. it. By considering a differentiable relaxation of ReLU, we show below an approximate result for a relaxed L -DUFM model.

Definition 7. We denote by ReLU ϵ (or σ ϵ ) a function satisfying the following conditions: (i) σ ϵ ( x ) = σ ( x ) , for x ∈ ( -∞ , 0] ∪ [ ϵ, ∞ ) , (ii) 0 < σ ϵ ( x ) < σ ( x ) for x ∈ (0 , ϵ ) , and (iii) σ ϵ is continuously differentiable with derivative bounded by a universal constant and strictly positive on (0 , ϵ ) .

Theorem 8. Denote by L -DUFM ϵ the equivalent of (1) , with σ replaced by σ ϵ . Let D = max { d 2 , d 3 , . . . , d L } and ¯ λ = λ H 1 λ W 1 . . . λ W L , with the regularization parameters upper bounded by 1 / ( L +1) . Then, for any globally optimal solution of the L -DUFM ϵ problem, the distance between any two feature vectors of the same class in any layer is at most

In words, as the activation function approaches ReLU (i.e., ϵ → 0 ), the within-class variability tends to 0 . The proof starts with a similar strategy as the argument of Theorem 6 and then explicitly tracks the error due to replacing σ with σ ϵ through the layers. The full argument is in Appendix A.2.

6 Numerical results

We employ the standard DNC1 metric tr (Σ W ) / tr (Σ B ) , where Σ W , Σ B are the within and between class variabilities. This is widely used in the literature [52, 43, 4] and considered more stable than other metrics [43]. We measure the DNC2 metric as the condition number of M l for l ≥ 1 [48]. We do not measure DNC3 here, as it is not well-defined for solutions that do not satisfy DNC2. For end-to-end DNN experiments, we employ a model from [48] where an MLP with a few layers is attached to a ResNet20 backbone. The output of the backbone is then treated as unconstrained features, and DNC metrics are measured for the MLP layers.

6.1 DUFM training

We start with the L -DUFM model (1), training both features and weights. In the top row of Figure 2, we consider a 4 -DUFM, with K = 10 and n = 50 , presenting the training progression of the losses (left plot), the DNC1 metrics (center plot) and the singular values at convergence (right plot).

The results are in excellent agreement with our theory. First, the training loss outperforms that of the DNC solution, and it is rather close to that of the SRG solution. Second, DNC1 holds in a clear way in all layers, especially in the last ones. Third, the solution at convergence exhibits a strong low rank bias: the ranks of intermediate layers range from 5 to 8, and they are always the same in all intermediate layers within one run. For comparison, we recall that the intermediate layers of the DNC solution have full rank K = 10 . Third, for a few runs, the Gram matrices of the intermediate class means resulting from gradient descent training coincide with those of an SRG solution. Finally we highlight that, similarly to our theory, the solutions found in all our experiments in the entire Section 6 have non-negative pre-activations in all intermediate layers of the MLP head except the last one.

Impact of number of classes and depth. For K = 2 or L = 2 , we recover the results of [48, 51] irrespective of other hyperparameters. The higher the number of classes, the more prevalent are lowrank solutions, while finding DNC solutions becomes challenging. The same holds for increasing the number of layers. For L = 3 and low number of classes ( K ≤ 6 ), we weren't able to experimentally find solutions that would outperform DNC, which aligns nicely with the fact that SRG outperforms DNC only from K = 10 for L = 3 . For large number of classes, the difference between the loss of low-rank solutions and the DNC loss is considerable already for L = 3 and becomes even larger for higher L. This is illustrated in the left plot of Figure 3.

For L ≤ 5 and moderate number of classes ( K ≤ 30 ), gradient descent solutions are as follows: until layer L -1 , feature matrices share the same rank and have similar Gram matrices; intermediate activations are typically non-negative, and the ReLU has no effect; then, the rank jumps to K after the final ReLU, as pre-activations are also negative. For large L or large K, the rank of the first few layers is low, growing gradually in the last couple of layers (see Figure 6 in Appendix B.1); the ReLU is active only in the final layers. This means that not only very low-rank solutions outperform DNC (as shown by our theory), but such solutions are routinely reached by gradient descent.

Impact of weight decay and width. While neither weight decay nor width influence Theorem 5 which shows that DNC is not optimal - both quantities influence the nature of the solutions found by gradient descent. In particular, the stronger the weight decay, the lower the rank, see the middle plot in Figure 3. For very small weight decay, DNC is sometimes recovered; for very high weight decay,

DNC1

Singularvalues

loss

103

10°

training_loss

DNC_loss

SRG loss

10°

layer_2

layer_1

layer_3

layer_4

2.0

1.5

layer_1

layer_4

layer_2

layer_3

loss

10-3

10-6

0.5

10-9]

0.0

10-1

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0

2

4

6

8

oda

1e5

1e5

index

DNC1

Singularvalues

loss

101

layer_1

2.0

layer_1

DNC_Ioss

training_loss

100

layer_2

layer_3

layer_2

layer_3

10-1

SRG loss

10-1

layer_4

1.5

layer_4

value

loss

10-2

1.0

10-3

10-4

0.5

10-1.5

10-5

0.0

0

i

2

3

4

5

0

1

2

3

4

5

0

2

4

6

8

1e3

1e3

index

1.0

L =3

14

1.0

L = 4

0.9

L = 5

rank

12

bability

0.8

atios

0.8

0.7

aver

6

era

10

8

DNC

proba

0.6

0.4

0.6

6

0.2

0.5

4

0.0

4

6

8

10

-12

-10

-8

20

40

60

80

r

log2weightdecay

width

it is never recovered. The width has an opposite effect, see the right plot of Figure 3. For small width, low-rank solutions are much more likely to be found; large width has a strong implicit bias towards DNC and, thus, rank K solutions. This means that, surprisingly, a larger width leads to a larger loss, since low-rank solutions exhibit a smaller loss than DNC. Thus, at least in DUFM, the infinite-width limit prevents gradient descent from finding a globally optimal solution, and sub-optimal solutions are reached with increasingly high probability.

6.2 End-to-end experiments with DUFM-like regularization

Next, we train a DNN backbone with an MLP head, regularizing only the output of the backbone and the layers of the MLP head (and not the layers of the backbone). This regularization is closer to our theory than the standard one, since we explicitly regularize the Frobenius norm of the unconstrained features. We also note that training with such a regularization scheme is easier than training with the standard regularization scheme. In the bottom row of Figure 2, we consider a ResNet20 backbone with a 4-layer MLP head trained on CIFAR10.

DNC1

Singularvalues

101

layer_1

layer_1

10°

layer_2

layer_3

2.0

layer_2

layer_3

C

0.7

layer_4

layer_4

10-1

layer_5

alue

1.5

layer_5

2

0.6

1.0

10-2

0.5

10-3

0.5

6

0.4

0.3

10-4

0.0

8

0.00

0.01

0.02

0.03

0.04

0.05

0

2

4

6

8

0.2

1e5

index

DNC1

Singular values

101

layer_1

3.0

layer_1

10°

layer_2

2.5

layer_2

2.5

layer_3

layer_3

2.0

10-1

layer_4

2.0

layer_4

10-2

layer_5

value

1.5

layer_5

2

1.5

1.0

10-3

1.0

0.5

0.5

0.0

10-4

10-5

0.00

0.01

0.02

0.03

0.04

0.05

0.0

0

2

4

6

8

8

0.5

1.0

1e5

index

The results agree well with our theory, and they are qualitatively similar to those of Section 6.1 for DUFMtraining. The DNNs consistently outperform the DNC loss, but still achieve DNC1. The ranks of class-mean matrices range from 5 to 6, and they are always the same in all intermediate layers within one run. Remarkably, the SRG solution was found by gradient descent also in this setting.

Both weight decay and learning rate affect the average rank of the solutions found by gradient descent. Varying the width can lead to unexpected results, as it changes the ratio between the number of parameters in the MLP and that in the backbone, so the effect of the width is harder to interpret. Similar results can be seen on MNIST.

6.3 End-to-end experiments

Finally, we perform experiments with standard regularization and the same architecture (i.e., DNN backbone plus MLP head) as in Section 6.2. In particular, in Figure 4 we consider a ResNet20 backbone with a 5-layer MLP head trained on CIFAR10 and MNIST with standard weight regularization.

Overall, the results remain qualitatively similar to those discussed above. This demonstrates that, in spite of a different loss landscape compared to previous settings, the low-rank bias is still responsible for DNC2 not being attained. Specifically, for CIFAR10, the rank in the third layer ranges between 8 and 9, and for MNIST ranges between 5 and 7; in contrast, the DNC solution has rank K = 10 . All DNNs display DNC1 across all layers. Remarkably, for the MNIST experiment the solution displayed in Figure 4 found by gradient descent is the SRG solution (compare the gram matrices in bottom right plot of Figure 4 with the right-most plot of Figure 1).

The difficulty of the learning task plays a significant role in this setting: when training on MNIST, it is rather easy to reach low-rank solutions and rather difficult to reach DNC solutions and the rank depends heavily on the regularization strength as shown in Figure 10 of Appendix B.3; when training on CIFAR-10, the weight decay needs to be high for the class mean matrices to be rank deficient. Moreover, the learning rate no longer exhibits a clear relation with the rank, since gradient descent diverges when the learning is too large. We also observe that the rank deficiency is the strongest in the mid-layer of the MLP head, creating a 'rank bottleneck'. This can be seen by a closer look at the tails of the singular values, which better match zero at intermediate layers (the green and red curves corresponding to layers 3 and 4 have tails slightly lower than the other curves). In a more precise manner, we further measured effective ranks of all the layers in Figure 4. For instance, the effective

ranks of CIFAR10 experiment layers are 8 . 96 , 7 . 46 , 6 . 88 , 7 . 04 , 7 . 73) , which shows the middle layer is closest to a low hard rank matrix. The rank bottleneck is also mentioned in [22, 21, 56]. In fact, these works also measure extremely low ranks, but [22, 21] do it on synthetic data with very low inner dimension, while [56] focuses on fully convolutional architectures trained with CE loss and including biases.

In summary, Figure 4 shows that both the low-rank bias and the optimality of DNC1 carry over to the standard training regime. This means that there are hyperparameter settings for which deep neural collapse, including in the very last layer , is not reached (and likely not even optimal). Although the sub-optimality of DNC in the last layer is not proved formally, this phenomenon is supported by evidence across all experimental settings and further corroborated by our theory where our SRG construction is far from being DNC2-collapsed in the last layer.

7 Conclusion

In this work, we reveal that the deep neural collapse is not an optimal solution of the deep unconstrained features model - the extension of the widely used unconstrained features model. This finding considerably changes our overall understanding of DNC, as all the previous models in simplified settings showed the global optimality of neural collapse and of its deep counterpart. The main culprit the low-rank bias - makes the orthogonal frame property of DNC, and thus DNC as a whole, too high rank to be optimal. We demonstrate this low-rank bias across a variety of experimental settings, from DUFM training to end-to-end training with the standard weight regularization. While the structure of the Gram matrices of class means is not captured by orthogonal matrices (or by the ETF), the within-class variability collapse remains optimal. Our theoretical analysis proves this for the DUFM problem, and our numerical results showcase the phenomenon across various settings.

Our analysis focuses on the MSE loss, but we expect similar results to hold for the cross-entropy loss and, in particular, that the same SRG construction proposed here would still refute the optimality of DNC. We leave as an open question whether DNC1 is strictly optimal across all layers. While proving this would likely require new ideas, we note that none of our experiments converged to a solution that would not be DNC1-collapsed.

Acknowledgments and Disclosure of Funding

Marco Mondelli is partially supported by the 2019 Lopez-Loreta prize. This research was supported by the Scientific Service Units (SSU) of ISTA through resources provided by Scientific Computing (SciComp).

References

Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpnessaware minimization leads to low-rank features. In Conference on Neural Information Processing Systems (NeurIPS) , volume 36, 2023.

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Conference on Neural Information Processing Systems (NeurIPS) , 2019.

Bradley T Baker, Barak A Pearlmutter, Robyn Miller, Vince D Calhoun, and Sergey M Plis. Low-rank learning by design: the role of network architecture and activation linearity in gradient rank collapse. arXiv preprint arXiv:2402.06751 , 2024.

Daniel Beaglehole, Peter Súkeník, Marco Mondelli, and Mikhail Belkin. Average gradient outer product as a mechanism for deep neural collapse. arXiv preprint arXiv:2402.13728 , 2024.

Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. In Topological, Algebraic and Geometric Learning Workshops , 2022.

Hung-Hsu Chou, Carsten Gieshoff, Johannes Maly, and Holger Rauhut. Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank. Applied and Computational Harmonic Analysis , 68, 2024.

Hien Dang, Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho. Neural collapse in deep linear network: From balanced to imbalanced data. In International Conference on Machine Learning (ICML) , 2023.

Hien Dang, Tho Tran, Tan Nguyen, and Nhat Ho. Neural collapse for cross-entropy classimbalanced learning with unconstrained ReLU feature model. arXiv preprint arXiv:2401.02058 , 2024.

Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. In Proceedings of the National Academy of Sciences (PNAS) , volume 118, 2021.

Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. Rank diminishing in deep neural networks. In Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the implicit bias towards minimal depth of deep neural networks. arXiv preprint arXiv:2202.09028 , 2022.

Tomer Galanti, András György, and Marcus Hutter. Improved generalization bounds for transfer learning via neural collapse. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML , 2022.

Tomer Galanti, Zachary S Siegel, Aparna Gupte, and Tomaso Poggio. SGD and weight decay provably induce a low-rank bias in neural networks. arXiv preprint arXiv:2206.05794 , 2022.

Connall Garrod and Jonathan P Keating. Unifying low dimensional observations in deep learning through the deep linear unconstrained feature model. arXiv preprint arXiv:2404.06106 , 2024.

Jarrod Haas, William Yolland, and Bernhard T Rabus. Linking neural collapse and l2 normalization with improved out-of-distribution detection in deep neural networks. Transactions on Machine Learning Research (TMLR) , 2022.

X. Y. Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations (ICLR) , 2022.

Hangfeng He and Weijie J Su. A law of data separation in deep learning. Proceedings of the National Academy of Sciences , 120(36), 2023.

Wanli Hong and Shuyang Ling. Neural collapse for unconstrained feature model under crossentropy loss with imbalanced data. arXiv preprint arXiv:2309.09725 , 2023.

Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research , 2022.

Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.08384 , 2022.

Arthur Jacot. Bottleneck structure in learned features: Low-dimension vs regularity tradeoff. In Conference on Neural Information Processing Systems (NeurIPS) , 2023.

Arthur Jacot. Implicit bias of large depth networks: a notion of rank for nonlinear functions. In International Conference on Learning Representations (ICLR) , 2023.

Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layerpeeled perspective on neural collapse. In International Conference on Learning Representations (ICLR) , 2022.

Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations (ICLR) , 2018.

Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin G Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for a large number of classes. In Conference on Parsimony and Learning (Recent Spotlight Track) , 2023.

Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. In Transactions on Machine Learning Research (TMLR) , 2023.

Vignesh Kothapalli, Tom Tirer, and Joan Bruna. A neural collapse perspective on feature evolution in graph neural networks. In Conference on Neural Information Processing Systems (NeurIPS) , volume 36, 2023.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

Daniel Kunin, Atsushi Yamamura, Chao Ma, and Surya Ganguli. The asymmetric maximum margin bias of quasi-homogeneous neural networks. In International Conference on Learning Representations (ICLR) , 2022.

Thien Le and Stefanie Jegelka. Training invariances and the low-rank phenomenon: beyond linear networks. In International Conference on Learning Representations (ICLR) , 2022.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11), 1998.

Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural collapse. In Conference on Parsimony and Learning (Recent Spotlight Track) , 2023.

Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis , 59, 2022.

Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Miło´ s, and Tomasz Trzcinski. The tunnel effect: Building data representations in deep neural networks. Conference on Neural Information Processing Systems (NeurIPS) , 36, 2023.

Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11619 , 2020.

Greg Ongie and Rebecca Willett. The role of linear layers in nonlinear interpolating networks. arXiv preprint arXiv:2202.00856 , 2022.

Leyan Pan and Xinyuan Cao. Towards understanding neural collapse: The effects of batch normalization and weight decay. arXiv preprint arXiv:2309.04644 , 2023.

Vardan Papyan, X. Y. Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. In Proceedings of the National Academy of Sciences (PNAS) , volume 117, 2020.

Liam Parker, Emre Onal, Anton Stengel, and Jake Intrater. Neural collapse in the intermediate hidden layers of classification neural networks. arXiv preprint arXiv:2308.02760 , 2023.

Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101.00072 , 2020.

Tomaso Poggio and Qianli Liao. Implicit dynamic regularization in deep networks. Technical report, Center for Brains, Minds and Machines (CBMM), 2020.

Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso Poggio. Feature learning in deep classifiers through intermediate neural collapse. Technical Report , 2023.

Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in tensor factorization. In International Conference on Machine Learning (ICML) , 2021.

Mariia Seleznova, Dana Weitzner, Raja Giryes, Gitta Kutyniok, and Hung-Hsu Chou. Neural (tangent kernel) collapse. In Conference on Neural Information Processing Systems (NeurIPS) , volume 36, 2023.

Fanhua Shang, Yuanyuan Liu, Fanjie Shang, Hongying Liu, Lin Kong, and Licheng Jiao. A unified scalable equivalent formulation for Schatten quasi-norms. Mathematics , 8(8), 2020.

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. In Journal of Machine Learning Research , volume 19, 2018.

Peter Súkeník, Marco Mondelli, and Christoph Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. In Conference on Neural Information Processing Systems (NeurIPS) , 2023.

Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. In Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu networks. In Algorithmic Learning Theory (ALT) , 2023.

Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning (ICML) , 2022.

Tom Tirer, Haoxiang Huang, and Jonathan Niles-Weed. Perturbation analysis of neural collapse. In International Conference on Machine Learning (ICML) , 2023.

Peng Wang, Huikang Liu, Can Yaras, Laura Balzano, and Qing Qu. Linear convergence analysis of neural collapse with unconstrained features. In NeurIPS Workshop on Optimization for Machine Learning (OPT) , 2022.

Zihan Wang and Arthur Jacot. Implicit bias of SGD in L 2 -regularized linear DNNs: One-way jumps from high to low rank. In International Conference on Learning Representations (ICLR) , 2024.

E Weinan and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Mathematical and Scientific Machine Learning , 2022.

Yuxiao Wen and Arthur Jacot. Which frequencies do CNNs need? Emergent bottleneck structure in feature learning. arXiv preprint arXiv:2402.08010 , 2024.

Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds. In Research , volume 6, 2023.

Jiawei Zhang, Yufan Chen, Cheng Jin, Lei Zhu, and Yuantao Gu. EPA: neural collapse inspired robust out-of-distribution detector. arXiv preprint arXiv:2401.01710 , 2024.

Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features. In International Conference on Machine Learning (ICML) , 2022.

Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. In Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. In Conference on Neural Information Processing Systems (NeurIPS) , 2021.

A Proofs

A.1 Low-rank solutions outperform deep neural collapse

We start by providing more detailed definitions of SRG and DNC solutions.

Definition 9. Let K = ( r 2 ) for r ≥ 4 . Then, a strongly regular graph (SRG) solution of the L -DUFM problem (1) is obtained by setting the matrices ( H 1 , W 1 , . . . , W L ) as follows. For all l , the feature matrices H l , ˜ H l are DNC1 collapsed, i.e., H l = M l ⊗ 1 T n , ˜ H l = ˜ M l ⊗ 1 T n . For 2 ≤ l ≤ L -1 , M l = ˜ M l and M l = A l T r , where each row of A l is a multiple of the standard basis vector of dimension r and the sum of squared multiples corresponding to one basis vector equals the same parameter α l . In other words, each row of M l is a multiple of a row of T r and the sum of squared norms of the rows of M l corresponding to a row of T r is the same for each row of T r . For l = 1 , W 1 , M 1 are any pair of matrices that minimize the objective conditionally on M 2 being defined as above. For l ≥ 2 , W l minimizes the objective conditional to the input and output to that layer. Let A (1) L be a K × r matrix where each row has exactly two ( -1) entries and exactly r -2 (+1) entries. Then, normalize the matrix A (1) L T r so that each row is unit norm and multiply from the left with A (2) L of dimension d L × K , where each of the rows of A (2) l is a multiple of a standard basis vector of dimension K and the total sum of squared multiples corresponding to each basis vector is α L . Then, ˜ M L is obtained via this procedure, and M L = σ ( ˜ M L ) . Finally, the parameters { α l } L l =2 satisfy

and the parameter q ≥ 0 is chosen to minimize the objective function in (1) .

Definition 10. A deep neural collapse (DNC) solution for any number of classes K of the L -DUFM problem (1) is obtained by setting the matrices ( H 1 , W 1 , . . . , W L ) as follows. For all l, the feature matrices H l , ˜ H l are DNC1 collapsed, i.e., H l = M l ⊗ 1 T n , ˜ H l = ˜ M l ⊗ 1 T n . For 2 ≤ l ≤ L , M l = ˜ M l and M T l M l = α l I K . For l = 1 , W 1 , M 1 are any pair of matrices that minimize the objective conditionally on M 2 being defined as above. For l ≥ 2 , W l minimizes the objective conditional to the input and output to that layer. Finally, the parameters { α l } L l =2 satisfy

and the parameter q ≥ 0 is chosen to minimize the objective function in (1) .

̸

Next, we define the SRG solution when K = ( r 2 ) for any r , and provide two constructions, each useful for different parts of the proof of Theorem 5.

Definition 11. A strongly regular graph (SRG) solution for K ≥ 6 of the L -DUFM problem (1) , is obtained in one of the two following ways.

First, we take the largest r s.t. K ≥ ( r 2 ) and construct the SRG solution ( ¯ H 1 , ¯ W 1 , . . . , ¯ W L ) as in Definition 9 setting the number of classes to ( r 2 ) . Next, we construct a DNC solution ( ˜ H 1 , ˜ W 1 , . . . , ˜ W L ) as in Definition 10 setting the number of classes to K -( r 2 ) . Then, to construct H 1 of the SRG solution, we create it as a diagonal block matrix with number of columns equal to K and number of rows equal to ¯ d 1 + ˜ d 1 , where these are the numbers of rows of the respective H matrices; the first block is ¯ H 1 , the second block is ˜ H 1 2 , and

the off-diagonal blocks are zero matrices. Similarly, we extend the weight matrices such that, for any l , W l is a block diagonal matrix where the number of rows is ˜ d l +1 + ¯ d l +1 and the number of columns is ˜ d l + ¯ d l ; the first block is ¯ W l , the second block is ˜ W l , and the off-diagonal blocks are zero matrices.

First, we take the smallest r s.t. K ≤ ( r 2 ) and construct the SRG solution as in Definition 9 setting the number of classes to ( r 2 ) . Then, we just remove the ( r 2 ) -K columns of H 1 that achieve the highest individual fit losses (in case of a tie choose arbitrarily), and define the SRG solution as the original solution without these columns.

We recall our main result and give the proof.

Theorem 5. If K ≥ 6 , L ≥ 4 or K ≥ 10 , L = 3 and d l ≥ K for all l , then L SRG < L DNC . Moreover, consider any sequence of L -DUFM problems for which K - →∞ so that 0 . 499 > L DNC for each problem. In that case,

Proof. We start by considering the case K = ( r 2 ) for some r. Without loss of generality we can assume n = 1 , because all comparisons are between solutions that are by definition DNC1 collapsed, and the ratio L SRG / L DNC in the theorem statement does not depend on n.

We first compute the loss of the SRG solution as in Definition 9 up to only one degree of freedom. Let us go term-by-term. The simplest to evaluate is λ Wl 2 ∥ W l ∥ 2 F for 2 ≤ l ≤ L -2 . Using Lemma 13 (which relies on Lemma 12), we get

Similarly, for layer L -1 , we use Lemma 14 (again relying on Lemma 12) to compute:

Combining Lemma 15 with Lemma 18 we get:

Finally, combining Lemma 17 with Lemma 16 we get:

The total loss of the SRG solution is just the sum of all these terms, which is expressed in terms of α 2 , α 3 , . . . , α L . We now verify that the choice in (6) minimizes the loss, having set q ≡ α 1 / 2 2 . To do so, we compute the partial derivatives of L w.r.t. the α l 's and set them to 0 :

For 3 ≤ l ≤ L -2 , we have

And finally, for layer L -1 , we have

Denoting by p = √ λ W 1 λ H 1 ( 2 + √ ( r -1)( r -2) ) , we can express these fractions as

which gives the expressions in (6). Finally, plugging this back into the loss function we get a univariate q -dependent function of the following form:

The loss of the DNC solution can be computed by a simple extension of the expression (17) from [48]:

At this point, we split our analysis for L = 3 and for L > 3 . We start with L > 3 , which is simpler. Analysis for L > 3 . As K ≥ 6 , the following upper bound holds:

Now we reparametrize L DNC ( q ) and the upper bound on L SRG ( q ) , so that they look as similar as possible. By replacing λ W L -1 q with q , we get

After the reparameterization, L DNC ( q ) and ¯ L SRG ( q ) have almost the same form except for the multiplier r -1 2 in the DNC case and

in the SRG case. Therefore, the inequality between L DNC and ¯ L SRG is fully determined by the inequality between these two terms. We can write:

We first solve it for L = 4 and any r ≥ 4 (which is guaranteed by K ≥ 6 ). We get the inequality r ( r -1) 2 > 2( r -1)( r -2) + 4 + 4 √ 2( r -1)( r -2) . This inequality is equivalent to ( r -1)( r 2 -3 r +4) > 4 + 4 √ 2( r -1)( r -2) , which holds for all r ≥ 4 .

Compared to the case L = 4 , for general L the LHS gets multiplied by ( r -1) L -4 and the RHS gets multiplied by 2 L -4 , which is smaller for r ≥ 4 . Hence, the inequality holds as well.

Analysis for L = 3 . Here, we need a tighter upper bound than ¯ L SRG ( q ) . Thus, we write

We equivalently re-write this by extending both of the ratios by r -2 r -2 r -1 r -1 and then moving r -2 r -1 to denominator. Thus,

Now, we perform the same reparametrizations as for L > 4 , with the only exception of treating ( r -1)( r -2) in the denominator of the denominator of the current ratios as ( r -1) 2 in the previous case. Then, we have

Assume that the following inequality holds:

Then,

The only difference between the expression considered here and the one considered in the L > 3 is that here we have ( r -1)( r -2) instead of ( r -1) 2

case within the expression. By comparing against ( √ ) 2

It remains to show that (9) holds, and it suffices to do so for the minimizer q ∗ of L DNC , as min q ≥ 0 ˜ L SRG ( q ) ≤ ˜ L SRG ( q ∗ ) ≤ ¯ L SRG ( q ∗ ) < L DNC ( q ∗ ) = min q ≥ 0 L DNC ( q ) . Note that this is equivalent to

Note that the minimum of the function in (7) (having the same reparametrization as ¯ L SRG ) - if it is not at q ∗ = 0 , in which case the statement of the theorem is trivial - must come after the unique inflection point of the function. A direct computation yields that this inflection point satisfies

Therefore, the minimum of (7) is attained at q ∗ for which

For r ≥ 5 , this implies that such a q ∗ satisfies (9), which concludes the argument for K = ( r 2 ) .

For a general K , the extension is rather simple. Note that the first type of SRGsolution in Definition 11 is constructed in a way so that the losses attained by the SRG and DNC parts sum up. Therefore, we can split the analysis for the SRG and DNC parts. The DNC part obviously attains equal loss to the DNC solution. For the SRG part, the analysis done above applies, and the argument is complete.

It remains to show the statement on the asymptotic relationship between L SRG and L DNC for K - →∞ when L ≥ 4 . Formally, we should consider sequences of the problems and label everything with an extra index corresponding to the order within the sequence. However, with an abuse of notation, we drop this indexing and switch to the O , Θ notations whenever convenient.

As before, we start by considering K of the form ( r 2 ) for some r . Let Λ = λ H 1 ∏ L i =1 λ W i and 1

that Ψ( K ) = Θ( K 3 -L 2 L ) . Since we are interested in the ratio L SRG L DNC , we do a few changes and reparametrizations to the expressions in (7) and (8): we multiply both by 2, divide all terms in the left summands by λ W L , rewrite r ( r -1) 2 as K, plug in the defined quantities, divide all the terms in the left summands by K and finally replace Λ -1 L K -1 L q with q to obtain the following expression

for the DNC loss, and the following expression

for the SRG loss. Using a similar trick as in the previous analysis for L = 3 , we have that the minimum of the function in (10) is achieved when q > 1 . Hence, we can lower bound (10) by

For this convex expression, we can find the optimal solution by setting to zero the derivative, which gives that the optimal solution is (1 + L )3 -1 L +1 K Λ 1 L +1 . Similarly, we can upper bound (11) by

and after finding the optimal solution we get that it equals (1 + L ) K Λ 1 L +1 Ψ( K ) L L +1 . This allows us to conclude that

To get the same formula when the number of classes is not of the form ( r 2 ) , we only need simple adjustments. For this part, we will employ the upper-index notation to denote the number of classes K to which the solution corresponds. First, note that the optimal value of (10) is continuous in the coefficient in front of the linear term q. Therefore, if L K DNC < 0 . 499 , then, choosing the smallest r for which K ≤ ( r 2 ) := ¯ K , we see that L ¯ K DNC < 0 . 5 for the same set of regularization parameters, ¯ K K - → ∞

as K -- - - → 1 . Since the argument above does not need L DNC < 0 . 499 but only L DNC < 0 . 5 , we can now use that L ¯ K SRG with the same regularization parameters is still O ( K 3 -L 2( L +1) ) . Finally, choosing the second construction in Definition 11, we construct the SRG solution for K classes from the SRG solution for ¯ K classes with the same regularization parameters (thus also the same regularization as for the DNC solution with K classes). To conclude, it just suffices to see that L ¯ K SRG ≥ L K SRG because we removed columns from H 1 , decreasing its norm and the fit loss is at most as big because the columns with the worst fit loss were removed and the fit loss is an average over the columns. This concludes the proof also for general K.

We conclude the section by stating and proving a few auxiliary lemmas that were used in the proof of Theorem 5.

Lemma 12. Consider the following optimization problem:

Then, the value of the optimal solution is

Proof. Multiplying the constraint with T T r ( T r T T r ) -1 from the right we get z T T T r ( T r T T r ) -1 = w T A l . Now, we can use that the minimum l 2 norm solution of such a system can be computed by multiplying with the right pseudoinverse of A l . Thus, we get w = A l ( A T l A l ) -1 ( T r T T r ) -1 T r z = 1 /α l A l ( T r T T r ) -1 T r z. Then the squared norm of this is simply:

Now, we know that

This can be seen by looking at the structure of K n where two vertices have exactly one edge between them. Now we can compute the the inverse of this matrix using the Sherman-Morrison formula

Putting this all together, the proof is complete.

Lemma 13. Let 2 ≤ l ≤ L -2 . Consider ˜ M l +1 , M l as in Definition 9 of the SRG solution. Then, the following optimization problem:

and the square is:

achieves optimal value of rα l +1 α l .

Proof. This is a direct consequence of Lemma 12. Denote γz 0 a row from ˜ M l +1 which corresponds to the first row of T r and such that ∥ z 0 ∥ = 1 . Then, T r γz 0 = γ (1 , 1 / ( r -1) , . . . , 1 / ( r -1)) T . Directly evaluating the expression in Lemma 12 will yield γ 2 α -1 l for a single row and thus for all rows we get the value from the lemma statement.

Lemma 14. Consider ˜ M L , M L -1 as in Definition 9 of the SRG solution. Then, the following optimization problem:

Proof. We first need to characterize the rows of ˜ M L . Note that ˜ M L comes from the multiplication of T r and A (1) L , see Definition 9. This operation can be seen as weighting vertices of K r (rows of T r ) and then looking at what the sum of the weights of adjacent vertices of each edge (column of T r ) is. We can easily see that the resulting vector has exactly one ' -1 ' entry (for the edge corresponding to the two vertices given negative weight) and exactly ( r -2 2 ) entries with ' +1 '. Therefore, it must be scaled with the inverse of s := √ ( r -2)( r -3) 2 +1 to be unit norm. Now, let z 1 be one of these vectors with the negative edge between first two vertices. Then,

achieves optimal value of

which can be easily derived if we imagine doing edge-wise dot-product between two vertex weightings, one with two ' -1 s' for z 1 and the other type with one ' +1 ' representing the rows of T r . For the other vectors in ˜ H L , the resulting vectors would be similar, except they would have the negative entries for different pairs of vertices of K r . Note that

Therefore, if we are optimizing for γz 1 , then an application of Lemma 12 gives

where w denotes a row of W . Simplifying this expression, we get

To conclude, it suffices to sum up r ( r -1) 2 such rows having total l 2 norm squared α L (and, hence, γ 2 = α L ), which gives

and concludes the proof.

Lemma 15. For 2 ≤ l ≤ L -1 , consider M l , ˜ M l as in Definition 9 of the SRG solution. Then, M l = ˜ M l and the eigenvalues of M T l M l are:

Proof. From the definition, it readily follows that M l = ˜ M l , so let us compute M T l M l . Looking at any row of M l , we see that it has non-negative equal entries of value √ α ij / √ r -1 , where ∑ j α ij = α l on all the r -1 edges of K r that contain the vertex corresponding to that row. Therefore, by definition of α l , the sum of squares of all entries corresponding to one row type within any column is α l / ( r -1) . Each edge (and, thus, column) contains exactly two vertices, thus the diagonal elements of M T l M l , which are the l 2 norms squared of the columns of M l , are simply equal to 2 α l r -1 . There are two possible off-diagonal values. One is for the pairs of columns that correspond to edges that share a vertex and one is for those pairs that do not share a vertex. The pairs of columns whose edges do not share a vertex do not have any entries which would both be jointly positive, because either a vertex does not belong to one edge or to the other. Therefore, the value of off-diagonal entries corresponding to such pairs is simply 0. On the other hand, there is exactly one vertex that has non-zero values for both edges corresponding to columns whose edges do share a vertex - it is the shared vertex. Therefore the value of off-diagonal entries of this type is α L r -1 . Crucially, the structure of the off-diagonal entries is fully determined by the graph T r , because two edges in K r share a vertex if and only if they are connected in the graph T r . Therefore, M T l M l can be written as a weighted sum of I K and the adjacency matrix G r of T r , where the weight of I K simply corresponds to the size of the diagonal term and the weight of G r to the positive off-diagonal term. In conclusion, we get

As T r is a strongly regular graph with parameters ( r ( r -1) / 2 , 2( r -2) , r -2 , 4) , G r has a single eigenvalue equal to 2( r -2) , r -1 eigenvalues equal to r -4 and r ( r -3) / 2 eigenvalues equal to -2 , which concludes the proof.

Lemma 16. Consider M L as in Definition 9 of the SRG solution. Then, the eigenvalues of M T L M L are:

Proof. Let us compute M T L M L . Looking at any row of M L , we see that it has non-negative equal entries of value √ α ij /s, where s = √ ( r -2)( r -3) 2 +1 and ∑ j α ij = α L on all edges in a subgraph of K r of size r -2 . Therefore, by definition of α L , the sum of squares of all entries corresponding to one row type within any column is α L /s 2 . However, not all row types have non-zero value on any particular column. Namely, a row type will only have non-zero value on a column, if the row-type corresponds to such a subgraph of K r , which is disjoint with the edge corresponding to the column. This is because all edges outside the complete subgraph of r -2 vertices corresponding to the row type are assigned 0 in M L . Therefore, the number of row types that assign non-zero value in a particular column is equal to the number of r -2 vertex sets. This corresponds to the number of edges in K r , which is equal to ( r -2 2 ) = s 2 -1 . Thus, the diagonal elements of M T L M L , which are the l 2 norms squared of the columns of M L , are simply equal to ( s 2 -1) α L s 2 . There are two possible off-diagonal values. One is for the pairs of columns that correspond to edges that share a vertex, and one is for those pairs that don't share a vertex. Let us compute the number of row types assigning positive value to both of these columns jointly. Using the same interpretation, the columns correspond to edges and only row types that correspond to r -2 vertex subsets disjoint with them assign positive value to the column of that edge. If we want this to be satisfied for both rows jointly, we need to take the intersection of those r -2 vertex subsets, which in this case will result in an r -3 vertex subset. Thus, exactly ( r -3 2 ) row types will jointly assign a positive value. Therefore, we have value ( r -3)( r -4) α L 2 s 2 on these off-diagonal entries. For the pairs of columns that correspond to edges with disjoint vertices, the same intersection will now yield a set of vertices of size only r -4 . Therefore, the value of this off-diagonal entry is ( r -4)( r -5) α L 2 s 2 . Crucially, the structure of the off-diagonal entries is fully determined by the graph T r , because two edges in K r share a vertex if and only if they are connected in the graph T r . Therefore, M T L M L can be written as a weighted sum of 1 K 1 T K , I K and the adjacency matrix G r of T r . The weights can be determined as follows: we first subtract a multiple of 1 K 1 T K to make the smaller off-diagonal entry of M T L M L zero, then we subtract what is left of the diagonal and we take the rest to be a multiple of G r . In conclusion, we get

As T r is a strongly regular graph with parameters ( r ( r -1) / 2 , 2( r -2) , r -2 , 4) , G r has a single eigenvalue equal to 2( r -2) , r -1 eigenvalues equal to r -4 and r ( r -3) / 2 eigenvalues equal to -2 . The summation with I K only shifts all the eigenvalues. The term 1 K 1 T K has only one non-zero eigenvalue, and the eigenvector is identical to that of the eigenvector corresponding to the dominant eigenvalue of G r . This concludes the proof.

Lemma 17. Assuming DNC1, let M L be the mean matrix of the last layer. Let M L = U Σ V T be the full SVD of M L and let σ i , i ∈ [ K ] , be the singular values of M L . Then, the following optimization problem:

attains the minimum of

Proof. The proof consists in a direct computation. Let W ∗ L denote the minimizer. Computing the gradient and setting it to 0 gives that

which readily implies that

thus concluding the argument.

Lemma 18. The optimization problem

attains the minimum of √ λ A λ B ∥ C ∥ ∗ , and the minimizers are of the form A ∗ = γ A U Σ 1 / 2 R T , B ∗ = γ B R Σ 1 / 2 V T . Here, the constants γ A , γ B only depend on λ A , λ B ; U Σ V T is the SVD of C ; and R is an orthogonal matrix.

A.2 No within-class variability is still optimal

Theorem 6. The optimal solutions of the L -DUFM (1) exhibit DNC1 at layer L , i.e.,

holds for any optimal solution ( H ∗ 1 , W ∗ 1 , . . . , W ∗ L ) of the L -DUFM problem.

̸

Proof. Step 1: Reduction to n = 1 . In the first step, assume by contradiction that there exists an optimal solution of (1) with regularization parameters ( λ H 1 , λ W 1 , . . . , λ W L ) denoted as ( H ∗ 1 , W ∗ 1 , . . . , W ∗ L ) which does not exhibit deep neural collapse at layer L . This means that there exist indices c, i, j s.t. h L ci = h L cj . Let us construct two solutions of the n = 1 L -DUFM. They will share the weight matrices which will equal ( W ∗ 1 , . . . , W ∗ L ) - the weight matrices of the original solution. To construct the features, for every class except the c -th, pick any sample and share it between both solutions. For the class c, take the samples h L ci , h L cj and put one in one solution and the other one in the other solution. Denote H (1) 1 , H (2) 1 the two n = 1 sample matrices. It is not hard to see that both ( H (1) 1 , W ∗ 1 , . . . , W ∗ L ) and ( H (2) 1 , W ∗ 1 , . . . , W ∗ L ) are optimal solutions of (1) with regularization parameters ( nλ H 1 , λ W 1 , . . . , λ W L ) . To prove it, assume by contradiction that, without loss of generality, ( H (2) 1 , W ∗ 1 , . . . , W ∗ L ) is not an optimal solution of the corresponding problem. Then, there exists an alternative ( H (0) 1 , ˆ W ∗ 1 , . . . , ˆ W ∗ L ) that achieves smaller loss for this problem. Let us duplicate all the samples of ˆ H (0) 1 for n times, thus constructing ˆ H ∗ 1 = H (0) 1 ⊗ 1 T n . The solution ( ˆ H ∗ 1 , ˆ W ∗ 1 , . . . , ˆ W ∗ L ) has the same loss under the L -DUFM problem with regularization parameters ( λ H 1 , λ W 1 , . . . , λ W L ) as the solution ( H (0) 1 , ˆ W ∗ 1 , . . . , ˆ W ∗ L ) for the L -DUFM with n = 1 and parameters ( nλ H 1 , λ W 1 , . . . , λ W L ) . This is easy to see from the separability of both ∥ H 1 ∥ 2 F and the fit part of the loss in (1) w.r.t. the columns of H 1 . For the same reasons, the loss

functions for ( H ∗ 1 , W ∗ 1 , . . . , W ∗ L ) in the original problem equals the loss function of the solutions ( H (1) 1 , W ∗ 1 , . . . , W ∗ L ) or ( H (2) 1 , W ∗ 1 , . . . , W ∗ L ) in the reduced problem. In fact, if this was not the case, there would need to be an inequality between the losses exhibited by two different columns of H 1 belonging to the same class, from which we could arrive at a contradiction by taking the better column and multiplying it to all columns within that class, thereby obtaining a better solution. This means that the loss of ( ˆ H ∗ 1 , ˆ W ∗ 1 , . . . , ˆ W ∗ L ) in the original problem is smaller than the loss of ( H ∗ 1 , W ∗ 1 , . . . , W ∗ L ) , which is a contradiction.

Step 2: Excluding an aligned case. By assumption we know that not only H (1) 1 and H (2) 1 differ in the c -th column (from now on we assume without loss of generality that it is the first column) but also H (1) L and H (2) L do. Denote for simplicity the first (differing) columns of H (1) L and H (2) L as x, y respectively. We now show that it is not possible that y = αx. First, α has to be non-negative since x, y are entry-wise non-negative given that they come after the application of σ. Assume w.l.o.g. α > 1 (otherwise, we can just exchange the roles of x and y ). Consider a reduced problem where we only optimize for the size of the first column of either H (1) L or H (2) L , focusing on that part of the problem (1) which is relevant for this column, being:

This problem is strongly convex, quadratic and simple enough to give the following: if α > 1 , then ∥ ∥ ∥ h 1(1) 11 ∥ ∥ ∥ 2 2 > ∥ ∥ ∥ h 1(2) 11 ∥ ∥ ∥ 2 2 and simultaneously ∥ W L x -e 1 ∥ 2 2 > ∥ W L y -e 1 ∥ 2 2 . This means that H (2) 1 is a strictly better solution than H (1) 1 - a contradiction.

Step 3: Contradiction by zero gradient condition. By optimality of both solutions we get

An application of the chain rule gives

where ˜ H L +1 is the output of our model. Plugging this back to the previous equation and using that W ∗ L is the same in both expressions, we get

Let us denote

Due to the separability of L F with respect to the columns of H 1 (and, thus, also of H l , ˜ H l for all l ≤ L + 1 ), we get that the matrices A,B can only differ in their first columns and are identical otherwise. We denote these columns a, b for A,B, respectively. This implies that ax T = by T .

̸

Now we exclude a few cases. First, neither a nor b can be zero, because by the exact formula that exists for them, this would mean that exact fit was achieved for either of the columns. This is impossible with non-zero weight-decay, because decreasing the norm of the column of H (1) 1 or H (2) 1 that achieves the exact fit by a sufficiently small value would necessarily lead to an improvement on the objective value. Moreover, if x = y = 0 , then this is a contradiction with the assumption that x = y. Finally, the case x = y = 0 or 0 = x = y is excluded already in the step 2 .

̸

̸

Thus we get x, y, a, b are all non-zero. Looking at any fixed row (column) of ax T and by T we see that necessarily x, y ( a, b ) are aligned. However, this case is already solved in step 2 and leads to x = y, which is the contradiction. This concludes the proof.

Theorem 8. Denote by L -DUFM ϵ the equivalent of (1) , with σ replaced by σ ϵ . Let D = max { d 2 , d 3 , . . . , d L } and ¯ λ = λ H 1 λ W 1 . . . λ W L , with the regularization parameters upper bounded by 1 / ( L +1) . Then, for any globally optimal solution of the L -DUFM ϵ problem, the distance between any two feature vectors of the same class in any layer is at most

Proof. In order not to mix approximation technical details with the gist of the proof, we split the argument into two parts: in the first part, we provide a heuristic for ϵ = 0 by assuming that ReLU is differentiable at 0; in the second part, we discuss what changes in the proof if we use the relaxation and then execute a technical computation that bounds the error based on the strength of the approximation (the size of ϵ ). It is useful to split the loss function L into the two terms L F and L R . The former represents the fit part of the loss (which penalizes for deviation of the predictions from Y ), and the latter represents the regularization part of the loss.

̸

Part 1: Heuristic for ϵ = 0 . We start identically to the proof of Theorem 6. As in Step 1 of that argument, if we have a globally optimal solution that does not exhibit DNC1 in the first layer, we can construct two different solutions of the n = 1 L -DUFM problem where the two solutions only differ in the first column of the H 1 matrix. Let us denote two such constructions H x 1 = H y 1 , where we denote x = y, to be the mentioned first columns, respectively. We emphasize that H x 1 , H y 1 both form optimal solutions of the n = 1 L -DUFM with the same tuple of weight matrices ( W ∗ 1 , . . . , W ∗ L ) . In particular, this means that

̸

An application of the chain rule gives

Plugging this back to the previous equation and using that the W ∗ 1 is the same in both expressions, we get

Let us denote

.

Due to the separability of L F with respect to the columns of H 1 (and, thus, also H l for all l ≤ L ), we get that the matrices A,B can only differ in their first columns and are identical otherwise. We denote these columns a, b for A,B, respectively. This implies that ax T = by T .

Now, we treat a few cases. First, assume a = b = 0 . Since it holds that ( H x 1 , W ∗ 1 , . . . , W ∗ L ) is the optimal solution, then necessarily

̸

Similarly, we get y = 0 , but that is a contradiction with x = y. Therefore, at least one of a, b is non-zero. Next assume x = 0 . Then ax T = 0 . If b = 0 then y = 0 and we have a contradiction. On the other hand, if b = 0 , then the row of by T that corresponds to a non-zero entry of b must be zero and thus y = 0 . Similarly, if y = 0 we can get x = 0 .

̸

̸

Let us therefore assume x, y are both non-zero, which also implies a, b are both non-zero. Looking at any fixed row (column) of ax T and by T we see that necessarily x, y ( a, b ) are aligned. Let us write x = αy and αa = b for some α = 0 . We will first show that if α > 0 then necessarily α = 1 and x = y. For this, let us fix any ray r ∈ R d 1 . The ray r represents a set of possible first columns in

H 1 . Let us fix any ( W 1 , . . . , W L ) . Since L is separable in the columns of H 1 , we can consider an optimization over β ≥ 0 to minimize L on ( βr, W 1 , . . . , W L ) . However, L F is convex by assumption and the mapping h 1 11 - → h L 11 is ray-linear, therefore L F is convex in β. Moreover L R is strongly convex in h 1 11 and therefore L is strongly convex in β. This means it has a unique optimal solution β ∗ .

We have just showed that, if α > 0 , then since x, y are aligned and thus lie on the same ray and are both optimal together with the same tuple of weight matrices, they must necessarily be identical and so α = 1 , x = y. This is a contradiction and thus we are left with the case α < 0 . Denote s = W ∗ 1 x , t = W ∗ 1 y. From linearity, we have s = αt. Note that if s = t = 0 , then necessarily x = y = 0 because they don't have any effect on L F and they minimize L at 0 . Thus, s, t are non-zero and negative entries of s are positive entries of t and vice-versa. This means that σ ( s ) and σ ( t ) have different sets of positive entries. However, from the optimality of the both solutions we know:

Again, using the chain rule we get:

Plugging this back to the previous equation and using that W ∗ 2 is the same in both expressions, we get

Let us denote

Due to the separability of L F with respect to the columns of H 1 (and, thus, also H l for all l ≤ L ), we get that the matrices C, D can only differ in their first columns and are identical otherwise. We denote these columns c, d for C, D respectively. This implies that cσ ( s ) T = dσ ( t ) T .

As above, if either c or d is zero, using the chain rule we would get that x or y (respectively) are zero too, which cannot happen. Therefore, both c and d are non-zero and σ ( s ) , σ ( t ) must be aligned. Since they are non-zero, non-negative and with different supports, we have reached a contradiction. This proves that α < 0 is also impossible and the only possible case is α = 1 , but that forces x = y, which is also a contradiction.

Part 2: Relaxation to ReLU ϵ . The difficulty in making the previous heuristic rigorous is that ReLU is not differentiable at 0 and, thus, we do not have the desired analytical statement that global solutions must necessarily admit zero derivative (because the loss might not be differentiable at them at all). If we tried to use the same proof as in part 1 with the differentiable relaxed version of ReLU ReLU ϵ , an issue would occur when showing that, if x, y are aligned and α ≥ 0 , then α must be 1. The reason is that the mapping h 1 11 - → h L 11 is no longer ray-linear and the corresponding optimization problem on any fixed ray is no longer quadratic and strongly convex. However, we can prove that the optimization problem admits a solution that is close to the solution of the corresponding ReLU optimization problem. For this, let us fix any direction r such that ∥ r ∥ = 1 and an optimization parameter t ≥ 0 , and define the following two losses:

We now bound max t ≥ 0 | L 0 ( t, r ) -L ϵ ( t, r ) | . For this, we fix any t ≥ 0 , and bound the accumulated l 2 error of using σ ϵ instead of σ throughout the layers. For this, a useful statement that we will need is the following:

This is true because, by a simple computation, we get that the terms must be balanced and the inequality comes from the fact that the solution (0 , 0 , . . . , 0) achieves full loss 1 / 2 in the L -DUFM as well as L -DUFM ϵ problems and, thus, L R is trivially upper-bounded by this. This implies that, for each W l (and H 1 ), ∥ W l ∥ ≤ ∥ W l ∥ F ≤ 1 √ ( L +1) λ Wl .

For ease of exposition, we set t = 1 (the same argument would work for all t ). We see that W 1 r does not introduce any l 2 error. Then, the l 2 error that is introduced in the application of the ReLU is trivially upper-bounded by ϵ √ d 2 . After applying W 2 , we can use the bound on the operator norm ∥ W 2 ∥ obtained above to upper bound the propagated l 2 error by

After that, we obtain an additive l 2 error of ϵ √ d 3 by applying σ ϵ , which gives a total l 2 error of

Then, again, we multiply this whole expression with 1 √ ( L +1) λ W 3 to account for the multiplication by W 3 . Inductively, the upper bound on the l 2 error in the output space is

This can be further upper bounded as

Using the triangle inequality, the upper bound on | L 0 ( t, r ) -L ϵ ( t, r ) | is this expression squared. On the other hand, the second derivative of L 0 ( t, r ) with respect to t is lower bounded by nλ H 1 .

Given two functions f 1 and f 2 , with f 1 strongly convex with second derivative at least c and f 2 everywhere at most d distant from f 1 , then the distance between their global minimizers is at most 2 √ d/c . Applying this to our case, we get that the distance between the minimizers t 0 and t ϵ of L 0 ( t, r ) and L ϵ ( t, r ) is at most

Since the ray r is unit-norm, this is also the upper bound on the distance between two feature vectors of any globally optimal solution in the first layer. To obtain the upper bound on the distance between two vectors in any layer, we proceed as follows.

Assume we have two input vectors of the first class (now we know they need to be aligned): x 1 , y 1 = α 1 x 1 , where α 1 > 1 . As before, ˜ x l , x l , ˜ y l , y l are the l -th layer representations of these vectors before and after σ ϵ . If we would compute ∂ L ∂W l with respect to any layer l ≥ 2 and used the same arguments as in part 1 (that still carry to the analysis for σ ϵ ), we would find out that in each layer, we need to have y l = α l x l . Moreover, we can assume that ˜ x l , x l , ˜ y l , y l are non-zero at all layers because otherwise the same argument in part 1 would trivialize the rest of the proof. By a simple inductive argument, since α 1 > 1 , we know that α 2 > 1 as well because σ ϵ is strictly increasing on [0 , ∞ ) . Similarly, α l > 1 for all l. Note that, if there exists at least one index i on which x l i is bigger or equal than ϵ, then necessarily α l = α l -1 , because σ ϵ is the identity on inputs of at least ϵ , and the α l can be uniquely determined from y l i /x l i = y l -1 i /x l -1 i . Having α l = α l -1 makes us have more control over the distance between x l and y l . If this fails to hold at some layer, we need a separate analysis.

For this, let l 0 denote the first layer (assuming it exists), where ˜ x l 0 does not have any entries that are bigger or equal than ϵ. This means that ∥ ∥ x l 0 ∥ ∥ ≤ ϵ √ d l 0 . We now compute the maximal possible norm of y l 0 . Since α 1 = α 2 = · · · = α l 0 -1 , for all 2 ≤ l < l 0 and for each index i, either ˜ x l i , ˜ y l i ≤ ϵ or ϵ ≤ ˜ x l i , ˜ y l i . Otherwise, since σ ϵ ( x ) < x ∀ x ∈ (0 , ϵ ) , y l i /x l i > α l , a contradiction. Thus, we get

∥ ∥ ˜ x l -˜ y l ∥ ∥ ≥ ∥ ∥ x l -y l ∥ ∥ . Therefore, ∥ ∥ x l -y l ∥ ∥ can only grow by applying W l . Thus, after applying W 1 , W 2 , . . . , W l 0 -1 and using the operator norm bound with all of the matrices, we get

Combining with ∥ ∥ x l 0 ∥ ∥ ≤ ϵ √ d l 0 and using the triangle inequality, we get

Applying the operator norm bound with W l 0 , W l 0 +1 , . . . , W L -1 , we get:

Since ∥ ∥ x L ∥ ∥ ≤ ∥ ∥ y L ∥ ∥ we can use the triangle inequality again to obtain the final:

Note that the same bound is an upper bound for any layer, since in each layer we are increasing the upper bound by at least a factor ( L +1) -1 / 2 λ -1 / 2 W l .

In case such a layer l 0 does not exist, the upper bound above is still valid, since we can simply use the error propagation through all the layers, as we did for l < l 0 in the previous case. We would arrive at the same bound as above but with 4 instead of 6 as a multiplicative factor (because we would not introduce the additive error ϵ √ d l 0 ).

B Further experimental evidence

B.1 L -DUFM experiments

Experiments on small number of classes and layers. For L = 3 , we weren't able to find any solutions that would outperform DNC for K ≤ 6 . For K = 7 , the difference in the losses is extremely small. For K = 6 , we did find some low rank solutions, but these were either slightly worse than DNC or the losses were the same up to four decimal places. Already for L ≥ 4 , we find low-rank solutions for all K ≥ 3 . We present some of them for K ∈ { 3 , 4 , 5 } in Figure 5. Note, that all these solutions were found automatically by gradient descent through the optimization process.

We specifically highlight the solution for K = 4 , as it represents another graph structure different from SRG. Namely, this solution is based on the square graph. We see that each column (edge) has positive scalar product with exactly two other columns (the corresponding edges share a vertex), and it is orthogonal to one other column (the non-touching sides of a square). Each row (vertex) has only two positive values, and these correspond to the two sides (edges) to which they belong.

̸

Experiments on large number of classes or layers. For large L or K, the behavior of the low-rank solutions is slightly different from the behavior when both L and K are moderate. When l is close to L , it no longer holds that ˜ M l = M l , and the rank of M l is larger as the singular values do not decay sharply to 0 . However, we note that the singular values are rather small and, thus, it is not clear whether they would disappear with a longer training (we trained for 200000 steps of full gradient descent with learning rate 0.5). In Figure 6, we take L = 7 , K = 15 , and weight decay 0 . 0025 in all layers. We highlight that ˜ M 6 = M 6 and the singular values of M 6 are small but non-zero after the 3 dominant values. We also highlight that the rank of previous layers is 3, which is remarkably small (and also smaller than the rank of the SRG solution).

Log losses step

0.0-

total_loss

fit_loss

2.5

0.0

0.35

0.40

0.0

2.5

0.4

0.3

0.5

0.0

0.5

DNC loss

5.0

0.30

5.0

0.2

0.4

l0g10 l0ss

0.5

7.5

0.25

7.5

0.1

0.5

0.3

1.0

10.0

0.20

10.0

0.0

1.0-

12.5

0.15

12.5

0.1

1.5

0.2

1.5

15.0

0.10

15.0

-0.2

2.0 -

17.5

0.05

17.5

0.3

2.0

0.1

0

200

400

600

800

1000

0.02.5

0.00

0.02.5

0.4

2.5

-0.50.00.51.01.52.02.5

0.0

step

SS01OT601

0.5

0.0

Log losses

total_loss

fit_loss

DNC loss

2.5

5.0

7.5

0.0

0.35

0.30

0.25

0.20

0.0

2.5

5.0

7.5

0.3

0.2

0.1

0.4

0.5

0.0

1.0

0.5

0.30

0.25

0.20

-1.0

10.0

0.15

10.0

0.0

1.5

0.15

1.5

15.0

12.5

0.10

15.0

12.5

0.2

0.1

2.0

2.5

0.10

2.0

17.5

0.05

17.5

-0.3

-0.4

3.0

0.05

0

200

400

600

800

1000

0.02.5

0.00

0.02.5

3.5

0

0.00

step

Log losses

0.0

0.0

0.6

0.0

total_loss

fit_loss

2.5

0.5

2.5

0.4

0

0.6

SSO1

0.5-

DNC loss

7.5

5.0

0.4

5.0

7.5

0.2

1

0.4

0.5

10T601

-1.0

10.0

0.3

10.0

0.0

2

0.3

1.5

12.5

0.2

12.5

-0.2

3

0.2

2.0

15.0

0.1

15.0

0.4

0.1

17.5

17.5

0

200

400

600

800

1000

0.02.5

0.0

0.02.5

-0.6

0.0

SRG solutions. In Figure 1 presented in the main body, we recover the SRG solution for K = 10 . In Figures 7 and 8, we show that solutions very similar to SRG are recovered for K = 6 and K = 15 , respectively. The only difference with SRG is in the construction of ˜ M L . We note that the losses of these solutions are slightly lower than the loss of our construction, which proves that the SRG solution itself is not necessarily globally optimal.

B.2 End-to-end experiments with DUFM-like regularization

We complement the experiments of Figure 3 with two extra ablation studies for ResNet20 trained on CIFAR-10 with a 4-layer MLP head. We focus on the dependence of the average rank on the weight decay and the learning rate, and present the results in Figure 9. The weight decay has a clear effect on the rank of the solutions found by gradient descent, similarly to the results in Figure 3 for the L -DUFM model. The effect of the learning rate is slightly less clear, but we still see a general downward trend.

B.3 End-to-end experiments with standard regularization

Finally, in Figure 10, we include the analysis of the average rank as a function of weight decay in the standard regularization setting for training on the MNIST dataset. The results confirm the trend from the previous experimental settings, showing that the weight decay strength is a crucial predictor of the final rank even in standard regularization setting, which has a different loss landscape compared to the L -DUFM.

Singular values of H_2

0.12

0

1.6

0.7

三

1.5

1.4

0.6

10

0.10

10

1.0

10

1.2

0.5

0.08

20

20

0.5

20

1.0

0.4

0.06

0.8

0.3

30

30

0.0

30

0.6

0.04

0.2

0.5

0.4

0.1

40

0.02

40

40

0.2

0.0

1.0

0.00

0.0

0

2

4

6

8

10

12

14

0

10

0

10

0

10

index

Singularvalues of H_6

Singular values of sigma_H_6

7

7

6

5

5

4

4

3

3

2

2

1

1

0

0

0

2

4

6

8

10

12

14

0

2

4

6

8

10

12

14

index index

0.35

0.30

0.30

0.4

5

0.25

5

1

0.25

0.2

10

0.20

10

2

0.20

15

0.15

15

0.0

3

0.15

0.10

20

0.10

20

0.2

4

0.05

25

0.05

25

5

-0.4

0.00

0.00

2

3

4

5

0

5

0

5

Singular values of H_3

Log losses

1.0

0.0

total_loss

fit_loss

0.8

DNCloss

-0.5

l0g10 l0ss

0.6

1.0 -

0.4

1.5

0.2

2.0

0.0

0

1

2

3

4

0

200

400

600

800

1000

index

step

0

0.4

0.35

口

0.4

0.3

10

0.30

10

4

0.2

0.3

0.25

6

20

20

0.1

0.20

0.0

8

0.2

30

0.15

30

-0.1

10

0.10

-0.2

12

0.1

40

40

-0.3

0.05

14

-0.4

0.0

0.00

0.02.55.07.510.012.5

0

10

0

10

Loglosses

0.0

total_loss

fit_loss

-0.5

........

theoretical_optimum

log10l0ss

-1.0

1.5

2.0

-2.5

0

200

400

600

800

1000

step

20

average rank

15

10

5

-26

-24

-22

-20

-18

log2 weight decay

10

9

rank

8

average

7

6

5

-10

-8

9-

-4

log2learningrate

10

9

rank

8

average

7

6

5

-11

-10

-9

-8

-7

log2 weight decay

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of the work performed by the authors.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provides the full set of assumptions and complete proofs in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper includes enough details so as to perform the same kind of experiments as the ones presented in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The contribution of our paper is mostly theoretical. The experiments are for illustrative purposes. The source code will be provided on request.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [No]

Justification: We do provide most of the experimental details in the manuscript. Additional (minor) parameter choices can be derived from the source code, which is available on request.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: The experiments do not require any specific hardware setup.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: no societal impact of the work performed

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provide references for the datasets used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the optimality of neural collapse in intermediate layers of a deep neural network on classification tasks with more than 2 classes.\n\n1. The paper shows that the NC2 property (formation of a simplex ETF/orthogonal frame) in the intermediate layer features is not optimal, and constructs a strongly regular graph (SRG) based lower-rank solution leading to lower loss than previously proposed deep neural collapse (DNC) optimal solutions.\n\n2. The NC1 property (variability collapse of features) is still shown to be optimal in these settings.\n\nSTRENGTHS:\nThe key contribution of this paper is the construction of a strongly regular graph (SRG) based lower-rank solution for a generic $L > 2$ layer neural network trained on classification tasks with $K > 2$ classes. The authors show that this construction leads to lower training loss than the deep neural collapse-based solution.\n\nWEAKNESSES:\n1. There seems to be an error in the Sherman-Morrison formula-based inverse in the final result of Lemma 12, i.e, $(I + \\frac{1}{r-1}11^\\top)^{-1} = I - \\frac{1}{2(r-1)}11^\\top$. (The authors can correct me on this). Since this result is used in Lemma 13 and affects the results in Theorem 5, verifying this result is important.\n\n2. Can Definition 4 be split into multiple parts for better readability? For instance, split it into parts w.r.t:\n\n- collapsed class means are unaffected by non-linearity.\n- The class means are scaled versions of $T_r$ where each row is scaled individually.\n- Choice of $A_L$ for the final layer.\n\nThis breakdown highlights a major assumption that the non-linearity is not in effect till the last layer. Additionally, the complete set of trainable and non-trainable parameters should be made clear. The full definition in Appendix A.1 mentions extra trainable terms corresponding to $q$ terms.\n\n3. In section 6.1 (DUFM training) Figure 2 (first row, last column), the plots for the singular values of class means seem to have a rank of around 7 or 8 for intermediate layers as well. The ideal rank $K=10$ might not exactly hold for intermediate layers but these observations should be discussed since it is one of the main claims in the paper. \n\n4. The experimental results in section 6.3 for MNIST datasets claim that the network trained using gradient descent is the SRG solution. A justification for this observation is missing. The heatmap for MNIST in Figure 4 corresponds to $M_5$ which as per notation indicates the class-means of the penultimate layer post-activations. However, shouldn't the SRG solution correspond to similar heat maps across intermediate layers? Also, a statement such as low-rank implying an SRG solution is not justified.\n\nQUESTIONS:\n1. Do numerical experiments with the 5-layer MLP head include a ReLU activation for L-DUFM and end-end training?\n\n2. The authors mention in section 6.3 that \"We also observe that the rank deficiency is the strongest in the mid-layer of the MLP head\". Can further justification be provided for this? Especially which aspect of the spectrum of Layer 3 or 4 in Figure 5 justifies this observation? \n\n3. Since the SRG solution relies on feature class means being non-negative, how valid is this assumption (based on your practical experiments)?\n\nnit: In the notation for $H_L = \\sigma(W_{L-1}X_{L-1})$ in section 3, shouldn't $X_{L-1}$ be replaced with $H_{L-1}$ ? The same applies to $\\widetilde{X}_l$.\n\nnit: Please fix the formula for NC1 in the first line of section 6. The pseudo-inverse is not required when dividing the traces.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDeep Neural Collapse (DNC) refers the the neural collapse phenomenon that has been observed on intermediate layers of a deep network (nb. whereas neural collapse (NC) focuses only on the penultimate layer). Similar to Unconstrained Feature Model (UFM) for analyzing NC, Deep UFM (DUFM) is the comparable framework to study DNC. However, as opposed to UFM for NC, it's been observed that once you go beyond two layers or two classes, DNC is no longer optimal for DUFM. \n\nIn this paper, the authors suggest that the reason for this is a low-rank bias of commonly used multi-layer regularization schemes, meaning the low-rank bias leads to optimal solutions that are lower rank than the neural collapse. \n\nThis paper extends the current research by looking at DUFM when there are more than two classes and the model is non-linear. Previous work has focused only on DUFM for binary classification with non-linear models or deep linear models or shallow models with multiple classes.\n\nMost notably, they show that in this more complicated setting, DNC is not optimal wrt DUFM. Namely, the class means fail one of the original conditions of NC; i.e., they fail to form an orthogonal frame. The authors suggest that the reason for this is the low-rank bias under L2 regularization. It is worth noting that the condition DFC1 (which states within class variability goes to zero) is still optimal and it's only the DNC2 property (which states that class means are orthogonal) that conflicts with the low-rank bias.\n\nTheir findings are supported with strong theoretical proofs and empirically on benchmark datasets such as MNIST and CIFAR10.\n\nOne of the main takeaways from this paper is that…if a DNC solution is found, it is not because of DUFM or global optimality. Instead, it is because of an implicit bias of the optimization procedure.\n\nSTRENGTHS:\nThe literature review is excellent and they do a great job motivating the problem they aim to address and how it compares with the existing literature. \nThe mathematical notation is clear and precise. The theory itself is very well developed with careful and detailed proofs. \nThe experiments they cover are solid and in agreement with their theory.\n\nThe authors pose some interesting open questions for future work related to the DNC1 condition.\n\nWEAKNESSES:\nBased on my understanding of the current work, I don't see any major weaknesses. I have some minor points of clarification or questions below but no major flaws.\n\nQUESTIONS:\nline 209 (clarification): Does the statement regarding K=2 or L=2 indicate that the argument of your theorem also be used to show DNC is optimal in this setting as shown in [51, 48] or are you just saying that as comparison when K=2 or L=2 by [51, 48] we know DNC is in fact optimal?\n\nline 343: why the MSE loss? I didn't catch that part.\n\n(typos/nits)\nline 206 (typo): according to Theorem 5, should L \\geq 3 be replaced with L = 3?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper theoretically explores the deep neural collapse (DNC) phenomenon within non-linear deep models for multi-class classification. Neural collapse (NC) is a phenomenon in deep overparameterized networks where, the last layer's feature vectors align with class means and their corresponding classifier vectors, while also maximizing separation between classes. Previous works have studied the propagation of this symmetric geometry to earlier layers and proved its optimality which is termed as DNC. These works focus on DNC's optimality in multi-class linear networks or binary non-linear networks. This paper, however, extends the analysis to multi-class non-linear deep networks with ReLU activation, using the deep unconstrained features model (DUFM), where input features are treated as unconstrained parameters in the training objective.\n\nWhile in the binary or linear case, DNC was shown to be optimal across all layers of DUFM, the authors show that in the multi-class case and in the presence of non-linearity, DNC is not always optimal. They prove this by explicitly constructing a solution, called SRG, that achieves a lower objective value than DNC. They show SRG is rank-deficit and the rank/loss gap between SRG and DNC increases as the network becomes deeper or the number of classes increases. They further numerically show on DUFM that, although DNC is not optimal, wider networks have an implicit bias toward finding the DNC solution. To further support their findings, they conduct additional experiments on benchmark datasets MNIST and CIFAR10 and ResNet architecture.\n\nSTRENGTHS:\nThis paper closes a gap in the theoretical analysis of the NC literature. The result is interesting as it challenges the optimality of DNC established in previous works and the accuracy of the abstract UFM for modeling the dynamics of deep overparameterized models.\n\nWEAKNESSES:\nThe proof of non-optimality of DNC in the paper relies on constructing a solution (SRG) that has a lower objective value. However, SRG might not be optimal. So, showing that SRG has a lower rank does not necessarily mean that the optimal solution also follows this low-rank structure. Still, the results and observations presented in this work remain interesting.\n\nQUESTIONS:\n1. As discussed in the paper, the last-layer feature vectors before the activation ($\\tilde{M}_L$), have a lower rank, and after activation ($M_L$) recover the full rank structure of NC. This is also clear from Fig. 2 (right). However, what I find missing is whether after activation $M_L$ follows the NC geometry or not, i.e., $M_L \\propto I_K$? Or does it follow a different rank $K$ structure?\n\n2. Line 263 mentions that only for a few runs the training solutions match SRG. In the rest of the runs, do the solutions achieve a lower objective value than SRG (thus proving that SRG is a local optimal at best)?\n\n3. What about $W_\\ell$'s? Do they also exhibit a similar low-rank structure in the SRG solution as well as the numerical experiments? \n\n    I also don't see why DNC3 is not well-defined in this setup (line 250). Still, at optimality, we might have the mean feature vectors $M_\\ell$ and classifiers $W_\\ell$ align in a low-rank structure. This alignment property doesn't hold in either the SRG or SGD solutions in the experiments?\n\n4. Is there a small typo in Fig. 4 caption? The bottom right heatmap cannot be $M_5$ since the features after ReLU are non-negative and the heatmap cannot have negative values.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
Papyan et. al that showed that at the terminal phase of training there are four phenomena called neural collapse on the last layer of a deep neural network architecture. The authors extended Papyan's work by considering the earlier layers of nonlinear deep neural networks. They proved the first neural collapse condition (convergence to class means) is optimal for DNNs (Theorem 6 in manuscript). They also proved the second neural collapse condition (within-class variability) does not hold if number of layers and classes are high enough by showing there exists a scenario with better performance (Theorem 5). The claims are also supported by the experiments.

STRENGTHS:
Authors extended our understanding of how deep neural networks work by analyzing the neural collapse for deep neural networks.

WEAKNESSES:
The results only hold under certain assumptions such as using multi-layer regularization.

QUESTIONS:
-

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13