The Persistence of Neural Collapse Despite Low-Rank Bias
Authors: Connall Garrod
Jonathan P. Keating
Venue: NeurIPS 2025 poster ()
Keywords: Neural collapse
low-rank bias
unconstrained feature models
loss landscape analysis
deep learning theory
optimization dynamics
Primary area: theory
TL;DR: We analytically show that deep neural collapse is suboptimal in deep cross-entropy unconstrained feature models and explain why it persists empirically despite this.

Abstract: Neural collapse (NC) and its multi-layer variant, deep neural collapse (DNC), describe a structured geometry that occurs in the features and weights of trained deep networks. Recent theoretical work by Sukenik et al. using a deep unconstrained feature model (UFM) suggests that DNC is suboptimal under mean squared error (MSE) loss. They heuristically argue that this is due to low-rank bias induced by L2 regularization. In this work, we extend this result to deep UFMs trained with cross-entropy loss, showing that high-rank structures—including DNC—are not generally optimal. We characterize the associated low-rank bias, proving a fixed bound on the number of non-negligible singular values at global minima as network depth increases. We further analyze the loss surface, demonstrating that DNC is more prevalent in the landscape than other critical configurations, which we argue explains its frequent empirical appearance. Our results are validated through experiments in deep UFMs and deep neural networks.

The Persistence of Neural Collapse Despite Low- Row-Rank Bias
Connall Garrod
Jonathan P. Keating Mathematical Institute University of Oxford
Mathematical Institute
University of Oxford
connall. garroddmaths....mmomommooo ox ac ac.uk. uk
Abstract
Neural collapse (NC) and its multi-layes variant, deep neural collapse (DNC), de- scribe a structured geometry that occurs in the features and weights of trained deep networks. Recent theoretical work by Sukenik et al. using deep unconstrained feature model (UFM) suggests that DNC is suboptimal under mean squared error (MSE) loss. They heuristically argue that this is due to low-rank bias induced by L2 regularization. In this work, we extend this result to deep UFMs trained with cross-entropy loss, showing that high-rank structures-incuuuee not generally optimal. We characterize the associated low-rank bias, proving a fixed bound on the number of non-negligible singular values global minima network depth increases. We further analyze the loss surface, demonstrating that DNC is more prevalent in the landscape than other critical configurations, which we argue explains its frequent empirical appearance. Our results are validated through experiments in deep UFMs and deep neural networks.
DNC-are are
Introduction
Many researchers have explored the geometric structures that emerge in well-trained deep neural networks (DNNs) applied classification tasks [35 11 36). Among these structures is neural collapse (NC) [36. which refers to the tendency of overparameterized neural networks to form a simplex equiangular tight frame (ETF) in their final-layer features and weights. Researchers have sought to explain the ubiquity of NC (24). A common approach involves the unconstrained feature model (UFM) 32 which treats the last-layer features as optimization variables and optimizes them jointly with the last-layer weights, effectively abstracting away the rest of the network. Within this
framework, NC has been shown be global optimum in various settings [32 21, and the associated
optimization function has been characterized as a strict saddle function 58 56 52 57].
While NC originally observed in the last layer, researchers have empirically verified that similar structure emerges in earlier layers the network 133 39 - phenomenon known as deep neural collapse (DNC). Theoretical investigations of DNC have been facilitated by extension of the UFM, referred to as the deep UFM, which isolates multiple layers from the network's 's approximated component. While DNC has been shown to be optimal in variety of restricted contexts 66 45| 43), a recent study by Sukenik et al. 42] demonstrated that in the ReLU MSE case, DNC is generally not optimal in the deep UFM. They argue that suboptimality arises due low-rank bias induced by L2 regularization. This is illustrated by constructing lower-rank solutions that achieve lower loss than any DNC configuration.
While this finding is significant, Sukenik et al. do not analyze whether DNC or their low-rank solutions are local minima. Consequently, the interplay between nonlinearity, the loss function, and low-rank bias remains poorly understood. Their work additionally raises a central question: why does DNC often emerge in practice despite being suboptimal? Sukenik et al. offer some empirical
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

evidence- DNC appears more often with increased width or reduced regularization but no theory yet explains its empirical persistence.
To address this question, we examine a deep UFM with cross-entropy (CE) loss and linear layers. This model strikes balance between analytical tractability and sophistication: the linear layers facilitate straightforward analysis, while the model remains sufficiently rich to admit low-rank solutions that outperform DNC. While Sukenik et al. 44] demonstrate the suboptimality of DNC by presenting an isolated better solution, we provide a complete theoretical characterization of how low-rank bias manifests across the loss landscape and why leads to the systematic suboptimality of DNC. Specifically:
We prove that any output matrix with rank higher than necessary to fit the data cannot be optimal when the number of separated layers L is sufficiently large. This provides a global classification of how low-rank bias shapes the loss surface.
We show that global minima must exhibit approximate low-rankness, with all but a small number of singular values decaying exponentially in L. This demonstrates how low-rank bias governs the structure of optimal solutions.
We establish that DNC, although suboptimal for general L, remains a critical point with a positive semi-definite Hessian when the regularization level is small. This explains why local search algorithms can converge to DNC solutions. It also contrasts with the L 1 case [56 58), where the loss surface is strict saddle function. This demonstrates that the fundamental geometry of the optimization landscape changes due low-rank bias.
We analyze the role of network width d and show that the dimensionality of the solution space associated with DNC grows more rapidly than that of lower-rank solutions. This suggests that width significantly influences the prevalence of DNC in the loss landscape, offering a heuristic explanation for why local optimization methods frequently converges to DNC despite its suboptimality.
We verify that DNC remains suboptimal in our model even when ReLU activations are used, suggesting that our findings for the linear model likely extend beyond linear layers and offer broader generalizability.
We provide empirical validation both deep UFMs and standard deep classification networks, confirming that our theoretical insights extend beyond linear models.
Together, these results provide a comprehensive theoretical characterization of low-rank bias and its implications for the loss surface, and offer the first explanation for the empirical persistence of DNC despite its suboptimality.
1.1 Related work
Neural collapse: Many researchers have studied the conditions under which NC emerges using UFMs 332 77, including across various loss functions [57 32 21 [2]. Landscape analyses have revealed favorable optimization structure, where the loss surface contains only global minima
and non-degenerate saddles [56 58 21. Extensions of NC research have explored its behavior in
settings with a large number of classes (33 and imbalanced class distributions [51 44 7 6 [4]
Generalizations from the hard-label setting have also been considered for the language setting [55] and multi-class regression [ The influence of dataset properties
NC has also been examined Jacot et al. 20 show that NC arises in wide networks with an architecture similar to our own, but they do not address DNC, low-rank bias, or use the UFM. NC has also been linked to various practical aspects of deep learning 8 30
15 25
18 50 (44, and the impact of different network For review of NC, see (24].
architectures has also been explored 29 49
To theoretically analyze DNC, researchers have employed deep UFMs, particularly for MSE loss. This includes two-layer networks (5). linear layers [6 I0]. binary classification [ and more general settings 42 Additionally, DNC has been studied beyond the UFM framework using unique layer-wise training procedures [3. Zangrando et al. (53) examine how DNC and weight decay lead to a low-rank bias. Our work, which considers solutions with lower rank than DNC, diverges substantially from theirs.
Low-rank bias: Low-rank bias has been studied for matrix completion where Schatten quasi-norm minimization [33 47 is used to recover low-rank matrices from partial observations.

---

Wang et al. 488] show that in linear networks trained for matrix completion, stochastic gradient descent (SGD) can jump to lower-rank solutions but back.
In classification settings, low-rank bias has been analyzed in architectures with linear layers. Ongie al. and Parkinson et al. 134 SX study networks with single nonlinearity in the final layer, demonstrating that effective rank tends to decrease as depth increases. A similar perspective across various architectures is explored in [5]. Low-rank bias is proven for separable data and exponential- tailed loss with linear networks in the binary setting by Ji et al. [22]. In nonlinear contexts, Galanti et al. [9] examine low-rank bias in mini-batch SGD, observing that smaller batch sizes, higher learning rates, and stronger weight decay promote lower-rank neural networks. Huh et al. [17] and Le et al. [27] study factors contributing low-rank behavior. Le al. [27] use networks with linear final layers, but focus on binary classification and do not use unconstrained features.
Background
We consider a classification task with K classes and n samples per class. We denote the 7th data point of the ct class by Iye Ro with corresponding one-hot encoded labels ye E RK _ A deep neural network [(x) Rdo - RK is used to model the relationship between training data and class labels. We write the network asf(x) = WWo(WL-1o (Wi (W1h(x)) (()x))..)), where Wz E Rxd
and W WL-1..... W1 E Rdxd are weight matrices. The function h IRdo - Rd represents highly expressive feature map, such as ResNet. The activation function o R - IR is applied elementwise. Network parameters are trained via variant of gradient descent on CE loss, along with weight decay.
We denote the image of the data under the map h by h((a))) = (1 and define the matrix H1 to be (...,,,,. [1] (1)
the image of the full dataset in class order: H, (1)
11,: -., E RdxKn, Similarly,
[11,11-..
let H2 W H1, Hi W2o ..., H W denote the output matrices of each separated layer before the nonlinearity, with columns (1)) Let Hi for l = 2, .... L be the
corresponding matrices after the non-linearity, with columns R(0) Define the class feature means out of each layer as pec Ave class feature means as columns in class order. Define the global means as Me (I) = Ave(ue) and (2) Ave( Also, define the simplex ETF matrix as S Ix
(8) } and =(t) Avi(h ). Let Mt and Mi be the matrices with
k
1 RRKK
DNC refers to the following observations, which are found to approximately hold in overparame- terized neural networks training continues, with adherence typically increasing the depth of the considered layer grows. By 'overparameterized, we mean the network capable of attaining approximately zero training error.
Definition 1: Deep neural collapse: the last layers obey DNC if. for 1..., L, they satisfy:
DNC1: Feature vectors collapse onto their means: H1 Mi @ 1 Hi Mi Ø 1T.
DNC2: The feature mean matrices Mj and Mr, after global centering, align with simplex ETF: -10) (M₂ 11--2 PG (T HG IT) (M - PG 1 - ( 11)1))))) x S.
DNC3: The rows of the weight matrices Wi are, after global centering, linear combinations of the columns of the matrix Mr PG (1) IT.
Deep unconstrained feature models: To define the deep UFM, we approximate the feature map h(x) as being capable of mapping the training data to arbitrary points in feature space, treating the feature vectors h (1) freely optimized variables. The loss function becomes:
1 1111
C(H,, Wi, WL) 9(%)
Will
(1)
2
1=1
where implements the CE loss
1
K
n
exp((Zic)c)
(
log
(2)
Kn
Ee= exp((zio)ct)
k
i=1
3

---

( )...) are the logit vectors, which make up the columns of the matrix Z E RRKKKn using the same ordering as in the feature matrices. Here, À > 0 is regularization coefficient, and for simplicity, we apply equal regularization to all parameters. Note that, due to the UFM approximation, weight decay is applied to Hd,-
and 2/c-
Low-rank solutions: Recent work by Sukenik al.
12 demonstrates that, when using ReLU
layers, DNC is generally not optimal in the deep UFM when using MSE loss. They provide useful intuition for why such solutions exist. Consider the deep UFM defined in Equation (I) the regularization component. A well- Schatten quasi-noma result [40] states:
and focus on
rank(
2/L 2 SalL
1 X
min
1
+
Σ 1=1
where
|X
Σ
Here, the minimization is over Wi.1,..., W1, H1 subject to X = Wi-1...W,HH, and {s m are the nonzero singular values of X. For large L, the Schatten quasi-norm approximates the rank of the matrix X, encouraging it to be low-rank. This is traded off against the fit loss, which requires that the network still fit the data, Since the DNC solution has rank K in the MSE case, sufficiently large K allows the existence of lower-rank solutions that outperform DNC.
3 The deep unconstrained feature model
We consider the deep UFM, defined in Equation (T. initially using linear layers, a = id, within the separated portion of the network. While linear networks lack the expressiveness of modern deep models, the unconstrained feature assumption effectively approximates an expressive network, mitigating this limitation. This model can be regarded approximating standard classification architecture with appended linear layers. We consider the full nonlinear case at the end of the section. All proofs appear in Appendix B
We now describe the specific DNC structure for this case, informed by previous works 58 6 First, we demonstrate a well-known simplicity in the singular value decompositions (SVDs) of the parameter matrices of a linear network at any critical solution:
Lemma I: Let the network width satisfy d 2 K. At any critical point of the deep linear UFM, there exists an SVD of the parameter matrices in the following form:
Wi = =U,UUL,, for 1ar1=1..., 1....,, L 1; WL ULÉUI-1 ===UVEV
(3)
where UL E RKKKK, Vo E RRKaKKn UL-1,... Up E Rdxd are all orthogonal matrices, and
E Rdxd, D kxd, Σ E RRdxKn have their top K x K block given by diag(s1,.... where 8; Z 0 l, K, and all other entries are zero.
This result indicates that at critical points, each parameter matrix has equal singular values, and the singular vectors of adjacent parameter matrices are aligned.
If we construct a solution that satisfies the structure of Lemma 1, we need only specify the logit matrix Z, since
Z= = W W1 UL diag(s(++ sx L++) 0xx(n-1)K] VoT,
(4)
and the only remaining free parameters are UI. Uo which do not affect the loss. Thus, the loss at any point with this structure can be expressed entirely in terms of the logit matrix Z. Our definition of DNC structure is consistent with this alignment between matrices the linear model and naturally leads to the DNC properties.
Definition 2: DNC in the deep linear UFM: A DNC solution in this setting is any set of matrices Wi...., W1, Hi that both satisfy the SVD properties stated in Lemma I and yield a logit matrix of the form Z a.S & 1T, for some a >
Proposition 1: Let the matrices WL..., W1, H2 form a DNC solution as in Definition 2. Then these (1)
matrices satisfy the DNC definition stated in Definition 1, with pgg 0 for I = 1,.., L.

---

3.1 Low-rank solutions
While the DNC structure is optimal for L == 58 it generally ceases to be optimal when L L21 2. 2 4, L 2 3, or
Theorem 1: Consider the deep linear UFM with network width d 2 K.
Kf K
K 6, I. = 2, then no solution with DNC structure can be a global minimum.
This result is proven by leveraging a low-rank structure observed in experiments. While the DNC structure is given by Z S @ 1 the low-rank structure we analyze, when K is even, takes the form
X
0
X
0
1 where X = - 1
Z x
2 1
(S)
0 0
X
We show that, when the scales are set equal, the low-rank solution outperforms the DNC solution.
Theorem I highlights that the L = 1 case, known as the UFM, fails to capture the full behavior of deep networks, as its optimal structure does not generalize the last layer in deeper models. While the proof relies on a specific structure, the underlying cause is low-rank bias, which is further explored the next two theorems. Furthermore, since DNC and NC are equivalent in our model due to layer alignment, this result also rules out the original NC phenomenon.
We can extend Theorem further: any fixed structure whose rank exceeds the minimal rank required to fit the data is also suboptimal when the number of layers L is sufficiently large. To formalize this, we introduce the following definition.
Definition 3: Diagonally superior matrix: A matrix M E KxKn, with columns denoted as mie in class order, is diagonally superior when (msc)c (mac)e for all è' +
This definition is useful because such matrices can achieve arbitrarily small loss in the fit term of our objective by setting the scale appropriately large. With this, we state the next theorem.
Theorem 2: Let X, M KxKK have ranks p and q < p. with M diagonally superior: Consider the deep linear UFM with network width d 2 K, where solutions are construct using the structure induced by a critical point, allowing the loss be written in terms of Z. Let (Z denote this loss, with L dependence made explicit. Then there exists Lo E IN such that for all L > Lo.
min (ox))) a20
min (BLM)), B20
with equality occurring only if the minimum is at a 3= 0.
Theorem asserts that for sufficiently large L, no structure can be optimal if its rank exceeds that of the rank-minimal diagonally superior matrix of size K x Kn, except the trivial zero solution when regularization is too large. This underscores the role of low-rank bias: when a matrix can achieve arbitrarily small loss the fit term, low-rank bias ensures that-once enough layers are separated-in will outperform any higher-rank solution. While this theorem requires large enough L, in practice a well-chosen M can outperform high-rank solutions even for small L, shown in Theorem 1.
The imbalanced case: Thus far, we have focused on balanced classes, but our arguments extend to imbalanced settings well. A concise description of the single-layer UFM in the imbalanced case is provided by Thrampoulidis et al. [4). Using their characterization of emergent structures in imbalanced scenarios, we show that the analogy DNC in these settings is also globally suboptimal in the deep linear UFM. A detailed discussion and supporting results are given in Appendix
3.2 Optimal structure
We now aim to characterize how low-rank bias influences the structure at global optima. To do so, we must explicitly account for the scaling of the regularization parameter as L increases. To clarify this dependence, we write X(L)= XL. Additionally, we note that the minimal rank of a diagonally superior matrix in K2KKn is the same as x Therefore, this minimal rank is independent of and can be denoted by ęk- The following theorem provides detailed information about the singular values of globally optimal solution.
5

---

Losses of DNC Low Rank Solution
0.700
0.40
bound
cor
0375
6375
0.00
6.15
0.130
10000 12500
Epscha
Figure I: Experiments in the deep linear UFM: Left: Loss curves for a solution that converges to DNC versus that converges the low-rank structure described in Equation (5). Middle/Right: Corresponding mean logit matrices at convergence. Hyperparameters: L 2, d 70, X 2 K 10, 5, learning rate 0.5.
Theorem 3: Consider a sequence of deep linear UFM optimization problems, indexed by the number of layers L = 1, 2, Assume the regularization parameter satisfies AL for all L, and that the network width is fixed at d 2 K. Denote each loss function by LL Let Z be the network output at global optimum of Lt. Also define the minimal rank diagonally superior matrix in R xx xxn be 9к. The following holds:
(i) If X is scaled such that XT = o(L)). there exists Lo such that for all L Lo. each singular value of ž is zero or converges to zero at a rate faster than exponential in L.
(ii) If X is scaled such that À o(L there exists an Lo such that for all L > Lu. the matrix Zi diagonally superior, and hence has rank least qK. Furthermore, at most 9к singular values of the normalized matrix Zí, are neither zero, nor converge to zero at a rate at least exponential in L.
The second part of this theorem further clarifies the role of low-rank bias. When regularization sufficiently weak, increasing L enhances the low-rank bias, causing most singular values of the normalized optimal matrix 2; to decay exponentially in L. As a result, 2; resembles a low-rank matrix, with at most exponentially suppressed perturbation.
This raises the question of how low rank a diagonally superior matrix can be. Consider matrix X E R2*K, where each column satisfies |xt/22 = 1 and no column is repeated. Then X E R k diagonally superior and has rank implying ęK < This demonstrates the significant gap between the rank of the optimal solution for large L and that of the DNC solution.
3.3 The prevalence of deep neural collapse
Although DNC is not globally optimal, numerical experiments show that models often converge to it e--peccccally when the regularization parameter À is small or the width d is large. We also observe DNC in real networks, even where our model predicts suboptimality. To explain this persistence, we show that there are DNC solutions that remain locally attractive despite being suboptimal.
Theorem 4: Consider the deep linear UFM with network width K. When the regularization parameter X 0 is suitably small, the following holds:
(i) There exists solutions with DNC structure that are critical points of the model. Moreover, a subset of these solutions have scale that diverges as X + O.
(ii) These scale-divergent solutions have Hessian matrices with no negative eigenvalues.
This result makes clear that although no DNC solution corresponds to global minimum, there are DNC solutions that correspond to local minima degenerate saddle points-bott of which are difficult for local optimization methods to escape. Consequently, gradient descent can readily converge to DNC solutions in this model.

---

1.-
0.8-
Empinial DNC Prbbabiity
0.4
0.4
0.
0.0
Networs Whath
(0g) Weight Decay
Figure 2: Experiments in the deep linear UFM: Left: Empirical probability of DNC versus width d. Right: Empirical probability of DNC versus regularization X. Averaged over 10 runs; same hyperparameters as Figure 1.
While this result necessary for the persistence of DNC, is not sufficient. We now turn our attention to the network width parameter d, and examine how changes in the loss surface structure may further incentivize DNC. We begin with the following definition:
Definition 4: Dimension of a critical point in the loss surface: Suppose Z denotes the network output at critical point of the deep linear UFM loss. Define the dimension of the space of network parameters (Wr,,.., W1, H1) that produce Z* to be Dz-.
In the specific case where the solution has DNC structure, denote this dimension by DoNc. We can compare the dimension of general low-rank solution to that of the DNC solution.
Theorem 5: Consider the deep linear UFM with network width d Z K. Let the regularization parameter À >> 0 be small enough that there exists DNC solution that is a critical point of the loss. let Z. be the network output at another critical point at some fixed scale, with rank( Z*) E [2, K 1). Define the ratio of the dimensions in the loss surface of Zunc and Z be
DDNc(d)
R(d) = Dz- (d)
This ratio monotonic increasing function on the set of natural numbers d 2 K, starting below I and tending towards (K - 1)/rr 1 as d + oo.
This theorem shows that, increases, the dimension associated with DNC eventually exceeds that of any low-rank solution. Since both (DNc(d) and Dz- (d) tend to infinity as - co, the difference DDNc(d)- Dz- (d) also diverges. This implies that, in the large- width limit, DNC structure becomes more prevalent in the loss surface relative to low-rank solutions, which helps explain why local optimization methods may converge to DNC more frequently. Although low-rank solutions are still observed even when R(d) > 1, suggesting that flatness and masin-of-attraction properties also play important roles, this result supports the growing persistence of DNC influenced by the parameter d.
Impact of the regularization parameter A: As - O, the loss of any diagonally superior structure at its optimal scale tends to zero. This implies that as À becomes small, the DNC loss becomes arbitrarily close to the optimal loss. We can also compare local flatness measures of the DNC solution those of the low-rank solution described in Equation (
Theorem Consider the deep linear UFM with network width d 2 K, where the number of classes K is even. Let the regularization parameter X be small enough that there exists both low-rank solutions as described in Equation (5), and DNC solutions, that are critical points of the loss with scales that diverge X + O. Define the local flatness of these solutions, measured by the Frobenius, spectral, or trace norm of their respective Hessians, as FDNC and FLR- Then, as -> 0, we have:
FDNC FLR
FDNc, FLR + 0, such that
+ C,
where C. is constant that depends the choice of flatness measure.
7

---

Regulerized
2,00
MNIST
CITAR1
0-
1.75
4
1.25
4-
22
0.75
1000
2000
3000
4000
5000
Loss of UFM Regularized Network
Singular ar Values
DMC bound
CITAR1O
50-
62
Hs
40
0.5
Loghs
L075
0.4
0.3
0.2
www.w.o.........................
0-
1000
2000
3000
4000
5000
Epochs
Figure 3: Experiments using UFM-style regularization: Top: Losses of low-rank solutions on MNIST and CIFAR-10 10 using linear layers in the fully connected head, along with mean logit matrix for CIFAR- 10. Hyperparameters: L 3, d = 64, Aw = 5 x 10 λи 10-6 learning rate 0.05. Bottom: Loss and singular values across layers for CIFAR-10 using ReLU in the fully connected head. Same hyperparameters as above, except J, = 4.
This result shows that both critical points have local flatness metrics tending to zero, with their ratio converging to constant. This suggests that flatness alone does account for the preferen- tial selection of DNC. However, when combined with the insight from Theorem 5, it supports a heuristic explanation for DNC's persistence: while all minima approximately equally flat and perform equally well, DNC structures dominate in number. We provide numerical evidence for this interpretation in the next section and leave a rigorous demonstration of such connection future work.
While this analysis sheds light the geometry of the loss surface, ultimately believe that fully understanding behavior the À + O limit will require accounting for the implicit biases of gradient descent and the role of initialization scale. In particular, whether initialization causes the network output to initially align with simplex-and thereby increases the likelihood of falling into the basin of attraction of DNC- apperrs more amenable investigation through gradient flow dynamics than through landscape analysis.
3.4 The deep ReLU unconstrained feature model
We show that low-rank bias extends to the deep UFM, defined in Equation 1 with ReLU activations. This provides evidence that the phenomena described previous sections persist more realistic settings. First, note how the definition of DNC structure changes.

---

Singul H1
R2
Legiss
30
20
0
Figure 4: Experiments with standard regularization on CIFAR-10: Left: Singular values of each feature matrix in the fully connected head. Right: Mean logit matrix. Hyperparameters: L 3. = 64, A = 10-2, learning rate 10-3
Definition 5: DNC in the deep ReLU UFM: A solution of the deep ReLU UFM has DNC structure if it satisfies the properties of DNC stated in Definition 1, and additionally, the distinct columns of the logit matrix directly align with simplex ETF: Z x S 1
The additional condition on consistent with experiments, matches the linear case, and is implied by the original NC observation of Papyan. We also make the following technical assumption:
For DNC solution in the deep ReLU UFM, the ratio of the of the global mean to the norm of the simplex component of Mt is not decreased by the application of ReLU. Specifically, for 2 < < if we define
Assumption 1:
11
1 ll
11) M MC
|| PG 11
(0)
then S Fi-
This technical assumption is used avoid solutions that begin with large global mean and gradually reduce it to amplify the simplex ETF component. Such behavior would result in early features approximating rank-1 matrix with small perturbation, rather than the typical K I-dimensional simplex ETF. These cases do not arise in our experiments, and Assumption is found to hold. It likely that such structure is prohibited by the properties of ReLU. With this, we present the following:
Theorem 7: Consider the deep ReLU UFM with network width d 2 K. If Assumption I holds, and K 2 10, I. 2 5, or K 2 16, 4, then no DNC solution can be globally optimal.
This theorem is proven by showing that the DNC loss in the linear model is a lower bound for the loss of DNC solution in the ReLU model, and then constructing a solution that outperforms this bound using method similar to that in Theorem 1.
Theorem confirms that the low-rank bias observed in the linear case also occurs the more realistic ReLU setting. Moreover, the similarity between the proofs for the two cases suggests that the linear model effectively captures key phenomenological aspects of the more complex nonlinear setting.
Numerical experiments
We empirically validate theoretical results in the deep UFM, and provide supporting experimental evidence on the MNIST (28) and CIFAR- 10 [26] datasets using both UFM-style and standard regularization. Additional experiments and details can be found in Appendix
Deep UFM experiments: We Examine the deep linear UFM, defined Equation T trained from random initialization using gradient descent. Figure compares the loss achieved by DNC solution

---

with that of the low-rank solution described in Equation 55. The final class mean logit matrices at convergence are also shown. As predicted by Theorem I, the low-rank solution outperforms the DNC solution. Figure 2 shows how the probability of DNC changes as network width d increases or regularization decreases. We observe that DNC is unlikely at small widths but becomes increasingly probable as the network widens, supporting the argument derived from Theorem 5. We also observe that the probability of DNC tends to increase as regularization decreases.
Full network experiments: We next consider a ResNet-20 feature map with a fully connected head, applied to the MNIST and CIFAR- 10 datasets using UFM-style regularization. The top row of Figure 3 shows the losses achieved by low-rank solutions when using linear layers in the fully connected component. We see that the DNC bound beaten by such solutions. We also display the logit matrix at output, which clearly is not deep neural collapsed. The number of singular values above 10 was 4 for MNIST and 3 for CIFAR-10. The bottom row of Figure uses ReLU activations in the fully connected head. Again, DNC is beaten by our solution, which is low rank, as is clear from the plot of its singular values. Finally, we consider network trained with standard regularization, where weight decay is applied to every parameter in the network. Figure 4 shows that the resulting structure is low rank, and the mean logits do not form a simplex. This demonstrates that low-rank bias has implications even outside the modeling context.
5 Conclusion
In this work, we investigate low-rank bias in deep UFMs, providing general results on how the loss varies with the rank of different structures and offering insights into globally optimal solutions. This constitutes the first rigorous account of how low-rank bias influences the solutions reached by local search algorithms in these models. We also extend the findings of Sukenik et al. 42 on the suboptimality of DNC to both linear and ReLU-based CE models. Furthermore, we examine how the loss landscapes of these models promote DNC solutions despite their suboptimality, suggesting that the size of the solution space plays a significant role. This is supported by experiments within the model as well as in deep neural networks trained on real data using standard procedures. Together, these results establish the first theoretical foundation for the empirical persistence of DNC.
In this paper, examined how variations in the parameters d and À reshape the loss surface and influence the likelihood of DNC solutions. However, we did not explore how the choice of local optimizer or the initialization scheme may bias convergence toward DNC. These factors are clearly relevant. For example, Sukenik et al. note that the learning rate can affect the probability of reaching a DNC solution. Regarding initialization, prior work has shown that small initializations can significantly influence the rank structure of the converged solution [19 31. Other hyperparameters, such as batch size, may also play a role. Investigating how these factors contribute to the persistence of DNC would be a valuable direction for future research.
Future Work:
We also only provide preliminary investigation into how the regularization parameter affects the frequency of DNC in our model. While the implicit bias of linear networks without explicit regulariza- tion has been widely studied [46), most of this work focuses either binary Plasification-whee the logit matrix reduces to a vector and the Schatten quasi-norm collapses to the Frobenius norm-or on two-layer networks, where the form of low-rank bias we describe does not arise. more comprehen- sive understanding of how implicit bias manifests in deeper networks without explicit regularization would likely clarify why the limit
0 increases the frequency of DNC.
Limitations: Our work focuses on a theoretical model of training dynamics and does not address downstream properties such as generalization robustness. Additionally, while our analysis of the deep UFM captures useful structural biases, it omits important practical factors like batch size, initialization scale, and optimizer choice. These elements may interact with low-rank bias in nontrivial ways. Additionally, the UFM assumes overparameterizzatonn and is well supported empirically for this setting- but its applicability may diminish in underparameterizeed or highly constrained regimes.
10

---

Acknowledgments and Disclosure of Funding
We are grateful to Christos Thrampoulidis for an extremely helpful discussion that stimulated our consideration of the imbalanced dataset case. CG also thanks Peter Sukenik for discussions regarding the minimal rank of diagonally superior matrix. CG is supported by the Charles Coulson Scholarship. The authors also acknowledge support from His Majesty's Government in the development of this research. For the purpose Open Access, the authors have applied CC BY public copyright license to any Author Accepted Manuscript (AAM) version arising from this submission.
References
[1] George Andriopoulos. Zixuan Dong, Li Guo, Zifan Zhao, and Keith Ross. The prevalence of neural collapse in neural multivariate regression. arXiv preprint arXiv:2409.04180, 2024.
[2] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[3] Daniel Beaglehole, Peter Súkeník, Marco Mondelli, and Mikhail Belkin. Average gradient outer product as a mechanism for deep neural collapse. arXiv preprint arXiv:2402.13728, 2024.
[4] Hung- Hsu Chou, Carsten Gieshoff, Johannes Maly, and Holger Rauhut. Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank. ArXiv, abs/2011.13772, 2020.
[5] Zhen Dai, Mina Karzand, and Nathan Srebro. Representation costs of linear neural networks: Analysis and design. In Advances in Neural Information Processing Systems, volume 34, pages 26884-2688 26886.. Curran Associates, Inc., 2021.
[6] Hien Dang, Tan Minh Nguyen, Tho Tran, Hung The Tran, Hung Tran, and Nhat Ho. Neural collapse deep linear networks: From balanced imbalanced data. In International Conference on Machine Learning. 2023.
[7] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 212343): 2021.
[8] Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. arXiv preprint arXiv:2112.15121, 2021.
[9] Tomer Galanti, Zachary Siegel, Apama Gupte, and Tomaso Poggio. Sgd and weight decay provably induce low-rank bias in neural networks. arXiv preprint arXiv:2206.05794, 2022.
[10] Connall Garrod and Jonathan P Keating. Unifying low dimensional observations in deep learning through the deep linear unconstrained feature model. arXiv preprint arXivv2404.061066 2024.
[11] Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace. ArXív, abs/1812.04754 2018.
[12] X. Y. Han, Vardan Papyan, and David Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations, 2022.
[13] Hangfeng He and Weijie Su. A law of data separation in deep learning. Proceedings of the National Academy of Sciences of the United States of America, 120, 2022.
[14] Wanli Hong and Shuyang Ling. Neural collapse for unconstrained feature model under cross- entropy loss with imbalanced data. arXiv preprint arXiv:2309.09725. 2023.
[15] Wanli Hong and Shuyang Ling. Beyond unconstrained features: Neural collapse for shallow neural networks with general data. arXiv preprint a7X1v:2409.01832, 2024.
11

---

[16] Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 1985.
[17] Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. Trans. Mach. Learn. Res., 2023, 2021.
[18] Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understand- ing generalization in deep learning. arXiv preprint urXiv:2202.88384 2022.
[19] Arthur Jacot, François Ged, Berfin Şimşek, Clément Hongler, and Franck Gabriel. Saddle-to- saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. arXiv preprint arXiv:2106.10 15033 15933, 2021.
[20] Arthur Jacot, Peter Súkeník, Zihan Wang, and Marco Mondelli. Wide neural networks trained with weight decay provably exhibit neural collapse. arXiv preprint arXiv:2410.0488) 2024.
[21] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie Su, An unconstrained layer- peeled perspective on neural collapse. In International Conference on Learning Representations, 2022.
[22] Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. ArXiv, 0tss/810.020033 2018.
[23] Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for a large number of classes. arXiv preprint 1rX5v:3310005351, 2023.
[24] Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: A review on modelling principles and generalization. Trans. Mach. Learn. Res., 2023, 2022.
[25] Vignesh Kothapalli and Tom Tirer. Kernel vs. kernel: Exploring how the data structure affects neural collapse. arXiv preprint vrXiv:2406.02100 2024.
[26] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[27] Thien Le and Stefanie Jegelka. Training invariances and the low-rank phenomenon: beyond linear networks. ArXiv, abs/2201.1 1968, 2022.
[28] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324 1998.
[29] Jianing Li and Vardan Papyan. Residual alignment: uncovering the mechanisms of residual networks. Advances Neural Information Processing Systems, 36, 2024.
[30] Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Srnnandez-Grandee Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural collapse. arXiv preprint arXiv:2212. /2206, 2022.
[31] Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. arXiv preprint arXiv:2012.09839, 2020.
[32] Dustin G. Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20, 2020.
[33] Feiping Nie, Heng Huang, and C. Ding, Low-rank matrix recovery via efficient schatten p-norm minimization. Proceedings of the AAAI Conference on Artificial Intelligence, 2012.
[34] Greg Ongie and Rebecca Willett. The role of linear layers in nonlinear interpolating networks. arXiv preprint arXiv:2202.00883 2022.
[35] Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 21(252):1-64, 2020.
12

---

[36] Vardan Papyan, Xuemei Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences of the United States of America, 117:24652 24663, 2020.
[37] Liam Parker, Emre Onal, Anton Stengel, and Jake Intrater. Neural collapse in the intermediate hidden layers of classification neural networks. ArXiv, abs/2308.02 027600, 2023.
[38] Suzanna Parkinson, Greg Ongie, and Rebecca Willett. Linear neural network layers promote learning single-and multiple-index models. arXiv preprint arXiv:2305. 15598, 2023,
[39] Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 18729-287455 PMLR, 23-29 Jul 2023.
[40] Fanhua Shang, Yuanyuan Liu, Fanjie Shang, Hongying Liu, Lin Kong, and Licheng Jiao. A unified scalable equivalent formulation for schatten quasi-norms. Mathematics, 838): 2020.
[41] Jingtong Su, Ya Shi Zhang, Nikolaos Tsilivis, and Julia Kempe. On the robustness of neural collapse and the neural collapse robustness. arXiv preprint arXiv:331..0444, 2023.
[42] Peter Súkeník, Marco Mondelli, and Christoph Lampert. Neural collapse versus low-rank bias: Is deep neural collapse really optimal? arXiv preprint 2rX.::2405. 14468, 2024.
[43] Peter Súkeník, Marco Mondelli, and Christoph H Lampert. Deep neural collapse is provably op- timal for the deep unconstrained features model. In Advances in Neural Information Processing Systems, volume 36, pages 52991-53024. Curran Associates, Inc., 2023.
[44] Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. Advances in Neural Information Processing Systems, 35:27225-27253 2022.
[45] Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning, 2022.
[46] Gal Vardi. On the implicit bias in deep-leaming algorithms. Communications of the ACM, 66(6):86-93, 2023.
[47] Wendong Wang, Feng Zhang, and Jianjun Wang. Low-rank matrix recovery via regularized nuclear norm minimization. arXiv: Numerical Analysis, 2019.
[48] Zihan Wang and Arthur Jacot. Implicit bias of sgd in 12-regularized linear dans: One-way jumps from high to low rank. ArXiv, abs/2305. 16038, 2023.
[49] Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models. arXiv preprint 2407: 2024.
[50] Yingwen Wu, Ruiji Yu, Xinwen Cheng, Zhengbao He, and Xiaolin Huang. Pursuing fea- ture separation based on neural collapse for out-of-distribution detection. arXiv preprint 2rXiv:2405.17816 2024.
[51] Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network? Advances in neural information processing systems, 35:37991-38002, 2022.
[52] Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu Neural collapse with normalized features: A geometric analysis over the riemannian manifold. In Advances in Neural Information Processing Systems, volume 35, pages 115600 Curran Associates, Inc., 2022.
[53] Emanuele Zangrando, Piero Deidda, Simone Brugiapaglia, Nicola Guglielmi, and Francesco Tudisco. Neural rank collapse: Weight decay and small within-class variability yield low-rank bias. 07X1v22402.03391, 2024.
13

---

[54] Jiawei Zhang, Yufan Chen, Cheng Jin, Lei Zhu, and Yuantao Gu. Epa: Neural collapse inspired robust dutto--disttiuution detector. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6515-6519, IEFE, 2024.
[55] Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis. Implicit geometry of next-token prediction: From language sparsity patterns to model representations. ArXiv, hhs/2408.15417,, 2024.
[56] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 27179-27202. PMLR, 17-23 Jul 2022.
[57] Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: neural collapse perspective. In Advances Neural Information Processing Systems, volume 35, pages 31697-31710. Curran Associates, Inc., 2022.
[58] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-29834, 2021.
14

---

Each claim in the abstract and introduction is evidenced by theorem in NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: paragraph on limitations is included in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
Each assumption is stated in the theorem, and all proofs appear in the appen-
dices.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The paper includes enough details so as to perform the same kind experi- ments as the ones presented in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our contributions are mostly theoretical, using experiments to illustrate the theory. The source code will be provided on request.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppllligggddeeeeeeeeeeee Coaee///ddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental details are provided in the main text and appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our experiments verify the sub-optimality of deep neural collapse, for which reporting single runs is sufficient.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: The experiments do not require any specific hardware setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
The research conducted in the paper conforms with the NeurIPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We provide references for the datasets used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were not used.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM for what should or should not be described.
21

---

A Imbalanced classes in the deep linear UFM
Our results on low-rank bias in the deep linear UFM extend naturally the case of imbalanced classes. This setting has been explored in several recent works [5] 7 44 14], though our focus will be on the studies by Thrampoulidis al. [44] and Hong et al. [14]. We begin by establishing
general properties of the low-rank bias in the imbalanced setting, before looking at what can be said specifically about DNC.
A.1 General results in the imbalanced case
Assume have K classes, where the number of data points in class is ne and the total number of data points is N Σ ne- The loss function for the deep linear UFM in this imbalanced setting is given by:
C = (() + 2
WilF + + M1H 12-
In this expression, the feature vectors represented by the matrix Hi E RaxN, which we assume is ordered so that the first nı columns correspond to class 1, the next columns to class 2, and so on. The function g computes the sum of cross-entropy losses between the column vectors of Z = WL W, H2 and the corresponding labels ye: that is:
K Ne
exp((2ic)c)
o(2)
= N c=1
log
Ee= exp((2ic)ee)
i=1
where (Zie) denotes the network output of the i-th sample of the corresponding class c.
Importantly, the result in Lemma 1 holds independently of the class structure; that is, it does not depend on the specific values of ne- While the dimensions of H1 are adjusted accordingly, the structural result remains unchanged. Thus, any optimal point of the loss, the same structural constraints from Lemma apply, allowing us to reduce the problem to specifying only the network output Z E R R The loss can then be expressed as:
((Z)=g(2) A(L + 1)
2
We now generalize the notion of diagonally superior matrix to the imbalanced class setting:
Definition: TK., ngg) Diagonally superior matrix: A matrix M E RKNN, is said to be (K; "i,....... Bg)
diagonally superior, where N Ee The if the first nn columns have their largest element in the first entry, the next n2 columns have their largest element in the second entry, and so on.
With this definition in place, we can now state a generalization of Theorem from the main text:
Theorem A1: Let X E KXNN be a matrix of rank p. and let M R K be a (K; ni. ne) diagonally superior matrix of rank q < p. Consider the deep linear UFM with K classes of sizes ni, ... T.K, and suppose d > K. Construct solutions using the structure induced by an optimal point that the loss can be written entirely in terms of the output matrix Z. Let L (Z) denote this loss, with its dependence on made explicit. Then there exists Lo E N such that for all L. > Lo. we have:
min a20
min (rin)),(3M)), B20
with equality only if the minimum attained at a 3
22

---

The proof closely mirrors that of Theorem 2 in the main text. Specifically, we identify a scale at which . achieves a lower loss than X on the fit term, and when enough layers are separated, the lower rank of .M ensures it also outperforms X on the regularization term. The full proof is provided in Appendix C.1. This establishes that the same low-rank bias persists in the imbalanced setting, with analogous implications to those in the balanced case.
We also present the following generalization of Theorem 3 from the main text:
Theorem A2: Consider a sequence of deep linear UFM optimization problems indexed by the 3,2,3.... Assume the regularization parameter obeys X > 0 ffror all L, that
number of layers there are K classes of sizes TI,...., RK respectively, and the network width obeys d 2 K. Let Lt denote the corresponding loss functions, making the dependence on L explicit. Let Zj, represent the network output at global optimum of LL. Define the minimal rank of diagonally superior matrix in I. RgKx X to be 9K- Then:
(i) If Xr is scaled such that X L.),, there exists Lo such that for all I, >> Lo. each singular value of Zi, zero, or converges to zero at a faster than exponential rate T..
(ii) If Xz is scaled such that X = o. there exists an Lo such that for all >>> Lo the matrix 71, (K; (K;n1,.. De) diagonally superior, and consequently has rank at least qK. In addition, when > Lo at most qu singular values of the normalized matrix 2i, are neither zero, nor converge to zero at rute at least exponential in L.
The proof follows the same structure as that of Theorem 3 in the main text and is given in Appendix C.2. As in the balanced case, these results illustrate how the number of separated layers L increases, the global optima become more biased towards low rank diagonally superior matrices.
A.2 Deep neural collapse results in the imbalanced case
We now turn our attention DNC in the imbalanced setting. Ideally, we would like a generalization of Theorem I from the main text. However, this is challenging because full characterizations of optimal structures under general class imbalance are not currently available in the literature. While Hong et al. [14] provide valuable insights for general imbalance, their results involve unevaluated constants, which prevent direct recovery rank information. Moreover, the phenomenon of minority collapse- collapse-where where minority classes receive vanishing representation- can result in solutions with lower rank than those observed in the balanced case.
Thrampoulidis et al. provide concrete result in the - 0 limit for specific class of imbalances. Their analysis enables us to draw conclusions about DNC behavior when regularization is sufficienlly small. Before stating our result, we recall two useful definitions adapted from their work:
on the other hand, focus on the no-regularization setting, but they do
Definition: (R, p) Step imbalance: In (R. PT-S imbalance setting with label-imbalance ratio R 2 1 and minority fraction E (0, 1) the following hold: all minority (resp. majority) classes have the same size Qmin (resp. RD-min). There are (1 p) K majority and pK minority classes. Without loss of generality, assume classes (1,..., (1 - pppppp K} are majorities.
The simplex-encoded label (SEL) matrix Š RKXN is a (K: nh)) diagonally superior matrix, in which all columns corresponding to the same class are iden- tical. The distinct columns of S are given by the columns of the standard simplex ETF RR
Definition: SEL matrix:
S= -
1 1
In Theorem of their work, Thrampoulidis al. show that, in the single layer UFM, with a (R. p)- STEP imbalance, there exists unique global minimizer Z for each X > O. However, unlike in the balanced case, they demonstrate matrix depends non-trivially on X, not just its scale. Moreover, they show (as consequence of their Proposition 2) that in the (R, PTPT imbalance setting, the global minimizer of the single-layer UFM aligns with the SEL matrix in the X + 0 limit.
Proposition 1 of their main text that the frame of this
23

---

While the structure of the optimal solution at non-zero regularization in the UFM case is not fully known, we treat the frame ž as the natural generalization of DNC in the deep UFM the imbalanced case when À > 0. This is justified by the fact that, in the balanced setting, the DNC structure is recovered as the global minimizer of the single-layer UFM. Therefore, in the presence of STEP imbalance, we define the DNC solution to be any matrix of the form Z 7 equipped with parameter matrices satisfying the SVD properties established in Lemma Here, "proportional" refers to allowing differences in the overall scale of Z, which may vary depending the depth of the network (i.e., number of separated layers), but we assume the frame structure of the DNC solution is determined entirely by z
We have the following theorem about this generalized DNC structure:
Theorem A3: Consider the deep linear UFM with K classes of sizes ni.... n. respectively, and with network width 2 K. Also assume we are in a (R, P)-STEP imbalance setting. Then there exists Ao Lo 0 such that for all < Xo, L Lo, and K 2 4, there is no generalized DNC structure that is global minimum of the loss.
The proof proceeds by noting that in order to align with the SEL geometry for small X, the generalized DNC structure must have rank K 1. since this is the rank of the SEL geometry. As consequence we can apply Theorem AI, where we use a generalization of the same structure from the proof of Theorem 1 in the main text as the explicit lower rank generalized diagonally superior matrix. Full details are provided in Appendix C.3.
We find that notions of DNC in the imbalanced setting are also no longer globally optimal in general when we have more than one layer separated in our model, reinforcing the fact that the suboptimality of DNC is not confined to balanced class settings. However, unlike in the balanced case, we do not obtain an explicit bound on Lo. and so could, principle, be large.
While we are currently limited to the (R, p) STEP setting, we expect this result to extend to more general imbalance structures when À sufficiently small and minority collapse does not occur.
B Deep UFM proofs
We begin by stating some useful definitions for the deep linear UFM model. Recall our loss function from Equation ( which when using linear layers becomes:
L = W.W.
W
11
1=1
where the function g is defined as:
K Kn
1
exp((Eic)c)
s(Z)
log
ΣK
Here, the matrix Z E KxKn is written in the form Z |a1 221, Enl, Z12; 2Kn], where Zío denotes the column corresponding to the i-th sample of class c.
We also define the following quantities:
for 0 S - 1: A:11 = W..,.W441 E Bxxd
for 1 < / 5 L: H₁ W, ddKKn
7 Wr...WHH E KxKn
where Pic = exp((zic)c)
P 1P11P211 Pat.P1 Pia..., pK]] E KxKnn
Σx= exp((zin)e))
= 1k 2 1me E kk
24

---

We also define for later notational convenience AL+1 = IK, Hu = IKn-
Lastly, we will use the notation that e Raxd D E Rxxd D RRdxKn D' E kR all represent matrices where the top K K block is given by 1, 1, 0), with all other entries being O.
B.1
Proof of Lemma 1
For this section, is convenient to denote H1 Wo to make the equations more compact. Using the quantities defined at the start of this appendix, have the following elementary caleulations:
aZzy (Al+1)za (Hi)by: a(W))ab
(6)
Og(Z)
(P-
0z
Kn
Hence, the set of first order differential equations becomes
ac
1 AP AW,
(7)
aw,
Kn
At any optimal point, clearly have
W OC
0C
W 0,
aw;
OWi-
and substituting the explicit forms of these derivatives yields
w7 Wi = Wi-w
This implies that at local optimum, all weight matrices have equal rank. Consequently, the rank is at most K, since W2 has at rank at most K.
We now express each weight matrix using its singular value decomposition (SVD) follows:
for = L...., 1: W₁ UYV, with U. Σ, Vi E Raxd
Wo = U..oVo with Un E Raxd En E RdxKn, Vo E RKKKKn
Wr VV,, with U, E R E, e kxd Ví, Raxd
where each Uj and Vi is orthogonal, and Σ, has non-zero entries only on its diagonal. Using the expressions derived from the gradients, we obtain for < I L- 1:
(MET - (..
Which implies
= -U-EE-1
(8)
From this, conclude that for 5-5L-1
Σ Σ1- = Σ,
where Σ diag($), 82, 8K, 0,... 0), and 81 ..., 8 denote the top K singular values.
We extend this to the cases and 1, giving:
25

---

V. U1.11 - V/ = 1020 Inlio.
(9)
(10)
This implies that Σz. Σ [diag($1, $1, sK), xx xx k)), and Eg E, where the top K x K block of Σ is diag(#1, 8k). and all other entries are 0.
Finally, we aim to show that there exists choice of SVD for the weight matrices where the left and right singular vectors align. There subtlety here, since Equations (8). and (TO) together imply that for any / such a decomposition exists, but does not guarantee a decomposition exists where it holds for all I once. Such a decomposition can be achieved by working iteratively. choose V1 Uo using Equation (TO), this places some constraint on the choice of U1. Next pick 1/2 so it matches U1 which places some constraint on U2 and so on. The outcome an SVD decomposition where Vi Ui-1 for 1 = 1, ...., L, which completes the proof.
B.2 Proof of Proposition
We consider a set of parameter matrices WL...., W1, H2 for which their SVDs can be expressed in the form described in Lemma 1. We also assume that the network output has the form Z = aS 6 1 where S T & xx 1 Our aim is to show this implies the parameter matrices obey the DNC properties Definition 1.
First observe that the matrix S admits an SVD of the form S QD'O". where D' diag(1, ..., 1,0) E R and Q is orthogonal. The SVD of the vector 1 is given by 1 ( )T >TRT RT, where R E gnx= is orthogonal, and (n) is the first standard basis vector in R". Hence the SVD of the Kronecker product S & 1 can be written as
vnQ (D' & (e) (Q 00 RYT. & 1
Using the SVD forms of the parameter matrices from Lemma 1, we have
W VT...WWH.. H1 UL [dliiggss f+1
$ J.1)), (xx(m-1)x]V
(11)
Comparing this with the SVD form of Z, we find that s; = (avr)2 zh for = I,.., K - 1, and вg O. Hence Equation (II can be written as
= (avn)UL (D's @ (c())r) Vo.
Using the remaining freedom in specifying the SVDs (as discussed in the proof of Lemma 1), we can choose Vi Q @ R.
Now consider the intermediate feature matrices Hi = W H4. Using the SVD structure from Lemma 1, compute:
H = UU-UUUU UVEV Ui-IE'''VV"
Notice that S = (avn) i. D @ (e, ( where D = [diag( (1, I,,,,,,,,,,,. 0xxx)kk e
k xd Therefore
Hi ( ( (i (el)))) vo (avn) (D & (e(
1 (av (DTQT 8 (1.))
vn
(a/n)) + (Ui- o11)) (1
26

---

and hence write H, M2 @ 1/2 claimed, where M, Ja (a Next note that
+ (Ui-)D"
Mf Mi & UD
S
and hence this matrix does align with a simplex ETF. Finally, note that up to scale the weight matrix is given by
w/ x DUI & 0o
Where O is the orthogonal matrix
Q
0
Uf
0 1
and hence if we denote the rows of Wi by w they satisfy
K w x
meaning they can be written as linear combination of the columns of Mj. Hence, all three properties hold.
We also note the property typically used in the linear setting as DNC3 holds as a consequence of our structure, though this does not generalize to the non-linear case and so was not incorporated as part of the definition:
Using the alternative formulation where Uz = Q, we find that
Wy...W ULEUU -UUU,, ULEE'U
(avi) T. 44 UDDUUI..
From which it follows
W......)(WL...... & ULDUT - =
B.3 Proof of Theorem
We begin by observing that, using the form of any critical point from Lemma 1, the loss at a critical point can be written entirely in terms of the output matrix
Z W.....Wii W.H. Ur. (iag((s++ ..... s I+1)
Okx(n-1)K| Vo.
The loss function then becomes
L =
[ + 1)
b+1
where 8, = = 8,.,..x = S L+1 are the singular values of Z. We will denote this second term by R(Z). Since our DNC solution already conforms to the structure described in Lemma 1, and we will construct a low-rank solution to obey the same properties, it suffices to compare the loss via this
27

---

reduced formulation in terms of Z alone. Note this is the exact same expression as one would have using the Schatten quasi-norm result from 40), but it for general critical points, not just global minima.
In this formulation the only relevant free parameter in the DNC solution is the scale To make this explicit we denote a DNC solution by (anc If the optimal scale occurs at a = 0, then DNC not optimal by definition. Thus, we assume the best performing scale satisfies a > We will similarly fix the structure in the lower-rank solution so its only freee parameter will be its scale 3, and denote this solution by (3)). The method of proof is then to show that the low-rank solution outperforms the DNC solution whenever their scales are set equal (and non-zero by definition), and hence this is true even when the DNC scale is optimized, and so no DNC solution can be optimal.
By definition, the DNC solution has DNC (a) = a(Ix k 1 & 12. This matrix has repeated columns, with the matrix formed of the distinct columns having K- a on the diagonal, and 1 a on the off-diagonal. Hence we compute the fit-term for this solution, which gives:
K
g (a)) log(1 + (K - 7))
In addition the singular values of ZDNe(a) are $1 ... 8K-1 an šK 0. So the regularization loss in this case is given by
1 (L+ 1)(K 1)A(avT) ...
=
We construct low-rank solution like so: If K 2m, then write the matrix Z in terms
of 2 x 2 blocks
X 0
0
(LR(B) 3
0 X
0
② 1 where X =
1 -1
0
0
X
If K 2m + 1 we simple use the same matrix as K 2m but with the extra row and column in the class mean matrix having 1 on its diagonal and 0 otherwise (up to the scaling by (3).
In the even case this matrix has singular values $1 ... 8., = 2.3 8m+1
82m
whilst for the odd case it has singular values 81 = =.. = 8m = /n, 8m+1 = B/n, 8m42 =
52m+1 1 O. The fit-term of the loss can be computed in both cases, giving
For K 2m::
g g(Z]x(B)) = loog (1 (K-KK220 -K-KKK2 2)e +e -202)
For K = 2m + 1:
1 g(((x(3)) = K
22)) log log (I+ - 20) + log (1+ (K - 1)E (p) ].
2m
From this expression we have that if we set the scales equal we arrive at g ZZ (()) < g (D)c((()) for both even and odd K. For example, in the even case this follows directly since 2a e and the fact that log is an increasing function. The odd case is then a weighted average of the even case and term that exactly equals the DNC term, and so is still strictly smaller.
We can also compute the regularization loss for our low-rank solutions, giving
1 (L + 1)mx(28v7)) Li
for K 2m: R(ZrR(3))
for K = 2m + 1: R 2 (3)) [ 1)A m(28V7) + (8VR)
Again, if we set the scales equal, then the inequality R(Z)(ar)) < R(ZDNc(a)) reduces in both cases to:
28

---

m(2ayn) I < (2m - ()
which is equivalent to:
2m -
2 14 <
This equation holds when m 2 2, T. 2 3 or m 2 3, L. = 2. equivalently stating this in terms of K,
this holds when K 2 4, L 2 3 or K 2 6, 2.
We can now put this all together, we have that for the given values of K and L
((Z)R(a)) < L(ZDNc(a)).
If we define Ommin argmin (((Dcc(()))))) then this implies
ming (Lnn(((Zzn(3))) 2 (( < min, < (ZDDDc(())),
and so the low-rank solution outperforms the DNC solution, so long as we have K 2 4, L 2 or K3 6, L 2.
Result by Dang et al.
A similar loss function was considered by Dang et al. 66 in Appendix A of their work. The model they study is essentially the same as ours, with the exception that they include bias vector in the final layer of the network. It is stated, in Theorem A. of their work, that at a global optima both W,...W, and H1 will align with simplex ETF shape, stated in the theorem, since the product of two simplex ETF's returns simplex ETF, this implies that also forms a simplex ETF at global optima, which appears to contradict with our Theorem 1. Whilst their model is slightly different, they do state that if the regularization term of the bias is non-zero, then the bias itself zero at optima and so the loss function reduces to the same as ours at global optima.
IK 1 1 Whilst isn' I explicitly K
The proof by Dang et al. proceeds similarly to the proof by Zhu et al. [58] for the L case. Specifically, they produce series of lower bounds for the loss arriving at an expression like
C > 8K, A5c1),
where $1, ......, are the singular values of H1, and is an arbitrary constant. They also show this bound is only attained when e takes specific value in terms of 81,..., 8K and constants of the problem, and the matrices Wi,..., Wi, Hi obey a series of properties that characterize DNC. They then show that the minimum of & oceurs at finite values of these 8,, and so the bound can be attained, and is only done so for DNC solution.
One issue with this specific bound is that for a simplex ETF the s; are not freely specified. For the simplex ETF we have šж = 0, and 8; for i o K, for some constant >> 0. If & has its minimum a point where this property does not hold, then this line of argument will not work.
To make this clear, let us assume 01 is written in terms of the $1, SK as they detail, and we suppress the À dependence in our notation, so { = {(11, .., 8K). Let the optimal DNC structure be {s,NC), and ÂDNC E(SPNC ..... sK DNC), with the loss of the optimal DNC solution being LDNC- Denote the global minimal structure for & by LR in the same way as we have for DNC. Whilst they show that Losc fuNc. it is entirely possible that Lise > LLR if ÉDNc ÂLR- In fact, if one considers the structure based on our low-rank solution from Theorem 1, then it can be shown that the minimal low-rank structure has a lower value of & than the minimal DNC structure, this concern actually holds.
In the L = 1 case Zhu et al. reduce to similar expression, but in terms of || rather than the Since the Frobenius norm is freely specified quantity for the DNC solution, the above concern does not arise, and so the L proof avoids this issuc.
29

---

B.4 Proof of Theorem 2
Suppose the non-zery singular values of and M are fi..... Hp and Vi....,y respectively. We choose to write 3 ya. We also write Ls for our loss, to make the dependence on the hyper- parameter L explicitly clear this section. The losses of solutions built from these matrices are given by:
((lx))=9(add + A(L 1)a T
Σ 14
2
i=1
Cr(raM) = (((aM)))))))))) + XLL + 1 Σ (74)
First We compare the fit term of each solution. We denote the columns of X Tie ordered by class as previously, and similar for M by mic- The fit terms are given by
K
2(yaM)
1
log
1 Σ
exp (7a((mie)) - (mic)e))
Kn
d'
K
g(aX)
log
1+ Σ
exp(a((<ic)) (zro)o))
Kn
e'zc
Since M diagonally superior, we know (mic)e (mic)e < for all c, d' # c and for all i, Choose a fixed 7 >> 0 so that
((mic)e (m4c)e) < (Ise)ee - (Ite)es for all c, d' # and for all i E (L....,m)
Hence, due to the monotonicity of exp and log, for this fixed 7 we have
S s(aX))
where equality only occurs when 0.
Next consider the regularization term. First note we can Taylor expand the exponential, since are only looking at non-zero singular values:
2 log(r)
= =1+ o(1),
n
L +1
R=0
where we consider this as an expansion in -- as L + cc. Hence, in the large I. limit we have
LiT
p ppp o(1),
==1
(m4) ofi = = +
Since q < p we have that there exists some Lo E N, such that for L Lo
(74) 2
30

---

and hence
1 AL 1)ati
p Σ (mm) t - 2 M(L + l)ati aii
Σ
14
2
where again equality only occurs when a = 0.
Putting this all together we have that for a 2 0
{ S (
with equality only at 0. Define m argmin a20 (x Then we know
min (BM))) & (y (L) M) L (a (L) 820
X)
min (ax)),
a20
and equality can only occur if the minimum is at a = 3 =
B.5 Proof of Theorem 3
In this proof we will relabel L+1 L and 1 - X. This change is purely for notational convenience and does not affect the underlying results.
Note, the deep linear UFM must attain its global minimum at finite-valued optimum, since the regularization term diverges as any parameter tends to infinity. We can define our optimization objective at critical point for given depth L as
K
(z(Z)= + LA
where { are the singular values of Z. We begin by proving claim (i).
Proof (i)
We consider the case X7, = of L). Since the minimizer of L is unchanged if we re-scale the loss by constant, we can consider the equivalent objective:
1 LL
LA (( +
Consider the solution Zz = 0, for which g(0) = llog( K), and all singular values are zero. This solution has loss
1 LA1
(,
log(K) (K) = o(1).
Denote by Zj an optimal matrix at layer number I., and suppose has rr many non-zero singular values. This solution must perform at least well as the zero solution, hence
Lr(Zit S (1)
Σ of. <0(1) - sf,
Writing Sn,1 exp(txx(L)))) for some value (L),,, we obtain:
exp(1x(L)) o(1),
31

---

and this only occurs if (L) + -oo. This tells us that each singular value of Z is either 0, or converges 0 at faster than exp(-3L) for any 3 > meaning faster than exponentially.
Proof of (ii)
Now consider the case that LAr = o(1). We first show that, for sufficiently large L, any global minimizer Z7, must be diagonally superior.
Let Z E IR Kxn be any matrix that is not diagonally superior. We consider the fit term for this matrix
12
K
(
log
1 Σ
exp((2ic)e" (zia)e)
Kn
d'fc
Since Z is not diagonally superior, there exists a value of i, d' + such that (Ziele - (Ziele 2 0. In this case we see from the above that we can bound the fit term below
1
g(Z)
2
log(2),
Kn
and the whole loss is then also bounded
1
(L(Z) 2 log(2).
Kn
Now let RxXK" be any diagonally superior matrix, and denote its singular values {s])/=]-
Consider the sequence of solutions for each L
K (L() = g(LZ') + LA
(
First looking at the fit term have (2(a)e (2/o)e for all i, # Hence
K
2(LZ') Kn log 1+ Σ exp(L((<(c)) -==())))
1
log(1 + o(1)) = o(1).
K
i=1
In addition, =1==100f o() and Ek (6)) (an) + o(1). Hence
CL(LZ') = o(1) + LAz(1 (rn)(()a))( + o(1))=
This sequence attains arbitrarily small loss. In particular, when L is large enough that this loss is less than 1_ 12g(2), it outperforms any matrix that not diagonally superior. Hence for larger than Re this the optimal Z must be diagonally superior.
It remains to show the condition on the singular values of normalized optimal solution žř-
Let X K scale that we have for all i, c' o that (zie)e (æıe)c -2. Suppose the optimal solution at each L is Zi. We can seperate the scale of Z from its frame by writing
lowest rank diagonally superior matrix, and label its rank by {K- Choose its
=
where az = 1z
Note in particular since the zero matrix not diagonally superior have that the scale is non-zero and this decomposition is well defined when L large enough.
2j.
zi
32

---

Note the elements of 2n are constrained such that ((1)az S 1, and so for all é, c, d' # e we have
(Ei, - (Ei 2 -2> (Ese)e - (Ita)e-
This means that for L large enough
gaa 2i) g(al
Now compare the losses the sequence Z with the sequence Xx,,X, Denoting the rank of Zi by its non-zero singular values by 8r,ü and the non-zero singular values of X by pii we have
Lr(ar - + LAa
Σ i=1
L
gk
- x(aax)) + LA
Using our inequality for the fit term, and the fact that Zř, is the minimum, we have that
qK
LAo
L
LAO
Σ
14
and hence
qN
qK + o(1).
Again write $1, = (L)() for some function (LL), we have
exp(ni()) 9K + o(1)
Now define the quantity 8 > 0 to be such that e 98 4x+1
9x
Suppose the number of (L) that are greater than -5 is pr., then
exp(7;(L)) 2 p7e
and hence
Pre < o(1) pr( + qk<gx (gx + o(1)
When the (1) term is less than 2. this equation is only satisfied if PL I ęк. This is since PL, 9K both integers, PL (gx + 1) is an increasing function in PL. and (gK + 1)2 qK(ạk + 2) + 1. Hence, when is large enough, all but at most ÿK singular value of 2j satisfy
(-20L)
1
< & & exp
where 5 is independent of This means that these singular values are either 0, or converge to at rate at least exponential in L.
33

---

B.6 Proof of Theorem 4
We begin by showing that DNC solutions at specific scale satisfy the critical point conditions the deep linear UFM when the regularization parameter > 0 is sufficiently small. Recall from Equation 7 in Appendix B.1 that the first order derivatives of the loss are given by:
ac 1 AP AW
aw,
Kn
Kn
where the matrices Ar+1, P.Y and Hi are defined at the start of Appendix B. Hence, the critical point condition is for 0 S L:
W Knd 1 P)H
(12)
Recall our DNC solution satisfies, for some > 0:
= aS 2T
where = 1K
K 1
As consequence, the softmax probability matrix P is given by
e
1
1 ea
1
P == (K - +) e ea
(13)
We can then compute the error matrix Y
P. which gives
K
Y-P= (K 1) + ea
(S (801
(14)
and the error matrix also aligns with simplex ETF.
We will now start simplify our critical point conditions in Equation (12.. starting with
L - 1, by writing each matrix in terms of its SVD. Recall that we define the following matrices
D E Rdxd D E IR Kxd D E dd Kn D' E IR K where each has its top K x K block given by diag(1, .... 1, 0), with all other entries being
First write the singular value decomposition of the simplex matrix:
s 1 = VOQDDR",
where DD e Kxnn and Q, R are orthogonal matrices. We also have for the DNC solution that UL = Using the definition of A441 and H,, as well Lemma 1, have for 1 --1- 1:
Wi
(15)
A411 UU (avn) tii (a U
(16)
U, UU (avn) i 1t:1U-- Ui-iDV
(17)
where we used that Σ, E and Σ are written in terms of the singular values of the parameter matrices, and for DNC the singular values are 81 = 8K-1 nn)n)) I 8K = 0.
Hence critical point condition in Equation for 1 < 1 becomes
34

---

U (avn) प 1 vn
VoD"
nd (K 1) ee
We can pre and post multiply by Ui U1-1 respectively, and use that U7.Q R² Vo give
Vn
D = (ay n2)
nd (K t (DTDDD".
Then using that D DDDT D, this reduces
vn D,
D - (aVT)
(18)
nd (K 1) ca
and this holds when
An Anth
1 (K - 1) e
a .t.
(19)
we write f(a) for the RHS of Equation (9). this function satisfies: f(0) 0. lima (f(a))
and f(a)) > for > O. In addition, f has unique maximum in a > 0, since
(
f'(a) - +
L+I (K 1)
L
which has exactly one solution in >> 0.
Therefore Equation 19 has two positive solutions when An 1 is smaller than the value of the maximum. One solution satisfies a + 0 as À + 0. This solution has high loss in this limit and lies very close to the zero solution, so won't consider it further. The other solution has - as + 0, which attains arbitrarily small loss in this limit.
This provides a solution that satisfies the critical point conditions for < L-1. The / = L, - 0 cases produce the same equation Equation (T8). but with 1) or D in place of D, which reduces to Equation (19 also.
Hence we have shown that there are DNC solutions that are critical points of the model, so long as suitably small, with scale that diverge as
It remains to show the positive semi-deiiniteness of the Hessian of this solution. We start by computing the second-order derivatives the loss. We then specialize to the DNC case. We will begin with the block diagonal terms. Recalling the form of the first-order derivative from Equation 77). we find
82
aPay a(W))od (HF)) ybb
O(W)e) = Adacdbd + Kn
E 2.
By the chain rule, we have:
0Pag a(Wi)ad
Σ 0Pay OZu @Zuo O(Wi)ed
uu
The partial derivative the logit matrix was calculated in Equation (6) It remains to compute the P partial derivative.
0p Pay
exp(Zuy)
exp(Zay)
Σ
expl (exp(Zer))0errr
az
Ee exp((.cy)
Enr exp(( (Z.ry)p
35

---

Paydoy 1an Puyl-
we choose to write Pay = (py)s as so that Py is the nth column of P, then
(Py)a diag(pp)zus (Pu)a (Pyp') wns
giving
0Pay dny (diig(()))) - PuPly Iza.
dZue
Hence
0
Adan + Kn
Σ
(Al+1)an (HF Deo (H)())) (diag(py) - -PyPy Jan.
8 ((Wi) (W))
Dda + Kn [digp((ig((p)) (diag(py)
(Hi)by (Hi)dy
We also choose write (Hi)by (hg )b, so that h is column y Hi. This gives
(1.I) Hess abod
O(i)dd ddd 1
OC
Kn
(20)
Next we compute the off-diagonal blocks. Taking wlog < L, we have
0CC
1 Kn
Σ
E(AAFriba Yy ((H)) yh O(W,)od
+ (Af+))
OP ay a(W,)oo
(H Jub
The second of these two terms compute in exactly the same way did for the second term of the diagonal blocks, giving
OPay Kn Σ (A az a(W,
(HH ybb
-opp)) Ar+1lac (1) ( bd-
Kn
We now compute the first term. Note that
a(HF)yh (WW))cc
a(WF) Yun yu O(W))ed
DwW,I..WW W, H
[W-11..WW WW-....r++l.,,
8,2
ve.T))l[(W:)) (W
and so the first term is
1 K
Σ aP
V)ay my a(W,) Kn
O O(H ybb
(P YIH (W
Combining both terms, arrive at for < I:
36

---

Hess (1,r)
(P -Y) ( .,,1 I2
abod
Kn
[ (diag(()) Puply Art [1g 1
(21)
Kn
This gives the general forms of the blocks of the Hessian matrix. We now show that the Hessian of the scale divergent DNC solution has no negative eigenvalues when is small enough. To show this can demonstrate that
Vc E RP Hess 2 0,
where Hess is the full Hessian after the blocks have been flattened into matrices, and p is the total number of parameters. Writing v² = (V.,.., UL)², where each component is of appropriate dimensions, this is equivalent
Σ of Hess of O.
Denoting the matrix X, to be the pre- tre-flattendd version of us this can be written
(.r) (1.r) Hess, abed abed (Xy)ed 2 0,
(22)
f, abed
or equivalently
Σ (Xi (1.1) abed
aboa (Xi)od od + 2 (X)))aaHssss (.)) abod (Xp)od 2 0.
r-l. abea
Both summations have two terms due the form of the on and off diagonal layer-wise Hessians described in Equations (20) and (21) We will make series of transformations of the matrices X that leave the range these matrices as X₃ varies unchanged, and show how this impacts each term individually.
Starting with the off-diagonal terms, and working with the second term arising from Equation 21 this gives a contribution
2 ΣΣ r<l
Tr(AF
x
(23)
Kn
y
where we now choose to denote Py diag(py) PoP
The first transformation is Xi + U,X;U Since the matrices Uj are invertible this does not affect the validity the inequality. Doing this, and using the expressions for A411, Hy in Equations (16) and transforms Equation 23 into:
rel
where By the standard basis vectors in Rkn Now we perform the transformation X₁ 0t X1O, where
Ut. 0 7 O
0
In this expression the dimensions the and 7 blocks are designed to match the dimensions X,- Again this is invertible. Using that
37

---

[S]
[ 1 0
OD
ODV
[
contributing term becomes:
26 (avn)
( Pols x
S 1 c T @ o)
Kn
rel
y
(Spps 0] X, (S (e 0
xx²²)
0 x7
EiTT (ayn)
2
Kn
= rel y
(avn) ETT Tr(Sp,, & ( (.))))))
Kn
where we define the top K K block of X₁ to be X/. Hence only the top K K block of the transformed matrices emerges in the expression. These steps are exactly the same for the the second term in the on on-diagonal block, just with r.
The only remaining difficult term is the first term in the off diagonal Hessian. This is given, before transformations, by
2
Σ
1 Kn
Performing both transformations, and using that the matrix Y is proportional to S @ 1TT vnU,DDV? as stated in Equation (4), and Wi has SVD given by Equation 15 a similar calculation to previous gives that this term is
-2 - r< Vn - 6a (aVn) A (XS(3333(Xx)
1
and so this term also only features the top K x K block. We also identify the coefficient in the sum X via Equation (19), reducing this to:
-2 Σ Tr(S(X))"'(x))"))
rct
This gives us finally the expression for the positive semi-definite condition of the Hessian, stated in Equation (22 is equivalent
Σ xTr(XFxx)+
1 Kn
(aV
ΣΣ
Tr(Sp, (5)
22 T((S(x))*(())))
2 Kn (avT)
26
ΣΣ
Tr 5X(50 (S2)) 2 0.
The only place the entries not belonging to the top K x K block of X; contribute is in the first term, and they clearly contribute quantity that is non- negaatvv, hence we can drop them and work with the reduced inequality. We do this, and now drop the primes from the matrices X/, using X denote just the top K x K block. In addition, notice the second term in both lines are the same and so can be combined into
over all r, I. This reduces positive semi-definiteness condition
Σ XTr(XF x1)+ (a Lt1
1 Kn
ΣΣ 15
Tr TTS., -2A Σ Tx(x 0.
rel
38

---

Now note that the sum over 1.... K'n can be replaced by sum over i = 1,.... 7 and I..... Additionally, we have
Pic Po: @ 11)ex Scr
where here we use P. diag(p=) - P.P where Pe are the softmax probabilities of the network output means, and are the columns of the matrix S (this is departure from the rest of the document where we use 8; denote singular values). Using this our condition simplifies to:
Σ xT((X 1
(avn)
(T 2X Σ Tr(SX, 2 0.
rel
Defining the matrix X = Ei X1, this then reduces further to
XTX(X + (av # ITnSppSXSsoxxX)) 2A Σ TT(SXT5X7) 0.
K
r<l
We can perform similar trick with the final term, but now must add the missing diagonal terms. giving us that our positive semi-definite condition is:
24
"XM 0.
(24)
We first show that the first two terms together non-negative. Note we use / = S 1 1 1 to write the first term as:
Σ Tr(SX(sx)) + TT 12 + 1 + ((Ix1
The last three terms are all non -egaatiie,,, since each can be written as the Frobenius norm of matrix. Hence show the sum of the first two terms in Equation (24) is nen-negative is sufficient to show the following is non-negative:
Σ XT(SSX'SXX) + A E SxF)
x Σ Tr (SXX + sxf
where used that S S. Note the right matrix in the trace is the symmetrization of the transpose of the left matrix (up to a factor of two). Since the trace of a symmetric matrix times an antisymmetric matrix is zero, this just becomes equal the symmetric part of 2ASX,S, and hence is clearly non-negative.
over of the Frobenius norms squared of the
It only remains to show that the last two terms in Equation 24 are non-negative. For this we need to compute Pe- We detail the case, the general case can be inferred from this. Using Equation
B3 we have
39

---

K 6
0
0
0 -1
1
0
K
1
P1 (K ea
0
-1
-1 K
2
0
1
-1 -1
K 2
1
(25)
Label the first matrix by p and the second by PK For general y we get a matrix similar in both cases, but erucially P" is always positive semi-definite. We hence have that the remaining terms in positive semi-definite condition are:
Sx") + (aVn) i (K- - 1+ eay
Σ
Tr(Sp(5Xxbb 8 x²)
K
(aVn)
1 (K- 1+ ea E In(SSSXe 20. 0.
K
Since D" is positive semi-definite, this last term is non-negative and can be dropped from the expression. We also can use the expression for in Equation 19)) to simplify the coefficient, leaving us with the following condition
-XT((SX SXT)+ + amd K-1+ TT(S 2 O.
We now use Lemma 2, detailed end the section, which gives us that Sp Kso + S. Also using that E. S.S S, the above condition becomes:
SXT) and K --1+ ea Tr(SXSX") +and K-1+ ga
ETr(sas X X xx) 2 0.
Since S is positive semi-definite, the last term again non-negative and can be dropped. The remaining two terms can be written as:
cm
K K--1tte -1) T(SXXXI) XT<(SX" - SX S)) 2 0. K
e
>
similar argument by considering the symmetric and antisymmetric parts of the matrix SXS shows that the last term is up to positive scale equal to the Frobenius norm squared of the antisymmetric part S.S.S which is always non-negative. The first term is non-negative, so long as we have:
am K KK-1-e 21. 1.
Since diverges À 0, this holds when À is small enough.
Hence there exists a Ag such that when < Ao there is a DNC solution that an critical point the model, with scale that diverges
+ 0, and with Hessian that is positive semi-definite.
It only remains to prove the following Lemma:
Lemma 2: The matrix de defined in Equation
25 satisfies
Sp.SS = Ks.s. + S,
40

---

where S = Ix K k and se is column of the matrix S.
Proof: Note that the entries of P are given by
K 1, if ==
1, =f
(
-1, if or c, but but
0, otherwise
Similarly the entries of K K5 and given by
K-1)² i-==jcc
R
(K (K5
K i or c, but ر
Kt otherwise
K
ifi=j
Sij
otherwise
K
Looking at this case by case, it is clear that P'e = Ks T under transformation by S, we have Sp..5 Pe.
+ S. Since the columns of are invariant
B.7
Proof of Theorem 5
Recall, from Lemma 1, we have the following SVDs for our parameter matrices at a critical point:
Wi UEEUT, for 1.....
Wr UVÊUf_
H1 UV.VV
where UL E RRKK Vo E KnnKn, UL-1...., Uo E Rddd are all orthogonal matrices, E E IRd Σ xxd and E E Id K"n all have their top K x K block given by diag SK), with all other entries being zero. Also, as a consequence, Z Wt... W.....W' W, H, has the form:
= W..... WiHH UL [diag (s 1++
L+1) Oxx(n-1)K] V.T.
Suppose we have some structure 7" which is critical point for some specific choice of scale, and we want to assess the dimension of the space of solutions that correspond this structure. Let us denote the rank of this critical point by rank( determines $1..... 8K, and as consequence E, Σ and Σ. It also specifies UL, Vo, up to some potential reparametrization of the singular spaces that we will account for later. Also note, for intuition purposes, that none of these depend on d when d K, the impact of this parameter ultimately is not the loss of any given structure.
First note that specifying Z" fully
The only remaining quantities that are not determined are Uz-1,..., Up E Rdddd The only constraint that they must orthogonal matrices. On the surface it appears that each choice of Z* has the same degeneracy, since each just requires orthogonal matrices for the U.. However, there are two forms degeneracy. The first leads to a different point the parameter space, whilst the second leads to different expression of the same point in the parameter space. We only aim to count the first. To demonstrate this let us look at single / {1,..., 1}. We have:
Wi = ---0
( 1-1)T s:
where u are the columns of U. We see that only the singular vectors corresponding to non-zero singular values contribute to the expression of Wi. So instead of specifying full orthogonal matrix,
41

---

we must specify orthonormal vectors, since changing the zero singular-vectors does not change the parameter matrix.
Whilst we must specify r orthonormal vectors, we must account for potential degenerate singular values, since this means there are transformations of the degenerate singular vectors that leave the matrix unchanged. Indeed this is precisely the case for the DNC solution. We consider the two extreme ends:
All singular values are different: When all non-zero singular values are different, we are picking an orthogonal r-frame in R The space of such solutions is known the Stiefel manifold St(r, d), and it is a standard result that this space has dimension given by:
dim(St(r, - rd - rr( +).
This is the largest the matrix Uj can have its degeneracy as function of r.
All singular values are equal: When all singular values equal, picking r-dimensional vector subspace. The space of such solutions is known as the Grassmannian manifold Gr(r, d). It is standard result that this space has dimension given by:
dim dim(Gr(r,dd) d)) rd r
This is the smallest the matrix Ul can have its degeneracy function of
This gives the degeneracy for a single Uj we have the same degeneracy for each of U2 (--1.... Ug- Denoting the dimension of the space of solutions corresponding to the structure Z as Dz (d), we have
(nd-fr(r+1))
1 2
M(rd -)) 12 (z) <
Where exactly it falls in this range depends on the number of degeneracies of the singular values, but all cases we get a value of the form rd Crr), for some function C(r) obeying r + 1) S C(r) S 22.
Also note that the reparametrizations of Ur, Vo are now accounted for since they correspond to reparametrizattions of the other matrices as are accounted for in the above.
In the case of DNC, the dimension is exactly
DDNc = -[(K (K- 1)2).
and so we can consider the ratio of this dimension to the dimension of our other critical point Z":
(K - 1)2
DDNC Dz- (d)
(K - 1)d (K 1)
rd rdr
rd r2
from which it immediately follows that as d + oo we have
DpNc(d) (K - 1) Dz. (d)
It remains to show that the ratio is monotonic increasing and starts below 1 initially. First note that our upper bound on the ratio is less than when:
(K - 1)d - (K rd 22
<1 <1 K K
and since K +r- 1> K, this gives that the ratio starts below For monotonicity, it sufficient to note that the function
42

---

f(d) (K - 1)d (K - 1 rd - Cr))
is monotonic on 2 K if Kf 1)r > C(r), which is the case since we know C(r) S <r(K-1).
B.8 Proof of Theorem 6
We begin by showing that the low-rank solutions described previously are indeed critical points of the model when the level of regularization À is sufficiently small. The logit matrix of our low-rank solution is
X
0
0
Z 3X @ 1 where X
0
X
0
with X
1 | -
-1
0
X
and assume the parameter matrices satisfy the structural properties of Lemma 1. Also, note the matrix Z has K/2 non-zero singular values, each equal to v
Recall from Equation 12 in Appendix B.6 that, at any critical point, we require for / 0,..., L:
Wi Knd -
(26)
We focus on the case 1 < 1, noting the boundary cases can be handled similarly.
From Lemma 1, and the definitions of A441 and Hi, we have:
Wi = DEU A++1 A+11 A
-1- -V,
Substituting into the criticality condition of Equation 26 yields:
= U
Pre and post multiplying by UT, UU,1,, respectively reduces this to:
KnAE= -H -
We now, for the purpose of this section only, define D E Raxd D E Rxdd D E RdxKn, D' e R x be matrices where the top K/2 K/2 block is the identity matrix, with all other entries being zero. We can now pull out the scales of the matrices in our previous equation, giving:
KnAD (2 -PV.D".
We now perform conjugate transform by the orthogonal matrix:
Un 0 0=
0
Ia-K
This reduces both sides of our criticality condition to only having their top K K block being non-zero. The reduced expression is then:
KnAU,D'Uf (23//) uu WW6DD",0xxxKo-1)PPU
We then note that
43

---

U = x,
X & 1. 2/n
and so this reduces further to:
KnAX= (23
I+T
X(Y
P)( 1n).
2/n
Calculating the probability matrix for this specific logit matrix, we get, written block form,
P
1
Pr
p'= [ el
e- eB
P
& 1
where
K 1-24
2
e-
1
P
Then:
E
11
Y-P= K-2tel 88 8
1
1
E
01TT
3
X
&
1
1
E
where
K - 2 0
E =
K -
Explicit computation then gives:
n (2(K - -2) + 46 4c-
X(Y - -) 2 11
X,
K-2 2 e0 -
and so our criticality condition reduces to
Antix X = K (23)
K 2 e - 22 8,
which holds when:
An 4: (2
2 8 + - (K 2
(27)
Plots of the RHS function in 3 for characteristic L, K reveal there a single solution 3 such that 3 - oo as À - 0. To see the existence of such solution theoretically, note that if 3 is large this equation reduces to similar one to what we saw for the DNC case in the proof of Theorem as described in Equation (19). A similar argument then shows that there is solution for which B diverges as À O. Thus, this structure is a critical point of the model for small enough X, with scale that diverges as À - 0.
We now turn the comparison of the Hessian of this low-rank solution versus that of a DNC solution, and analyze the asymptotic scaling behavior in the limit À + O.
Recall from the proof of Theorem 4 (Equation flattening) are given by:
20 that the on-diagonal Hessian blocks (pre-
1 E(AA Jods K
((W)))) Adadded +
while the off-diagonal blocks for r < I are given by (Equation
21
44

---

We argue the first term each block is O()), whilst the second (BA), and so in the À + limit the first term becomes small compared to the second and does not contribute any flatness metric. I (A [Wd((W lest [AA 1g K
8
1
(WW)
K
where Py diag(py) PoPy. with Py the yth column of the softmax probability matrix P.
Clearly for the on-diagonal Hessian blocks, the first term is O()). For the off-diagonal, note, all dependence on À to leading order is given by the scaling constant, with that scaling being
8 Ki B bi1
gi-r-1
K- 2-2 + - eB e 1+1 X.
K-2+
where used Equation
27 to relate 3 to A. Henee, this is also ( as claimed.
To evaluate the scale of the second terms of the block Hessians, we need to know how Py depends on A. Doing the computation explicitly for gives:
K
2
0
-1
e 2 (K -K-24 8 B
0
P1
-1
0
1
0
O (c)-2)).
0
This scale is representative for all y. The scale of the second block Hessian term for any l, then:
(K - 2 e t
+ g t B B
te- क. BA.
K-2 el
Since diverges, this term dominates, and we can safely ignore the first terms the block Hessians in the + 0 limit.
We have found that each block of the Hessian is, leading order, fixed object (i.e. dependence), multiplied by 3X. Since each of our norms allow a scaling constant to be pulled out, we arrive at the scale of the Hessian being asymptotically proportional 8A the limit 0.
The exact same steps for DNC lead to the same implications, with each leading term in the block Hessian being proportional to at, and so the scale of the DNC Hessian is also asymptotically proportional to a to leading order.
We now need only understand how the two scales and 3 depend on the parameter X. This is described in Equations (1)), (27), which when written as asymptotic relations give:
A K 2-2 Antir ~ aite
K
Starting with the B equation, taking logarithms gives:
log(A) L 1 log(3)
L 1-1
log(2) log
K
1
log((n))
3
L+1 B 3
L+1
K-2
K
L.
and since 3 - 0o, we arrive at B ~ log(A).
Similarly for the equation:
log(A)
L 1 log(a) (a)
1 log(n)
a
L+1
a
a L ++1
and since also diverges, find -log(A) also. Hence the ratio of the scales is:
45

---

aA 3X
1.
Since the two deterministic matrices are different, the constant of proportionality is not but we conclude that the ratio of any norm of the Hessian must tend to constant as X 0. This constant is clearly dependent on the choice of norm. We also note that since log(A) + 0 as À + 0, both these scales do tend zero in the limit. This completes the proof.
B.9 Proof of Theorem 7
We aim to show that solution with DNC structure in the deep ReLU UFM can be globally optimal for the values of K and L stated in Theorem 7, under the given technical assumption. We will first demonstrate that the loss attained by DNC solution in the deep linear UFM serves as a lower bound for the loss of a DNC solution in the ReLU case.
Recall under the DNC structure, defined in Section 3.4, the matrices Hı and Hi, (as defined in Section 2) take the form:
H1 = M1 ② 1 1 Mr 8 1.
and the matrices Mr - PG M, - ne align with the simplex ETF. We will denote these globally
centered class mean matrices as M(() = Mi ( and M/ Mi Pg ( respectively.
QDQQ", Where Q is an orthogonal matrix that diagonalizes S, and We write $ = Jx 1x = D' diag(1 1, ..., 1,0) E BR
The alignment of ( M(()) with the simplex ETF implies that can write their SVDs
M o1 QT, MO
where R₁, R₁ E Rddd are orthogonal matrices, D = [D. x kY) Rxd and ag, 3t are scales that give the non-zero singular values of each matrix respectively.
Recall, also by definition of DNC solution, we can write the logit matrix Z Wr, (...Wo.. H))..,, as Z a141S & 1s. Inputting this into our loss function, we find the loss of DNC solution is given by
1 DNC log(1 + (K - 1)e 1eeaattt) + = A| + -
Will
(28)
We now seek to express 01++1 in terms of the singular values of the parameter matrices.
First note that for each / we have the equation Hi = Wi - By using repeated columns and global centering both sides, deduce:
M WW-
(29)
Let the first K singular values of Wi-1 1 be denoted by wi (1-1) for i = L,.., K, where as usual these in decreasing order. Note all other singular values of Wi must be zero by the third DNC property.
We quote the following lemma from Horn & Johnson [16] about the singular values of matrix product in terms of the singular values of the components.
Lemma: Given two matrices A E Rank and E Rkxn, and denoting the tth singular value of matrix by se(-) in descending order, we have
46

---

s((BB) S (i(())1(B).
are ar with multiplicity K 1, and O with multiplicity 1, and the singular values of M 3:-1 with multiplieity K- and O with multiplicity 1. Therefore, applying the Lemma to Equation (2)) this gives us the following inequality for 2 < L + and 1 S K 1
Recall that the singular values M, (G)
(1-1) 04 S w, B1-1-
In particular since w! 1(11)
for Sis K- the strongest of these inequalities is w WK-1
(1-1)
(1-1) a S w K-1 31-1-
This gives an inequality relating oy 81-1, will get inequality that relates 3:-1 04-1 using the technical assumption.
From the definition il we have
10 + 1 = o(M + -
Taking Frobenius norms on both sides and using the fact that the ReL.U activation satisfies [M)()) & | y for any matrix M, we get:
IM + 1 12 S { ( + 11
Now using the fact that M, M 1K
(0) 1 M
0, the above becomes
(0 K 1 < | + K 1
10
Now define the ratios
|| 1
71 =
M
F
Using these ratios, and the expressions for the Frobenius norms in terms of the singular values, can write:
1 Kr?
8 of
1 KP
and using the technical assumption this gives
81 $ ag for 2 2 L.
If we define H1 = Hi, then this inequality extends to / = 1. Using our two inequalities involving a4, Bi we can recursively derive bound the output scale:
on S w (-) B Wh--o---
((-1)
Iterating this repeatedly yields:
47

---

(T)
0L+1 S an K-1
We can now return to the DNC loss given by Equation (28) Using the fact that the function
g(z) log(1 + (K -)ee 1)e="
is strictly decreasing function, we have that:
L II 1=1
L [
DNcC 2 9
+ H
2
Next, we construct a lower bound on the Frobenius norm terms.
From the fact that M (c) ( = O, we have:
M || = M ( 13 + 11 |
= (K - 1)a
and hence:
H1 1 M 2 (K 1)na²
For the weight matrices, recall that w!") 2 w (1)
-1 for l.... K 1, giving that
Σ (4 2 (K 1) (w
Will
Combining everything, conclude that the loss of DNC solution satisfies:
LDNC 2 0
II 1=1
WK K-1 + 1 2 A(K- - 1) Σ 2 (2)2))
We will now show that this lower bound is attainable by a linear model DNC solution, and thus, the loss attained a DNC solution the non-linear model cannot outperform the global minimum of DNC solution in the linear case.
First consider the function on the right hand side of the previous inequality and for notational convenience, relabel
a avvn + Do, (1) - xz-
Then the DNC loss lower bound becomes:
RHS
II 2
A(K 1) Σ o7
Note any global-minimum of this structure occurs for finite x,, since the regularization term diverges otherwise. Hence the sum E/- 2y is finite at all global minima. Now note, since ğ is a decreasing function, it is minimized when all si equal given that their sum is finite, this follows from the AM-GM inequality. In addition since the quadratic function is convex, Jensen's 's inequality says that the regularization term is minimized when all the xę equal. Hence the total loss is minimized
when all x take the same value. Setting Ii = (a I we find the minimum DNC loss lower bounded by the following
LNsc 2 min g(a) / AAKK - 1)(L + ()(an)) :2 }.
az0
48

---

Note if the minimum occurs at = O, then this is attainable by setting all the parameter matrices zero, which is DNC solution, and so DNC not optimal in this case. Hence we assume minimum occurs at a * 0.
This is precisely the loss attained by the deep linear UFM the best performing DNC solution, derived in the proof of Theorem in Appendix B.3. Thus, we can compare to the following loss parameterized by scale a 0
DNC (a) log(1 11 + (K 1)e A(L + -)
We will now construct a low-rank solution that can beat this loss, for any even for ReLU non- linearity included.
Initially let us assume K is even. In constructing our low-rank solution, we can no longer use used in the linear model. This is since for any choice of the orthogonal
the same structure matrices - Un there will be negative entries that are zeroed out by ReLU. To address this, we modify the construction by adding a rank-l positive matrix ensure all entries remain non-negative throughout the network.
Define the matrices X, Y.Y E BK
follows:
X
0
07
0
X
1
-1 1
X
where
0
0
X
Y 1K1
We observe the following key properties
X = vX =0
= =²=
| 2K,
|
these algebraic identities will be useful in analyzing the propagation of signals throughout the layers. We construct our solution that for 2 < S the intermediate features are of the form:
0x + Y) Hi
@ 1 E Rd
(30)
0id-Ki*K
for scalar Φi 2 0.
We'll also construct our weight matrices so that for 2 S 1 they are of the form:
Wi = 0(d-KiXK
Ji + x
Oxx(d-K) ((--K)x(-KK))
E Rdxd
for some scalars v/1, XI 2 0.
Since all entries of H, are non-negative by construction, the ReLU activation has no effect, meaning H1t1 Wio(H) W,H, Using this, and the properties of X and Y stated above, we have for
2 < 5
22rri + K
H
30 1
O(d-K)xK
To preserve the structural form, we require that both components scale equally. This is achieved by setting X: k t/. Hence we arrive at the following recurrence relation for 2 l -1
49

---

11+1 24/101.
In addition, for the last layer we set WT = 11, (x Rxxd
We want to set the scales of each of our parameter matrices to be equal, and so we need to also calculate the appropriate choices of W1, H1. We have our given form for H2 stated in Equation (30)
suppose its SVD is H2 = U2EV2 Then we set
Wi = U,ΣLū H1 =
where U is any orthogonal matrix of appropriate dimensions. We then find
1 1W Σ sH()),
where s; (H2) are the singular values of H2. using the forms of X, Y. the non-zero singular values of H2 are K /ndz with multiplicity 1, and 2 2/no with multiplicity K
Hence we find that
2
H1 P| = |W113 2K 2
For W, with 2 2 we can also simply calculate the norm
|W = +I +
M = 2(K 224?
and also clearly for the final layer have
W 1 = 27
We now enforce equal Frobenius norms for each parameter matrix. We see for 2 < 1 that if the scales are equal then 1h = vb. addition setting the scale of W1 and W, equal to the scale of W; for 2 < 1 gives
K +2 2
K + K
62
WT.
V.
K Vn
This determines all the scales of our parameter matrices up to single scale sb. We now want to compute how this scale arises in the fit term through the output of the network.
Using the previously established recurrence relation: for 2 < S I. we have 01+1 24/rch, this gives that for 2 < S T.
Φi+1 = (216)(-102)
and hence
Φr. = (2 L-2 2/ = 2-2 +
K nn
Since Wno(Hz) 2n (82 @ 1,),, we then have
2 1 2 11 1+1
K +
K
We now compare this low rank solution to the DNC solution, beginning with the fit term characterized by the function g X - IR that was definied the start of Appendix B. In the proof of Theorem in Appendix B.3, we saw that for any scale 7 >> 0:
50

---

@ 11) < (
9
1
@ 1
Hence if we match the scales of the output matrices, the low rank solution will strictly outperform the DNC solution on the fit term. To match the scales, set 1/ to have the following relationship with the scale a of the DNC solution:
1 26-1 1 (1)
K 2 K
=
This guarantees that our low-rank solution performs better on the fit term. We now assess when will do better on the regularization term. Computing the regularization term for the low rank solution, we require that:
2 ML 1)(K 1)(av)) ML + 1)|2(K + 2)
Removing the common factors, since they are positive, and using the form of in terms of v, this is equivalent to
L. 1)
K + 2 K
(K (K-1)
2 TTTI + 2 2(K + 2)
Eliminating 222 since we assume it is non-zero, and cleaning the expression slightly gives:
K K+ K
2 K 1) K2
and this holds when = K, 2 14, or T. 5, K > 10. Hence we find that in these cases our low-rank solution outperforms the DNC solution when the scales of the output matrices are set equal. Hence this is true at the optimal scale of the DNC solution, and DNC cannot be optimal. In this proof, we assumed when minimizing the scale that a / at the minima, but if the optimal a occurs at 0 then DNC is not optimal, since we required a > 0 in its definition.
Hence we find that the DNC solution cannot optimal in the deep ReLU UFM when is even, subject to these conditions on L and K.
remains cover the odd K cases. Let = 2m 1. We similarly define our matrices X, Y to before, with slight change to account for the odd dimension.
X
02x2 X
02x2
02x2
02x2
[ -1
X
where X =
02x2
02x2
X
0
2
2m
02mx1
V
0 01x2m
0
These matrices now satisfy
X = =0:0,
X = 2X, 82= =KK 1)8,
|| (K 1)2
As before, define the intermediate features and weights to have the following form:
51

---

K + 3 2/2 for 2 21 H, = [i(( + +)
@ 1 E Rdxkn
Oid-KixK
for 2 1:15--1:1 Wi-
1 + + O(d-K)xK
OKx(d-K) (d--K)((-K))
E Rdddd
To maintain the desired form of the H; matrices we now require XI K-I the and we find o1+1 2 as before. For the final layer, define:
2
WE (VLXX,xx(dd-)) (xx(
As before, define the Wi, Hi matrices terms of the components of the SVD of H2. This yields
H 1ト W 2K 2K
Computing the norms of the other parameter matrices gives
2 2515L-1:
| 2(K + 3)02
[wll 2(K 1)v7
Again setting each parameter matrix norm equal we find that for
K 3
2 2ILL-1 d = VL
V K
d2
K vn
We then recurrence relation for oi get or matrix Z is
(2 L-2
222
We hence find that the output
(K+3)
K K+1)
33
K K33
Z 24-1
bl
K
Again setting the scale of the DNC solution equal to the scale of this Z guarantees it outperforms on the term, whilst the regularization term reduces to
K + 3 ri:
K 3
K+35 - (K-
K
K
and this holds for L = 4, 17, or, 2 11. Hence by the same arguments the DNC
solution cannot be optimal for odd K in these cases.
Putting these two together, we find that the DNC solution not optimal when K, L. satisfy 4, K 16, or L > 5, K 10.
C Proofs in imbalanced deep linear UFM
The proofs in this section closely follow those of the corresponding theorems in the main text. Here, we highlight the key differences that arise due class imbalance, while referring the reader to the previous appendix for all other details that remain unchanged.
52

---

C.1 Proof of Theorem A1
We follow the proof of Theorem and begin by showing that there exists constant y such that ayM achieves strictly lower fit loss than oxX for any O. Let the column of corresponding the 7t sample of class be denoted by Zic, and let the corresponding column for M be mic- Then the respective fit losses are:
K ne
s(yaM,Y)
1
log
1
Σ c'
exp (ya((mseee - (mac)ol)
N c=1
g(aX,Y)=
1
Th
log
Σ
exp (al(xic)e - (die)el)
e'
To ensure yaM yields a lower fit loss than a.X, it suffices that:
Ve,d' #CE (1,-.,,, KJ, Vi (I,..., neJ 7((mea)e - (mic)el < [(Fsc)e (x)
Since all The are finite, and the differences (mic)e (mec)e < 0 for all d' * due the generalized diagonal superiority of M, we can choose y large enough that this inequality holds for all d' Therefore, for this fixed 7, we have:
g(aM) S g(xX) for all a > O,
where equality only occurs when a The remainder the proof is identical that of Theorem found in Appendix B.4.
C.2 Proof of Theorem A2
As in the proof of Theorem relabel L+ 1 + and 1 + X, and similarly apply Lemma to reduce the loss function to the following form:
((()) LA
where ( are the singular values of Z.
The proof of claim (i) follows through exactly as in the proof of Theorem found in Appendix B.5, and we refer the reader there for details.
Proof of (ii)
Assume LA = o(1). We show that, for large enough L, any optimal solution Zj must be IK.,,.... nx) diagonally superior. Let Z e kxN be any matrix that is not (K: ni, ng) diagonally superior. Denote the columns of corresponding the thh sample of class e by Zic- Then the fit term is given by
K
The log
oZ, Y)
1+
Σ exp((zic)e (zic)le e'
N
Since Z is not (K; RK; nkk) diagonally superior, there exists values c' + c {1,..., KJ...,,K, (1...., Na) such that (Zie)er (Zie)e 2 This implies that the term bounded below
1 N
g(Z,Y)
log(2),
53

---

and hence the whole loss is also bounded
1 N
(r(Z) 2 1og(2).
Now let 7' be any (K: diagonally superior matrix, with singular values {s]] Let z/e denote the column of Z' corresponding to the jth sample of class Consider the sequence of solutions LZ' for each L. The loss becomes:
Lr,(Z')) - g(()),)) LA (
For the fit term, note by definition of diagonally superiority, we have (z/cle (#/cle < for all e' c {1...., K}. i {l,..., neJ. Thus.
K N
K Σ d'vc
K
g(LZ', =
log
1+
exp (L((2c))) - (Hic)e) =
log(1+((1)) = o(1).
N
c=1
1=1
Moreover, since L = 1 o(1) and Σx (5 rank o(1), we have
LL(ZZ') o + L11 (ro)) o(1)) = o
Thus, for sufficiently large L. the loss of LZ' is strictly less than + log(2), which is lower bound on the loss of any matrix that is not (K; ,.., nk) diagonally superior. Therefore, any optimal Zi, must be (K; nx) diagonally superior.
The remainder of the proof of claim (ii) proceeds exactly as in the proof of Theorem 3 found in Appendix B.5, once it is established that the minimal rank g of diagonally superior matrix in K X the same the minimal rank of (K; diagonally superior matrix in K First note clearly q S q since we could just column repeats of the solution in the diagonally superior case. Also q & G. since can just select individual columns from the (K R1,... nx) diagonally superior case, and the resulting matrix will have at most the same rank and will be diagonally superior in k
C.3 Proof of Theorem A3
Suppose have some fixed (R, p)-STEP imbalanced dataset, and denote the unique global minimizer of the corresponding single layer UFM problem by Z KgK'., By Proposition 2 Thrampoulidis al. 44), we know that the normalized matrix
2 = Z
converges the corresponding normalized SEL matrix as X 0.
Note that any SEL matrix has rank K - 1, since its column span is identical to the standard simplex ETF. This implies our corresponding normalized SEL matrix, has K non-zero singular values, suppose the smallest is μ Then, by convergence, there exists a constant λo > such that when < Ag, the matrix 2 has K - 1 non-zero singular values, all greater than _
Now let M E TR K be a diagonally superior matrix of rank r < K 1. Using the structure described in the proof of Theorem we know this is achievable whenever K 4. Additionally, the scale of M so that for all c, #
Mee Moc < -2
54

---

Construct the matrix M E RK by simply repeating the columns of M so that the first nı columns of M are the first column of M, and so on. We now compare the loss of a to the solution a M, with the goal of showing the latter outperforms the former. Again work with the loss written in terms of only the logit matrix:
=(a2))= 9((2))) + XL + 1)a ||
LFI 2
2
[(aM)= g((MM) + 2 AT + 1)
デी +
Since 27 is normalized matrix, its margins cannot be greater than 2, and so we must have g(a M)a and so our low rank solution does perform better on the fit term, then sufficient that we also outperform on the regularization term, meaning
A(L + 1)a i |M in
- L71 Mc + 1)a 1112111
1
2
after dropping positive constants, this
| M
<
L 241
For this hold when A < Ao, it sufficient that
c < (K -1] (
|M
( 2HK-1
Denoting the singular values of M by vi, 1, r. this becomes
Σ V < (K - 11
The LHS of this inequality tends to T. x0, whereas the RHS tends to K - r, hence there exists an Lo such that when L > Lo. this inequality is satisfied, and hence L(aM) < L()Z)))
Hence no generalized DNC structure in the imbalanced setting globally optimal for 2 4, when À < A and L > Lu. as claimed.
D Further experiments
Further numerical details: In most of our experiments, we present a single indicative run of the network model. Since aim is to demonstrate the suboptimality of DNC, we consider single run sufficient. The behavior we describe is representative when using high levels of regularization, as did in experiments and is detailed in the figure captions. For the standard regularization experiments, we apply weight decay equally at each layer in both the ResNet-20 backbone and the fully connected head. For experiments on MNIST, we subsample 5,000 examples per class match the class balance of the CIFAR-10 dataset. Input data is preprocessed by subtracting the mean and dividing by the standard deviation. We use batch gradient descent with batch size 10,000 so to approximate gradient descent, which is used in the model. However experiments with other batch sizes still lead low rank solutions when regularization was high.
We also provide here additional experiments that support our theoretical findings:
Deep UFM experiments: The left panel of Figure 5 illustrates how the rank of the solution found by gradient descent depends on the width the network in the deep linear UFM. predicted by our theory, the rank increases with width. Solutions with ranks between and represent a midpoint
55

---

9.-
8.-
8.0
7.5-
7.0
6.5
6.0
3.0
20
40
50
70
80
90
100
Network: Width
logu Weight Decay
Figure Experiments in the deep linear UFM: Left: Rank of converged solution versus width d. Right: Rank of converged solution versus regularization A. Averaged over 10 runs; same hyperparameters as Figure
Singular Values
Loss of Low Rank Solution
16
0.55
R
R
Logits ts
0.45
10
1055
0.35
10000
10000
Epachs
Figure 6: Experiments in the deep ReLU UFM: Left: Loss curve for a low-rank solution against the DNC lower bound. Right: Singular values of each feature matrix in the fully connected head. Hyperparameters: L = = 3, 70, X = 2 2-9, K = 10, n = 5, learning rate = 0.5.
between DNC and our low-rank solution, where subset of the classes forms restricted simplex ETF and the remaining classes pair up in a manner similar to the low-rank structure. The right panel of Figure 5 shows that lower levels of regularization
also associated with higher ranks.
Figure presents the experiment the deep UFM with ReLU activations, again demonstrating that low-rank solutions outperform DNC in this setting.
Full network experiments: The corresponding experiment using UFM-style regularization with ReLU activations on the MNIST dataset is shown in Figure 7, with similar implications to those observed for CIFAR-10 in the bottom of Figure Figure 8 displays the analogous experiment on MNIST using standard regularization, again with implications consistent with those of Figure 4 in the main text.
56

---

Singular
Regularized
MAITT
R
0.24
30-
0.22
0.20
10-
0.14
0.12
1000
2000
3000
4000
5000
Epochs
Index
Figure 7: Experiments using UFM-style regularization on MNIST: Left: Loss curve for low-rank
solution against the DNC lower bound. Right: Singular values of each feature matrix in the fully connected head. Hyperparameters: 4, d 64, Am x 10-6, - Aw = x 10-3 learning rate
= =10-3
Singular
M:
6
Ra
Logis
2
40
volue
-2
-4
8
-8
Index
Figure 8: Experiments with standard regularization on MNIST: Left: Singular values of each feature matrix in the fully connected head. Right: Mean logit matrix. Hyperparameters: I= 3, d 64, 10-2, learning rate 10-3
57


Reviews:
[{"review_id": "556XBByrLj", "reviewer": "Reviewer_9eBN", "summary": "This paper investigates why deep neural collapse (DNC)—a symmetric geometric structure often seen in trained deep networks—frequently emerges, despite being theoretically suboptimal. Building on prior work that showed DNC is not globally optimal under MSE loss due to low-rank bias, the authors extend this result to the more realistic setting of cross-entropy loss, including models with ReLU activations.\n\nThey prove that as network depth increases, optimal solutions become increasingly low-rank, with most, but not all, singular values of the output matrix suppressed exponentially. Although DNC has full rank (or rank-one deficiency for simplex ETFs) and thus incurs higher regularisation cost, it remains a frequent outcome in training. The authors explain this paradox by showing that DNC corresponds to stable critical points with its solution space grows faster with width than that of low-rank alternatives.\n\nEmpirical results on synthetic models (UFMs) and real datasets (MNIST, CIFAR-10) confirm the theory: low-rank solutions achieve lower loss, but DNC still frequently arises, especially with small regularisation and wide layers. The work offers the first rigorous explanation for the persistence of DNC in deep learning, linking it to geometry, low-rank bias, and network overparameterisation.", "questions": "1) Theorem 1 addresses also the $L=2$ case. However, the work in [43] proves that DNC is optimal for $L=2$ under MSE loss, and further suggests that the result could extend to CE loss. In contrast, your result appears to contradict this, showing that DNC is suboptimal for CE even when $L=2$. Could you clarify whether this discrepancy is entirely due to the change in loss function (MSE vs CE), or if there are additional structural or technical reasons that break the equivalence? It would be interesting to better understand how the choice of loss affects optimality—potentially highlighting why the generalisation of [43] to CE (and potentially other losses, as suggested in [53] for the L=1) may not hold. A brief discussion on this as the authors do wrt the result from [6] for Theorem 1 would be helpful. \n\n2) The authors mention that the single-layer UFM case ($L=1$) cannot capture the optimal structure of the last-layer for deep networks, since the last-layer weights have an optimal structure different from what NC claims (a simplex ETF). Am I correct in assuming that this is specific to CE? For instance, under MSE, the fit-optimal solution would need to have the same rank as $Y$ (true labels) to fully minimise the loss, implying that a lower-rank solution, specifically in the last-layer, (as allowed under CE) wouldn’t suffice. This suggests that optimality is strongly loss-dependent—particularly given CE’s focus on margin maximisation versus the regression-type MSE. Also, for the CE case, I can see this as an attribute of a deep linear network (from Lemma 1), but not sure how/if this generalises when non-linearity takes effect. \n\n3) The authors present a construction of diagonally superior matrix that achieves lower DUFM objective value than DNC. In the discussion around Theorem 3, do you claim that the optimal structure for $Z^\\star$ under DUFM is necessarily a rank-2 diagonally superior matrix? Or is rank-2 just an illustrative example of suboptimality of DNC, rather than a necessary property of all global optima? Some clarification would be helpful. To be clear, I’m focusing here on the rank of $Z^\\star$, rather than the specific construction itself. Also, I suspect it’s not hard to show that rank-1 solutions cannot be diagonally superior. Is that correct?\n\n4) Just to confirm: in Figure 1, are the optimisation curves plotted with respect to scaling parameters applied to fixed DNC and low-rank (LR) constructions? That is, do the curves reflect optimisation over the scalar factors (e.g., $\\alpha$ and $\\beta$) for a fixed geometry of $Z_{DNC}$ and $Z_{LR}$ respectively?", "limitations": "The authors addressed the limitations of their analysis.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- For deep linear cross-entropy unconstrained feature models (CE DUFMs), the paper identifies configurations of $K$ and $L$ where DNC does not constitute a global optimum.\n\n- The authors establish that diagonal superior matrices are a necessary condition for global optimality, tying it directly to achieving perfect classification, and proceed to investigate how such matrices can also exhibit reduced rank.\n\n- A notable insight is the contrast between the prevalence of DNC (in regimes of small regularisation and large width) and the existence of lower-rank global solutions that achieve lower objective values, showed by a dimension-counting argument. This is not really unexpected, at least in my opinion, since as $\\lambda \\to 0$, we are moving to the unregularised case of UFM where only the fit loss matter, so any diagonally superior matrix (simplex ETF construction being one) is equally optimal. The more interesting thing here is the large width regime, since it indicates a generalisation in some way (here the proper NTK-regime analysis is missing) the result of [19] in its deep NC counterpart.\n\n- The results are extended to the deep ReLU UFM case, showing that low-rank, diagonally superior solutions also arise in networks with nonlinearity, although the costruction here is handled in a similar way as in [40], meaning making the features non-negative for ReLU to not have any effect. Nonetheless, I believe it's an important negative result against the optimality of DNC in non-linear networks.\n\n- The analysis is further generalised to imbalanced datasets, demonstrating robustness of the framework beyond the balanced case.\n\n- Finally, the experimental findings support the theory, validating that networks trained in relevant regimes indeed with the constructed low-rank, diagonally superior structures indeed achieve lower loss.\n\nWeaknesses:\n\n1) While the paper demonstrates that DNC is suboptimal under CE DUFM and presents low-rank constructions that outperform it, it stops short of fully characterising the global minimisers. It remains unclear whether the proposed constructions are exhaustive or whether other classes of solutions exist. A complete description of the optimal solution set is still missing.\n\n2) In the ReLU setting, the paper establishes that DNC is not globally optimal, but provides no analysis of what the optimal structures might be. This result is strictly negative, and the paper offers no insight into how nonlinearity shapes the optimal logits or feature structure. The nonlinear case remains theoretically underdeveloped.\n\n3) Theorem 5 argues for the prevalence of DNC-type solutions by comparing the manifold dimensions of the DNC set and the set of low-rank diagonally superior solutions. This is done by accounting for certain degeneracies and essentially bounding the dimension of low-rank solutions between the Stiefel and Grassmann manifolds. Then they compare those bounds to the fixed Grassmannian dimensions of DNC. However, the interpretation of these dimension bounds as evidence of prevalence is still a heuristic step. Dimensionality alone does not account for the topology, measure, or dynamical accessibility of these sets under training. More refined tools from differential geometry would be needed to rigorously support claims about which solutions are likely to be attained. Thus, while the dimension argument is a reasonable first-order heuristic, it should not be equated with likelihood or abundance in a training context. This is something that the authors also acknowledge.\n\n4) A minor detail: Line 77: Zangrando et al., I assume the authors refer to the relevant arxiv paper, but the reference is missing.", "was_revised": true, "final_justification": "I believe this line of work in deep learning theory is important and advances the knowledge we have in the field. Thus I have increased my score (from 4 to 5) to account for the merit of this paper. It would be good for the authors in their revision to incorporate the comments and suggestions from the reviewer to increase the delivery and clarity of the work.", "rebuttal": "We thank the reviewers for their thoughtful and detailed feedback. The following responses address each of the raised questions and weaknesses.\n\n**The paper stops short of fully characterizing global minimisers**\n\nWe appreciate this important observation. While we agree that a complete characterization of global minima would be highly desirable, our paper does not aim to provide such a result. Rather, our goal is to highlight the emergence of low-rank bias in deep networks and analyse its consequences for the geometry of the loss surface and on optimization outcomes.\n\n* Theorem 2 shows that, for sufficiently large depth L, any lower-rank, diagonally superior matrix eventually outperforms any higher-rank structure (including DNC), revealing a systematic low-rank bias in the landscape.\n* Theorem 3 further establishes that all global minimizers must exhibit approximate low-rankness, with all but a few singular values decaying exponentially in depth. This imposes strong constraints on the structure of global optima, even without a closed-form description.\n\nWe believe this perspective offers valuable and broadly applicable insight into the geometry of deep learning with regularization.\n\nWe also note that fully characterizing the global minimizers in this setting is inherently difficult: the problem becomes similar to minimizing cross-entropy loss under simultaneous hypersphere and rank constraints—a class of non-convex problems known to be highly analytically challenging.\n\n**There is no analysis of what the optimal structure in the ReLU setting might be**\n\nThank you for raising this point. We agree that the nonlinear setting remains less theoretically developed than the linear case, and that understanding the structure of global optima under nonlinearity is a challenging and important open problem.\nOur main result in the ReLU case—that DNC is not globally optimal in the deep UFM with weight decay—is intentionally scoped as a first step. While this is a “negative” result in form, it is nontrivial: proving suboptimality even under ReLU required overcoming challenges that do not arise in the linear setting, such as the possibility of increasing rank across layers.\n\nThat said, we expect several qualitative features of the linear case to persist under ReLU, including a preference for low-rank diagonally superior structures at large depth (the results of Theorems 2 and 3). We see our result as an early indicator of this shift in behaviour and a foundation for future analysis.\n\n**the interpretation of these dimension bounds as evidence of prevalence is still a heuristic step. Dimensionality alone does not account for the topology, measure, or dynamical accessibility of these sets under training. More refined tools from differential geometry would be needed to rigorously support claims about which solutions are likely to be attained.**\n\nThank you for this insightful comment. We agree that interpreting dimension as evidence for which solutions are actually attained during training is heuristic in nature and does not capture many important factors such as those you mention. We appreciate the opportunity to clarify both our intent and terminology in Section 3.3.\n\nThe primary goal of our theoretical analysis was to highlight a geometric asymmetry in the loss landscape—namely, that DNC solutions occupy a higher-dimensional region than lower-rank alternatives, with this gap being influenced by network width and regularization. Based on this structural observation, we propose a heuristic explanation for the frequent appearance of DNC during training.\n\nWhile a rigorous dynamical explanation would require more advanced tools—such as gradient flow analysis or differential geometric methods—our contribution is to identify a structural bias in the landscape that favours DNC. To support this perspective, we provide empirical results in Figure 2, which show that DNC becomes more common with increasing width and decreasing regularization—consistent with our heuristic interpretation.\n\nAfter reviewing your comment, we recognize that the manuscript does not always consistently distinguish between two related but distinct notions of prevalence: (1) prevalence in the geometry of the loss surface, and (2) prevalence as an empirical outcome of training. In particular, some of our language may have inadvertently suggested stronger claims about optimization dynamics than we intended.\n\nWe thank the reviewer for making us aware of this and will revise the manuscript to enforce the following terminological distinction:\n\n* Prevalence will refer strictly to geometric properties of the loss surface (e.g., the volume or dimension of solution sets).\n* Persistence will refer to empirical outcomes under gradient-based optimization.\n\n**Zangrando et al., I assume the authors refer to the relevant arxiv paper, but the reference is missing**\n\nThank you for pointing this out, we will correct this in our revision.\n\n**The work in [43] proves that DNC is optimal for L=2 under MSE loss, and further suggests that the result could extend to CE loss… Could you clarify whether this discrepancy is entirely due to the change in loss function (MSE vs CE), or if there are additional structural or technical reasons that break the equivalence?**\n\nYes, the discrepancy is entirely due to the change in loss function. Your next question includes an accurate description of why this is: under MSE, the network output must have the same rank as the matrix $Y$ (i.e., rank $K$); otherwise, the fit loss is bounded below by a positive constant. The equivalent of Lemma 1 from our paper then forces the network to be rank $K$ throughout, and DNC ends up being the best among solutions at that rank. The CE case is different: to fit $Y$, the network output only needs to be diagonally superior, which imposes a much weaker rank constraint. Other loss functions will impose different structural conditions on the network output to achieve arbitrarily small fit loss. These differences in constraints affect how low-rank bias manifests—and ultimately influence the structure of global minima when $L>1$.\n\n**The authors mention that the single-layer UFM case (L=1) cannot capture the optimal structure of the last-layer for deep networks… Am I correct in assuming that this is specific to CE? … Also, for the CE case, I can see this as an attribute of a deep linear network (from Lemma 1), but not sure how/if this generalises when non-linearity takes effect.**\n\nYes, you are correct—different losses impose different conditions on the network output to achieve arbitrarily small fit loss, which in turn leads to different outcomes in how low-rank bias manifests. The deep UFM case when using MSE loss and linear layers is covered by Dang et al. [6] who show that the optimal solution still must be DNC, and this is for the reason you point out.\n\nWhen nonlinearity is introduced, this becomes theoretically much harder to evaluate, as ReLU can both raise and lower the rank of a matrix in nontrivial ways. For example, we can construct rank 3 matrices that become full rank after ReLU application by using a diagonally superior matrix that has its only positive entries on the diagonal. As a consequence, there is no clear analogue of Lemma 1 in this setting. \n\nIn experiments, solutions tend to use positive feature matrices, in which case ReLU acts like a linear function and the network effectively behaves as a linear model. However, due to the complexity of the loss surface, we suspect these solutions do not represent global optima—and without further theoretical advances, we do not yet know what the global optimal structures are in the ReLU case.\n\n**do you claim that the optimal structure for Z⋆ under DUFM is necessarily a rank-2 diagonally superior matrix?**\n\nThe result of Theorem 3 applies as we increase the number of layers, so we expect that as $L$ becomes large, the optimal structure will have only two singular values above an arbitrarily small threshold. A key reason for this is that the Schatten quasi-norm, which captures the regularization terms, approximates the hard rank of the matrix—and the hard rank does not penalize overall scale. \n\nFor small or moderate $L$ the picture is different: the overall scale affects the loss non-trivially, and different diagonally superior matrices can have different fit losses at a fixed scale. Hence, at small $L$ low rank bias is traded off against the fit loss, and we do not expect the optimal solution to necessarily be a rank-minimal diagonally superior matrix. \n\nThis trade-off is also evident in Theorem 2: if we consider setting $M$ to be a rank-minimal diagonally superior matrix, it only becomes preferable when the low-rank bias is strong enough to outweigh the loss from reduced fit. Therefore, to be clear, we do not believe that for small or moderate $L$, $Z^*$ must be a rank-2 diagonally superior matrix.\n\nAlso, yes—it is straightforward to show that a rank-1 matrix can only be diagonally superior when its size is 1 or 2.\n\n**in Figure 1, are the optimisation curves plotted with respect to scaling parameters applied to fixed DNC and low-rank (LR) constructions?**\n\nThe results in Figure 1 show the losses of deep UFMs trained from random initialization. It is only after training that we identify them as $Z_{DNC}$ and $Z_{LR}$, based on the recovered logit matrices, as well as the features and weights matching the respective definitions. \n\nThe dotted lines indicate the losses achieved by DNC and LR solutions at the optimal choice of α and β. We apologize that this was not clear and will revise the caption accordingly.\n\n\nWe appreciate the reviewer’s insightful questions and suggestions, which have helped us identify several areas to refine and clarify in our manuscript. We will incorporate these revisions to ensure the claims are more precisely framed, and the scope of our results more clearly communicated. Thank you again for your constructive and thoughtful evaluation."}, {"review_id": "MTugdiLzh7", "reviewer": "Reviewer_esms", "summary": "The paper analyzes the deep neural collapse phenomenon, showing it is not globally optimal under cross-entropy loss. The proof relies on theoretical results on the implicit bias of deep matrix factorization with L2 regularization. The paper then claims that DNC emerges in practice due to large width, as it is more prevalent in the loss surface relative to low-rank solutions. Overall, the paper manages to connect three separate ideas in deep learning theory: (1) neural collapse, (2) implicit bias in matrix factorization, and (3) large-width behavior.", "questions": "1. In the definition of DNC, you assume that all layers have fixed equal rank. Isn’t DNC characterised through gradual collapse towards lower rank matrices as we progress through the layers? Isn’t that what we actually see empirically in deep networks?\n2. What’s the motivation for studying diagonally superior matrices?\n3. In what sense is the relu analysis/results different than the deep linear case?\n4. Have you explored whether low-rank solutions generalize better, worse, or similarly compared to DNC solutions in practical networks?\n5. How do optimizer choices (say SGD vs. Adam) affect the low-rank bias? Isn't the low-rank bias due to SGD? (see for example \"Implicit Regularization in Matrix Factorization\")", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper is well-written and easy to follow, both the theoretical part and the experiments.\n2. The theory is accompanied by experiments both on toy data and real networks.\n\nWeaknesses\n1. The last two paragraphs before the related works subsection are a little too dense and hard to digest. Perhaps the authors might want to summarise their contributions in a bullet list. \n2. The experiments are somewhat simple.\n3. The relation between DNC and low-rank implicit bias was already established for MSE by Sukenik et al.\n4. The analysis relies on the Unconstrained Feature Model, which assumes full expressivity (but almost all prior work do that as well).", "was_revised": true, "final_justification": "The authors have addressed all of my concerns and I retain my very positive score.", "rebuttal": "We thank the reviewer for their constructive and thoughtful comments. We appreciate the time and care taken in reviewing our submission and respond to your points in turn below.\n\n**The last two paragraphs before the related works subsection are a little too dense and hard to digest**\n\nWe appreciate the suggestion to improve readability at the end of the theory section. We agree that the current presentation would benefit from improved clarity. In the revised manuscript, we will restructure this portion using bullet points to better summarize our key contributions and improve readability.\n\n**The experiments are somewhat simple**\n\nThank you for this observation. Our primary aim in the experimental section was not to provide a broad empirical study, but rather to validate specific theoretical predictions made by our model—namely, the suboptimality of DNC, its increased prevalence with width, and the role of regularization strength. We intentionally designed the experiments to isolate these effects in controlled settings, highlighting the theoretical phenomena without confounding variables.\nWhile the experiments are deliberately minimal to remain closely aligned with the theory, we agree that extending the analysis to more complex architectures and training regimes would be a valuable direction for future work.\n\n**The relation between DNC and low-rank implicit bias was already established for MSE by Sukenik et al.**\n\nThank you for highlighting this point. We fully agree that the connection between low-rank bias and the suboptimality of DNC under MSE loss was first introduced by Sukenik et al. [40], whose work we greatly respect. Their central theoretical contribution—Theorem 5 in their paper—shows that DNC is outperformed by a specific low-rank construction they term the SRG solution. However, their analysis does not characterize how low-rank bias shapes the broader structure of the loss landscape, nor do they make claims about the rank or geometry of global minima.\n\nOur work significantly extends this line of inquiry—not only to CE loss, but also in terms of the generality and theoretical completeness of the results:\n\n* Theorem 2 proves that for sufficiently large depth $L$, any lower-rank, diagonally superior structure eventually outperforms any higher-rank structure (including DNC). This generalizes the single comparison result (SRG vs DNC) of [40] to a full structural characterization.\n* Theorem 3 shows that global minima must be approximately low-rank, with all but a few singular values decaying exponentially in $L$. This demonstrates that low-rank bias governs the structure of global optima, not just isolated examples.\n* Theorem 4 establishes that DNC remains a local minimum or degenerate saddle point when regularization is sufficiently small. As a result, the loss surface for $L>1$ no longer satisfies the strict saddle property, which does hold for both MSE [52] and CE [54] when $L=1$.This reflects a fundamental shift in the geometry of the optimization landscape with increasing depth.\n\nWhile both works reveal a tension between DNC and low-rank structures, we believe our results offer a more general and theoretically complete picture. We are happy to revise the manuscript to ensure this relationship and distinction with [40] is more clearly explained.\n\n**The analysis relies on the Unconstrained Feature Model, which assumes full expressivity.**\n\nWe agree that the assumptions of the unconstrained feature model must hold for our theoretical results to apply directly to real neural networks. To support this connection, we included experiments using highly expressive architectures that approximate the UFM regime. In addition, there is broad empirical evidence in the literature suggesting that the implications of the UFM hold in practice when regularization is matched to the assumptions of the model. Indeed, most of the UFM-based works cited in our related work section numerically validate its predictions on real data using expressive networks.\n\nThat said, moving beyond the UFM and relaxing its assumptions would be an important and valuable direction for future work.\n\n**You assume that all layers have fixed equal rank. Isn’t DNC characterised through gradual collapse towards lower rank matrices as we progress through the layers? Isn’t that what we actually see empirically in deep networks?**\n\nYes, your description is correct. In empirical experiments with overparametrized neural networks, the level of DNC—measured by various metrics—typically increases as one moves deeper into the network (see references [13,35,37]). The feature matrices at each layer tend to exhibit decreasing effective rank, eventually approaching that of a simplex ETF, which is either $K$ or $K-1$ depending on whether global centring occurs.\n\nIn the deep UFM, the modelling assumptions treat each layer as effectively ‘infinitely deep’, meaning it can collapse exactly. As a result, there is no need to model the gradualness of collapse across layers; it is implicitly captured by the framework. Some prior works, such as Perturbation Analysis of Neural Collapse by Tirer et al., aim to make this more realistic by introducing additional constraints and observing increasing collapse depth by depth.\n\nWe also provided a definition of DNC for real neural networks in Definition 1 of the background section. This definition is intended to apply in the overparameterization and training limits, and is therefore compatible with the gradual emergence of DNC in networks that do not literally attain those limits. We would be happy to revise the manuscript to clarify this point.\n\n**What’s the motivation for studying diagonally superior matrices?**\n\nDiagonally superior matrices arise naturally when considering cross-entropy loss on hard labelled data. In the unregularized case, any diagonally superior logit matrix can achieve arbitrarily small loss by making the matrix scale sufficiently large. This property emerged in our theoretical analysis, motivating us to study the rank structure of these matrices.\n\n**In what sense is the relu analysis/results different than the deep linear case?**\n\nThe inclusion of the ReLU activation function makes the analysis significantly more challenging. This is because ReLU can raise and lower the rank of a matrix, and as a result, there is no analogue of Lemma 1 in the ReLU setting. In particular, the rank of the feature matrix can vary as one move through the network, unlike in the linear case where rank is preserved. \n\nIn terms of results, the ranges of $K$ and $L$ for which we can guarantee that DNC is suboptimal (as shown in Theorems 1 and 7) are narrower in the ReLU case, reflecting the reduced analytical tractability. However, the essential conclusion—that DNC is not globally optimal—remains the same in both the linear and ReLU settings for those parameter regimes.\n\nThe other results (Theorems 2–6) are not directly available in the nonlinear case, again due to the greater analytical difficulty. That said, we anticipate that the insights and structural intuitions from the linear case still carry over and may inform future theoretical developments in this more complex setting.\n\n**Have you explored whether low-rank solutions generalize better, worse, or similarly compared to DNC solutions?**\n\nWhen training networks on CIFAR10 and MNIST, we found that both DNC solutions and low-rank solutions achieve similar performances on the test set. However, we did not conduct a systematic study of generalization behaviour. Exploring how generalization—and other properties such as robustness—compare would be an interesting direction for future work. This is particularly relevant in light of the original claims by Papyan et al. [34], who suggest that neural collapse improves deployment properties; low-rank solutions provide a natural control group for testing such claims.\n\n**How do optimizer choices affect the low-rank bias? Isn't the low-rank bias due to SGD?**\n\nWe conducted experiments using both batch SGD and Adam, and in both cases, we observed that low-rank solutions can emerge. However, we did not perform a systematic analysis of how optimizer choice affects the frequency or precise rank of converged solutions. In the limitations section of our paper, we note that several important practical factors—such as optimizer choice, batch size, and initialization scale—merit further exploration.\n\nRegarding the second part of your question: there is a distinct line of work studying implicit bias, where optimization dynamics alone (typically of SGD or gradient flow) shape the solution in the absence of explicit regularization. The work by Gunasekar et al., which you reference, falls in this category.\nIn contrast, our analysis focuses on bias that is explicit in the loss function, arising directly from the L2 regularization term, as made clear in our theorems and proofs. We chose to focus on this regularized regime, but there are interesting follow-ups related to what happens in the absence of explicit regularization.\nFor example, Ji et al. [20] study the single-layer CE UFM without regularization and show a bias toward minimizing the nuclear norm of the logit matrix. Similarly, the analysis in Appendix G of *Gradient Descent Maximizes the Margin of Homogeneous Neural Networks* by Lyu and Li suggests that for deeper models, this implicit bias transitions to a Schatten quasi-norm—closely resembling the explicit bias in our formulation.\n\n\nOnce again, we sincerely thank the reviewer for the encouraging assessment and valuable suggestions. We will revise the manuscript to clarify our contributions relative to prior work, improve the exposition in dense sections, and more explicitly communicate the theoretical and empirical implications of our results. We appreciate the opportunity to strengthen the paper and are grateful for the reviewer’s time and insight."}, {"review_id": "ztdEqEqtAQ", "reviewer": "Reviewer_6xQX", "summary": "This paper studies the optimality of DNC in deep UFM, and prevalence of DNC udner cross-entropy loss. The authors first show that DNC solutions are not in general optimal under this setting, and then prove properties of the loss landscape that intuitively indicates the prevalence of DNC.", "questions": "1. Please also refer to the Strengths and Weaknesses part. \n\n2. Since the results in section 3.3. requires the regularization factor $\\lambda \\rightarrow 0,$ I wonder if it is possible to prove implicit bias results for GD/GF as in [4,5], and understand whether when those training dynamics are implicitly biased towards NC/DNC solutions?\n\n3. While the DNC solutions are not optimal as proved in this paper, is it possible to show that DNC are partially optimal,  e.g. DNC1 is still optimal?\n\n*References:*\n\n[4] \"Gradient Descent Maximizes the Margin of Homogeneous Neural Networks.\" Kaifeng Lyu, Jian Li\n\n[5] \"An unconstrained layer-peeled perspective on neural collapse.\" Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, Weijie J Su", "limitations": "There's no potential negative societal impact of this work in my opinion. The authors adequately addressed the limitations in Section 5.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strength\n\n1. This paper studies DNC for cross-entropy loss, which I think is a well-motivated problem, and is new in the literature.\n\n2. The paper provide high level intuitions on the non-optimality of the DNC solution, and the cause of prevalence of DNC, which I think is interesting. \n\n3. The paper is well-written, and is easy to follow the main points.\n\n### Weaknesses\n\n1. While I think the study of  DNC for cross-entropy loss is interesting, this paper focus more on studying the optimality of DNC, which I feel is less significance than explaining the prevalence of DNC. In particular, the non-optimality of DNC under MSE loss is already studied in [1], and cause of non-optimality of DNC in this paper is the same as the one in [1], which is the low-rank bias of the problem. In my opinion, this results do not provide much fundamentally new insights compare to [1]. Besides, the study of  the optimality of DNC is not that well-motivated in my opinion, as NC or DNC is just an experimental phenomenon observed empirically, and not much empirical evidence indicates the global optimality of NC or DNC in practice to my best knowledge. \n\n2. I'm not entirely sure the non-optimality of DNC is due to the low-rank bias as claimed in the paper, or is due to the fact that the strong definitions.  (1)In Definition 1, DNC is defined as all the last $L$ linear layers are collapse, this seems a bit strong to me, since it is possible that e.g. the level of collapse is inceasing by concatenating more linear layers, as observed in [2] for MSE loss. (2) Also, in Definition 1, it also requires the features after applying the activation functions also satisfies NC2, which seems a bit strong, since a feature that is NC2 may not be NC2 any more after apply e.g. ReLU, and in that case other structure such as orthogonal frame could be optimal as discussed in [3] for MSE loss.\n\n3. The paper discuss the prevalence of NC1 in section 3.3, however, those results are not direct cause of the occurrence of the DNC solutions. In particular, Theorem 4 only shows the existence of DNC solution as a non-supirious local minima, and Theorem 5,6 only provide high-level intuitions. I wonder it is possible to provide some more direct evidences, such as claims like: all local minima with small loss are DNC solutions, or gradient descent/gradient flow is implicitly biased towards DNC solutions.\n\n\n*References*:\n\n[1] \"Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?\" Peter Súkeník, Marco Mondelli, Christoph Lampert\n\n[2] \"Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse.\" Arthur Jacot, Peter Súkeník, Zihan Wang, Marco Mondelli\n\n[3] \"Extended Unconstrained Features Model for Exploring Deep Neural Collapse.\" Tom Tirer, Joan Bruna", "was_revised": true, "final_justification": "This paper studies the optimality of deep neural collapse (DNC) solution under cross-entropy loss. It provides interesting results on the loss landscape, showing the non-optimality of the DNC solution, and the intuitive cause of the prevalence of DNC, which I've pointed out in the original *Strengths* part. \n\nThe first two points in my origin *Weaknesses* part include the comparison with [1], and the definition of DNC, and these issues are addressed in the authors' rebuttal. In particular, this paper provides a structural characterization of the global optima of the deep UFM under cross-entropy, and technically it could also be interesting to potential future works in this direction.\n\nThe final concern is that this paper still does not provide any direct evidence on the prevalence of the DNC through its analysis, but from the authors' rebuttal, we know that all local minimas are DNC1 with some other structural properties, and this compensates some of the weaknesses in my opinion. \n\nThus, I would like raise my score to 4, and recommend an acceptance.", "rebuttal": "We thank the reviewer for their thoughtful and detailed feedback. Below, we address all raised weaknesses and questions. We hope this resolves many of their concerns, and kindly ask that they consider raising their scores if satisfied.\n\n**the non-optimality of DNC under MSE loss is already studied in [1] … In my opinion, this results do not provide much new insights compare to [1]**\n\nWe appreciate the opportunity to clarify the distinction between our work and that of Sukenik et al. [40], which we regard as foundational.\n\nSukenik et al. introduce the notion of low-rank bias and offer heuristic arguments and empirical observations supporting its role in DNC suboptimality under MSE loss. Their primary theoretical result on this topic is their Theorem 5, which constructs a specific low-rank solution—termed the SRG solution—that achieves lower loss than DNC.\n\nTheir result is example-based, relying on a single explicit construction, without analysing global minima or how the loss landscape changes. Crucially, their work does not preclude the possibility that the global minimum is not a low rank solution, making it unclear theoretically whether low rank bias is the cause of suboptimality.\n\nIn contrast, we provide a complete theoretical characterization of how low-rank bias manifests across the loss landscape, and why it leads to the systematic suboptimality of DNC. Specifically:\n\n* Theorem 2 shows that for sufficiently large depth L, any lower-rank, diagonally superior matrix will eventually outperform any higher-rank structure—including DNC. This generalizes the insight in [40] from a specific pairwise comparison (DNC vs. SRG) to a global classification of how low-rank bias affects the loss surface.\n* Theorem 3 shows that global optima must exhibit approximate low-rankness: all but a small number of singular values decay exponentially with depth. This demonstrates that low-rank bias governs the structure of global minima, not just the existence of an isolated better solution.\n* Theorem 4 proves that DNC remains a local minimum for small regularization. Thus, the loss surface for L>1 is not a strict saddle function, in contrast to the L=1 case shown in prior works [52,54]. This indicates that the fundamental geometry of the optimization landscape changes with depth, beyond just a shift in which solution is optimal.\n\nNone of these deeper structural or global results appear in [40]. While thematically related, our work turns the intuition in [40] into a rigorous theory. We also go beyond their scope by analysing the prevalence of DNC, which we discuss in response to another of your questions.\n\nWe thank the reviewer for pointing out that the distinctions between our work and [40] could be more clearly highlighted, and we will revise the manuscript to emphasize these differences more explicitly.\n\n**The study of the optimality of DNC is not that well-motivated in my opinion, as DNC is just an experimental phenomenon … not much empirical evidence indicates global optimality.**\n\nWe agree that prevalence is an important direction. We also agree that NC and DNC have not had their optimality explored directly via experimentation. This is likely because they arise in highly overparameterized neural networks, where the loss surface is highly non-convex and empirically identifying optimal solutions is not a realistic goal.\n\nHowever, we believe exploring optimality remains a valuable research direction, particularly in theoretical studies. The optimality of NC and DNC has been a central topic in numerous theoretical works—just among the references in our paper, this includes [1,3,6,7,14,20,22,28,30,40,41,42,43,48,49,52,53,54]. In many of these works, the optimality of NC/DNC within specific models is used as evidence for, and a theoretical explanation of, its prevalence. Until the work by Sukenik et al. [40] and our own, optimality and prevalence were treated as closely intertwined.\n\nMoreover, the extent to which DNC is suboptimal appears to directly impact its prevalence. This is demonstrated empirically in the right panel of Figure 2: as regularization increases, the gap in performance between DNC and lower-rank solutions increases, and the figure shows the prevalence of DNC correspondingly decreases.\n\nIf desired, we can also elaborate on the further research directions made possible as a direct consequence of our observation of suboptimality, though we believe the above arguments sufficiently justify studying optimality.\n\n**In Definition 1, DNC is defined as all the last L linear layers are collapse, this seems a bit strong to me, since it is possible that the level of collapse is increased by concatenating more linear layers**\n\nThank you for this perceptive comment. Definition 1 is to be interpreted as emerging ‘in the limit of an overparametrized network as training continues’. What we mean by this is it will only approximately hold in real networks, since they are not infinitely expressive or trained for infinite time. This is similar to the description of NC in Papyan et al. [34] which is stated as a limit rather than something that is literally achieved. We agree with your observation that in practical settings, collapse typically intensifies with depth—this is compatible with our interpretation of Definition 1 as an asymptotic phenomenon.\n\nIn contrast, Definitions 2 and 5, which formalize DNC within the deep UFM, do require collapse at each layer. This is since the deep UFM, due to the freely optimized features, represents the overparameterization limit, and consideration of critical points represents the training limit.\n\nThis is formalized in Lemma 1, which shows that at any critical point the rank of any intermediate layer's features must match that of the logit matrix. Therefore, either all layers will be approximate, or none will be. Empirically, this is precisely what we observe in deep UFMs: when DNC occurs, all layers collapse up to a numerical threshold. \n\nWe hope this makes clear that this definition is not restrictive under the modelling assumptions. We will revise Definition 1 to make clear that for real networks this is considered as an approximate relation.\n\n**Definition 1 requires the features after applying the activation functions also satisfies NC2, which seems a bit strong, since a feature that is NC2 may not be NC2 any more after apply ReLU, and in that case other structure such as orthogonal frame could be optimal**\n\nOur Definition 1 is adapted from empirical studies of DNC [35,37], which explicitly examine features after ReLU and report consistent evidence of the DNC2 property. To the best of our knowledge, all numerical experiments studying DNC and NC have considered the output of the activation function. Additionally, in both our experiments and prior work on deep UFMs [40,41], whenever DNC2 emerges, it remains preserved after nonlinearity. \n\nWe also note that the only theoretical result in our paper where this distinction is relevant is Theorem 7, as the issue does not arise in the deep linear UFM setting.\n\nRegarding your comment on orthogonal frames: our definition of DNC2 is intentionally broad and is compatible with both simplex ETFs and orthogonal frames. Any orthogonal frame, once globally centred, becomes a simplex ETF. In particular, the work by Tirer et al. [43] shows orthogonal frame structure before centring—these still satisfy our DNC2 condition after centring. This is why we impose no constraints on the centring vectors $μ_G^{(l)}$, $μ ̃_G^{(l)}$ in Definition 1.\n\nWe would be happy to revise the manuscript to clarify these points, and we are of course open to further discussion if any part of our definition or response remains unclear.\n\n**The paper discuss the prevalence of NC1 in section 3.3 … those results are not direct cause of the occurrence of the DNC solutions … I wonder it is possible to provide some direct evidences**\n\nWe thank the reviewer for this important comment. While our current results do not constitute a direct dynamical explanation for why optimization converges to DNC, they aim to provide landscape-level insight into why DNC frequently emerges during training.\n\nSpecifically:\n\n* Theorem 4 shows that DNC is a local minimum, making it reachable by local search methods.\n* Theorem 5 demonstrates that the dimension of the parameter space corresponding to DNC grows faster with width than that of low-rank alternatives, suggesting DNC occupies a larger portion of the landscape.\n* Theorem 6 shows that both DNC and our low-rank minima have comparable local flatness as regularization decreases.\n\nThough not a formal justification, our results reveal structural biases favouring DNC under first-order methods in wide, regularized networks. We support this argument with empirical findings in Figure 2, where DNC becomes more frequent as predicted.\n\nWe agree that a full dynamical explanation would be stronger, we see our work as a step toward such rigorous understanding. We are happy to revise the manuscript to clarify the heuristic nature of our argument.\n\n**Is it possible to prove implicit bias results for GD/GF, and understand whether those training dynamics are implicitly biased towards NC/DNC solutions?**\n\nFirst, we clarify that only Theorem 6 concerns the $λ→0$ limit; Theorems 4 and 5 hold for $λ<λ_0$ for some $λ_0$.\n\nJi et al. [20] show for L=1 that the UFM will tend to a KKT point of a constrained minimal nuclear norm problem. Appendix G of the work of Lyu and Li implies when using more layers the nuclear norm is replaced with a Schatten quasi-norm, mirroring our analysis. A similar result to [20] likely could be attained in the unregularized setting.\n\n**Is it possible to show that DNC are partially optimal**\n\nYes, we believe DNC1 remains optimal for CE, and arguments from [40] in the MSE setting likely extend.\n\nWe appreciate the reviewer’s detailed feedback and hope our responses have addressed their concerns. If there are further questions, we’d be happy to discuss."}, {"review_id": "lOkgPwNhLZ", "reviewer": "Reviewer_34uB", "summary": "This work makes use of a theoretical tool called Unconstrained Features Model (UFM) to studying the phenomenon of neural collapse (NC). Deep versions exist. Namely deep UFM (DUFM) to study deep neural collapse (DNC). \n\nDNC is not globally optimal once a network trained with weight decay under cross-entropy (CE) loss is deeper than two layers (unless the number of classes is very small, see Theorem 1). Instead, lower rank solutions can be found that achieve lower loss than DNC and Theorem 2 shows how there is a low rank bias for DUFM. The authors then explain why sub-optimal DNC, which is a local minimum or degenerate saddle point, are quite \"prevalent\" (this is theorems 4 and 5): DNC remains a benign local minimum whose attraction basin grows rapidly with width, whereas the low-rank minima are fewer but achieve smaller objective value. \n\nThe finding gives a theoretical explanation for the prevalence of DNC despite the fact that they are not optimal.", "questions": "1. UFM has a dizzying number of qualifiers. When these different names refer to the same thing, I cannot be entirely sure. I counted at least six variants:\n* Deep MSE UFM\n* Deep Linear UFM\n* Deep UFM\n* Deep CE UFM\n* Deep Linear CE UFM\n* Deep ReLU UFM\n2. The title does not quite reflect the actual message conveyed. It's more that neural collapse persists in spite of a bias for lower-rank solutions, which neural collapse solutions are not. \n3. I'm never quite sure what people mean when they say overparametrization as this has become an overloaded term. Do you there are more parameters than sample size? It doesn't sound like it as you don't speak much of sample size. Or do you mean, as I've sometimes seen, that there is a sub-network that is functionally identical? Where exactly do you use the mathematical notion of overparametrization?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The paper tackles a timely, well-posed extension of the neural-collapse literature: it asks whether DNC remains optimal under cross-entropy training with weight decay in deeper architectures—an incremental but natural progression beyond existing one- and two-layer analyses.\n\n**Weaknesses**\n\n* The claim that optimisation *prefers* the sub-optimal DNC over the lower-rank solution is not convincingly supported. The authors only show that DNC occupies a higher-dimensional region of parameter space (i.e.\\ is more “prevalent”), but they do not establish a causal link between that prevalence and the trajectories taken by practical optimisers. In other words, dimensional dominance alone does not explain *why* gradient-based training should consistently land in the DNC basin.\n\n---\n\nOverall, my confidence in this assessment is limited because I am not deeply familiar with the theoretical NC/DNC, UFM/DUFM literature.", "was_revised": true, "final_justification": "I maintain my positive score.", "rebuttal": "We thank the reviewer for the thoughtful feedback and constructive suggestions. Below, we address the main concerns and outline the specific changes we plan to make in the revised version of our paper.\n\n**The claim that optimisation prefers the sub-optimal DNC over the lower-rank solution is not convincingly supported**\n\nWe appreciate this important point. We agree that our current analysis does not establish a causal link between the prevalence of DNC and the behaviour of particular optimization algorithms such as SGD. Rather our goal is to provide a heuristic description of how first-order properties of the loss landscape encourage the frequent occurrences of DNC during training.\n\nTo clarify:\n* Theorem 4 shows that DNC solutions are critical points with positive semi-definite Hessians. While this does not imply convergence, it confirms that such solutions are reachable by local descent methods.\n* Theorem 5 shows that the dimension of the parameter space corresponding to DNC configurations grows faster than that of low-rank alternatives as network width increases.\n* Theorem 6 shows that both DNC and our low-rank minima have comparable local flatness, with their flatness measures converging to a constant ratio as regularization decreases.\n\nTaken together, these results provide a heuristic argument that DNC occupies a larger volume in the loss landscape and is therefore more likely to be encountered by local search methods in overparameterized regimes—assuming no strong inductive biases. Of course, many other factors affect optimization trajectories in practice, including the choice of optimizer, initialization scale and batch size. There are also higher-order landscape properties to account for such as dynamical accessibility. A rigorous dynamical explanation would likely require tools such as gradient flow analysis that can model these influences explicitly. \n\nThat said, we believe our landscape-based approach remains impactful: it highlights how one specific property of the optimization problem—geometry— can encourage convergence to DNC, and represents a first step in this direction within the literature. To support this heuristic view, we also include empirical results in Figure 2, which show that the likelihood of DNC increases with width and decreases with regularization, consistent with the predictions of our geometric analysis.\n\nWe will clarify this point in the paper and revise the relevant discussion to better communicate the scope and limitations of our conclusions. Specifically, we will emphasize that:\n\n* Our results provide landscape-level insight, not a rigorous theoretical claim.\n* Our empirical findings support the idea that DNC becomes more likely under conditions predicted by the landscape analysis.\n\nWe would welcome any suggestions on how to make this distinction more transparent in the revised manuscript.\n\n**UFM has a dizzying number of qualifiers… I counted at least six variants**\n\nThank you for highlighting this. We agree that the terminology has become confusing due to the number of variants. In the revised manuscript, we will streamline the naming convention and clearly distinguish the two central models analysed: the deep UFM with cross-entropy loss, using either linear or ReLU activations. References to MSE-based variants will be reduced and redundant qualifiers will be removed unless essential.\n\n**The title does not quite reflect the actual message… it's more that NC persists in spite of low-rank bias**\n\nWe appreciate this insight. We agree that the central message is that neural collapse persists despite a bias toward lower-rank solutions. The phrase \"under low-rank bias\" in the title was intended to convey this tension, but we now see that it may be too ambiguous. In response, we are happy to revise the title to more directly reflect the paper’s core message. If the reviewer has a specific change in mind, could they please let us know.\n\n**What exactly do you mean by overparameterization? Is it sample size? Or functional redundancy?**\n\nWe appreciate the reviewer raising this point. “Overparameterization” is indeed an overloaded term and we will clarify our usage. In our paper, we adopt the definition used in the original neural collapse work of Papyan et al. [34]: A network is overparameterized if it has enough capacity to fit the training data to near-zero training error. \n\nThis aligns with our use in both theory and experiments:\n* In the UFM, overparameterization is built-in, since features are treated as free optimization variables, and perfect fitting is always achievable.\n* In real-world networks, such as our experiments with ResNet-20, overparameterization means the model achieves near-zero training loss on the considered dataset, such as CIFAR-10 and MNIST in our case.\n\nIn the revision, we will explicitly define our usage of the term early in the paper.\n\n\nWe hope these clarifications help reinforce the contributions of our work and the care we have taken in interpreting and presenting our findings. We thank the reviewer again for their valuable feedback and for helping improve the clarity and impact of our paper."}]

Author rebuttal:

Meta-review: This paper studies the interaction between low-rank bias and Neural Collapse (NC), building upon previous work that showed that NC is not optimal in large depth networks with weight decay. This paper shows a similar result for linear networks with the cross-entropy loss and nonlinear networks. This paper also provides a few arguments that even though NC is sub-optimal, GD might still converge to such a solution because of there are comparatively many critical points that exhibit NC that GD might converge to. This paints a more subtle picture, where NC could still be proven if one can track the GD dynamics closely enough.

In agreement with the reviewers, we are happy to accept this paper as a poster at NeurIPS.
Decision: Accept (poster)