Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
Authors: Peter Súkeník
Christoph H. Lampert
Marco Mondelli
Venue: NeurIPS 2025 poster ()
Keywords: neural collapse
ResNets
transformers
LayerNorm
weight regularization
Primary area: deep_learning
TL;DR: We prove that neural collapse is approximately optimal in deep regularized ResNets and transformers end-to-end.

Abstract: The empirical emergence of neural collapse---a surprising symmetry in the feature representations of the training data in the penultimate layer of deep neural networks---has spurred a line of theoretical research aimed at its understanding. However, existing work focuses on data-agnostic models or, when data structure is taken into account, it remains limited to multi-layer perceptrons. Our paper fills both these gaps by analyzing modern architectures in data-aware regime: we prove that global optima of deep regularized transformers and residual networks (ResNets) with LayerNorm trained with cross entropy or mean squared error loss are approximately collapsed, and the approximation gets tighter as the depth grows. More generally, we formally reduce any end-to-end large-depth ResNet or transformer training into an equivalent unconstrained features model, thus justifying its wide use in the literature even beyond data-agnostic settings. Our theoretical results are supported by experiments on computer vision and language datasets showing that, as the depth grows, neural collapse indeed becomes more prominent.

Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
Peter Súkeník
Christoph H. Lampert"
Institute of Science and Technology (ISTA)
Institute of Science and Technology (ISTA)
Austria
Austria
peter. suberr.skenikkistaaacccc
chl@ista.ac.
Marco Mondelli*
Institute of Science and Technology (ISTA)
Austria
marco www.mmmeiiiist....mmmmmm ac.at
Abstract
The empirical emergence of neural collapse- surprising symmetry in the fea- ture representations of the training data in the penultimate layer of deep neural networks has spurred a line of theoretical research aimed at its understanding. However, existing work focuses data-agnostic models when data structure is taken into account, it remains limited to multi-layer perceptrons. Our paper fills both these gaps by analyzing modern architectures in data-aware regime: prove that global optima of deep regularized transformers and residual networks (ResNets) with LayerNorm trained with cross entropy or are approximately collapsed, and the approximation gets tighter as the depth grows. More generally,
squared error loss
formally reduce any end-to-end large-depth ResNet or trans- former training into equivalent unconstrained features model, thus justifying its wide use in the literature even beyond data-agnostic settings. Our theoretical results are supported by experiments on computer vision and language datasets showing that, as the depth grows, neural collapse indeed becomes more prominent.
Introduction
In 2020, Papyan et al. discovered a surprising geometric structure in learned representations of deep neural networks (DNNs) convergence. This strueture-duubbed "neural collapse" (NC)- was present in various architectures trained on many computer vision datasets, and it concerns the representations of the training samples in the last layer of the network: the feature vectors of the samples from the same class converge to the respective class-mean (NC1); the class-means form a simplex equiangular tight frame (ETF), maximizing the pairwise angles (NC2); finally, the class- means align with the rows of the weight matrix of the last layer (NC3). Similar structures were also subsequently discovered for class-imbalanced classification (5I) regression [ and language modeling (62), demonstrating that neural collapse is ubiquitous when training deep models.
The NC phenomenon raised significant interest in the machine learning community from both theoreticians and practitioners, due to its high relevance both areas. Theoreticians use it to improve generalization understanding [64 12 55) both in-distribution and in transfer learning. OOD detection
113. imbalanced learning understanding 677 theory of feature learning [28 40), robustness [47]. as
well representation learning itself 41
56
In practice, neural collapse has implications
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

transfer learning B2 55 61, OOD detection 65 63 37 compression (26], performance improvement
[55 9 and other aspects 770 33 35}.
In accordance with the high relevance of NC, a plethora of works aimed at understanding its origins in DNN training. To this goal, Mixon et al. 42 introduced simplified mathematical framework, called the "unconstrained features model" (UFM), In the UFM, one assumes the features of the last layer be free variables and optimizes over them together with the weight matrix of the last layer. Using the UFM, the optimality of the NC has been proven, as well as its emergence during gradient descent training various settings, see Section [2 for details. However, the UFM has since been criticized [23] for being too simplistic: the gradient dynamics in the UFM are inconsistent with those in the entire DNN trained end-to-end, and the global optima might be misaligned due to the difference between plain Frobenius norm regularization and the representation cost of the features, influenced by the training data. This led to attempts of proving NC in end-to-end training. However, the results so far cover shallow (up to three layer) networks 19 61 or come with strong (see Section 2). Moreover, with the exception of [58], all works only focus on multi-layer perceptrons (MLPs). However, NC is equally present and important in modern architectures, such as ResNets (18) or transformers ((54)) (88 62.. The addition of modern DNN components, such as residual connections, layer normalization or attention layers, makes the loss landscape significantly different and thus it is unlikely that the theoretical tools developed so far will be easily adjustable to these newer architectures.
assumptions [64 44 46 58
In this work, we fill both mentioned gaps at once. First, we analyze ResNets with LayerNorm and transformers. We are the first to theoretically analyze NC in transformer architectures, while also significantly extending the knowledge on ResNets. Second, our results prove end-to-ent approximate optimality of NC in training with weight regularization. This has only ever been done for MLPs with deep linear heads in [23] To be more precise, our contributions are summarized below.
For ResNets and transformers with one linear layer per MLP block and constant regularization strength, we prove that NC is the asymptotically optimal solution as the number of blocks goes to infinity. Moreover, all global optima in deep-enough networks must be approximately collapsed and the distance from perfect collapse is non-asymptotically upper-bounded in terms of the depth. These results hold for both cross entropy (CE) and mean squared error (MSE) loss, under minimal assumptions on the data.
We prove the same set of results for ResNets and transformers with two linear layers per MLP block and vanishing regularization strength.
We support these findings by experiments on computer vision datasets with both ResNets and vision transformers, which show that the amount of collapse increases with the depth of the architecture, as predicted by our theory.
More generally, we provide a formal connection between deep Res.eetsttrasffommers and uncon- strained features models: prove that, these architectures become deeper, their global optima converge to those of an equivalent UFM. This result holds for a wide class of continuous losses.
Let us highlight the conceptual relevance of the last contribution, which reduces trained DNNs to an equivalent UFM. As a consequence, if one can solve the underlying UFM and identify its global optima (which we do for CF and MSE loss), these optima will be provably approached by globally optimal ResNets and transformers trained end-to-end, as long as they are deep enough. This provides theoretical justification for the use of UFM in the analysis of these architectures and, in fact, it is the first such justification with a theoretical backing appearing in the literature.
2 Related work
Unconstrained features model (UFM). First introduced in 42 10], the UFM has been widely analyzed in the literature. The optimality of NC in the UFM has been proved for CE loss [60] 39 30 MSE loss [68] and other losses 66). A line of work [10
7 has focused on the class- imbalanced setting, formulating generalized NC geometry and proving its optimality. The loss landscape of the UFM was shown to be benign in [71
[T
20
24
68 and the emergence of NC in the
UFM through gradient descent training was proved in 42
16
24
57
Several extensions of the
UFM to non-standard settings have been considered, including GNNs (28, large number of classes
25 unconstrained features regressed to the input data [53] and regression
[ Recently, the UFM

---

has been used to describe a form of NC in language modeling, where each context (sample) can be followed by multiple continuations, making the labels effectively stochastic
50
66
NC has been
considered also for more layers following empirical observations 17
45
22
and accordingly.
UFM was generalized to multiple linear layers R 14 36
two non-linear layers in [52] and
multiple non-linear layers in [48, 49 13].
Beyond UFM. Going towards the analysis of neural networks trained end-to-end, conditions on data that make NC feasible in the shallow case are identified in [9] Two-layer networks are considered
in 277. which uses NTK theory and other kernel methods to conclude that NC in this regime is rather
restricted. To the contrary, in the mean-field regime, positive results about NC1 are given in [6]
for certain three-layer networks. In the deep case, convergence to NC is studied in [46] 44 64 23
However, a block- struut-tdt empirical NTK is assumed in [46], and symmetric quasi-interoolation is The former does not justify this assumption, while the latter requires an unusual
required in 44 64
weight regularization and interpolators with a given norm. Wide networks are considered in [23]
which proves the emergence of NC1 requiring at least the last two layers to be linear (and even deeper linear heads for NC2 and NC3).
Closer to the scope of the current work, NC is studied in ResNets in [58] Two main claims are proved: the monotonicity of NCI-NC2 metrics across layers of ResNets, and a negative result about collapse in a variant of UFM similar to the one considered in [53]. However, the monotonicity is proved under the strong assumption that the data evolves across layers on a geodesic, which is not possible in general since one can construct configurations where samples from different classes would collide. Moreover, the UFM taken into account is based on a heuristic derivation (a link between representation cost and transport cost of the features) that does not hold exactly in practice.
3 Preliminaries
Notation. We study two different data formats and architectures. For ResNets, the input data and one-hot labels are Xo E RdoxN and Y K where dg is the input dimension, N the number of samples and K the number of classes. For transformers, the input data and one-hot labels are Xa NXVX and Y E IR xxxxxxxC where C is the context length (number of tokens in the prompt) and V the vocabulary size (number of distinct tokens). We take C = 1 when the third dimension of Xo. Y is not used. If index a matrix with three abstract indices, the last one is implicitly equal to 1. We assume a blasnebalanced setting, i.c., NC = Kn, where n is the number of samples per class. Unless stated otherwise, we use Ty: to indicate the i-th sample of the k-th class. For transformers, a sample corresponds to the position of each individual token and, thus, T&i corresponds to token position labeled as class k with samples ordered arbitrarily. For additional notation regarding vision transformers, see Appendix B
ResNets and transformers. Let a denote the ReLU function. Denote by LN(-) the output of a normalization layer that first subtracts the mean of each column of the input from itself and then divides each column by its standard deviation (if the imput is vector, it returns the normalized vector; if the input is a matrix or tensor, it returns the matrix or tensor with centered and normalized columns of the inner-most dimension matrices). Define also i.) as the identity mapping.
Definition 3.1. An L-block ResNet with LayerNorm and one linear layer per block (later referred to as L-RNI) is defined as
fo = llin,, o LN o(id + tolooooolliiioole o(id too LN o(id to lino, (1)
where lin,(x) = Wfx + by for all E Lo....,-- 1] and for I = L we remove the bias term.
0 is the collection of all learnable parameters. We denote as X1 LN(W, bo). X1+1 IN(X, + (W,X, + bi)) (1 E (1..... L 11), (x,) = XL+1 := W,X, the intermediate representations of the training data stored in matrix form. We assume that all intermediate representations Xi (1 E (I,...,,, L)) are of dimension d. Analogously, L-RN2 denotes ResNet with two linear layers per block defined as
fo -inn o LN o(id + lin or lin LN o(id + lin oa lin1,1) o LN ling, (2) with x, = TN, + 1o), X = IN(X, + W1.0 X1 + by,1) + (r,)) (1 E (1, 1})
and fo (Xo) = X1+1 == W.X1.
3

---

Definition 3.2. An L-block transformer with one or two linear layers in the attention sub-bloch and one or two layers in the MLP sub-block (later referred as L-TI1, L-T12, L-T21, L-T22 based on the number of linear layers in attention and MLP sub-blocks, respectively) is defined as
fo(Z) linn+1 o oN o BL ..... B1 Embed
Here, l W1+1& Z is the last layer; Embed(Z) WeZ / + W p is the embedding layer with W. being the loken embedding and p (having the same shape as We Z) the positional embedding; and the 1-th block is given by
Bi = MLP; o L o ATTN, LN
(4)
Such block consists of the normalization layers 1, LN1,2 the MLP
MLP = Z + oWi + b.), or MZP,(Z) = Z + W1,20 (Wi, 2+, + b1.2
(5)
respectively for the architecture I.-Tx/ and T.-Tx2, and the single-head attention
(T Z W WvoZA,(Z), A( softmax (M 2" Z"WokkW WQK 2/vd),
(6)
or (TTN((Z) = Z + WoWv A(ZZ) = softmax + Z" W 2 Woz/Vd),
respectively for the architecture L-Tlx and L-T2x. The matrix M the masking matrix whose entries are -co on the lower triangle and O on the upper triangle and the diagonal.
Remark 3.3. Both of the above definitions consider the post-LN versions of ResNets and transformers. where the LayerNorm acts in between residual connections. We work with this version here because the arguments are cleaner, but the results do not qualitatively change if we used pre-norm ResNets or transformers instead. We discuss pre-LN architectures and their proof Appendix B
Neural collapse metrics and generalized unconstrained features model (GUFM). Regardless of the model, let h-(+) (-) be the output of the corresponding architecture before the last layer, i.e., the feature on which neural collapse is defined. We denote by ah, the i-th sample of the k-th class in the 1-th layer. We define 14
E rhe as the class-means in the 1-th layer and μ/a := R ΣK u as the global Knn mean. Let En := N Exim k2 (=h,, v/) / )² and Σ/ K Σ Ek=1 (ut pl) - ,()² be the within- and between-class variability matrices in the 1-th layer, and M be the matrix of class-means stacked column- wise. Let Ex Ix 1x 1x1 be the un-rotated ETF matrix. We define below neural collapse and its metries for generic matrices.
Definition 3.4. Any pair (W, X) of matrices s.I. W has at least many columns rows, X has N = Kn columns and they can multiply as W X has the following NC metrics:
NC1 tr(Ew) T((E) Le., the ratio of within- and between-class variability.
.
min W - -F ll
NC2A(W,X) =
i.e. the distance of WWT from the closest (scaled) ETF. IWWTIF
min WW -cIK l1r
NC2B(W;
.ie. the distance of WWT from the closest (scaled) identity.
WWF/|p I|e
N E cos(E Ws:), i.e., one minus the average cosine similarity between the samples and the corresponding row of W.
NC3(W, 1
A model is said to exhibit NC if all metrics are and approximate NC if all metrics are close to zero NC2A is defined for CE loss or MSE loss with unregularized bias in the last layer, and NC2B is defined for MSE loss with bias-free last layer.
We consider the following optimization problem:
min ((fo(X), A |012
where À > O, 0 is the subset of parameters that excludes biases and the parameters in embedding layers (W., Wp, Wo), and a continuous, non-negative loss. Let LCE, LMSE be CE and MSE loss, and Lt,, (0) be the loss of the L-RNm, L-Tlm, or L-T2m architecture with parameters 0 (it will

---

be clear from the context whether this refers to ResNet or transformer). We denote by L the } the set of parameters e-close to
optimal such loss value and by M = {0: Lm. (0) S LL. + the optimum. We denote by Mr the set of all pairs (WL. hx())) s.t. @ (including Wt) is in M In our theoretical analysis, we will reduce the end-to-end problem 7 into a simpler unconstrained features model, which we define below.
Definition 3.5. Given a continuous loss L 2 O and an equivalence relation R. on fI...., N), the generalized unconstrained features model (GUFM) refers to the following optimization problem:
min C(WX, Y) + X 2 |WW]2
(8)
W.X
S.t. |2:||| = Vd, 1/ 00, for i E (1, 2; Zj. for NJ,, i~R.. J.
where W E Kxd x= == IN| E RRxN and E KRN Let LoUrM W X) be the loss of the feasible pair (W, X) under this model, Courss the optimal such loss and MGUFM ((W, X) M LOUFM(W, X) S Caurss + t}, with M the set of feasible solutions.
The mean-zero constraint æ.-1e O comes from the application of this model to ResNets and transformers having LayerNorm before the last layer, which allows the model to represent only zero-mean solutions. We note that this is without loss of generality for CE/MSE loss, since for those losses the optimum is zero-mean. The equivalence relation constraints are introduced to account for potential hard constraints from the input data where we may have identical samples or contexts that may or may not be in the same class. Again, for CE/MSE loss this is without loss of generality, given that all identical contexts are always labeled with the same class (see Assumption 4.4).
4
Main results
4.1 Analysis of the generalized unconstrained features model
We start with a lemma showing that nearly-optimal solutions of the GUFM problem above must necessarily be close to the global optima.
Lemma 4.1. Denote as distmax(A, sup dist(z, fo any sets A, B. Then, we have
.EA
lim sup Distmax(MMune GUUM, MoUEM M = 0.
Proof. Assume by contradiction there exists a sequence (Xn, W )= of points such that lim (W X,) = CūUMM but lim limsup dist( n. X,), M 700
> 0. Then, since the feasible set of GUFM is compact (for W,, take large-enough ball around 0 that must contain the global optimum), we can choose a subsequence (Xay- Wn, ) having an accumulation point (W.)) in the feasible set and s.t. dist((W; X). > O (first picking a subsequence for which the limsup above is realized and only choosing subsequence with accumulation point from this subsequence; then using the continuity of the distance to conclude). From the continuity of the loss function, it must follow LOUTM (W, X)= LGUFM: which also implies (W, X) M MGUFM However, this is contradiction because the distance of this point from GGFFM is both and bigger than O. D
n n-o
Next, we focus on CE and MSE loss, showing that the optima of the corresponding GUFMs (denoted by UFM-CE and UFM-MSE) exhibit NC.
Lemma 4.2. Assume that only the samples within the same class are in relation R. Then, the global optima M UFM-CE and M MFFM-MSE are all perfectly collapsed, i.e., for all (W. X)
O and for all (W.X) E MMFM NC1(W, X) = NC2B X) NC3 [W.) X) O. Conversely, for any feusible pair (W,X² X) s.f.
M MLFM.CF NC1(W. = NC2A(W, X) NC3(W.
NCI X) = NC2A(W, X) NC3 W. X) O, there exists unique scular e.t. (cW, E
M MUFM-CE, and for any feasible pair (W, X) s.l. (C, NC2B NC3(W, X)
there exists unique scalar s.t. (cW, X) MFFM-MSE
5

---

The proof is deferred to Appendix A For the CE loss, it is based on an adaptation of the results in 77T. For the MSE loss, we compute the global optima by lower-bounding the loss, solving the problem for the lower-bound and showing that the loss and its lower-bound agree at these optima.
4.2 Deep single-layer architectures are collapsed at the global optimum
We first consider ResNets/transformers with one linear layer per MLP block.
Theorem 4.3. Let the architecture be L-RNI or L-Txl for x E {1, 2). Assume the inner dimension of the L-Txl is at least 2V+2 and the inner dimension of L-RNI is at least Consider the optimization problem 7 with X independent of the number of layers. Consider also its corresponding GUFM (8)
with the same loss L and the equivalence relation defined by pairs of samples in X that coincide (for transformers, these correspond to pair of identicul contexts). If GGUMM > 0, then
lim sup distmax(Ms MMGUFM _MGUMM= = 0.
(10)
The result above provides a reduction of the end-to-end training objective of a deep-enoughh archi- tecture to GUFM using the same loss. This has two important implications. First, it shows that optimal deep ResNets and transformers can represent the optimal solution of the corresponding GUFM problem. As formalized in Corollary 4.5 this gives a precise characterization of the structure of feature representations in the last layer at the global optimum - the first result of this sort for modern architectures beyond MLPs. Second, it provides theoretical justification for using the UFM to explain the emergence of NC, showing that the UFM does not oversimplify the problem even when dealing with ResNets and transformers. We note that the lower bound of 2V + 2 on the dimension of transformers is for technical convenience, and it can be loosened to a lower bound that does not depend on V. We now give proof sketch deferring the complete argument to Appendix A
Proof sketch. We start with the sketch for the L-RNI model. Notice that (0) CWURM(W1,, XT.) + ELS [[il The goal is to show LGUFM- = lim Li,1, which implies - that LCUFM (Wr, Xx) ). for (Wr, Xt) E Mr converges to LGUPM- Thus, (Wr, Xx) E Mr must also belong to MGGUMM for e small enough, which by Lemma 4.1 guarantees the convergence as in (10).
Note that we can represent a mee-block-deepe by simply adding an identity block with zero weight matrices/biases and residual connection left untouched. Thus, Li, is non-increasing in L and it suffices to prove the limit for any sequence of L's going to infinity. We will prove it by explicitly constructing sequence of L-RN1 ResNets s.t. their losses converge to LGUFM as I. + cc. This crucially relies on the fact that it is possible to almost perfectly fit the training data X, with ResNets so that the sum of Frobenius norms of all their layers converges to O. This is a special property of residual networks that qualitatively differs from MLPs.
ResNet that perfectly copies the original ResNet
To build the intuition on why this is possible, consider a ID example where we want to fit the label exp(a) when the input is with a ID ResNet. Let x be a shared weight across all layers. Then, we need exp(a)=
(1 + )z)", which can be asymptotically achieved by setting x = a/ L. Importantly, the sum of Frobenius norms El (%)² = r. vanishes as L - co. In other words, by "splitting" the mapping done by a single ResNet layer into L layers with smaller weights, the total cost is smaller. A similar intuition was also used in A.
For multi-dimensional ResNet and general data, the idea of the construction is to split the blocks of the ResNet into N groups, with each group moving only a single sample (for simplicity we assume all samples are distinct in this proof sketch). In this way, it is possible to split the layers within one group into several layers implementing the same mapping with a smaller Frobenius norm. Each sample has predefined smooth trajectory from its initial position to the near-optimal position under the GUFM, and the group of blocks responsible for moving this sample approximates a smooth movement along this trajectory. As the depth increases, the total cost of these layers decreases, since each of them gets smaller with rate 1/ LL, as in the 1D computation above, thus giving the desired result.
Next, 1et us consider the L-Txl model. The key observation is that transformers are basically strict extension of ResNets, with attention layers being the only extra component. However, setting the attention layers to 0 and directly applying the result above for the I-RN1 model does not immediately work. In fact, for each token, we need attention layers to acquire information from previous tokens, which may be useful to fit the label. At the same time, if we want the sum of Frobenius norms of all

---

the layers to converge to zero, so must all the key and query matrices, which makes attention scores in all layers necessarily converge to uniform across the entire past.
The solution is to design the embedding layer and the first transformer block so that distinct contexts of Xo remain distinct and their distances do not converge to 0 too fast (this would disrupt our construction for ResNets which implicitly assumes that the initial distances between samples are constant w.r.t. L). The embedding layer and first attention layer encode the contexts so that the j-th entry contains the history (encoded in binary) of all the tokens belonging to the j-th class from the past. At this point, a slight adjustment of the construction for ResNets finishes the proof.
O
Note that the uniform attention which we use in the proof, although asymptotically optimal, is not expected to be optimal for any moderate number of layers and only use it in lnite-layer constructions for mathematical convenience and consistency with the asymptotic case. We highlight that Theorem 4.3 holds for any continuous loss. By considering CE or MSE for which the global optima of the corresponding GUFMs are collapsed by Lemma 4.2 the emergence of collapse in ResNets and transformers readily obtained, assuming the following about training data:
Assumption 4.4. For the ResNet architecture, we assume all training samples in X to be umique. For the transformer architecture, we assume the labels Y to be uniquely determined by the context, Le., two identical contexts in two different input sequences will be assigned the same label.
Corollary 4.5. Let the architecture be L-RNI or L-Txl for x E (1.2). Assume the training data (X, satisfies Assumption 4.4 and all the assumptions of Theorem 4.3 Using CE or MSE loss, all global optima of the optimization problem 7 exhibit approximate neural collapse which gets tighter as L increases.
We make several remarks about this result.
Rate of convergence. While the results are stated asymptotically for simplicity, one can readily recover a convergence rate of the global optimum NC from the argument. In particular, since the total regularization of the layers scales as L-1, the global optima can only be suboptimal w.r.t. the GUFM objective with the same scaling. Then, assuming a differentiable loss (e.g.. CE or MSE), the distance from the optima scales as the inverse of the power in the Taylor approximation of the loss at the global optima the flattest direction, up to logarithmic factors that come from making a finer approximation. Now, for the CE loss, the leading term is quadratic: by using the chain rule, the slope of CE at the optimum is non-zero, and the sum of exponentials of dot-products between X, W is quadratic as we approach the ETF. Thus, the convergence distance is ÖI, where Ô omits logarithmic factors. For the MSE loss we compute by error analysis in the proof of Lemma 4.2 that the convergence rate is also (I2-
Language modeling. When considering the transformer architecture, require the labels to be unique given a specific context. While this is a realistic assumption in vision or language classification tasks (c.g., sentiment analysis, harmful content classification, spam detection), does not apply to language pretraining, where single context may have many different continuations. In fact, in the setting of non-unique continuations, neural collapse is not to be expected, and the optimal structure was discussed [50] (66) by using a form of UFM. We remark that Theorem 4.3 shows that the optimal solutions identified in these works are exhibited by transformers, as long as they satisfy the conditions in (8). This is the case, for instance, in some symmetric settings, see Proposition 2 in [66] with slight modification in the underlying UFM (the authors consider weight decay instead of norm constraints the features), where the optimal limiting solution is indeed collapse. In non-symmetric cases, while NC is not expected to be optimal (as the case with class imbalance (5I), transformers still represent the optimal zero-mes pero-mean solution of the underlying UFM, whatever that is. This allows future work to focus on solving the application-relle relevant UFM in the corresponding setting and then use Theorem 4.3 to conclude that the solutions are globally optimal end-to-end.
Deep neural collapse. Although our theory focuses on last-layer geometry, the analysis sheds some light on the collapse in the earlier layers as well. In particular, one can readily obtain that any finite number of layers at the end of the network converges to neural collapse (with the exception of NC3 which has a different formulation for multi-layer collapse). Note that adding residual connection (as in ResNets and transformers) resolves the inconsistency of deep UFMs pointed out in [49, where it is shown that the global optima of the deep UFM in the multi-class setting do not exhibit neural collapse. In fact, the optimal solution of deep UFM with residual connections obtained by simply copying the shallow UFM in the first layer and setting all remaining layers to O. We also remark
7

---

that, from the argument of Theorem 4.3 network (L being a constant independent of L) converge to the global optima of the corresponding deep GUFM with residual connections and depth L.
it follows that the global optima of the last layers of the
In contrast, understanding the emergence of neural collapse for small, but constant fraction of the final layers of the network appears to require a different approach. Intuitively, the network starts processing all samples at once from some layer onwards (which is expected to improve the loss w.r.t. our construction), then the collapse is progressive and occurs to some extent already in a constant fraction of the final layers, also the discussion in [58]
4.3
Deep double-layer architectures collapsed at the global optimum with vanishing regularization
Let us now consider ResNets with two linear layers per block and transformers with two linear layers per MLP sub-block (the number matrices in the attention sub-block does not affect the result). Then, show that neural collapse is globally optimal, provided that the regularization strength in all layers except the last one decreases with the depth L.
Theorem 4.6. Let the architecture be L-RN2 L-Tx2 for E (1,2). Assume the inner dimension of the L-TxI is at least 2V +2 and the inner dimension of L-RNI is at least 4. Consider the optimization problem
min ((fo(X), 0
AL 2
A(L) 1 - +
2 1001
(11)
where XL is a regularization on the weight matrix of the last layer that does not depend on and XLL is a depth- dependet regularization s.t. (L) o(log(L)))) Consider the corresponding
GUFM with regularization NL. If COUFM > 0, then
sup distmax ML AMM MMUFMM = 0. 1+00
(12)
The reason why the regularization is required to be vanishing can be already seen from the ID example mentioned in the proof sketch of Theorem 4.3 in order to ensure that (1 + x²)² converges to exp(a) as L + oc, one needs to pick x /I' which implies that the sum of squares El= is of constant order w.r.t. L. In fact, the requirement vanishing regularization is necessary for the statement to be true, and the result also cannot hold if both X(L) and X are vanishing. An additional discussion this point, together with concrete dataset for which collapse cannot be reached, are provided in Appendix C Understanding the structure of the optimal representations for double-layer architectures in the regime of constant regularization represents exciting future direction.
va
The proof of Theorem 4.6 is similar to that of Theorem 4.3 In particular, the first layers of the blocks are defined in the same way, and the second layers are to act as a projection matrix on the space spanned by the output of the first layer, which has rank 1. Furthermore, the scalings of these layers are split in identical square roots of the scaling of the original layer. Thus, the sum over the squared Frobenius norms is constant w.r.t. L, which requires X(L) to vanish in L. The detailed proof is deferred to Appendix() [A] We conclude the section by stating the approximate optimality of NC in the global optima of double-layer architectures under CE or MSE loss.
Corollary 4.7. Let the architecture be L-RN2 or L-Tx2 for E [1,2). Assume the training data (X, Y) satisfies Assumption 4.4 and the assumptions of Theorem 4.6 Using CE or MSE loss, all global optima of the optimization problem (77) exhibit approximate neural collapse which gets tighter as L increases.
5
Experimental results
Our theoretical results suggest an improvement of the NC metrics the global optima as the depth increases. To empirically verify whether this effect is already present at moderate depths and for solutions found by gradient descent, we train ResNets and transformers on MNIST 29] CIFARIO 10 31 and IMDB 21 with increasing depths in (2, 3, 5, 8, 13, 21, 21, The hidden dimension is 64, the learning rate 0.005 for vision and 0.001 for language and the (constant) regularization 0.005 for
architectures having one linear layer per block and 0.005/ L for architectures having two linear layers
8

---

-0.2
-0.
-15
Log30 MC2
-0.3
Logi0 NCJ
-16
Logão
0.4
-0.4
-17
-18
-0.6
-0.5
-19
-0.8
13
21
34
-2.0
13
blocks
blocks
-0.8
-0.2
1.0-
-0.3
Log10 NC2
NC3
-0.4
-0.3
0.4
-0.6
-0.5
-188
13
21
5
21
34
13
23
number of blocks
number of blocks
number of blocks
-0.2
-0.4
0.8-
-0.4
-0.6
NC1
Lag10 NC2
-0.6
NC3
-0.8
-1.2
-1.0
0.8-
12-
-1.
1.2
-1.6
13
21
3
5
21
34
13
21
numbeer
number blocks
numbeer
7-
-0.6
2.0
-1.0
Logio NC2
-0.8
Logio NC3
-2.5
-1.2
-10
-1.
-3.0
-1.4
-1.4
-1.6
13
34
5
13
number of blocks
number blocks
number of blocks
Figure log10 of NCI, NC2 and NC3 metries respectively in the left, middle and right column, as function of the number of blocks L. First row: L-RNI on CIFAR10; second row: L-T11 on CIFAR10; third row: pre-LN L-T11 on IMDB; Fourth row: T.RN22 on MNIST with X x I.-1.
per block. Each setting trained for 5 different random seeds for 5000 epochs on CE loss, the results are averaged, and the error bars at one standard deviation are reported. We use pre-LN transformers for language experiments and, due to training instabilities, only report the runs which converged by the end of the training. See Appendix D for additional experimental results.
function of the depth of the architecture. The results are in agreement with the theory developed in Section 4 across different datasets and architectures. NC metrics improve with depth, even when the solutions are obtained via gradient descent. Furthermore, for large enough depth, the plots roughly follow log- figelinear trend with an average slope of around -0.335, especially for ResNets. This suggests polynomial dependence between NC metrics and depth L, which is also consistent with our theory, see the remark on the rate of convergence in Section 4.2 including the quantitative estimate of the slope being between 11/2 and 1/4. The metrics are generally a bit larger (meaning less strong collapse) than the ones usually measured in MLPs [48), but this because collapse ResNets approached more slowly due to different loss landscape. Finally, we remark that [17 45 consider the effect of depth, but instead focus on the progression of NC metrics across layers, rather than evaluating such metrics in the last layer as function of the overall depth.
Figure T shows the three NC metrics convergence,

---

Conclusion
This work provides global optimality guarantees for neural collapse in two modern architectures: ResNets and transformers. Besides [23] for simplified MLPs, this is the first end-to-end global optimality result for NC in deep networks. Our approach involves reduction to general form of unconstrained features model that holds for any continuous loss. This provides formal justification for the validity of the UFM as a modeling principle and motivates future work on in new settings, such as language modeling [50] 66]. Experimental results confirm our theoretical predictions on standard datasets trained via gradient descent, thus providing simple recipe for practitioners thriving to achieve a strong collapse in applications [4 38] just increase the depth.
Although the analysis covers wide range of models, the behavior of global optima for architectures with two linear layers per block and constant regularization remains open. While we know that NC is not asymptotically reached for all datasets, studying the tradeoff between representation cost and fit loss (and, thus, the extent of NC in global optima) is an important open problem. Beyond that, our work suggests several interesting future directions. First, by improving the constructions used to prove Theorems (43 and 4.6] one could obtain more refined bounds on the convergence rate in terms of the depth L, leading sharp NC guarantees already for a moderate number of layers. Second, it would be very exciting to adjust our results to describe deep neural collapse and quantify the evolution of NC metrics across depth, thereby refining the results in [58]
Acknowledgements
M. M. and P. S. are funded by the European Union (ERC, INF2, project number 101161364). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. This research was supported by the Scientific Service Units (SSU) of ISTA through resources provided by Scientific Computing (SciComp).
References
[1] George Andriopoulos, Zixuan Dong. Li Guo, Zifan Zhao, and Keith Ross. The prevalence of neural collapse in neural multivariate regression. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
[2] Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Conference on Neural Information Processing Systems (NeurIPS), 2019.
[3] Daniel Beaglehole, Peter Súkeník, Marco Mondelli, and Mikhail Belkin. Average gradient outer product as a mechanism for deep neural collapse. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
[4] Enric Boix-Adsera. On the inductive bias of infinite-depth resnets and the bottleneck rank. arXiv preprini arXiv:2501. 19149, 2025.
[5] Yang Cao, Yanbo Chen, and Weiwei Liu. Prevalence of simplex compression in adversarial deep neural networks. Proceedings of the National Academy of Sciences, 122(17))e2221553122, 2025.
[6] Mayee Chen, Daniel Y Fu, Avanika Narayan, Michael Zhang, Zhao Song, Kayvon Fatahalian, and Christopher Ré. Perfectly balanced: Improving transfer and robustness of supervised contrastive learning. In International Conference on Machine Learning (ICML), 2022.
[7] Hien Dang, Tho Tran Huu, Tan Minh Nguyen, and Nhat Ho. Neural collapse for cross-entropy class-imbalanced learning with unconstrained relu features model. In International Conference on Machine Learning (ICML), 2024.
[8] Hien Dang, Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho. Neural collapse in deep linear network: From balanced imbalanced data. In International Conference on Machine Learning (ICML), 2023.
10

---

[9] Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy S Liang, Improving self- supervised learning by characterizing idealized representations. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[10] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. In Proceedings of the National Academy of Sciences (PNAS), volume 118, 2021.
[11] Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the implicit bias towards minimal depth of deep neural networks. arXiv preprint arXiv:2202.09028. 2022.
[12] Tomer Galanti, András György, and Marcus Hutter. Improved generalization bounds for transfer learning via neural collapse. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML, 2022.
[13] Connall Garrod and Jonathan P Keating. The persistence of neural collapse despite low-rank bias: An analytic perspective through unconstrained features. arXiv preprint arXiv:2410.23169, 2024.
[14] Connall Garrod and Jonathan P Keating. Unifying low dimensional observations in deep learning through the deep linear unconstrained feature model. arXiv preprint arXiv:2404.061066 2024.
[15] Jarrod Haas, William Yolland, and Bernhard T Rabus. Linking neural collapse and 12 normal- ization with improved out-o--distriuution detection in deep neural networks. Transactions on Machine Learning Research (TMLR), 2022.
[16] X. Y. Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In International Conference on Learning Representutions (ICLR), 2022.
[17] Hangfeng He and Weijie J Su. A law of data separation in deep learning. Proceedings of the National Academy of Sciences, 120(36), 2023.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
[19] Wanli Hong and Shuyang Ling. Beyond unconstrained features: Neural collapse for shallow neural networks with general data. arXiv preprint arXiv:2409.01832 2024.
[20] Wanli Hong and Shuyang Ling, Neural collapse for unconstrained feature model under cross- entropy loss with imbalanced data. Journal of Machine Learning Research, 25(19)):1-48, 2024.
[21]
reviews.
[22] Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understand- ing generalization in deep learning. arXiv preprint arXiv:2202.08384 2022.
[23] Arthur Jacot, Peter Súkeník, Zihan Wang, and Marco Mondelli. Wide neural networks trained with weight decay provably exhibit neural collapse. In International Conference on Learning Representations (ICLR), 2025.
[24] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer- peeled perspective on neural collapse. In International Conference on Learning Representations (ICLR), 2022.
[25] Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu. Dustin G Mixon, Chong You, and Zhihui Zhu, Generalized neural collapse for a large number of classes. In Conference on Parsimony and Learning (Recent Spotlight Track), 2023.
(26) Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? In International Conference Machine Learning (ICML), 2023.
11

---

[27] Vignesh Kothapalli and Tom Tirer. Kernel vs. kernl: Exploring how the data structure affects neural collapse. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
[28] Vignesh Kothapalli, Tom Tirer, and Joan Bruna. neural collapse perspective on feature evolution in graph neural networks. In Conference on Neural Information Processing Systems (Neur[PS), volume 36, 2023.
[29] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[30] Daniel Kunin, Atsushi Yamamura, Chao Ma, and Surya Ganguli. The asymmetric maximum margin bias of quasi-homogeneous neural networks. In International Conference on Learning Representations (ICLR), 2022.
[31] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-bart basedd learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998.
[32] Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Gernandez-Grande Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural collapse. In Conference on Parsimony and Learning (Recent Spotlight Track), 2023.
[33] Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[34] Tong Liang and Jim Davis. Inducing neural collapse to fixed hierarchy-aware frame for reducing mistake severity. In Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
[35] Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong. Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distribution generalization. In International Conference on Learning Representations (ICLR), 2024.
[36] Haixia Liu. The exploration of neural collapse under imbalanced data. arXiv preprint arXiv:241..17278. 2024.
[37] Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax based out-of-distribution detection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[38] Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing neural collapse in deep long-tailes learning. In International conference on artificial intelligence and statistics, pages 11534-11544 11544. PMLR, 2023.
[39] Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59, 2022.
[40] Jiawei Ma, Chong You, Sashank Reddi, Sadeep Jayasumana, Himanshu Jain, Felix Yu, Shih- Fu Chang, and Sanjiv Kumar. Do need neural collapse? Learning diverse features for fine-grained and long-tail classification. openreview, 2023.
[41] Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Mitoš, and Tomasz Trzcinski. The tunnel effect: Building data representations in deep neural networks. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[42] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):11, 2022.
[43] Vardan Papyan, X. Y. Han, and David L Donoho. Prevalence of neural collapse during the terminal phase deep learning training. In Proceedings of the National Academy of Sciences (PNAS), volume 1 117, 2020.
[44] Akshay Rangamani and Andrzej Banburski-Fahey. Neural collapse in deep homogeneous classifiers and the role of weight decay. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
12

---

[45] Akshay Rangamani, Marius Lindegaard. Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference on Machine Learning (ICML), 2023.
[46] Mariia Seleznova, Dana Weitzner, Raja Giryes, Gitta Kutyniok, and Hung-Hsu Chou, Neural (tangent kernel) collapse. In Conference on Neural Information Processing Systems (NeurIPS), volume 36, 2023.
[47] Jingtong Su, Ya Shi Zhang, Nikolaos Tsilivis, and Julia Kempe. On the robustness of neural collapse and the neural collapse of robustness. Transactions on Machine Learning Research (TMLR), 2024.
[48] Peter Súkeník, Marco Mondelli, and Christoph H. Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[49] Peter Súkeník, Marco Mondelli, and Christoph H. Lampert. Neural collapse versus low-rank bias: Is deep neural collapse really optimal? Conference on Neural Information Processing Systems (NeurIPS), 2024.
[50] Christos Thrampoulidis. Implicit optimization bias of next-token prediction in linear models. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
[51] Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[52] Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference Machine Learning (ICML), 2022.
[53] Tom Tirer, Haoxiang Huang, and Jonathan Wiles-Weed Perturbation analysis of neural collapse. In International Conference on Machine Learning (ICML), 2023.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[55] Haoqi Wang. Zhizhong Li, and Wayne Zhang. Get the best of both worlds: Improving accuracy and transfcrability by grassmann class representation. In Conference Computer Vision and Puttern Recognition (CVPR), 2023.
[56] Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, and Qing Qu. Under- standing deep representation learning via layerwise feature compression and discrimination. arXiv preprint 1..iv:22311.029600 2023.
[57] Peng Wang, Huikang Liu, Can Yaras, Laura Balzano, and Qing Qu. Linear convergence analysis of neural collapse with unconstrained features. In NeurlPS Workshop on Optimization for Machine Learning (OPT), 2022.
[58] Sicong Wang, Kuo Gai, and Shihua Zhang. Progressive feedforward collapse of resnet training. arXiv preprint arXiv:2405.00 009855 2024.
[59] Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pre- trained models are from neural collapse on the target dataset informs their transferability. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[60] E Weinan and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Mathematical and Scientific Machine Learning, 2022.
[61] Diyuan Wu and Marco Mondelli. Neural collapse beyond the unconstrainted features model: Landscape, dynamics, and generalization in the mean-field regime. arXiv preprint 20114 2025.
13

---

[62] Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models. arXiv preprint arXiv:2405.17767, 2024.
[63] Yingwen Wu, Ruiji Yu, Xinwen Cheng, Zhengbao He, and Xiaolin Huang. Pursuing feature separation based on neural collapse for out-of-distribution detection. CoRR, 2024.
[64] Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds. In Research, volume 6, 2023.
[65] Jiawei Zhang, Yufan Chen, Cheng Jin, Lei Zhu, and Yuantao Gu. Epa: Neural collapse inspired robust out-of-distribution detector. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6515-6519, IEEE, 2024.
[66] Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis. Implicit geometry of next-token prediction: From language sparsity patterns to model representations. In First Conference on Language Modeling, 2024.
[67] Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, and Jiaya Jia. Understanding imbalanced semantic segmentation through neural collapse. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[68] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features. In International Conference on Machine Learning (ICML), 2022.
[69] Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: neural collapse perspective. In Conference Neural Information Processing Systems (NeurIPS), 2022.
[70] Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Bal- anced contrastive learning for long-tailed visual recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[71] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our claims exactly match the presented results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the concluding remarks.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

All assumptions are clearly stated or referenced in the statement of the theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide enough information in the description of experiments make our results reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We do not find it necessary release the code. Our experiments concern the training of rather standard architectures and they can be readily reproduced without needing upload the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /nips.cc//ppp http:ppblligggiieeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide the necessary details. All the other hyperparameters are irrelevant to the claims made in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report confidence intervals in all our numerical experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Our experiments require only modest computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have read and conform to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The paper is theory-oriented and no potential societal impact is foreseen.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This is not relevant to our paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
This is not relevant to our paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This is not relevant to our paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This is not relevant to paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This is not relevant to our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: This is not relevant to our paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy ottp:e/s//eerrppcccc/o/oootrooos for what should or should not be described.
21

---

A Deferred proofs
Lemma 4.2. Assume that only the samples within the same class are in relation R. Then,
the global optima M MUFM-CE and MDFM-MSE are all perfectly collapsed, i.e., for all (W. X) E
M MUFM-CE NCI(W, x) NC2A(W, X) NC3(W. = -0 and for all (W. E Mo M 0. Conversely, for any feasible pair (W,X) s.f.
NC X) = NC2B X) NC3(W, X)
NC ) NC2 X) = NC3 W. (.X) X) = O, there exists a unique scalar e s.J. (cW, X) E
MUFM-CE, and for any feasible pair (W, X) s.l. (C NC2B(W, NCW X) = 0,
there exists unique scalar e s.I. (cW, X) E MO MURM-MSE
Proof. For both losses, we will relax the problem and ignore the constraints coming from the equivalence relation R. Then, we prove that NCI holds in all of these cases, which grants equivalence between the relaxed and original problem.
For the CE loss, we apply Theorem 3.1 of [71]. In particular, from this theorem it follows that the optimal solutions of the regularized UFM-CE (not a-priori equivalent to (8) because of the feature constraint) exhibit neural collapse. From their proof. it is also clear that not only does the ratio between the sizes of the optimal we and F4: only depend the ratio of the regularization terms, but also that the absolute size of these vectors is an increasing function of the regularization strength, with the limit À + oo being infinity. Therefore, let us pick Xw from the paper be À in (8) while we find Au s.t. the optimal solutions of the problem in 711 have norm vd. Then, the global optima of the regularized UFM-CE are exactly those of the UFM-CE we consider in (8)
To the last statement, assume by contradiction that there is a global optimum of the problem in (8) which is not a global optimum of the regularized UFM-CE. Then, we can plug this solution into the regularized UFM-CE. Since it is not global optimum, there exists solution with strictly lower loss, and this optimum is guaranteed to have unit norm features. By plugging this optimum into (). we must obtain a loss that is better than the optimal one, since the objectives are equivalent in this case. This leads to a contradiction. Similar arguments give that there cannot exist a global optimum of the regularized UFM-CE which is not a global optimum of (8). thus proving the desired equivalence.
For the MSE loss, we perform a direct computation which includes a perturbation analysis. To simplify the loss landscape, we start by defining lower bound on the UFM-MSE loss, which we will analyze first. Denote
K,n Σ R
A [ (w) TH TKI - 1)2 + 2
CUFM-MSE = 2NN
(13)
and MMFFMMMM MSF the corresponding near-optimal set. Note that 13 is separable in the index k, thus we are facing K identical, independent optimization problems. We will now do a series of partial conditional optimizations and comment the cost of deviating from these conditional optima. First, conditioning on any wk (corresponding to the k-th row of W), we can almost exactly specify the optimal values of T&a for any i. In particular, if [|wx||| < d= then the optimal solution is The = -vd wk// ||wx||| If wxll > d=$ then the optimal solution is any vector on hypersphere such that w/ "f= mzi 1. the former case, for each Tki, a deviation from the optimal value of the dot- dot-product W mu. = /d ||w results in a quadratic increase in the loss around the optimal point (the cosine function has zero linear term in the Taylor expansion and non-zero quadratic term) or quartic if | = d= (because the loss at optimum would be 0 and being itself quadratic function, the effects would multiply). In the case || >> d -=3 the loss increase around the optimum again quadratic. Therefore, in all cases the maximum allowed deviation from the optimum given an extra loss of is at most (c4/4) and, thus, goes to 0 as goes to zero
Now, denote |wsl|| The loss of the k-th group only depends on and Tais but plugging-in the optimal value after solving for Ty: we arrive at a single-dimensional objective that only depends on 2:
2K - (1-2Vd) max(1 2vd,0) 0) 2
From the form of this optimization problem, it is clear that the unique global optimum reached on (0, d d-) The solution is simply vā(--K]) First, we note that, for fixed X, K, this solution is strictly
22

---

smaller than d' with non-zero margin. Second, any deviation from this optimal solution will result in quadratic increase in the loss function, therefore for fixed extra loss of e, the maximum allowed Moreover, since its optimal value (and also maximum allowed deviation for small-enough) is strictly smaller than d' 1, we know that the optimal value of the 22s is indeed vd w&/ /|/lel and the maximum allowed deviation is also (c21/3)
deviation of || from its optimal value is ( which also goes to with e going
The function value in 11 cannot be optimized any further, thus we know what Mo MUFM-MSE is. In MUFM-MSE must satisfy the NCI and NC3 properties. Now, if the global optima of (8) with MSE loss and 13 are equal, then MUFM-MSE C Mu M.FMMMSS and thus the optimal solutions of with MSE loss must also satisfy the NC1 and NC3 criteria from the lemma statement.
particular, the solutions in M/
To show that the global optima are equal and to argue about NC2, we turn back to the original problem 8 with MSE loss. Since we know that the optimal solutions agree, can focus directly on MD MST. After plugging any optimal solution of (13) into LUEM we see that the regularization
part is constant, so we are left with optimizing the fit part. Analyzing the loss incurred by 2ki on position * i we see that it is (wf raa)² (w] wx ) )d + >X)2 Summing this over all indices and samples (using the symmetries) we see that the total loss is proportional to the Frobenius norm of the off-diagona elements of W WW Therefore, lower-bound on the loss is O, which is achievable provided W has at least as many columns as rows, as assumed in the lemma. Let us simply choose W to be scaled orthogonal matrix, and note that the loss cannot be optimized any further. Thus, we see that CÜRM MSE = LÜFM MSE and the solutions of (8) with MSE must satisfy NC2. Any deviation of W from an orthogonal matrix will result in an increase in the loss which is at least quartie: given a fixed extra loss of c, the solution in UFMMMME must be (4t//4) close to an orthogonal matrix.
Finally, the converse statements also readily follow from the above computations.
Theorem 4.3. Let the architecture be T.-RNI or T--xxl for x E (1, Assume the inner dimension of the L-Txl is at least 2V +2 and the inner dimension of L-RNI is at least 4. Consider the optimization problem 7 with independent of the number of layers. Consider also its corresponding GUFM (8)
with the same loss L and the equivalence relation defined by pairs of samples in X that coincide (for transformers, these correspond to a pair of identical contexts). If CGUPM- > 0, then
lim sup distmax(M,, MMGUFM .MGUFM) 0. I. 1.0
(10)
Proof. We first discuss how to deal with the equivalence relation R. The argument is identical whether we take individual samples if all samples are distinet, or we treat the equivalence classes as individual samples. Thus, for simplicity of notation we assume, without loss of generality, that the samples are all distinct.
We start with the proof for the L-RNI model. Notice that (0) = LOUPM (W2, XL) + A M ||Wall?- The goal is to show CoUrM = lim Li, In that case, LW Xy. must -ox
converge to LoUTM. Therefore, (WL, X2) induced by 0 E Mb must also belong to MGGUTM for e arbitrarily small, which evoking Lemma 4.1 guarantees the convergence as defined in (10).
Note that we can represent a one-block-deeper ResNet that perfectly copies the original ResNet by simply adding an identity block with zero weight matrices/biases and residual connection left untouched. Thus, Li,, is non-increasing in L and it suffices to prove the limit for any sequence of L's going to infinity. We will prove it by explicitly constructing sequence of L-RN1 ResNets s.t. their losses converge to LūURM as L + co.
Pick any (WL, Xt) E MM and relabel H = X,, Thus, hyg is the feature representation of the sample ki in the penultimate layer. Define H as the matrix of unique points hys and let us index them with a single index as hy. Denote the number of these unique points as K. If we write jki) we mean the index such that hy = hgs Before starting the construction, we need define a key data-dependent quantity. First, take X, = LN(W.,Xo + bu) for by and Wo sampled from a continuous distribution. Since points in X, are all disjoint, this property holds also for X, with probability 1. Moreover, with probability zero, any sample in X1 is identical to his for any of the vectors in H. For simplicity, we will refer to X, and its samples as X and drop the index. Fix an ordering of the points Zgg the lexicographical ordering of (k, i). For each (k, i) find a smooth oriented curve Ga.
23

---

connecting z4: with hug on the set of feasible points (vd norm hypersphere with zero-sum entries) such that all of the following holds:
The curvature of GRv defined as the Lipschitz constant of the unit-norm oriented tangent function Thi is bounded by B.
2. For all (1,j) (k, i), max=ffa I d(1 dd1-m) m.) for some 0, ie., all the subsequent points
=Eth: Fiy are far enough from the curve Gki-
3. There is precisely one point Txi E Gk: such that Thg = d(1 cm), where > 1 is chosen large enough. Denote GRi as the set of points on Guo between DAs and Tss- Then, we assume that, for all (l, < (k, JEEI
x" S d(1 m).
max
4. The length of GR. is no more than 2r
5. m. is chosen small enough s.t. 10cm & (d- max hT (Jkks) Jas stp))// j(ki)F((pp)
It is clear that a construction satisfying these properties exists, since the constants B, m are chosen with respect to X, H and the number of points we consider is finite. We also note that this requires the inner dimension of the representations to be at least 4 since this would not be possible on a 2D circle.
The idea of the construction is as follows. Take L large enough and divide the layers into N + K + 1 blocks. The first N blocks are of the same number of layers Lı, and the depth of the last will be specified later. Each of the first N blocks of layers will focus on single sample, while not changing the representation of the other samples at all. The goal of the ki-th block is only move the ki-th sample on its curve towards hgs, until it hits Txg. Then, the K next blocks of depth L2 will move all the samples corresponding the same j(ki) at once, ever closer their respective (1)(k)) vectors. Finally, the very last block which consists of the very last layer will simply be chosen as the optimal WL corresponding H.
We will now construct explicitly all the layers. Denote by W! Ris byg the parameters the I-th layer of the kitth block and define I to be the feature representation the ki-th sample as an input to that
layer. Consider a sphere with center o/e and radius aL.mvd where abe is a small-enough number 2 2
whose role will be clear soon. Since this sphere is small enough and GRv has bounded curvature, there exists exactly one point ./+1 on the intersection between Gk: and the considered sphere which is
closer to byg x- Denote dh.
The weights are constructed follows:
2/d1m274
1
2 dl (xf m.
Wt: = dbi
(14)
Vd + m2/ Vd
6/. = (1- m
a Vd
1,
dd + m²
if Thgs S d(1 cm), otherwise W! 0; b/. = 0. The af is an optimizable parameter and since the form above is also W's SVD, it is its singular value. Thus, o(W + bla)) 2/d4m2/1 definition. Therefore, before zyi hits its final destination, have
al. (1 + d'a), while a(W w. + t(a)) 0 for any (s,t) * (k) thanks to our margin
vdl
The + al.mya
d/is
aym (1 + dla)) 2 m²/4
2
1+1 LN 1/ss ki
2411 ks
26
x//s + 2 m4/44
From this, it is clear that æti is moving along and the curve, while the other samples stay stationary.
24

---

It remains to compute how fast travels along the geodesic with this construction. To this end, denote $/ := <1 kes Bxt) as the spherical angle between d and T.i. Let A == Pk B i.c.
the angle shift of z in the 1-th layer of the ki-th block. Using simple trigonometry we can compute:
A arcsin
ammm 2 + m2 /4
m
ki
4 4vd
where the inequality holds for di small enough.
Therefore, it suffices to choose L and Lı large enough and set a and 0 otherwise. In this way, the total regularization cost of the layers in the first N blocks can be upper bounded
4v
Lim if (T S d(l-em)
K.n.,
32dm
Σ
[ [
Lim
We see that this cost goes to 0 as L1 goes to infinity.
After N blocks, all the samples now lie within the e-multiple of margin (< his 2 d(1-cn)) cm)) of their respective optimal hyj features. The goal of each of the K blocks move the corresponding samples in the j-th group all together ever closer to these final vectors. Since this time the construction will be equivalent for all the samples within one group. we will refer these samples simply as single j-th sample in the /-th layer of the respective block, using the notation z/j. We define all layers in the j-th block follows:
+ emhj hy 1 cmhy Vd'
W = df
0² Va + cmh,l
6) -1 ---1-ecm
1,
where again af is an optimizable parameter. By similar computations as above, the above construction
makes sure that or (W)ar) + +b,) [ || (0h at - d 2cmd)1 + cmhţ xr) while oWWja!
6)) for i + j. After subtracting the mean in the layer norm we are adding a aiemh? at hy, which is 1I+mh,
at least af cm multiple of hy. Denote $ a(a), h2) and A - 8/. 2vd
Using trigonometry again, we get:
alem sin (33)
em (in(
A
arctan
2vd(1 + cm cos( (3))/(2V))
4v/(1 + a cos (os(8))
16vd
where incqualities hold from basic properties of trigonometric functions for small-enough angles.
a.em,,, If we choose ag = at constant Thus, the angular shift is lower bounded as follows: A 2 16vā
La
across layers, get 8/- <
ar cm 16v7
$9.
We will choose aj 16 T.2), Then, the total regularization of the layers in the penultimate La
blocks is upper bounded as follows:
L
27 Nd m²Lz
Σ
2
1J=1
This goes to zero linearly up to poly-log factors as Le goes to infinity. Finally, we have that the final positions z of the samples converge fast to their optimal counterparts hag with L. To see this, plugging our choice of aj into 3/2 <
a,em 16vā
La
39 we get 8/. s log(Lz) L2
1.2 $39 S 1
20
so the samples converge lincarly to their optimal positions L1, L2 co. From the continuity of
25

---

the fit part of the loss, we see that the total loss of this construction indeed converges to GGUFM of the corresponding GUFM problem. Therefore, our upper bound on the loss of globally optimal solutions converges to GGUFM and evoking Lemma 4.1 we know that (Wr. L.; Xr.) optimal for (7) is nearly optimal for (8) and thus exhibits the required convergence.
Next, we continue with the proof for L-TxI. Notice that, if we can ensure after the end of the first block that all the different contexts have different representations and that two representations of different contexts don't t lie on a line with some of the final positions logs, then by setting all the weights in attention layers of the subsequent blocks to O, the rest of the transformer becomes a ResNet with LayerNorm and we can apply an identical construction as in the L-RNI part to conclude. The only caveat (except making sure that the margin positive) is that, since the total regularization loss of the construction for L-RNI goes to 0 with L going to infinity, we must make sure that the same is true for the first block. However, as we will see, this will make the margin m a function of I. that slowly goes to O. To compensate for this, we will need to set the layers in the subsequent blocks accordingly bigger, and we will make sure that the margin goes to O slowly enough so that this adjustment will not qualitatively change the results. Another issue we have to deal with is that if the norm of the Wok matrix has to go to O, the attention weights must necessarily converge to uniform. Thus, our construction must withstand this burden.
We will start with the construction of the embedding matrices. The embedding matrix We E &ddd will just lift the dimension to the dimensinn of the transformer di 2 2V + 2, i.e., the w-th column of W. is e, in dy-dimensional space. Then, the (C' - i))th column of W, E BaxcC will be +2 + b- <2v++2 where a, b > O are the unique solutions of the following two equations: a + b = -1 and a² + 62 = 22(i+1) - 1. Thus, after the embedding layer, the sum of the entries of the entire embedding 0 and after the first normalization layer, the j-th token at the (C i)-th position
will have (d2-((+11 on its j-th entry and the only other non-zero entries will be at positions 2V + and 2V + 2.
Let us construct the first block. Here, all MLP layers will be set to so that they have zero effect. Moreover, due to the constraints discussed above, attention matrices Wx, WQ or WoK will also be set to zero Finally, the value and output matrices will be set as Wv Wo Wvo = (L) where A shifts all entries from the range 1, ..... V to the range V + ,,, 2V, and (T.) is decreasing function converging 0 infinity that will be defined later. This ensures that the representations before and after attention are summed in the residual connection on different positions, which will be technically convenient later. Since the attention matrices are identically zero the attention weights corresponding to the e-th token will just be uniform 1/c for all the tokens up to this one. Therefore, the representation of the e-th token after the attention layer and before the residual connection is the (L)) of the average of all the representations of the previous tokens and itself from an input to the first block shifted by V positions.
(L)A or
We now show that two different contexts must necessarily have different representations, which gives that the margin after block is non-zero. If we compare two samples (contexts) with different context lengths, then they will necessarily have different numbers of distinguishable summands (i.e. various negative powers of 2, divided by the sample's context length) present in the entries between (V + 1)-th and 2V-th. Since there is a different number of summands, there must exist at least one entry where the number of summands disagree, and the numbers in this entry must have different numbers of ones in their binary representation, which guarantees that samples with different context lengths must have different representations. Furthermore, two samples with the same context length but different contexts will be divided by the same averaging number, but then they can be distinguished since the map from contexts to representations (without dividing by the context length) is injective due to the uniqueness of the binary representation of the summands.
Therefore, all non-identical contexts have different representations and, in addition, the previous argument also shows that every pair of representations of two different contexts is linearly independent. This remains true after the residual connection. If we choose 7L) small enough for all L, then the original encodings of the current token will not mix up with the much smaller summands from the attention layer. The relative size of all the summands stays the same also after normalization and the MLP block has no effect, so all different contexts have different representations after the first layer. The only issue we could face is that the representations end up coinciding with one of the has's s. To avoid this, Wo or Wvo can implement a tiny rotation. Since the number of tiny rotations is uncountably infinite, there is at least one for which there is no intersection. Let v dmm
26

---

be the minimal distance between representations of any two samples after the attention mixing, before the multiplication by value and output matrices and before the residual connection. Note that vi is positive and independent of X, Y, I,, because the different contexts are all pairwise linearly independent. Then, after the multiplication by the value and output matrices, such distance will be ((L))) dm.. For small enough >LL, the worst-case addition in the residual connection corresponds to the case in which the two samples with the same latest token also realize the margin minimum. However, if (L) S 0.1, then the difference of the samples after the residual connection and after the normalization is at least equal to the distance between the representations on positions V + 1 to 2V, which is least ( Thus, this is the minimum pairwise distance of the data after the first attention block.
Next, we can apply the construction for L-RNI if we set all the remaining attention layers to zeros, since then the remainder of the network will be functionally equivalent to L-RNI. The only remaining issue is that the margin after the first layer is function (L) of the total number of layers. To choose a good scaling of (L), we need to consider the elements of the construction for L-RNI that depend
on the margin, which is the sum of the Frobenius norms of the layers in the first N + K blocks. This is
by 32- XNd + 128 L2)2" Therefore, we choose (L) = ( / = O(L)
L Lim
m²Lg
then both the sum of Frobenius norms of the layers in first N layer blocks, as well as the Frobenius
norms of Wv, Wo Wvo in the first block of the transformer will go to 0 L1 L2 - co. This
concludes the proof.
O
Corollary 4.5. Let the architecture be L-RNI or L-Txl for (X,Y) Y) satisfies Assumption 4.4 and all the assumptions of Theorem 4.3 Using CE or MSE loss, all global optima of the optimization problem as L increases.
(1.2). Assume the training data
an4
7 exhibit approximate neural collapse which gets tighter
Proof. This is a straightforward combination of Lemma 4.2 and Theorem once we use that identical contexts for transformers are only labeled by one class, which allows to directly apply the lemma.
Theorem 4.6. Let the architecture be L-RN2 L-Tx2 for E {1,2]. Assume the inner dimension of the I--xxl is at least 2V + and the inner dimension of L-RNI is at least 4. Consider the optimization problem
min ((So(X), dL 1|221 + A(L)
(11)
where dL is regularization on the weight matrix of the last layer that does not depend on L and XLL is a depth-depender regularization s.t. (LL) o(log(L) (L))( Consider the corresponding
GUFM with regularization X1. If fGUFM > 0, then
lim sup distmax(MM. M .MGUFM) = 0. 1
(12)
Proof. The proof follows that of Theorem 4.3 The only difference is that the construction of the weight matrices changes that W and b have Vale in place of ag. The second layers' weight matrices W of the first sub-layer on sample a so that the total mapping will be identical to the single-layer construction. Using analogous computations as above, we get:
ki are defined as / Val. mulliilpss of the projection matrix on the span of the output
X(L)
K.n.ly
w ki
W ki
16vdT\(L)N
+
2
m
and for the second part of the blocks we get:
L
|| 16N (6N
m
レ-=1
In order for the sum of these two components to go to zero, we need X(L) (llog(T2) and we can choose L1 = (L2)). The rest of the proof is identical to that of Theorem 4.3
D
27

---

Corollary 4.7. Let the architecture be L-RN2 or L-Tx2 for E {1.2}. Assume the training data (X, Y) satisfies Assumption 4.4 and the assumptions of Theorem 4.6 Using CE or MSE loss, all global optima of the optimization problem (7) exhibit approximate neural collapse which gets lighter as I increases.
Proof. This is a straightforward combination of Lemma 4.2 and Theorem 4.6 once we use that identical contexts for transformers are only labeled by one class, which allows to directly apply the lemma.
B Alternative architectures
B.1
Vision transformers
For vision transformers, the data is tensor-like X, Nxxxxxx where C' now denotes the number of patches and dg is the dimension of the patch. However, the labels remain two-dimensional Y E BNNK What is considered as a sample depends on how labels are produced in the transformer. The simplest option (w.rt. the rest our paper) is to generate the prediction on the last patch of the sequence, keeping the causal mask. This will, however, change the definition of "samples" and the NC metries, since we only need to focus on the last patch. Therefore, samples will only considered as the last patch, and the NC metrics will only be defined over the representations of the last patches. Similarly, the equivalent DUFM will also correspond to the last patches.
Theorem 4.3 and 4.6 and, thus, also Corollary 4.5 and 4.7 hold for vision transformers too, as long as we do the following changes the proof of Theorem (the other statements are adjustable trivially once this is established).
Necessary adjustments to the proof. Together with the uniqueness of the labeling function, we will also assume that the samples are taken from continuous distribution (which is reasonable in the vision domain). This guarantees that the feature representations of the final patches are unique also after the first transformer block, as the event that averages over patches of two different samples coincide has zero probability. The rest of the proof is similar to that of Theorem 4.3] but the subsequent MLP layers only focus on the movement of the last patches' representations and the movement of the other patches is irrelevant.
B.2 Pre-LN ResNets and transformers
Unlike the post-LN ResNets (Definition 3.1 and transformers (Definition 3.2). the pre-LN archi- tectures apply the LayerNorm directly before the attention and/or linear layers, but only within the residual connection, leaving the main residual stream untouched. While this potentially makes the features initialization grow linearly with depth, it makes for more stable gradients thanks to the direct residual path, avoiding LayerNorms that can serve as error propagation channels. This significantly simplifies the training dynamics and therefore the pre-LN transformers are currently being predominantly used. For this reason, we fully define the pre-LN architectures here and then discuss in sufficient amount of detail how adjust the proof for this setting, since the results are qualitatively the same.
Definition B.1. An L-block pre-LN ResNet with LayerNorm and one linear layer per block (later referred to pre-T.-RNI) is defined as fo ling LN o(id aoling o o (id +oooling-2 o (id tooling oN o ling.
(15)
where lin:(x:) Wjx + by for all E (0...., L.) and 0 is the collection of all learnable parameters. We denote as X1 = LN X, + bp), X = X₁ + LWLLN(X + by) (1 E {1....., L 1}), fo(X,) X 1 := W, LN(X,) the intermediate representations of the training data stored in matrix form. We assume that all intermediate representations Xi (1 E (l,..., L}} are of dimension
d. Analogously, L-RN2 denotes ResNet with two linear layers per block defined
So = =liu,, LN o(id + lin oa lin o LN) o...o o (id + +lin,, oa o lini, o oLN) o LN o lings
(16)
with x, = T. TN, + by), X = X (W1, LN(X,) + b,,)) + b, (1 E (I,...,,. T. 1))
and fo (X,) = XL+1 Wz LN
28

---

Definition B.2. An L-block pre-LN transformer with one or two linear layers in the attention sub-block and one or two layers the MLP sub-block (later referred to as pre-L-TI1, pre-L-T12. pre- T.-T2I, 12-2 based on the number of linear layers in attention and MLP sub-blocks, respectively) defined as
(Z) linZ+1 o LN o BL 0... B1 o LNo Embed
(17)
Here, lin = WL+12 + b1,+1 is the last layer (bI+11 is a matrix with the same number of columns as 7. that are all identical); Embed (2) W Z W is the embedding layer with W being the token embedding and Wp (having the same shape as W Z) the positional embedding; and the 1-th block is given by
B = (id + MLL, o LN1,2) o (id + ATTN, LNL.).
(18)
Such block consists of the normalization layers 1 L. the MLP
MLPi(Z)= - oWi + bi), or MLP: respectively for the architecture Tre- and Lre-L- and the single-head attention
W1,2o(Wi,, + by,1) + b1.2,
(19)
ATTN( = WvoZAA(Z), A(Z)) softmax(M (M + 7" QK 7/ 2/va), or (Z) = WoWwZA((2), ArZZ) softmax(M + zz WZ WaZIva,
(20)
respectively for the architecture pre- L-T-- and L-e-L- The matrix M is the masking matrix whose entries are -co on the lower triangle and 0 on the upper triangle and the diagonal.
We note that the first LayerNorm right after the embedding layer, which might not be used in practice often, is introduced for technical convenience but does not change the results qualitatively. Theorem 4.3 and 6 thus, also Corollary 4.5 we do the following changes the proof of Theorem 4.3 (the other statements are adjustable trivially once this is established).
4.5 and hold for pre-LN architectures too, as long
Necessary adaptations to the proof. This architecture has the disadvantage that it does not immediately absorb deviations from the zero-sum sphere and therefore, technically, the single linear layer architectures can only add non-negative changes to the residual stream. However, we argue that an almost identical construction to the one in proof of Theorem works here as well. Note that the construction from this proof, see (14):
1 1 Th dd (T W/: = alss
V m²/4 Vd.
a/aVa d + m2
b/g -1 (1-1)
1,
will result in shift in oa that can be written alan (1 + d(..) and the 1 will not get absorbed 2
in the residual stream, but is orthogonal to the zero-sum component of the movement of ax and will get absorbed in the next LayerNorm within the next residual stream. This allows us to copy the entire first part of the post-L.N proof by mimicking the trajectories of the unit ball, while adding the
constant amount of al.mv/d
2//dmm²/1 -ulliple of all-ones vector in each round. Therefore, after the first
N blocks, the projections of all the samples on the zero-sum hyperplane are identical to those in the post-LN proof. Each sample, however, has different component in the direction of the all-one vector. This will, however, be absorbed by the last LayerNorm. Moreover, by triangle inequality, the margin of the trajectories in this extended space is at least big the margin of the trajectories zero-mean ball. The construction for the next K blocks works by the same reasoning as well. Thus, after these layers, the projections of the samples on the zero-sum ball are identical to the post-LN proof and the last LayenNorm will absorb the component along the all-ones vector.
the
As for the transformers, although after the first block the samples are not centered and do not all have norm vd, after applying the LayerNorm in the first subsequent MLP block, they will all be distinct (except the ones with identical contexts). Therefore, we define the same trajectories as in the ResNet construction with the centered and normalized features, but we will perform an equivalent movement on the aero-mean ball with the radius equal to the norm of the projection of the particular
29

---

sample onto the zero-mean hyperplane, while ignoring the all-ones component completely. As result, each sample moves on its own cylinder, a projection to the zero-mean hyperplane following the trajectories on the normalized zero-mean ball, while moving arbitrarily along the all-ones direction. As before, a triangle inequality guarantees that the margin defined on the zero-mean normalized ball is not violated in the wider space during this process. The only caveat is that, if the norm of the ball along which a sample is traveling is larger than that of the Vdd-normed ball, we need to upscale oi;; by that ratio. Note that the size of the vector after the first block is upper bounded independently of the number of layers, therefore such an upscaling will only multiply the cost of weight matrices by constant. The rest of the argument follows that of the adaptation for pre-LN ResNets.
C Two linear layers per block with non-vanishing or uniform weight decay
Here, we intuitively describe why the NC metrics in general do not approach the perfect NC in architectures with two linear layers per residual block as the depth goes to infinity, if the regularization non-vanishing or vanishes uniformly across all layers. The key is the simple inequality ||AB]] S ||A|| [BI] We can interpret these matrices as features, weight matrices and the change on the features added to the residual. In particular, in a ResNet with a single linear layer per block we have ||Xill S | I| (AX; is the outcome of the residual branch added back to the residual stream) and, importantly, this inequality can be made equality in some cases. Even if the inequality does not hold as an equality, we still have that, for fixed Wi, X,, if AX; o (W; o O, then due to homogeneity cáX; = (c. This makes the total change Wi makes to Xr scale linearly with but its cost is quadratic. Therefore, if the directional derivative of the loss w.r.t. AX; at layer / is strictly positive, then there exists e > 0 for which cWi will make an improvement against Wi- = 0. However, if two linear layers involved, we have
| S (i S //44 (IWLa +|WLall+
Therefore, any change to the features will scale linearly with the regularization cost of the matrices that were responsible for this change. In this case, the opposite to the previous statement holds: if the directional derivative of the loss w.r.t. AX, is small enough. then for small c, the c-scaling of weight matrices will necessarily worsen the loss compared to doing nothing.
As we have seen in the proof of Lemma 4.2 for the MSE loss (but this also holds for the CE loss), an D(e)-sized perturbation around the global optimum of neural collapse causes only an OEe2) increase in the loss. Furthermore, the derivative is zero at NC and locally Lipschitz around that point, which implies that the size of the derivative is Oe)). For any input dataset X that is not yet collapsed, the points Wr,X, in the set of global optima/ Mr. did approach NC in the limit, we could, by contradiction, take an optimum that is c-close NC (for sufficiently small e) and zero-out all the last layers that were responsible for moving the samples by a total amount of (c) shift (this would need care in rigorous proof because of the possible discontinuity of the eoyer-to-feature mapping). The change in the fit part of the loss would be O(E"), but thanks to the above inequality, the total regularization cost saved by this would be le)), so the loss would improve and we would arrive at a contradiction.
[
The above argument holds for constant regularization A. However, even if the regularization was vanishing, but it was the same for WL and for the rest of the network, the NC would still not be approached. To
this for MSE loss, consider a perturbed perfect scenario where the input data is X= @ 1 + E and F is a perturbation matrix of size O(c). X is already c-close to NC. To move X ((c) closer to NC, we need ()),c) cost in terms of the weight matrices. Let now compute the improvement in the corresponding GUFM objective that results from doing so. The DUFM objective with MSE is 2N [| - 1 + 1 [W]] If we simplify the problem to just fitting a single row of W (the optimization problem is separable, so this is w.l.o.g.). we have a simple ridge regression solution for w. In particular
w² = (nIK + ALLK + +(()) @ 1 + +(e))
Therefore, the distance from the unperturbed fit (nIx + - @ 1 is itself O(e) and plugging this in the loss, we see that the change in the loss function O(A which, for sufficiently small e, is less than the price in terms of weight regularization.
30

---

-0.4
LRN1
LT11
-0.6
LRN2
Log 10 NC1
0.8
-1.0
-12
-1.4
2
3
13
21
34
number of blocks
0.4
-0.6
Log10 NC2
-0.8
-1.0
LRN1
-12
LT11
LRN2
-1.4
2
13
21
34
number blocks
-1.50
-1.75
OTGor NO
-2.00
-2.25
-2.50
LRN1
-2.5
LT11
LRN2
33.00
2
3
13 number of blocks
5
8
21
34
Figure 2: MNIST training. Iog10 of NCI, NC2 and NC3 metries respectively in the upper, middle and bottom row, as a function the number blocks L. The architectures are L-RN1 with 0.005. L-T11 with À 0.005, and L-RN2 with 0.0025.
D Additional experimental results
In Figure 2 we provide additional experimental results which complement Figure with MNIST training of both ResNets and transformers with one linear layer per MLP block. The results and the message are consistent with those of Section 5 Furthermore, we consider a ResNet with two linear layers per block trained MNIST, but with constant weight decay of 0.0025. As we can see, the NC metrics are almost constant across multiple depths, which is consistent with our claim that NC is not approached in this regime.
Additionally, in Figure 3 we show the sum of Frobenius norms all the weight matrices of ResNets trained on MNIST. We can see that as the number of layers increases, the total norm of the weights decreases. We note that this is the same phenomenon leading to neural collapse our theory. Similar results hold for the other training scenarios well.
31

---

Log10 regularization losses
-0.9
-1.0
-1.1
2
3
5
8
13
21
34
number of blocks
Figure 3: 1o of total regularization loss of ResNet trained on MNIST as function of the depth of the network.
32


Reviews:
[{"review_id": "LlafpTOAeZ", "reviewer": "Reviewer_s3Hv", "summary": "The paper establishes a theoretical connection between the generalized unconstrained feature model (GUFM) and deep post-LayerNorm residual networks and Transformers.\n\nThe authors first show that all global optima of GUFM, under CE and MSE losses, exhibit vanishing neural collapse metrics. They then prove that sufficiently deep L-RN1 and L-Tx1 networks can approximate any such GUFM solution, and hence the collapse properties. Later on, the authors extend their analysis to show in L-RN2 and L-Tx2 (2 layers in each MLP block) also share the same property as the regularization strength goes to 0.\n\nEmpirical results incorporated in the paper demonstrate that as depth grows, neural collapse metrics consistently decrease.", "questions": "* Is it possible to design some experiments to show that the constructed optima in the paper are achievable? \n* It seems like the NC metrics shown in the experiment section are not very small given the very long training time (5000) epoch. What would be the cause of this? Based on my previous experience, it would be better to include some kind of learning rate annealing techniques to encourage NC. I wonder whether the observed results still hold when adding this kind of more subtle optimization strategy.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The focus of the paper covers a very interesting yet not well-established theoretical question. Many previous NC theoretical results focuses on proving using UFM but the connection between UFM and modern deep networks remain elusive. This paper steps towards closing this gap.\n* The thretical results are quite comprehensive, cover both residual based network and Transformers, also consists of different architecture choice within these models.\n\nWeaknesses:\n* The notation is a bit un-organized, and it hinders the readibility of the paper. For example, $lin$ is defined as a affine transformation but in the last layer $lin_L$, the bias term is dropped. \n* The global optima constructed in the paper is not verified: we don't know whether gradient based optimization method would converge to such solutions, it would be nice to design some experiments to check this.\n* Related to the previous point, the paper does not address whether all global minima exhibit neural collapse, this would something valuable to check perhaps in future work.", "was_revised": true, "final_justification": "This is an interesting paper within the Neural Collapse community, and I recommend acceptance. Given the complexity of the problem being analyzed, I would think the use of constructed solutions (while questioned by several reviewers, including myself) to be a reasonable choice.", "rebuttal": "We thank the reviewer for the comments and the overall positive evaluation. We address concerns and questions below.\n \n---\n\n**Weakness 1. Unorganized notation.**\n\nThanks for pointing out the notation inconsistency. We will make this explicit in the revision. \n\n---\n\n**Weaknesses 2. Are global optima constructed in the paper achievable by gradient descent?**\n\nWe note that the optimality we show is in the limit $L\\to\\infty$: our result proves neural collapse for global optima as $L\\to\\infty$. This means that, when $L$ grows, as long as gradient descent finds a global optimum, the penultimate layer of the gradient descent solution will necessarily coincide with that of the solution we propose. Now, there may be different ways to converge to the limit. In the paper we describe one that is convenient to analyze, in which samples are moved one at a time, i.e., different layers are used to move different samples. Gradient descent may choose a different solution and, in fact, we would expect architectures trained via gradient-based optimization to move several samples simultaneously. \n\n---\n\n**Weakness 3. Do all global optima exhibit neural collapse?**\n\nYes, Corollaries 4.5 and 4.7 show that **all** global optima exhibit approximate neural collapse (Corollary 4.5 is for deep single-layer architectures, and Corollary 4.7 is for deep double-layer architectures). Thus, **any** solution which does not exhibit neural collapse cannot be a global optimum. \n\n---\n\n**Question 1. Is it possible to design some experiments to show that the constructed optima in the paper are achievable?**\n\nAs mentioned above, we do not necessarily expect the constructed solutions to be achievable by gradient descent. While our construction moves one sample at a time, we expect solutions obtained via gradient descent to be more efficient in the way the network uses the depth. However, we note that the convergence rate predicted by our theory is in line with what we observe in numerical experiments. In fact, the slopes of the NC curves in Figures 1 and 2 for networks with one linear layer per residual block (computed starting from depth 5, since the shallower depths are too weak to follow the trend) average at around -0.335. This is comparable to our theory, which ranges between $-1/2$ and $-1/4$, depending on the setting. \n\nWe also note that the main takeaway of our proof remains true in practice: as the depth grows, layers get smaller, splitting the total work, so that their total norm is decreased. To support this claim, we have computed the sum of all layers’ squared Frobenius norms multiplied by a constant regularization parameter for ResNet trained on MNIST (corresponding to the experiment of Figure 2 in the Appendix), as a function of the depth of the network.\n\nNumber of layers: [2, 3, 5, 8, 13, 21, 34]\n\nMeans: [0.1467, 0.1471, 0.1360, 0.1182, 0.0986, 0.0835, 0.0711]\n\nStandard deviations: [0.0015, 0.0014, 0.0004, 0.0005, 0.0010, 0.0006, 0.0002]\n\nThis clearly shows that, as the number of layers increases, the total norm of the weights decreases. We note that this is the same phenomenon leading to neural collapse in our theory. Similar results hold for the other training scenarios as well. We will add a figure in the revision.\n\n---\n\n**Question 2. The NC metrics are not very small, considering the large training time. Why? Would LR annealing help?**\n\nThe main reason why the NC metrics are not very small for shallower networks is that the collapse is not *exactly* optimal in shallow ResNets and transformers. However, as our theory predicts, the precise collapse gets approached more and more closely by global optima as we increase the depth. It is visible from Figure 1 that this is indeed the case. Moreover, the slopes of the NC curves in our experiments are comparable to our theory, see the answer to the previous question. We believe LR annealing techniques would not help decrease NC metrics further, because we already employ a ten-fold LR drop at epoch 4000 in all our experiments, and the NC formation is not heavily affected by this (in fact, it is slowed down, as opposed to being speeded up). We will clarify this point in the revision."}, {"review_id": "qMrO7V1UL8", "reviewer": "Reviewer_PA89", "summary": "This paper addresses a central theoretical question in modern machine learning: Is the prevailing neural collapse globally optimal (in the training data)? The authors prove that NC is globally optimal in deep ResNets and Transformers trained end-to-end with weight decay under cross-entropy or MSE loss. The authors successfully demonstrate that the core mechanism driving collapse is depth and regularization, not architecture details. \n\nThe core of their analysis involves a reduction from the end-to-end deep optimization problem to a Generalized Unconstrained Features Model (GUFM). In GUFM, last-layer features are treated as free variables constrained to a unit sphere with zero mean. Previous works proved NC optimality in GUFM for MLPs or in simplified regimes, but this paper rigorously shows that deep ResNets and Transformers approximate GUFM behavior at global optima as depth grows, even when data structure is taken into account.\n\nI particularly appreciate the key conceptual innovation in transformers: by carefully designing the embedding and first attention block, the authors ensure that even with uniform attention weights (zeroed key/query matrices), the necessary context information is encoded into token embeddings. Said another way, Transformers can be shown to approach GUFM-optimal NC solutions using only very weak attention mechanisms.", "questions": "1) Is LayerNorm necessary? Could your results be extended to BatchNorm or to ResNets without normalization? \n2) Do you anticipate low-rank bias (as discussed in recent work) to appear in your framework under different regularization conditions?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength: \n\n1) Novel theoretical result for ResNets and Transformers:\nThe first proof of NC optimality in architectures with skip connections, LayerNorm, and attention layers.\n\n2) Rigorous reduction to GUFM:\nBridges the practical training setup of deep networks with unconstrained theoretical models previously considered too simplified.\n\n3) Elegant use of depth as capacity control:\nShows that residual architectures can split large transformations into small norm-preserving steps, enabling global optimization.\n\n4) Unified treatment of CE and MSE loss:\nApplies to standard losses used in practice, with theoretical guarantees for both.\n\nWeakness: \n1) The assumption of uniform attention might seem unrealistic to readers unfamiliar with theoretical proof constructions. A clarification that this strengthens the claim by showing NC even in the worst-case attention regime would help.\n\n2) Not enough discussion of finite-depth limitations: While convergence rate is discussed (e.g., $O(L^{-1/2})$), readers would benefit from an example\n\n3) For readers not specializing in optimization, could the authors elaborate, in intuitive terms, why the regularization cost in Theorem 4.6 must vanish, whereas in Theorem 4.3 it can remain constant, even though both constructions distribute feature movement across many layers? In particular, can they clarify how the composition of two linear layers per block introduces a “norm floor,” and how this affects the feasibility of reaching GUFM-optimal solutions at finite cost?", "was_revised": true, "final_justification": "The authors' responses to my questions and other reviewer's questions are convincing. I congratulate the authors for their great work", "rebuttal": "We thank the reviewer for an insightful review and we are delighted by the reviewer's positive evaluation of our paper. Below, we address questions and concerns.\n\n---\n\n**Weakness 1. The assumption of uniform attention might seem unrealistic. Please clarify that this strengthens the claim, being the worst-case scenario.**\n\nThank you for pointing this out. We will explain more clearly in the paper that the uniform attention only serves as a theoretical construction tool and it is not expected to be present in global optima of moderate-depth networks.\n\n---\n\n**Weakness 2. Provide an example for the convergence rate and discuss the finite depth limitations.**\n\nWe have measured the slope of the NC metrics in our experiments of Figures 1 and 2 for architectures with one linear layer per residual connection (computed starting from depth 5, since the shallower depths are too weak to follow the trend), and we have found that its average is at around $-0.335$. This is in agreement with our theory, which predicts a convergence rate between $L^{-1/2}$ and $L^{-1/4}$ (depending on the setting) and, hence, a slope between $-1/2$ and $-1/4$. We will report this observation in the revision.\n\n---\n\n**Weakness 3. Elaborate, why the regularization cost in Theorem 4.6 must vanish, whereas in Theorem 4.3 it can remain constant.**\n\nThis is an important point, which can be best explained through our 1D example briefly discussed in lines 239-244 and 330-333. If we want to fit a constant label, say $\\exp(a)$, from an input of size 1, we need to scale our 1D ResNet’s $(1+x)^L$ parameter $x$ as $a/L$, which converges to the label $\\exp(a)$ as $L\\to\\infty$. This linear scaling in $L$ makes the sum of Frobenius norms scale as $1/L.$ However, this is not the case in the second type of ResNet with two linear layers per residual block. This ResNet is of the form $(1+x^2)^L,$ forcing $x$ to scale as  $\\sqrt{a}/\\sqrt{L}$. This root scaling w.r.t. $L$ makes the sum of squared norms constant w.r.t. $L$. Thus, we need to pick a vanishing regularization term to make the sum of Frobenius norms negligible as the depth grows. The way in which this affects the feasibility of achieving GUFM-optimal solutions at finite cost is explained in Appendix C. There, we give evidence towards the fact that the global optima of the neural network do not approach the global optima of GUFM exactly. To the contrary, the global optima of the neural network are expected to stay within a possibly small, but roughly constant distance from a perfect NC. \n\nWe will incorporate the discussion above in the revision. \n\n---\n\n**Question 1. Is LayerNorm necessary? Do the results hold if BatchNorm was used instead? What about LN-free ResNets?**\n\nThis is an interesting question. The LayerNorm is necessary in the first and last layers. It is not necessary in the intermediate layers, though. Therefore, our results hold for ResNets, which only employ the LayerNorm in the embedding and unembedding layers. For ResNets that are fully normalization-free, qualitatively different results have recently been shown in [1], where low-rank bias is demonstrated instead. As for the BatchNorm, we believe our results would still hold, since the presence of normalization is more important than its precise form. However, our proofs would need to be adapted significantly, since they rely on the samples having the same norm throughout the forward pass.\n\n---\n\n **Question 2. Do you expect low-rank bias to appear in your setting with different regularization?**\n\nThis is a very insightful question. Indeed, as discussed in the previous point, low-rank bias has been demonstrated in some settings for ResNets [1], and it is even responsible for NC not being optimal in MLPs [2]. However, in our setting with LayerNorms and residual connections, a rigorous answer to your question is still open. We do believe, though, that the low-rank bias is *not* present (beyond the low-rankness already guaranteed by neural collapse itself) in this setting. The intuition is that low-rank bias is mostly present in deep architectures because maintaining high-rank features requires high-rank weight matrices, which is costly in terms of Frobenius norm. However, in the ResNets and transformers we consider here, maintaining high-rank features can be done for free by setting the weight matrices to 0. From this point of view, there is no obvious reason for low-rank bias to hold in our setting. \n\n---\n\n**References**\n\n[1] Enric Boix-Adsera. \"On the inductive bias of infinite-depth ResNets and the bottleneck rank.\" arXiv preprint arXiv:2501.19149 (2025).\n\n[2] Peter Súkeník, Christoph Lampert, and Marco Mondelli. \"Neural collapse vs. low-rank bias: Is deep neural collapse really optimal?.\" Advances in Neural Information Processing Systems 37 (2024): 138250-138288."}, {"review_id": "yIis1edokr", "reviewer": "Reviewer_EMH7", "summary": "This article provides a theoretical analysis that proves that under weight regularization and standard loss functions (MSE and CE),  in Transformer and Resnet architectures, the optimal solution is approximately equivalent to Neural Collapse. And as depth increases, this approximation gets tighter.", "questions": "As a practitioner, I find it hard to quantify the importance of this theoretical work. Could you expand on why the modeling of Neural Collapse is important and what takeaways a practitioner should take? Do we always want to have Neural Collapse when training neural networks? Does this work tell us that with more depth, we can expect better generalization on these main architectures?", "limitations": "yes", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ The article is well written\n+ connects the impact of depth and neural collapse on 2 well-established architectures\n+ The empirical results follow the theory\n\nWeaknesses:\n+ As a practitioner, from reading this paper alone is hard to obtain the main takeaways from this theoretical work, for example, on what neural collapse is and why it is important.", "was_revised": true, "final_justification": "I maintain the positive review as I think this paper builds on the understanding of the interesting phenomena of neural collapse. However due to the theoretical nature of the paper that is not in my area, I found it hard to follow the theory, as such my confidence score is low.", "rebuttal": "We thank the reviewer for the positive evaluation of our paper. We are happy to address the reviewer's practical concerns about the applicability of neural collapse and we will extend our related work section in the revised version to include a more elaborate explanation concerning this point.\n\n---\n\n**Why is it important to model the neural collapse and what are the practical implications?**\n\nNeural collapse is a high-impact phenomenon, with the seminal paper [1] that introduced it reaching over 700 citations, the majority of which are practical applications of neural collapse. These include (but they are not limited to) transfer learning, OOD detection, compression and performance improvement. Neural collapse also helped theoreticians establish compelling results in generalization, OOD detection, feature learning, robustness and other areas. Please refer to lines 25-31 of our introduction for citations.\n\nThe vast applicability of neural collapse makes it highly desirable to characterize. This has prompted a line of theoretical work aimed at providing a precise understanding of the architectures/conditions under which we can expect to achieve neural collapse (and, conversely, for which architectures/conditions, neural collapse is unlikely to appear). \n\n---\n\n**Do we always want to have the neural collapse when training neural networks?**\n\nThis is a great question. While works that apply neural collapse need it and some theoretical works suggest it can have a positive impact on the performance of neural networks [1, 2, 3, 4, 5], neural collapse is not always helpful. In particular, NC is only fully achieved if the training loss is very close to zero. Therefore, in scenarios where overfitting could be an issue and we would need to apply early stopping strategies or deliberately short training, we in fact want to avoid the emergence of neural collapse. Understanding under which conditions neural collapse is desirable is important and theoretical analyses can help us answer this question.\n\n---\n\n**Does this work tell us that with more depth, we can expect better generalization on these main architectures?**\n\nWhile some works claim a connection between neural collapse and better generalization [1, 2, 3, 5], a formal, rigorous link between collapse and generalization has not yet been established. Therefore, our work does not conclusively tell us whether better generalization can be expected in deeper architectures. \n\n---\n\n**References**\n\n[1] Vardan Papyan, X. Y. Han, and David L. Donoho. \"Prevalence of neural collapse during the terminal phase of deep learning training.\" Proceedings of the National Academy of Sciences 117.40 (2020): 24652-24663.\n\n[2] Vignesh Kothapalli. \"Neural collapse: A review on modelling principles and generalization.\" Transactions on Machine Learning Research, 2023.\n\n[3] Tomer Galanti, András György, and Marcus Hutter. \"Improved generalization bounds for transfer learning via neural collapse.\" First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022. 2022.\n\n[4] Jingtong Su et al. \"On the robustness of neural collapse and the neural collapse of robustness.\" Transactions on Machine Learning Research, 2024.\n\n[5] Peifeng Gao et al. \"Towards demystifying the generalization behaviors when neural collapse emerges.\" arXiv preprint arXiv:2310.08358 (2023)."}, {"review_id": "Rjjli2F7BG", "reviewer": "Reviewer_agMG", "summary": "This paper investigates neural collapse (NC) in deep regularized ResNets and transformers with LayerNorm, trained end-to-end with cross-entropy or mean squared error loss. It proves that global optima of these architectures exhibit approximate NC as depth increases, extending prior work limited to multi-layer perceptrons. By reducing the training objective to a generalized unconstrained features model (GUFM), it shows that NC is asymptotically optimal, with the distance to perfect collapse bounded by depth. The results hold under minimal data assumptions and are supported by experiments on vision datasets, demonstrating increased collapse with depth. This work provides a theoretical justification for using UFMs to analyze modern architectures.", "questions": "No", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**: The paper offers a groundbreaking theoretical analysis of neural collapse in modern architectures like ResNets and transformers, significantly advancing beyond prior MLP-focused studies. Its rigorous reduction of end-to-end training to a GUFM elegantly connects complex architectures to simpler models, providing novel insights into NC optimality. The innovative proof techniques, and the clear justification for UFM applicability make this a contribution to understanding feature representations in deep learning. \n\n**Weakness**: \nWhen I saw that the global optima of deep models with CE loss could be formulated, I was thrilled. I had once tried to characterize the gradient descent dynamics of MLPs, where NC1, NC2, and NC3 gradually decrease with increasing layers and training iterations, but gave up due to its complexity. However, I was somewhat disappointed to see this paper’s results rely on constructive proofs. While not a flaw, I believe this limits the work’s practical utility. Due to the well-known implicit bias, when multiple optimal solutions exist, optimization algorithms tend to favor certain solutions with better generalization. The authors should add experiments to verify if the training dynamics of cross-entropy loss in deep models align with the constructed solutions (which I doubt) and explore, through experiments, how training dynamics select intermediate layer parameters to achieve NC if they deviate.\n\nGiven that this addition isn’t central to the paper’s main contribution and rebuttal time is limited, I believe the authors should, at minimum, \nmake some experiments to verify if the constructive solution will be selected by optimization algorithm  and discuss these shortcomings thoroughly in the conclusion and highlight corresponding challenges to inspire future work.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the in-depth review and the overall positive evaluation. We address the reviewer’s concerns and questions below. \n\n---\n\n**Verify if the constructive solution will be selected by SGD and discuss these shortcomings.**\n\nWe do not necessarily expect the solutions constructed in our proofs to be achievable by gradient descent. The construction moves one sample per layer, and for finite depths it serves as an upper bound to the loss exhibited by global optima. In contrast, we expect both the global optima, as well as the architectures trained via gradient descent, to move multiple samples simultaneously. However, as experimentally measured in Figures 1 and 2, we *do* expect the solutions found by gradient descent to exhibit neural collapse ever more exactly with increasing depth. In fact, we expect an asymptotic rate of convergence to neural collapse which is similar to that exhibited by our constructive solution. To support this claim, we have computed the slopes of the NC curves in Figures 1 and 2 for networks with one linear layer per residual block (computed starting from depth 5, since the shallower depths are too weak to follow the trend), and we have found that such slopes average at around $-0.335$. This is comparable to our theory, which ranges between $-1/2$ and $-1/4$, depending on the setting. We will clarify this in the revision.\n\n---\n\n**Explore, through experiments, how training dynamics select intermediate layer parameters to achieve NC.**\n\nThis is an interesting question. An extensive answer is an open problem and a great future direction (we will mention this in the conclusion). We can still give partial observations, though. First, due to the limited number of layers ($\\le 34$) compared to the number of samples ($50000$), it is clear that the layers are moving all or at least a high proportion of samples in each layer. Second, although the solution we construct is likely not achieved by gradient descent, the main takeaway of our proof (layers getting smaller, splitting the total work, so that they decrease their total norm) remains true in practice. To support this claim, we have computed the sum of all layers’ squared Frobenius norms multiplied by a constant regularization parameter for ResNet trained on MNIST (corresponding to the experiment of Figure 2 of the Appendix), as a function of the depth of the network.\n\nNumber of layers: [2, 3, 5, 8, 13, 21, 34]\n\nMeans: [0.1467, 0.1471, 0.1360, 0.1182, 0.0986, 0.0835, 0.0711]\n\nStandard deviations: [0.0015, 0.0014, 0.0004, 0.0005, 0.0010, 0.0006, 0.0002]\n\nThis clearly shows that, as the number of layers increases, the total norm of the weights decreases. We note that this is the same phenomenon leading to neural collapse in our theory. Similar results hold for the other training scenarios as well. We will add a figure in the revision."}]

Author rebuttal:

Meta-review: This paper investigates neural collapse (NC) in deep regularized ResNets and transformers with LayerNorm, trained end-to-end using cross-entropy or mean squared error loss. The authors prove that the global optima of these architectures exhibit approximate NC as depth increases.

The theoretical analysis extends prior work on the unconstrained features model (UFM) and MLP-focused studies. Reviewers highlighted that the rigorous reduction of end-to-end training to a generalized UFM (GUFM) provides an elegant bridge from complex architectures to simpler models, yielding novel insights into NC optimality. On the other hand, the analysis relies on constructive proofs, and therefore provides limited explanation of why NC emerges under gradient descent dynamics in practice. Overall, the reviewers consensually recognize the paper as a technically solid and insightful theoretical contribution.
Decision: Accept (poster)