Optimizer Choice Matters For The Emergence of Neural Collapse
Authors: Jim Zhao
Tin Sum Cheng
Wojciech Masarczyk
Aurelien Lucchi
Venue: ICLR 2026 Poster (2026)
Keywords: neural collapse
implicit bias
deep learning theory
classification
adaptive optimizers
training dynamics
adamw
sgd
weight decay
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We show theoretically and empirically that the optimizer choice affects the emergence of neural collapse (NC). In particular, we show that the dynamics of NC metrics differ qualitatively depending whether coupled or decoupled weight decay is used.

Abstract: Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the optimizer, thereby suggesting that NC is universal across optimization methods.
In this work, we challenge this assumption and demonstrate that the choice of optimizer plays a critical role in the emergence of NC. The phenomenon is typically quantified through NC metrics, which, however, are difficult to track and analyze theoretically. To overcome this limitation, we introduce a novel diagnostic metric, NC0, whose convergence to zero is a necessary condition for NC. Using NC0, we provide theoretical evidence that NC cannot emerge under decoupled weight decay in adaptive optimizers, as implemented in AdamW. Concretely, we prove that SGD, SignGD with coupled weight decay (a special case of Adam), and SignGD with decoupled weight decay (a special case of AdamW) exhibit qualitatively different NC0 dynamics.
Also, we show the accelerating effect of momentum on NC (beyond convergence of train loss) when trained with SGD, being the first result concerning momentum in the context of NC.
Finally, we conduct extensive empirical experiments consisting of 3,900 training runs across various datasets, architectures, optimizers, and hyperparameters, 
confirming our theoretical results.
This work provides the first theoretical explanation for optimizer-dependent emergence of NC and highlights
the overlooked role of weight decay coupling in implicit biases of optimizers.

Published as a conference paper at ICLR 2026
OPTIMIZER CHOICE MATTERS FOR THE EMERGENCE OF NEURAL COLLAPSE
Jim Zhao; Tin Sum Cheng
Wojciech Masarezyk
University of Basel (j fimm zhao, Sinsum.chent @uni bas ch
Warsaw University of Technology IDEAS Research Institute wo joiciech www.chch.amaaaraaeeaaaaaaa com
Aurelien Lucchi
University of Basel
aurelien. Lucchi aanniiiunbbas... ch
ABSTRACT
Neural Collapse (NC) refers the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the optimizer, thereby suggest- ing that NC is universal across optimization methods. In this work, we challenge this assumption and demonstrate that the choice of optimizer plays critical role in the emergence NC. The phenomenon is typically quantified through NC metrics, which, however, are difficult to track and analyze theoretically. To overcome this limitation, we introduce a novel diagnostic metric, NCO, whose convergence to zero is necessary condition for NC. Using NCO, we provide theoretical evidence that NC cannot emerge under decoupled weight decay in adaptive optimizers, as implemented in AdamW. Concretely, weight decay (a special case of Adam), and SignGD with decoupled weight decay (a special case of AdamW) exhibit qualitatively different NCO dynamics. Also, we show the accelerating effect of momentum on NC (beyond convergence of train loss) when trained with SGD, being the first result concerning momentum in the context of NC. Finally, we conduct extensive empirical experiments consist- ing of 3,900 training runs across various datasets, architectures, optimizers, and hyperparameters, confirming our theoretical results. This work provides the first theoretical explanation for optimizer-dependent emergence of NC and highlights the overlooked role of weight-decay coupling in shaping the implicit biases of optimizers.
prove that SGD, SignGD with coupled
INTRODUCTION
Neural networks have driven many of the recent breakthroughs in artificial intelligence, yet the mechanisms underlying their success remain only partially understood. A key empirical clue neural collapse (NC) first documented by Paapyan al. (2020) in which the last-layer feature vectors and classifier weights self-organise into a highly symmetric configuration during the terminal phase of training (TPT). While the reasons for the emergence of NC are still not fully understood, its impact on the behavior a model is evident. For instance, Liu et al. (2023) induce NC to improve generalization in iaaa--imbllaneed training and Galanti et al. (2021) show that the emergence of NC
improves transfer learning as well. Furthermore, the presence of NC has been connected to better out-of-distribution detection (Liu & Qin 2023).
Theoretical explanations for NC have primarily relied on simplified models and assumptions (Mixon) al. 2022 Zhu al. (02I)) that have largely ignored the role the optimizer, thereby suggesting that NC is universal across optimization methods. In this work, we challenge this assumption and
First authors share equal contribution.

---

Published as a conference paper at ICLR 2026
demonstrate that the choice of optimizer plays a critical role in the emergence of NC. Concretely, we show that training with Adam (Loshchilov & Hutter (2019) does not lead to an NC solution, whereas training with SGD or Adam (Kingma & Ba122014) does. Through extensive experiments, trace this back to how weight decay is applied in both optimizer and identify the coupling of weight decay as a necessity for the emergence of NC.
One major challenge in studying NC lies in the original metrics, which are difficult to track and analyze theoretically. These metries were designed to quantify the progressive geometric alignment associated with NC and are expected to converge to zero in the idealized setting where NC holds as training time approaches infinity. However, under realistic training regimes, such finite training epochs and learning rate decay, these metries typically plateau at small but nonzero values. As result, there no rigorous criterion for determining whether NC has truly occurred.
This limitation motivates to introduce novel diagnostic metric, NCO, whose convergence to zero necessary (though not sufficient) for NC. Unlike previous metrics, NCO enables more definitive assessment: if NCO diverges during training, we can conclude that NC can not occur-eyy in cases where other NC metrics misleadingly converge to small positive values, creating an illusion of collapse. We discuss the peculiarity of interpreting NC metrics in practice later in Section 4.] Furthermore, NCO allows us to go beyond loss landscape analysis and theoretically derive convergence rates with which NCO converges to zero.
Contribution In this paper, we conduct extensive experiments spanning over 3,900 training runs investigate the role of coupled weight decay in the emergence of NC. We identify coupled weight decay as a key driver of NC in realistic settings, extending recent theoretical insights (Pan) Cao 2024 Jacot et al. 2024) that were limited quasi-optimal solutions simplified models. In particular, we show that the form of weight decay used in adaptive optimizers such as Adam (Kingma & Ba 2014) and AdamW (Loshchilov & Hutter, (219) critically affects whether NC emerges. Strikingly, while networks trained with Adam often exhibit NC, AdamW - despite its algorithmic similarity -fails to produce NC, with the corresponding metrics failing to converge to zero over time (Figure D. This subtle yet consequential distinction has been largely overlooked in prior work. An overview of our theoretical contributions can be found Table T
In summary, we make the following contributions:
1. Across a wide range of experiments, we find that coupled weight decay is a necessary condition for NC to emerge in adaptive optimizers, such Adam and Signum.
2. Furthermore, we show the accelerating effect of momentum on NC (beyond convergence of train loss) when trained with SGD, being the first result concerning momentum in the context of NC.
3. We support our empirical findings with the following theoretical statements on the new NCO metrie:
with SGD (with both coupled or decoupled weight decay), NCO converges to zero at an exponential rate proportional to the weight decay;
with sign gradient descent (SignGD) with decoupled weight decay, a special case of
AdamW, NCO converges some positive constant;
with SignGD with coupled weight decay, special of Adam, NCO exhibits non-monotonic trajectory, increasing before eventually decreasing. Using learning rate decreasing zero, we show that NCO also vanishes.
Organization This paper is organized as follows. In Section 2 we recapitulate the four properties to characterize NC and introduce novel NC property NCO. In Section 3 we present our main experimental results with theoretical support. Finally, Section provides insights and discussions on the implications of our results.
Notation We use [K] (1, K} to denote the index set for any integer K E N. For matrix W, we let Vec (W) denote the vectorization of W obtained by stacking its columns. The Frobenius inner product between two matrices W. W is denoted by (W, W") Tr(WT W'). With slight abuse notation, we write |W| = || || r for the Frobenius norm when W is a matrix, and ||v| ||v||2 for the Fuclidean norm when vector. In other words, ||]| || Vec(W)]].
2

---

Published as a conference paper at ICLR 2026
Figure NCO (left) and NC3 (right) met- rics at the end of training. Lower values in- dicate stronger NC. Adam shows consis- tently higher metrics than Adam. Averages computed over runs with varying learning rates and momentum: shaded regions show #1 standard deviation. X-axis is log-scaled. Note that there
aptimizer
Acanii
0.8
Acam
0.6
16+4
values for Adam for WD larger than 0.05 as the model did not train due over regularization.
Alam
Wesggn decay
Wiight decay
We denote by I the identity matrix, by 1 the all-ones column vector, and by J the all-ones matrix, i.e.,
J 1=11.
Table 1: Overview of our theoretical results on NCO.
Result
Optimizers
Model
Convergence to 0?
learning rate
Theorem
3.1
SGD with DWD
Any
yes, exponential yes, exponential
constant constant step-wise decay
Theorem
SGD with CWD
Any
Theorem
3.3
SignGD with DWD
UFM
yes no
Theorem
SignGD with CWD
UFM
2
NEURAL COLLAPSE
Neural collapse (NC), observed during the terminal phase of training (TPT) in deep neural networks (DNN), manifests itself through several geometric properties involving the last-layer features and weights in the K-class classification task:
X |
A Vec(t)]12
min ((Who(x)). vn)
(1)
where (X,, yn ) C RD [K] is the training set, W kxx is the last- weights, he (x(x)) R² is the last-layer feature as the output of some backbone parameterized by 0, é R x [K] +
[0, 0o) is the loss function, and X > 0 the 122reeularization constant.
These properties, formalized by their corresponding metrics in the original paper Papyan et al. 2020 are:
1. NC1 Variability Collapse:
Features collapse their respective class means, indicating that within-class variability vanishes.
2. NC2 Convergence of Centered Class Means to Simplex ETF: Centered Class means converge simplex equiangular tight frame (ETF).
3. NC3 - Convergence to Self-Duality: Rows of the last-layer weight W E R align with the columns of the class means, creating dual relationship between weights and features.
4. NC4- Simplification to Nearest-Classe Class-Center: The classifier's decision boundaries are simplified to those of nearest-class-mean (NCC) classifier.
A solution satisfying all of these properties is referred to as a NC' solution. In addition to these prior NC properties, we introduce another novel NC' property NCO, whose convergence zero is necessary condition (though not sufficient) for NC.
NCO Zero Row Sum Last-Layer Weight:
The row sum the last-layer weight W in the model
converges to zero.
The first observation is that NCO is a necessary condition for NC2 and NC3:
Proposition 2.1. NC2 and NC3 implies NCO.
3

---

Published as a conference paper at ICLR 2026
Proof. For each class ls [K], we define the class mean 14: [n:y.=k)
-n:g.=hh ho(xx) E R
and the centered class mean jux Mk N En= hx(x)). We concatenate them into a matrix M (f+) PPP with M1 0. since we centered the class means, By NC2, M converge simplex ETF in the ambient space IR=", meaning M/||M]] F + QM* where M E RR is unit matrix with columns forming K -ismllex EFT in R and Q E R PxX is the isometric injection map into the ambient space. Since M1 0 and Q is injective, the unit matrix M' has be in the form: M
WK-I (I- J) for some orthogonal matrix P. But it can be absorbed into Q the matrix QP is still an isometric injection. Hence, without loss of generality, we assume M def
P
1
VK-I [I J) and hence
M M/MM MI - (QM") 'QM" = (M²*) = -M"
On the other hand, NC3 states that M/|[M| WT /|| - 0 as - 00. Hence we have
www M + 0 as co. Now note that 1 ''' = hence | 1 1 1 WW + 0.
Note that the last line holds if and only if NCO holds.
NCO offers two key advantages. First, it serves as a diagnostic tool: if NCO does not converge, then at least one of NC2 or NC3 must fail, providing clear signal that neural collapse cannot occur Second, NCO is more mathematically tractable than the original NC metrics, whose dynamics are difficult to analyze and remain underexplored. As we demonstrate in Section 3 NCO's evolution during training can be reliably tracked and used to explain empirical trends observed across different optimizers. In addition, NC metrics, particularly for small learning rates (see Figure 2). For a more detailed explanation and formal definitions of NC properties and their metrics, we reter the reader to Section B
extensive experiments also show that NCO is correlating well with prior
0.001
Ir 0.005
Ir 0.01
Ir 0.0679
optimizer
AdamW
NC
Adam
SGDW
SGD
SignumW
10-2 10a
10-2
10a
10-2
102
10-2
10a
Signum
NCO
NCO
NCD
NCD
Figure 2: NCO weakly correlates with NC3 across different optimizers and learning rates. Details on the regression fit can be found in Appendix D.3
3 MAIN RESULT
3.1 EXPERIMENTAL SETUP
We conducted extensive experiments training a ResNet9 and VGG9 using various optimizers, includ- ing Adam, AdamW, SGD, SGD with decoupled weight decay (SGDW), Signum (Bernstein et al. 2018), and Signum with decoupled weight decay (SignumW) W) trained on MNIST, FashionMNIST and Cifar10. Every optimizer is trained with three different learning rates (LR), six different values of momentum, and six different values of weight decay to also control the effect of hyperparameters on the emergence of NC. This resulted in total of 2 x 3 x 108 3, 888 training runs. Note that we only keep runs with reasonably high training accuracy. Too large weight decay over regularize the model and the model does not train anymore. Thus, the number of valid training runs actually smaller than 3,888. All networks were trained for 200 epochs using a batch size of 128, with the learning rate being decayed by a factor of 10 after one-third and two-thirds the training duration, as described in the original work by Papyyan et al. (2020). In addition, we conducted ablation studies to control for the number of training epochs and to verify that the results also hold for unconstrained feature models (UFM)) leading to a total of over 3,900+ training runs. Further details and all experimental results can be found in Appendix D Ablation studies on the effect of training epochs can be found in Appendix D.4.I]
Appendix C.5 for introduction UFM.
4

---

Published as a conference paper at ICLR 2026
Table 2: Final NC metrics for the same setting in Figure following the setup of Papyan et al. (2020). Lower values (L) indicate stronger neural collapse. Values in parentheses represent percentages relative to the metric at initialization.
Optimizer
NCO
NC1,
NC2,
NC3
SGD
2.14c-04 (< 99)5%
0.05 (-99.3%)
0.29 (-63.0%)
0.35 (-75.1%
SGDW
0.55 (-68.9%)
0.26 (-96.3%)
0.46 (-42.4%)
0.80 (-43.5%)
Adam
0.34 (-80.6%)
0.29 (-63.9%)
0.29 (-79.5%)
AdamW
5.33 (>B 100%)
(-
0.54 (-32.4%)
0.78 (-45.2%)
Signum
0.78 -55.3%)
0.13 (-98.1%)
0.50 (-36.8%)
0.58 (-9.0%)
SignumW
3185.69
( 100%)
0.30 (-9.7%%
1.15 (+44.2%%
1.40 (-1.2%)
3.2 WEIGHT DECAY IS ESSENTIAL AND MOMENTUM ACCELERATES NC
Our experiments show that weight decay is necessary to reduce the NC metric across all optimizers and hyperparameter settings, as shown in Figure for Signum and SGD, and earlier in Figure T for Adam and Adam W as well as in our ablation studies in Appendix D.4.1 and Appendix D.4.6 While the experiments cannot fully exclude the possibility that NC be achieved eventually the
asymptotic limit without weight decay, we argue that WD is essential to observe the emergence of NC in practical finite-length training settings on realistic model
3.4
Cpt mizer
1.2
13
octminer
332
13a
21-
6.5
opimierr
Sgganil
13a4
32a
1344
-2-
Wr detiy
We dedyy
Weigh deCN
WHITTE Secay
Figure 3: NCO and NC3 metrics at the end of training for ResNet9 trained on FashionMNIST for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers one standard deviation across all trainings run with corresponding optimizer. Note that there are fewer values for Signum and SGD as the model did not train due to over regularization for too large WD.
From the figures, can conclude that larger weight decay leads to a stronger decrease of NC metrics. particular, we show that adaptive optimizers with decoupled weight decay have much larger NC metrics, which are strictly away from zero, showing sign of NC. In addition, we show empirically that momentum amplifies the effect of weight decay on the decrease of NC metrics in SGD, as shown in the heatmap in Figure metrics both by increasing weight decay for fixed momentum or by increasing momentum for fixed non-zero weight decay. The effect of momentum on the NC metrics becomes larger for larger values of weight decay. We remark that this goes beyond the acceleration of convergence of the train loss, as we study in an ablation study in Appendix D.4.5 In particular, we show in Figure 4 that two training runs with different momentum and otherwise same hyperparameters can reach the same train loss, while reaching different NC metries. This indicates that they have converged solutions with very different geometric structure.
This implies that one achieves decrease in the NC
The experimental results are complemented by Theorem 3.1 and Theorem 3.2 showing that NCO converges to 0 with an exponential rate trained with SGD, which is proportional to momentum and weight decay, highlighting that NC cannot be achieved without weight decay and that momentum accelerates the convergence of NC metrics.
Theorem 3.1 (SGD with decoupled weight decay promotes NCO). Assume a model of the form f(W, 0,x) = Wha (r) is trained using cross-entropy loss with stochastic gradient descent (SGD) and momentum 3 (0,1), weight decay À [0, 1), and learning rate on all parameters 0, W.
We note that Ji et al. 2021 show both theoretically and empirically the emergence of NC on the uncon- strained layer-peeled model (UL.PM) objective under gradient flow without weight decay.
5

---

Published as a conference paper at ICLR 2026
109
D.7
0.9
1.00
0.9
1.25
6.7
0.9
10-1
1.50
0.75
10-4
0.65
4x 10
0.50
3x 10"4
100
150 200
50
203
100
Figure 4: Train loss, train accuracy and NC metrics for fixed WD=0.005 and mom=0.7 and 0.9. Although both runs converge to almost exactly the same train loss, the final NC metries differ considerably. Plots including NC1 and NC2 can be found in Figure 24
LK-0.503
LS=1,001
NC3 Lite 2,00
1:03
1.4
0.22
0.03
0.22
0.72
0.73
03
1/5
1.40
1.4
0.5
0.55
U51
-65
05
0.10
00
0000
00
porroun
1.8
0.21
8
8
R
515
-0.4
677
8
10
0.01
0.50
3
031
037
8
8
-03
6.1
63
22403
0.900
6.30
6.30
63
4000
0.003
0,303
23603
6:30
Figure 5: Heatmap of NCO, NC2 and NC3 for varying values of momentum and weight decay ResNet9 trained on FasSionMNIST with SGD.
For instance, the last-layer weights W are updated according to:
V t-1 VV. + Vw, LCE.
W Wo+1- (1 TT V/
If0 < n7 < 2, then the NCO metric at IW, 1 decays exponentially zero in t.
Proof. The key observation is that the largely simplifies the NCO metric to only be dependent on the weight decay À and momentum B3. For the details of the proof, please refer to Subsection E in the Appendix.
of the loss gradient VLCE( 1x zero, which
Theorem 3.2 (SGD with coupled weight decay promotes NCO). Assume a model of the form f(W, fr Who(x) (x:) is trained using cross-entropy loss with stochastic gradient descent (SGD) and momentum 3 E 1), weight decay e [0, 1), and learning rate on all parameters 0, W. For instance, the last-layer weights W are updated according to:
V = 3V, + Vw, LCE + AWW,
W++ W. v
If0 < n/ < 2(1 + 9), then the NCO metric
1 k W 1 decays exponentially to zero in t.
Proof. Similar the proof of Theorem 3.1 For the details of the proof, please refer to Subsection the Appendix.
3.3 WEIGHT DECAY COUPLING MATTERS
While weight decay has been theoretically shown to be essential for NC in prior works (Pan & Cao 2024 Jacot al. 2024), these works ignore how weight decay is applied by treating Le-regularization of the gradient and applying weight decay directly on parameters as equivalent. However, we note that this equivalency only holds for vanilla SGD and not for adaptive optimizers, such as Adam Adam nor when momentum is applied. In particular, our experiments reveal that NC does not emerge under SignumW W and Adam W under realistic settings. This highlights the crucial role of coupled weight

---

Published as a conference paper at ICLR 2026
101
10a
NCO
10"
1.0
10"
10-1
0.5
0.5
10
100 epoch
200
100 epoch
200
100 epoch
200
100 epoch
200
SGD
SGDW
AdamW
Adam
Signum
SignumW
Figure NC metrics throughout training on a ResNet9 trained on MNSHi
decay - that is Le-regularization applied directly within the gradient update as requirement for NC. This subtle yet important distinction has been largely overlooked in prior literature.
Importantly. tracking the evolution of the NC metrics (Figure 6 and the singular values of centered class means M and the last- last-layer weight W (Figure throughout training (here shown for a ResNet9 trained on FashionMNISTT, one can see that using adaptive optimizers with decoupled weight decay leads to fundamentally different dynamies of the NC metrics and singular values despite all models reaching TPT, where training error is (almost) zero.
www.
20
Singular
1.
SG0
SCDW
Acamiv
Adam
Sigrum
Figure 7: Singular values of last- weights w (left) and centered class means M (right) through- out training. The dotted line corresponds to the smallest singular value and the full line corresponds the average singular value, excluding the smallest singular value. Singular values for SignumW are out-of-range and are shown in Figure 29 in the appendix.
Specifically, Figure shows that the smallest singular value of W increases during training with AdamW and SignumW, indicating failure to satisfy NC3. Additionally, NCO and the nonzero singular values of M grow throughout training and exhibit high variance, suggesting that NC2 is also less well-fulfileed in these settings.
In Figure 6 we further observe that SGD and Adam achieve the lowest NC metric values, while Adam W. SignumW, W, and SGDW saturate early at much higher levels. Although the NC metrics for Signum are slightly larger than for SGD and Adam, they continue to decrease over time, suggesting potential convergence to NC under longer training.
Finally, our experiments in Figure and Figure demonstrate that the NCO and NC3 metrics of AdamW W and SignumW remain significantly larger than those of Adam and Signum, even when using weight decay several orders of magnitudes higher. This indicates that models trained with AdamW or SignumW are consistently farther from achieving NC. Note that the NC metrics for SGD and SGDW remain relatively close, consistent with our theoretical results in Theorem 3.1 and Theorem 3.2, while the gap between coupled and decoupled weight decay has more pronounced effect adaptive optimizers than SGD. This suggests the effect is not simply due to greater weight decay accumulation through momentum but stems from deeper interaction with the optimization dynamics.
3.4 INTERPOLATING ADAMW AND ADAM
To further investigate why AdamW fails exhibit neural collapse (NC) while Adam does, con- ducted an ablation study by "interpolating" between the two optimizers. Specifically, we implemented variant that combines both coupled weight decay (as in Adam) and decoupled weight decay (as
7

---

Published as a conference paper at ICLR 2026
in AdamW/ W). For each varied the strength of the coupled weight decay while adjusting the decoupled component such that the total weight decay remained fixed at 0.0005. The momentum was set to 0.9 across all configurations.
As shown in Figure g increasing the coupled component Ieads to smooth improvement NC metrics particclaaa NCO, NC2, and NC3- NC3-while the validation aceuracy remains largely unaffected. This experiment suggests that coupled weight decay a critical factor in enabling neural collapse, yet it is not strictly necessary for achieving strong generalization performance, all configurations yield similar validation accuracy. This strengthens a point raised earlier about the limitations of NC understand generalization Hui al. (2022).
33,000
9.995
0.6
0.8
5.6
8.985
3,900
opach
eB0ch
cooch
spoch
coupled Wi gright decay
6,0003
0.0003
0,0922
10003
0.0034
5.0005
Figure Interpolating Adam and AdamW W by varying the coupled and decoupled weight decay. Total weight decay was fixed to 0.0005. Note that coupled weight decay 0 is equivalent to AdamW and coupled weight decay 0.0005 equivalent Adam. Experiments trained on ResNet9 with MNIST.
This observation is supported by theoretical results in Theorem 3.3 and Theorem 3.4 which show that SignGI with decoupled weight decay fails to satisfy NCO and therefore cannot converge to a neural collapse solution, whereas SignGI) with coupled weight decay exhibits different behaviour. We note that SignGD corresponds to special case of Adam and Adam W when the parameters $1, 82, and are zero.
Theorem 3.3 (Sign GD with decoupled weight decay avoids NCO). Consider sign GD with (decoupled) weight decay À > and step size 1 > on the UFM loss Lce (HH, En Lce(W (Wh,, en), where the feature H = M is fixed to an NC solution and only the weight W is trained:
W W. w L.CE) XW,)
Then the NCO metric a = [ 1x increases monotonically from zero to the limit:
(K 2)
lim or (+000
X2
particular, an does not vanish as 0c.
Proof idea: The key observation is that the signed loss gradient sign( V TCE (W.) in this setting is constant in f, simplifying the following computation. See Appendix for the full proof.
Theorem 3.4 (Sign GD with coupled weight decay lead to NCO). Consider sign GD with (coupled) weight decay > and step size r > on the UFM loss LeE( (HH, = Enn LCE(Wi (Wh,, en), where the feature H = M is fixed to an NC solu- tion and only the weight W is trained:
W W. - (iiiin(iinnnnnn I.CE + XW.))
We initialize Wo = 0 B K and define the covariance matrix C, = W,W, and the scalar ae 3)r where
K 11 Then there exists a learning rate decay scheme v7(1) ビt
such that at t-400
0.
Proof. See Appendix
E
The key difference between the results of Theorem 3.3 and Theorem 3.4 lies in how coupled weight decay affects the signed gradient during training. As the weight norm [W] increases, the coupled decay term eventually flip the sign of the gradient, altering the trajectory of the NCO metric oz.
8

---

Published as a conference paper at ICLR 2026
500
Adam
Acami
SignSoD
SinSsow
2074
10
9.63x.0
9.23--20
0.6 10
5d-50-04
wd-1e 03
Wd 04
Wd-1e 03
MCO
wd-
Wd--5e
WC---5 04
Wd-Sc 04
NC 1e 03
wd-1e
0.
10
Wd-5e
Step
Stop
Stop
Stao
Step
Figure 9: Training dynamic of NCO with optimizers SGD, Adam, AdamW, Adamo (51 B2 0), AdamWo WO (31 = 32 = 0). For A and SignSGD the inlay shows the NCO metric more detailed for the last 2000 steps. Note that 5 steps correspond to training epoch.
Initially, Or grows at a similar rate in both cases, but their behaviors diverge once the decay term becomes dominant.
To illustrate this effect, we conducted small-scale experiment using simple MI.P on separable dataset with various optimizers. As shown in Figure SignSGD displays non-monotonic dynamics in at while SignSGDW exhibits steady convergence to a positive value. Similar patterns appear in Adam and AdamW, though more smoothed due to their adaptive updates.
4 DISCUSSION AND LIMITATIONS
In this section, we discuss new insights, additional considerations and limitations from the main results in Section 3 Additionally, we explore potential follow-up research directions that could provide theoretical explanations extend experiments to broader settings.
4.1 INTERPRETING NC METRICS IN PRACTICE
While NC is defined by the convergence of all NC metrics to zero in the limit, practical experiments never achieve exact zeros. Since NC inherently a continuous rather than discrete phenomenon, becomes necessary to define what constitutes the presence of NC in practice. This important issue has not been thoroughly addressed in the existing literature.
A further complication is that different NC metrics operate on different scales and these scales vary across settings of architectures and datasets. For example, our experiments, the smallest observed values for NC2 and NC3 are the order of 0.1, whereas NCI can reach values an order of magnitude smaller.
In this work, we therefore refer to the emergence of NC: in terms of relative strength. Specifically, we use the NC metric values at initialization as a bascline for models that do not exhibit NC, and use the smallest values achieved across all experiments allows us to discuss the strength of NC emergence across different optimizers and settings.
reference point for models that do. This framing
4.2 THE REDUNDANT NC4 PROPERTY
Readers may notice that we omit NC4 from the results in Section 3
This is because we observed
that NC4 is consistently satisfied whenever the training accuracy approaches 100%. regardless of whether the other NC metries (NCI-NC3) exhibit collapse. As shown in Figure 57] NC4 largely uncorrelated with the other metrics. To maintain a clearer and more focused presentation, we therefore exclude NC4 from our main analysis.
4.3 PARTIAL NEURAL COLLAPSE
Another subtlety observe is what term partial neural collapse. As shown in Table 3 AdamW can achieve minimal values for NC1 and NC2 among all optimizers, even while NCO diverges and NC3 is not satisfied. This indicates that NC properties may not always emerge jointly, contrary the original claim in Papyan et al. (2020). Understanding the theoretical conditions under which only a subset of NC properties holds remains intriguing open question.

---

Published as a conference paper at ICLR 2026
Table 3: Final NC metries for the run with the smallest absolute NC3 metric and > 99% training accuracy for each optimizer. Lower values (4) indicate stronger neural collapse. Values in parentheses represent percentages relative to the metric initialization. Hyperparameters used for each optimizer can be found in Table
Optimizer
NCO,
NC1,
NC2,
NC3,
SGD
1.53e-05 (< -99.5%)
0.02 (< -99.5%)
0.19 (-75.8%)
0.13 (-90.9%)
SGDW
1.54e-04 (< -99.5%)
0.01 (< -99.5%)
0.15 (-81.7%)
0.10 (-92.7%)
Adam
0.12 (< -93.2%)
0.04 (-99.5%)
0.23 (-71.6%)
0.17 (-8..2%)
Adam W
8.09 (> 100%)
0.01 (< -99.5%)
0.14 (-82.1%)
0.49 (-65.1%)
4.4
LIMITATIONS OF THEORETICAL SUPPORT
Our experiments on Adam and Adam W are conducted on realistic models and datasets, whereas our theoretical results (Theorem 3.3 Theorem (344) focus on a simplified setting: SignGD applied the unconstrained feature model. While this restricted setup already demonstrates that AdamW fails to achieve NC, it does not fully capture the complexity deep neural networks or adaptive optimizers in practice. Nevertheless, we believe our proof techniques could be extended to explain why Adam may lead to NC in more general settings. Moreover, our theoretical analysis is limited to the training dynamics of NCO, chosen for its analytical tractability and strong empirical correlation with other NC metrics. A full theoretical understanding of NC1-NC3 under realistic optimization dynamics remains an open challenge, and leave this direction for future work.
4.5 FUTURE RESEARCI
Other than the topic we have discussed in the previous subsections, findings also open other intriguing avenues for future research.
Empirical studies should be expanded to include larger models, such as Vision Transformers (ViTs) and DenseNets, well more diverse datasets, to assess the broader generality of our findings. Our preliminary results on ViT are available in Appendix 1.rrnndxxDD.4.1.. D.4.10 and largely confirm our findings also extend Transformers.
Due to computational constraints, our study only analyzed NC properties the last layer. However, previous works (Masarczyk et al. 2023 Rangamani et al. 2023) suggest that these properties may also manifest in intermediate layers. Investigating NC behavior across different depths could provide further insights into hierarchical feature representations.
In addition the optimizers (SGD, Adam, Adam W. Signum) studied in this work, novel first-order methods such as Lion (Chen et al. (003)) and Mars (Yuan et al. (004)) and second- order methods, such as Shampoo (Gupta et al. 2018). SOAP (Vyas et al., 2024) and Muon (Jordan et al.) demonstrated promising improvements in convergence and generalization. However, their effects on NC remain largely unexplored.
5 CONCLUSION
In this paper we have conducted an extensive number of experiments to elucidate the role of the optimization algorithm in the emergence of the neural collapse (NC) phenomenon. In particular, our experiments consistently show that coupled weight decay is necessary for achieving small NC metrics. While the role of weight decay the context of NC has been studied in the literature before, this is the first paper distinguishing between coupled and decoupled weight decay. Moreover, our theoretical results show that the resulting training dynamics differ considerably and one needs to take this into account. These findings underscore the limitations of existing theoretical frameworks, which have studied NC mainly under gradient flow or gradient descent, and highlight the need for further investigation into the interplay between optimizers and NC.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
WO acknowledges that this research was partially funded by National Science Centre, Poland grant no 2022/45/N/ST6/040088
REFERENCES
Mouin Ben Ammar, Nacim Belkhir, Sebastian Popescu, Antoine Manzanera, and Gianni Franchi. Neco: Neural collapse based out-of-distribution detection, 2024.
Tina Behnia, Ganesh Ramachandra Kini, Vala Vakilian, and Christos Thrampoulidis. On the implicit geometry of cross-entropy parameterizations for abbelimbalancced data. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 10815-10838. PMLR, 25-27 Apr 2023.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp. 560-569. PMLR, 2018.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoe V Le. Symbolic discovery of optimization algo- rithms. In Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 49205-49233. Curran Associates, Inc., 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Tomer Galanti, András György, and Mareus Hutter. On the role neural collapse in transfer learning. arXiv preprint arXiv:2112.15122 2021.
Connall Garrod and Jonathan P. Keating. The persistence of neural collapse despite low-rank bias:
An analytic perspective through unconstrained features, 2024.
Vincet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimiza- tion. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1850. PMLR, 10-15 Jul 2018.
X. Y. Han, Vardan Papyan, and David I.. Donoho. Neural collapse under mse loss: Proximity and dynamics on the central path, 2022.
Md Yousuf Harun, Jhair Gallardo, and Christopher Kanan. Controlling neural collapse enhances out-of-distribution detection and transfer learning. 2025.
Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.08384 2022.
Arthur Jacot, Peter Súkeník, Zihan Wang, and Marco Mondelli. Wide neural networks trained with weight decay provably exhibit neural collapse, 2024.
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie Su. An unconstrained layer-peeled perspective on neural collapse. arXiv preprint arXiv:21 10.02796, 2021.
Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for large number of classes. arXiv preprint arXiv:2310.0535., 2023.
K Jordan, Y Jin, V Boza, Y Jiacheng, F Cecista, L. Newhouse, and Bersstenn Muon: An optimizer for hidden layers neural networks, 2024b. URL https///ellerjjoddn. github. io/posts/muon.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 44/266980 2014.
11

---

Published as a conference paper at ICLR 2026
Vignesh Kothapalli. Neural collapse: A review modelling principles and generalization, 2023.
Litian Liu and Yao Qin. Detecting nut-of-distribunt through the lens of neural collapse. arXiv preprint arXiv:2311.01473 2023.
Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing neural collapse in deep long-tailed learning. In International conference on artificial intelligence and statistics, pp. 3534-11544 PMLR, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Milos, and Tomasz Trzciński. The tunnel effect: Building data representations in deep neural networks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 76772-76805. Curran Associates, Inc., 2023.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2): 2022.
Kaouther Mouheb, Marawan Elbatel, Stefan Klein, and Esther E Bron. Evaluating the fairness of neural collapse in medical image classification. In International Conference Medical Image Computing and Computer-Assister Intervention, pp. 286-296. Springer, 2024.
Leyan Pan and Xinyuan Cao. Towards understanding neural collapse: The effects of batch normaliza- tion and weight decay, 2024.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso Poggio. Feature learning in deep classifiers through intermediate neural collapse. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28729-28745. PMLR, 23-29 Jul 2023.
Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. In S. Koyejo, S. Mohamed, A. Agarwal, Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27225-27238. Curran Associates, Inc., 2022.
Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409. 1121,, 2024.
Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models. arXiv preprint arXiv:2405.1777 2024.
Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. Mars: Unleashing the power of variance reduction for training large models, 2024.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 29820-2929 29834. Curran Associates, Inc., 2021.
12

---

Published as a conference paper at ICLR 2026
Appendix
A LLM USAGE STATEMENT
We disclaim that we have used Large Language Models to refine few sentences and additionally as a proxy of a search engine to retrieve additional related work.
The appendix is organized as follows, In Section we formally define the neural collapse (NC) phenomenon and introduce the metrics used in the experiments presented in the main text. In Appendix C we review prior works related to our paper. SectionD)jrrviin detailed descriptions and additional observations from experiments. In Section E present the full proof of the theorems stated in the main text.
B
NC METRICS
Neural collapse (NC), discovered by Papyan et al. (2020), is a striking phenomenon observed during the terminal phase of training (TPT) deep neural networks (DNN) for multi-class classification tasks, particularly when trained with cross-entropy (CE) loss. Formally, let the (trained) last-layer features of the DNN be denoted by h,, and concatenate them into a matrix H E T&px,, where p is the width of the last layer and N is the number of training samples indexed by n. The output logits of the network are then computed as W, H E KxN, where Wt K x is the last-layer weight, b R is the bias vector, and K is the number of classes.
The DNN is trained using the CE loss computed on the logits:
exp(W Lhn)ya
CB(W L. H)
Σ log
k= exp (W W.hn)k
where Yn. [K] denotes the class label index of the feature vector h, Let Ck E [M] k be the index set of data points belonging to class & E [K]. In this paper, we assume that the classes are balanced, i.e., (Ca) is equal for all k E [K]. For the effects of class imbalance on NC, we refer the reader Han et al. (2022); Thrampoulidis et al. (2022) ; Behnia et al. 2023)..
Let ph
[.. Enec, h, be the class mean for each class h. The global mean of all classes is given by NG k Ek Mh and centered class means defined f4 44g pG- Let the between-class covariance Eg E Rexp and the within-class covariance Ew R be:
K
Σg
P&Pko
K
K
1
Ew
(h
pk)( Mx)
KN
N
k=1n=1
where by correspond to the feature vectors of class k.
We also concatenate the centered class means into a matrix M def. (Pi,..., E B
With these definitions in place, we now conceptually outline the NC properties and introduce corresponding metrics to quantitatively measure these properties our experiments.
For simplicity, interchangeably refer to an input IRd and its corresponding last-layer feature h E TRP after the parameters of the network have converged during TPT and the mapping h fixed.
13

---

Published as a conference paper at ICLR 2026
NC1 Variability Collapse The first property of neural collapse (NC1) describes the collapse of features to their respective class means. Formally, this means that the distance between feature vector h, and its corresponding class mean M approaches zero:
||h,, Mell2 0, Ak [K], Ck.
A corresponding metric is defined Zhu al. 2021);
Kothapalli (2023) Ammar et al. (2024):
NC1 def
Tr Ew Eg
K
where j denotes the Moore-Penrose pseudo-inverse.
NC2 Convergence of Class Means Simplex ETF The second property of neural collapse (NC2) describes the convergence of class means to simplex equiangular tight frame (ETF), where the angles between the means are maximally symmetric. Formally, this property can be expressed as:
j ll2
| Ae
- 0
Vj,k [K].
A:
K K-
55
1 K-1
To measure this property, we define two metrics capturing the equinormality and equiangularity of the centered class means Papyan et al. (2020); AAmmar et al. (2024):
std { 11
NC2, avg; 1111211122
(3)
NC2, = avg
K
Here, (-) and avg.(-) denote the standard deviation and mean, respectively, over the specified index.
An alternative metric for NC2, introduced by Kothapalli (023 the centered class means from a simplex ETF:
directly measures the deviation of
M
M
NC2
M
(5)
K2
M M
F
where
M def.
VK
Ik
K
JK)
Is Rxk is the identity matrix and RR is the matrix of ones. Note that NC2,,, NC2 +
NC2W- Convergence of Weight Rows to Simplex ETF In addition NC2, define related property, NC2W, which describes the convergence of the rows of the last-layer weights W. E Kxx simplex ETF, If the third NC property, NC3 (described later), holds, then NC2 and NC2W equivalent. However, to study partial NC, it is essential to decouple these properties and measure NC2 and NC2W separately.
To measure NC2W, Zhu et al. 2021 introduced the following metric:
W
NC2W
M
(6)
K²
[
While this metric measures the overall alignment of W, 1. with simplex ETF, it does not account for the equinormality and equiangularity the rows W r. To address this, we introduce the following metrics:
NC2W,, std& (//W& [
(7)
avg. (/ [[
NC2W, = avg
K
(wall
Wk
WP
1
(8)
K -
where W. E IRP is the k-th row of WT
14

---

Published as a conference paper at ICLR 2026
NC2M Convergence of Product to Simplex ETF Finally, Zhu et al. 2021
Kothapalli 2023
proposed a metric that interpolates between NC2 and NC2W:
NC2M
del.
1 K
WLM
M
I F
F
Note that NC2,, +0 = NC2M - 0 but the converse does not hold.
NC3 Convergence Self-Duality The third property of neural collapse (NC3) describes that the rows of the last-layer weight align with the column of the class means, that is,
WL
M
0:
W L
M
F
the corresponding metric is an obvious one Papyan et al. (2020):
Garrod & Keating (2024):
AC3 def.
1
WL
M
(10)
Kp
W
[M
NC4- Simplification of Nearest-Class-Centee (NCC) (NC4) describes that the classifier decision boundaries become equivalent to those derived by nearest-class-net classifier, that is,
The fourth property of neural collapse
arg max Wky h) arg min |h 14
k
for any test feature R"; hence we can fix a test set of features (h/ J==1 define the metric:
(hlesay Nex
NC4 def
Σ
(argmax
hust) arg min h mellzt
(11)
Viest
where 1 is the indicator function.
The above NC properties hold if their corresponding metrics approach zero (except for NC4, which approach one) as the training step t + oo. A solution W, L, H satisfying these properties is referred to as an NC solution.
To observe the interpolation between partial and full NC, we introduce a weaker property:
NCO - Zero Row Sum of Last-Layer Weight This new property describes that the rows of the last-layer weight W, y sums up to zero with the corresponding metric
NCO
def.
W
(12)
2
Note that NC2W + ACCO 0 but the converse does not hold.
The analogous property for the last-layer features, Zero Column Sum of Last-Layer Features, holds automatically because the columns M are centered class means:
K
Σ
(u - MG) = 0.
A=1
Thus, NCO for the last-layer weights already represents a form of duality similar to NC3.
'In the original works, this metric was used to evaluate self-duality. However, in this paper, decouple the NC properties study the effects of implicit hiases each individually.
15

---

Published as a conference paper at ICLR 2026
C ADDITIONAL RELATED WORK
C.I WEIGHTT DECAY AND NEURAL COLLAPSE
Weight Decay has been shown to be essential for NC in prior works, like (Zhu et al. 2021: Pan & Cao 2024, Jacot et al., 2024). However, their statements on weight decay are solutions in oversimplified models, which ignore the complex interaction between non-convex loss landscape and optimizers. Please see Appendix C.3 for an example.
C.2 EMPIRICAL STUDIES ON THE EMERGENCE OF NEURAL COLLAPSE
Neural collapse has also been studied beyond the original problem setting, which assumes few balanced classes as well as noise-free labels. Notably, Wu & Papyan (2024) studied the occurrence of NC for large language models, which do not satisfy any of the original assumption. Jiang et al.] (2023) studied neural collapse for a large number of classes, while Mouheb et al. (2024) studied the influence of imbalanced in medical image classification NC.
C.3 APPLICATIONS OF NEURAL COLLAPSE
The observation of neural collapse (NC) has inspired growing body of follow-up work that applies NC metries across various settings. In the context of out-of-distribution (OOD) detection, AAmmar et al. (2024) propose a novel post-hoc detection method based on the geometric properties of NC, while Harun et al. (2025) show that explicitly controlling for NC1 can enhance OOD detection performance. Notably, the latter also claim that Adam W leads to NC, based on empirical results where NC3 values hover around 0.5 across different models-mirroring the misleading metrics reported in Table 3 As we demonstrate in the main text, however, this does not indicate true NC. This discrepancy underscores the need for more precise and systematic framework for evaluating NC one of the central contributions of this work.
In a separate line of inquiry, Liu et al. (2023) study the impact of class imbalance on NC and propose explicit feature regularization terms to induce NC under imbalanced distributions, resulting in improved model performance.
C.4 COUPLED WEIGIT DECAY THE CONTEXT OF NEURAL COLLAPSE
To the best of our knowledge, no prior work has investigated the role of optimizer choice in the context of NC. When minimizing the objective in Equation [ or Equation (13). the weight decay induced by the 1.2-regularization parameter À is coupled with the training loss. However, with the introduction of AdamW Loshchilov & Hutter (2019). decoupled weight decay has become the default many modern optimizers. This paper aims bridge this gap by systematically examining the impact of coupled versus decoupled weight decay on the emergence of NC.
C.5 UNCONSTRAINED FEATURE MODEL
The unconstrained feature model (UFM) Mixon al. (2022); Zhu (202T) is simplified theoretical framework commonly used to study neural collapse (NC). In UFM, the last layer feature is
replaced by a trainable matrix (h,, > > referred to as the unconstrained feature, which mimics the role of feature extraction layers in deep neural networks (DNN). For analytical simplicity, the
layer following the unconstrained feature is often assumed to linear W, making UFM a special
case of deep linear networks (DLN):
N Σ 1=1
A 2 H
((Wh,, yn)
(13)
H
simplifying the minimization problem in Equation (T. In this paper, the loss E is always assumed to be the cross-entropy (CE) loss, because it is the standard loss used in multi-classification tasks.
Zhu al. 2021 has reported positive results on NC using UFM. Informally it holds that:
16

---

Published as a conference paper at ICLR 2026
Theorem C.1 (Theorem 3.1 and 3.2 in Zhu al. (021)), Any global optimal solution of UFM is an NC solution, while all other critical points are strict saddles. As a result, for random initialization, is almost surely that gradient descent finds an NC solution.
Zhu et al. 2021) also experimented NC on realistic models with optimizers like SGD and Adam, concluding the universality of NC across different optimizers.
17

---

Published as a conference paper at ICLR 2026
D EXPERIMENT
The experiments of this work, particularly regarding computing the NC metrics, based code in Wu & Papyan (2024), which can be found at Github repository https:: / thub com/thubarbww/ http://wwwneuuaaa-olllooage
which was published under the MIT license. The implementation of VGG9 was based on Code taken from Intpp:: httt::://gitub.... com/ ctt.be...cm////ettt// jerett/
PyIorch-CIFAR1O The author granted explicit permission use the code.
An overview of the experiments that were conducted in this work can be found in Table 4 which resulted a total number of 36 different experimental settings of (arehitecture x optimizer x dataset) combinations. Fach optimizer optimizer was trained using three different learning rates, six different values of momentum and six different values of weight decay, resulting in 108 training runs per optimizer and 3.888 training runs in total. Some of the runs diverged or only achieved suboptimal training performance, which were then discarded. In total we had 2.500 "valid" training runs, which reached at least 99% training accuracy, which were considered for for the subsequent data analysis.
Table 4: Overview of experiments conducted in this work.
Architectures
Optimizers
Datasets
SGD, SGDW, Adam,
ResNet9, VGG9
MNIST, FashionMNIST, CIFAR1O
AdamW, Signum, SignumW
D.1 DETAILS ON CHOICE OF HYPERPARAMETERS
Every model was trained over 200 epochs with a batch size of 128. The learning rate À was chosen to be (0.001, 0.01, 0.0679] for SGD and SGDW (the last learning rate was also reported in the original work by Papyan et al. (2000) ) and À E {0.001, 0.005, 0.01} for Adam, Adam W. Signum, and SignumW because most trainings diverged with larger learning rates during initial experimental train- ing runs. The learning rate was decayed by factor of 10 after one third and two third of training as has been done in original work by Paayyan et al. (2020). Momentum (or 31 for Adam, AdamW, Signum, and SignumW) was chosen to be in the range μ {0, 0.5, 0.7, 0.9, 0.95, 0.98) for all optimizers and weight decay WD was chosen to be in the range WD E {0, 5e 5 5e- 5e 0.05, 0.5) } for SGD, SGDW, Adam, and Signum and WD (0,5e 0.05, 0.5, 5, 10] for SignumW and Adam W. The main motivation for using AdamW W and Signum W with much larger weight decay values was based on the hypothesis that the effect of weight decay reduced due to decoupling. The B3 parameter in Adam and Adam W was left to its default value of 0.999.
D.2 DETAILS ON COMPUTATIONAL RESOURCES
All experiments, including preliminary experiments as well as the final 3.888 experiments were run on 5 NVIDIA RTX4090 GPUs with 24 GB RAM. Since the models and the batch size was comparably small, actually only 3 GB GPU memory per training
required. Each training took between 8 and 16 minutes, leading to a total of 500-1000 GPU hours of training.
Table 5: Hyperparameters for each optimizer to achieve the smallest NC3 metric shown in Table
Optimizer
Learning rate
Momentum/3,
Weight decay
SGD
0.01
0.9
0.05
SGDW
0.0679
0.5
0.05
Adam
0.005
0.98
0.05
AdamW
0.005
0.95
Signum
0.001
0.9
0.05
SignumW
0.001
0.98
10
18

---

Published as a conference paper at ICLR 2026
Table 6: Summary of regression fit between NC3 and NCO
Experiment
i SE(
t-value
p p-value
95 % C1
R Adj R
F-statistic
LR
170
0.1903
0.008
24.262
0.000
[0.175, 0.206]
0.778 0.777
588.6
1R
74
0.2017
0.012
16.252
0.000
[0.177, 0.226]
0.786 0.783
264.1
LR=0.01
114
0.1439
0.007
19.892
0.000
[0.13, 0.158]
0.779 0.777
395.7
LR=0.0679
0.1771
0,012
14.367
0.000
[0.152, 0.202]
0.841 0.837
206.4
all
399
0.1582
0.005
32.760
0,000
[0.149, 0.168]
0.730 0.729
1073
D.3 DETAILS ON REGRESSION FIT BETWEEN NC3 AND NCO
In this subsection we provide additional details regarding the regression fit between NC3 and NCO. For the sake of completeness, we show the regression fit in Figure again below. In addition, have also computed regression fit across all training runs, which converged, and all learning rates, shown Figure IT] summary of the regression fit can be found in Table showing that more than 70% of the variation in NC3 can be explained by NCO.
0.001
Ir 0.005
0.01
0.0679
optimizer
AdamW
NC3
Adam
SGDW
SGD
SignumW
10
10a
10
102
10
10a
10
10a
Signum
NCO
NCO
NCO
NCO
Figure 10: Figure shown again for ease of reading. NCO weakly correlates with NC3 across different optimizers and learning rates (here shown for ResNet9 trained on FashionMNIST).
AdamW
1.0
Adam
NC3
SGDW
0.5
SGD
SignumW
0.0
Signum
10-3
100
103
NCO
Figure 11:
NCO correlates with NC3 even when considered across all learning rates together (here shown for ResNet9 trained on FashionMNIST).
D.4 ADDITIONAL. EXPERIMENTAL RESULTS
D.4.1 ABLATION STUDY ON TRAINING EPOCHS
As Neural collapse occurs at the terminal phase of training, natural to control for the effect that the number of training epochs has the final NC metrics. After all, it is possible that the emergence of NC occurs at different speeds for different optimizers.
For this reason, we conducted two ablation studies, in which prolong the training in two settings: We train a ResNet9 in FashionMNIST. which corresponds to the setting which is shown in Figure [ for 2000 epochs with LR=0.000s 0.00 and momentum=0.9 for both optimizers. We note that in this setting, Adam reaches 100% training accuracy already after around 700 epochs for all training runs with WD 0.05. The results can be found in Figure 14 While this leads some improvement of the final
19

---

Published as a conference paper at ICLR 2026
Ir 0.001
Y 0.005
I 0.91
Ir 0.0679
123
127
SCDW
20
50D Sgrani Sgram
10'4
100
10
10-
122
NCD
NCO
Figure 12: NCO vs. NC1 across different optimizers and learning rates (here shown for ResNet9 trained on FashionMNIST).
20001
1.0
optimiter
Acami
0.6
SCD
Siguow
10th
30
102
122
101
10th
20n2
10t
1.0
124
12n
10th
10F
204
20f
10
12.0
107
100
NCO
NC0
NCO
NC0
Figure 13: NCO vs. NC2 across different optimizers and learning rates (here shown for ResNet9 trained on FashionMNIST).
NC1 and NC2 metric for A for some values of weight decay, this has barely an effect on NCO and NC3.
Furthermore we extend training to up to 2000 epochs for selected runs from Figure Concretely, these runs trained with LR of 0.001 and the following combination of WD and momentum (mom, WD) E {(0, 0), (0.97, 5e-5), (0, 5e 5e-4) -), (0.9, 5e 5e-"), (0.9, 0), (0.95, 0.0025)) which corresponds to different parts in the heatmap. The results can be found in Figure 15 While one observe general decrease the NC metrics in all cases, the overall trend for increasing weight decay remains unchanged. Both figures indicate that training the models considered in this work for 200 epochs is sufficient to draw the conclusions that we make about the necessity of coupled WD for the emergence of full NC.
10a
0.2
0.4
10
10
10"
10
10"
10'
10"
10 10"
16"
decay
weight decaay
weight decay
denay
Adantty
Acmm
2000
Figure 14: ResNet9 trained on FashionMNIST with Adam and Adam W for more epochs.
D.4.2 UNCONSTRAINED FEATURE MODEL.
We also validated our results on the unconstrained feature model (UFM) (see Appendix C.5 for
reference) with width d = 512, K = 10 classes and N = 10.000 samples. The UFM was trained with Adam, AdamW and SGDMW with momentum=0.9 and varying Ire {0.1, 0.5, 1.0] and weight decay ranging from 0.0 to 0.05. We then filtered the results, by only including models which achieved 100% training accuracy. The results in can be found in Figure [6 The plots show that the NC metrics, in particular NCO and NC3 remain at least one magnitude of order larger than the same metrics for Adam and SGDMW, highlighting that Adam converges to different solution than Adam, which is not NC.
20

---

Published as a conference paper at ICLR 2026
0.15
0.6
10"
0.6
0.10
g04
0.4
10
0.2
0.06
02
16"
10"
10"
10
wiight deaay
e20-dr
200
2000
SGDMW
Figure 15: Selected runs from Figure trained for more number of epochs.
10
16'
10
SCOW
10
16"
Figure 16: NCO (left), NC3 (center left), NC1 (center right), and NC2 (right) for increasing weight decay.
D.4.3 TRAINING DYNAMICS OF MINIMAL NC3 RUNS
In this section we provide the dynamies of the NC metrics as well as the singular values from the training runs which reached the smallest final NC3 metric reported in Table The purpose is to disentangle the effect of using first-order optimizers (such as SGD and SGDW) vs. second-order like optimizers (such Adam and AdamW) from the effect of applying coupled decoupled weight decay. The main question we try to answer here is: Is the difference between Adam and SGD with respect to the emergence of NC larger than the difference between AdamW and Adam? Figure 17 shows that all runs reach perfect train accuracy well before the end of training, such that they have reached the terminal- ferminal-phase of training (TPT) at epoch 200. Looking the NC1-NC3 metrics in Figure and and Figure 20 one can see that the NC metrics for SGD and SGDW are close to each other. It is harder judge whether AdamW Adam
closer to NC, NC3 is considerably larger for AdamW, while NC1 is slightly larger for Adam, compared to the other optimizers. Nonetheless, the NCO metric in Figure 18 and the evolution of the singular values of W in Figure 19 (left) indicate that AdamW has considerably different training dynamics than Adam, as both NCO as well as the smallest singular value increase instead of converging to zero for Adam W. but not for Adam. While the NCO metric of Adam is still orders of magnitude larger than for SGD and SGDW and the smallest singular value of W converges to a small, but non-zero value, Adam shares similar trends as SGD and SGDW and such converges to solution which is arguably closer to NC3 than AdamW. Whether the solution found by Adam can already be classified as NC or not is an inherent problem of interpreting the NC metrics in practical settings, we have also discussed in Section 4.1
SGD
SGDW
Adam
AdamW
10
1.00
10-1
0.95
train loss
train acc
10-3
0.90
10-5
0.85
10-7
0.80
50
100
150
200
0
50
100
150
200
epoch
epoch
Figure 17: Train loss (left) and train accuracy (right) for training runs with smallest final NC3 metric for different optimizers.
21

---

Published as a conference paper at ICLR 2026
SGD
SGDW
Adam
AdamW
10
10
NCO
NC3
10
0.0
200
50
200
epoch
epoch
Figure 18: NCO (left) and NC3 (right) for training runs with smallest final NC3 metric for different optimizers.
Singular values of w
Singular values of M
2
30
20
1
10
100
200
0
100
200
Epoch
Epoch
SGD
Adam
SGD
Adam
SGDW
AdamW
SGDW
AdamW
Figure Singular values of last-layer weights W (left) and centered class means M (right) throughout training for runs corresponding Table The dotted line corresponds to the smallest singular value and the full line corresponds to the average singular value, excluding the smallest singular value.
SGD
SGDW
..B..
Adam
+
AdamW
0.08
0.06
NC1
0.6
0.04
0.4
0.02
0.2
50
100
150
200
0
50
100 epoch
150
200
epoch
Figure 20: NC1 (left) and NC2 (right) for training runs with smallest final NC3 metric for different optimizers.
22

---

Published as a conference paper at ICLR 2026
10
NCO. to__oommalize
10
10
NCO
10
10
10-5
0
50
100 epoch
150
200
0
50
100 epoch
150
200
Figure 21: NCO (left) and normalized NCO (right) for training with smallest final NC3 metric for different optimizers.
n_epochs
1.7544
10"
2.7542
3.7540
NCO normalized lizeed
1.7542
2 2./530
1.7538
1.7540
300h
1.7538
10-1
1.7536
4-4-0000-
epoch
epoch
Figure 22: NCO (left) and normalized NCO (right) for training runs with zero weight decay from the ablation study in Appendix (.44.d Note that the x-axis is in logarithmic scale and that the point at epoch 1 corresponds to the model at initialization.
D.4.4 ABLATION STUDY ON NORMALIZING THE NCO METRIC
We evaluate whether measuring a normalized NCO metric affects the conclusions that we draw in our work. Concretely, we compute the normalization as
1 NCO P
W W
(14)
We compute both NCO well normalized NCO for the setting of minimal NC3 that we studied in Appendix D.4.3] which we show in Figure 21] While the absolute values differ slightly between NCO and NCO both the trends as well the final values are almost the same.
For zero weight decay, one would expect to see more difference between the dynamics of NCO and normalized NCO, which we show in Figure 22 While one can observe the monotontic effect of momentum on normalized NCO, but not on NCO, we point out that in this case normalized NCO does not correlate with NC1-NC3 anymore. On the contrary, NCI-NC3, while still comparably large, are smaller with less momentum.
As the dynamics of NCO and normalized NCO are almost the same for larger values of WD or normalized NCO is not consistent with NC1-NC3 for zero WD, we are tentative to conclude that the normalization will not affect the conclusions that draw in this work.
D.4.5 ABLATION STUDY ON EFFECT OF MOMENTUM ON NC EMERGENCE
We conduct another ablation study to further evaluate the effect of momentum on the NC emergence. The main question that we try to answer with this ablation is whether the effect of momentum on smaller NC metrics can be simply traced back to the fact that momentum accelerates convergence or it affects the emergence of NC beyond this.
Concretely, track the evolution of the NC metrics together with the train loss and accuracy for the | of momentum. This is because the final train loss value varies for different values of WD due
same setting as Figure over time for fixed value fo weight decay=0.005 and varying values
23

---

Published as a conference paper at ICLR 2026
1.00
0.0
100
0.5
mor
train loss
0.9
0.7
tran acc
0.95
0.0
0.5
0.95
0.90
0.7
0.97
0.9
10
0.98
0.85
0.95
0.98
0.80
50
100
150
200
50
100
150
200
epoch
epoch
100
0.0
1.25
0.5
10-1
1.00
NCD
0.7
0.75
0.95
10
0.9
0.97
0.95
0.50
0.98
10
0.97
0.25
50
100
150
200
50
100
150
200
epoch
epoch
1.0
0.0
0.0
0.5
0.5
0.7
0.8
0.7
0.9
0.9
NC1
0.95
0.6
0.95
0.97
0.97
0.98
0.4
0.98
0.2
10-2
50
100
150
200
50
100
150
200
epoch
epoch
Figure 23: Train loss, train accuracy and NC metrics for fixed WD=0.005 and different values of momentum on ResNet9 trained with SGD with otherwise same hyperparameters Figure
its regularizing effect. The results can be seen in Figure 23
There are two things to be observed: While the accelerating effect of momentum is mainly visible in the early phase of training (up to 50-100 50-100 epochs), modulo some loss spikes for high momentum, the final train loss is not smallest for the largest value momentum. While this is not surprising per se, as too large momentum can lead to overshooting of the training trajectory, the NC metrics show a clear monotonic behavior with respect to the momentum. Furthermore, while the training runs with momentum=0.7 and 0.9 reach almost the exact same final train loss, the disparity in NC metrics indicates that they converged to solutions with very different geometric structure. This can be seen more clearly in Figure 24 Both observations suggest that momentum affects the emergence of NC beyond simply accelerating the speed of convergence. To the best of our knowledge, connecting the magnitude of momentum to NC: is novel and not been discussed in prior work.
D.4.6 ABLATION STUDY ON NC EMERGENCE UNDER ZERO WEIGHTT DECAY
To investigate whether WD is necessary or not for the emergence of NC, we track the NC metrics while training a ResNet9 on FashionMNIST (Note that this is the same problem setting as in Appendix D.4.5) using SGD with zero WD and varying values of momentum with an initial LR=0.01 for 200 epochs. Additionally, we train the model also with zero momentum and high momentum=0.99 for 2000 epochs, with LR decay after 1/3 and 2/3 of training. Importantly, all training reach perfect train accuracy after 40 epochs. The training dynamics can be found Figure 25 two conclusions from this ablation study:
We draw
24

---

Published as a conference paper at ICLR 2026
1.00
0.7
10"
0.9
train loss
train acc
10-1
0.90
10-2
0.85
0.7
0.9
0.80
50
100
150
200
0
50
100
150
200
epoch
epoch
0.7
1.4
0.7
0.9
1.2
0.9
10"
NCO
1.0
x 10
0.8
0.6
x 10
3 10
0.4
50
100
150
200
0
50
100 epoch
150
200
eooch
0.7
0.7
10"
NC1
0.6
10-1
0.5
50
100
150
200
0
50
100
150
200
epoch
epoch
Figure 24: Train loss, train accuracy and NC metrics for fixed WD=0.005 and mom=0.7 and 0.9. Although both runs converge to almost exactly the same train loss, the final NC metrics differ considerably.
25

---

Published as a conference paper at ICLR 2026
Table 7: Smallest NC metries achieved with and without weight decay for training ResNet9 on MNShi
SGD with WD (2000 epochs)
NC1
NC2
NC3
0.2
R3 0.55
0.7
WD (200 epochs)
~ 0.02
~ 0.2
= 0.13
1. The final NC metrics NCO-NC3 after 2000 epochs are slightly smaller than after 200 epochs, consistent with our ablation study in D.4.1. that longer training reduces the NC metrics. This decrease however fairly small.
2. The final NC metrics (both for 200 epochs and 2000 epochs of training) remain considerably higher than what is achieved by the "best" run of SGD in terms of NC metrics with 200 epochs of training for all NC metrics, even with 10 times longer training. See Figure and Figure 20 for comparison.
The final NCI-NC3 metrics achieved with WD after 200 epochs and without WD after 2000 epochs can be found in Table 7 While the experiments cannot fully exclude the possibility that NC can be achieved eventually in the asymptotic limit, we argue that WD is essential to observe the emergence of NC practical finite-length training settings.
D.4.7 MORE DETAILED PLOTS ON COUPLED VS. DECOUPLED WEIGIIT DECAY
As we average across different values of momentum and learning rates in Figure and Figure 3 we provide more detailed plots here in Figure 26 Figure 27 and Figure 28 It can be seen that for the adaptive optimizers and SGDW the variance for varying values of momentum is comparably small for each fixed learning rate, with the variance generally increasing with larger weight decay. For SGD the variance for NCO higher for large values of weight decay, consistent with what is shown in Figure 3 (right) and what is shown in Figure 5
D.4.8 MISSING PLOT: SINGULAR VALUE OF W AND M WITH SIGNUMW
The missing plot of the evolution of the singular values of the last-layer weights W and feature matrix M can be found in Figure 29
D.4.9 COUPLED VS. DECOUPLED DECAY ON OTHER DATASETS
The comparison between coupled and decoupled decay SGD, Adam, and Signum other combinations of (architecture x dataset) can be found in the following pages below, which confirm observations made earlier the ResNet9 trained FashionMNIST. While NCO (visually) correlates well with NC3, it correlates considerably less with NCI and NC2, although general trend still visible across all experiments.
ResNet50 on ImageNetIK We also conducted experiments a ResNer50 trained on ImageNetIK Deng et al. (2009). The model was trained with Adam and AdamW for 90 epochs. We left out other optimizers due to limited resources. For both optimizers the learning rate was chosen as 0.0003 with step-wise decay after 1/3 and 2/3 of training. momentum was chosen from {0.0, 0.5, 0.9} and weight decay was chosen from {0.0, 1e -e 1e_ 1e 11.... The resulting NC metrics can be found in Figure 30 and Figure 31 and confirm the conclusion that Adam W does not have full NC emergence.
VGG9 on FashionMNIST The comparison between coupled and decoupled weight decay on SGD, Adam, and Signum on a VGG9 trained on FashionMNIST can be found in Figure 32 and Figure 33 The relation between NCO and NC3 can be found in Figure 36 between NCO and NC1 Figure
and between NCO and NC2 in Figure 35
ResNet9 on Cifar10 The comparison between coupled and decoupled weight decay on SGD,
Adam, and Signum a ResNet9 trained on Cifar10 can be found in Figure 37 and Figure 38 The relation between NCO and NC3 can be found in Figure between NCO and NC1 in Figure and between NCO and NC2 in Figure 40
26

---

Published as a conference paper at ICLR 2026
PLepOchs
1.000
100
0.975
0.950
train loss
10
0.925
0.900
10-3
0.875
10
0.850
10-5
0.825
0.800
10"
10a
10a
10a
10a
10a
10a
10a
epoch
epoch
r_spocts
1.7544 10
1.4
1.7543 x 10"
1.3
1.7542 10
1.2
1.7541 10"
1.1
Tugenee
NCO
1.754 109
1.7539 10
1.7538 10
1.7537 100
1.7536 109
10a
101
102
10a
10
10a
10a
103
epech
epoch
2.00
2000
1.0
1.1
1.0
0.9
NC1
0.4
0.8
0.6
0.0
10a
10a
10a
10a
0
109
10a epoch
107
10a
epoch
Figure 25: Training loss and train accuracy (top row), NCO and NC3 (middle row), and NCI and NC2 (bottom row) for ResNet9 trained on MNITT with SGD without weight decay for varying values of momentum and number of epochs. Note that the x-axis in log-scale to improve readability.
27

---

Published as a conference paper at ICLR 2026
Ir 0.001
Ir 0.005
0.01
AdamW
Adam
NCO
100
0.0
0.5
0.7
10-1
0.9
0.95
010-1
10-1 10-3 10-2 10-2 10 10
010-4
10-1 10-2 10-2 10-1
100 10-
010-4
10-110- 10-3
10-2 10-1 10f
10
weight_decay
weight_decay
Weight decay
0.01
optimizer
AdamW
momentum
0.6
0.4
0.7
0.2
0.98
010:5
10 10 10-1 10-1 10t 10!
010-4
10 10-210-2 10-1
10"
10t
010-5
10-110--1 10-1100 10"
10t
10-
weight_decay
weight_decay
Weight-detay
Figure 26: NCO metric (top) and NC3 metric (bottom for different values of weight decay, momentum and LR for Adam vs. W.
0.001
= 0.005
Ir- 0.01
10
optimizer
SignumW
10a
Signum
10a
momentum
0.0
10a
0.5
10c
0.9
10:4
10-110-110--10 weight_decay
30-110'1001100 100-100- weight decay
020-5 10-4 10-110--1100
Weight decay
Ir 0.001
Ir 0.005
Ir 0.01
optimizer
SignumWW
Signum
3.0
momentum
0.0
-8 0.5
0.9
0.4
0.95
10 "00 '1
10
10" 10'
90 '10 20 weight_
10'
10 $10 100 100 1 1000
Weight decay
Figure 27: NCO metric (top) and NC3 metric (bottom for different values of weight decay, momentum and LR for Signum vs. SignumW.
28

---

Published as a conference paper at ICLR 2026
Ir 0.001
Ir 0.01
0.0679
optimizer
10-4
SGDW
SGD
10a2
momentum
10
0.5
0.0
10
0.7
0.9
10
10-1 10-3 10
10-110-110-10 10-1
010-3 10-110'10--10- 10 10-
weigh_deccay
Weight_decay
Weight detay
Ir 0.001
I-- 0.01
Ir 0.0679
160
optimizer
5GDW
SGD
NC3
0.0
-8- 0.5
0.9
0.95
010-3 10 110-410"1 30
10" 10f
010
10 10 10 10t
10t
010
10 100110000000 20-
10" 102
beight_decay
weight_deray
Weight decay
Figure 28: NCO metric (top) and NC3 metric (bottom for different values of weight decay, momentum and LR for SGD SGDW.
600
SignumW
SignumW
Singular values of
400
100
200
0
50
100 Epoch
150
200
100
150
200
Epach
Figure 29: Singular values of last-layer weights W (left) and feature matrix M (right) throughout training for SignumW W on ResNet9 trained on FashionMNIST. Dotted line corresponds do smallest singular value and full line corresponds to the average singular value excluding the smallest singular value.
VGG9 on Cifar10 The comparison between coupled and decoupled weight decay on SGD, Adam, and Signum can be found in Figure 4 and Figure 43 The relation between NCO and NC3 can be found in Figure 46 between NCO and NC1 in Figure 44 and between NCO and NC2 in Figure
ResNet9 on MNIST The comparison between coupled and decoupled weight decay SGD, Adam, and Signum on a ResNet9 trained on MNIST can be found in Figure /47 and Figure 48 The relation between NCO and NC3 can be found in Figure ST between NCO and NC1 in Figure 49 and between NCO and NC2 in Figure 50
VGG9 on MNIST The comparison between coupled and decoupled weight decay on SGD, Adam, and Signum on a VGG9 trained on MNIST can be found in Figure (52 and Figure (53] The relation between NCO and NC3 can be found in Figure 56 between NCO and NC1 Figure 54 and between NCO and NC2 in Figure 55
29

---

Published as a conference paper at ICLR 2026
1.0
0.9
NC
10a
0.7
optimizer
optimizer
Adam
0.6
AdamW
Adom
Adam
0.5
10
10 Weight decay
10-3
10-5
10 Weight decay
10
Figure 30: NCO (left) and NC3 (right) metrics plotted against weight decay on a ResNet50 trained on ImageNetIK K for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
AlantW
Adam
1.33
1.32
2
1.30
optimizer
AdamW
1.29
Adam
10-3
10- -4
10-2
10
10-4
10-2
Weight decay
Weight decay
Figure 31: NC1 (left) and NC2 (right) metrics plotted against weight decay on a ResNet50 trained on ImageNetIK K for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
D.4.10 10 PRELIMINARY EXPERIMENTAL RESULTS ON VISION TRANSFORMER
We have also conducted preliminary experiments pretraining small Vision Transformers (ViT) on Cifar10 from scratch. Given that training ViTs is computationally much more expensive given the larger size of the model, we had to limit ourselves to a more restricted number of experiments. Specifically, we chose to train the ViT with Adam, A, and SGD for 200 epochs with batch size of 512 with momentum in the range E {0, 0.8, 0.9, 0.95) and weight decay WD {0, le -5 14 -4- 1e-3 0.05, 0.5} for Adam and SGD and WD E {0, 1e-4 0.05, 0.5, 1, 4] for AdamW. We discarded all runs, which did not achieve a training accuracy of at least 50%. This mainly corresponded training runs of SGD and Adam either with momentum=0 WD> 0.05.
The VIT implementation is based on code from from www.ttttttteeeeeeeteeceeeeaaeeet /main MIT license. Specifically, the transformer model was chosen with a hidden dimension of 512, 6 hidden layers, and attention heads, with no dropout applied.
http::////itttu.coo/.o./ which published under the
Compared to the training procedure used in other settings, we employ a cosine-decay learning rate schedule with warm-up, where 5% of the total training steps are allocated to warm-up, and the base learning rate is set to 1 x 10-3, Weight decay is applied to all layers except for LayerNorm and biases, which standard practice.
The highest final test accuracy across all trainings was achieved by AdamW (3 0.95, WD 0.5) with 83.67%, with a final test loss of 0.895. Notably, higher accuracy levels can be attained by increasing the network size and applying data augmentation by using pre-trained model as in Ammar et al. (2024). However, to ensure consistency with the experiments in the main study, we do not perform data augmentation due limited computational resources. This likely explains the relatively lower test accuracy. Investigating the impact of data augmentation on the convergence to NC remains an interesting avenue for future work.
30

---

Published as a conference paper at ICLR 2026
optimizer
Adamw
Adam
NC3
10
0.4
- Adam
10-1
02
0.0
10-4 10-3
10
10
10?
10a
010
10"
10
Weight decay
Welght decay
Figure 32: NCO (left) and NC3 (right) metrics plotted against weight decay on a VGG9 trained on MNSHT for Adam and AdamW. Shaded area refers one standard deviation across all trainings run with corresponding optimizer.
1.4
opi mine SCEW
1.2
332
uslimer
1.5
opdm
113
159
22a
500
Wr (i detay
No woight.coo cecky
Weight
WeigTt
Figure 33: NCO and NC3 metrics plotted against weight decay VGG9 trained FashionMNIST for Signum and Signum (left side) and SGD and SGDW (right side). Shaded area refers one standard deviation across all trainings run with corresponding optimizer.
While we observe the general trend of decreasing NC metrics with increasing values of weight decay for SGD (Figure ) we note that in the case VITs the NCO metric for both Adam and Adam W first increases betore decreasing (Figure [58b] left), while the NC3 metric for both Adam and Adam W has U-shape (Figure 588) right). We also note that the VIT is much more sensitive the choice of weight decay and the training and validation accuracy degrades quickly due to overregularization as can be seen Figure 58c further investigation of these observations is left for future work.
31

---

Published as a conference paper at ICLR 2026
Ir 0.001
Ir 0.005
Ir 0.01
Ir 0.0679
optimieer
10c
AdamW
NC1
Adam
10-1
SGDW
SGD
10-2
SignumW
10-2
10
10-2
10a
10-2
10a
10-2
10a
Signum
NCO
NCO
Figure 34: NCO vs. NCI on VGG9 trained on FashionMNIST. Note that the x-axis is plotted in log-scale.
Ir 0.001
0,005
0,01
Ir 0.0679
optimizer
1.0
AdamW
NO
Adam
0.5
SGDW
SGD
SignumW
10-2
102
10-2
10?
10-2
10?
10
10?
Signum
NCO
NCO
NCO
NCO
Figure 35:
NCO NC2 VGG9 trained FashionMNIST. Note that the x-axis is plotted in
log-scale.
0.001
0.005
0.01
0.0679
optimizer
00
AdamW
1.0
NC3
Adam
SGDW
0.5
SGD
SignumW
10 2
10
102
102
10 2
102
Signum
NCO
NCO
NCO
NCO
Figure 36:
NCO vs. NC3 on VGG9 trained on FashionMNIST. Note that the x-axis is plotted in log-scale.
optimizer AdamW Adam
0.8
10a
NCO
0.6
Adami
Adam
0.5
10-
10-4 10-3
10-2
10-1
10
10-4
10-3
10-2
10-1
Weight decay
Weight decay
Figure 37: NCO (left) and NC3 (right) metrics plotted against weight decay on a ResNet9 trained on Cifar10 for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
(p mizr
+ SCOW
13th
3.4
Sggur
3300
10-4
13-1
0:10 10
13th
16-1
2.5-4
30-4
15-4 10-1
30-1
1.0
10-4 1p-1
Figure 38: NCO and NC3 metrics plotted against weight decay on a ResNer9 trained on C0 for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
32

---

Published as a conference paper at ICLR 2026
Ir 0.001
0.005
Ir 0.01
0.0679
10a
optimizer
AdamW
NC1
Adam
SGDW
10
+5 %
SGD
SignumW
10-2 NCO
102
10-2
102
10-2
102
10-2
102
Signum
NCO
NCO
NCO
Figure 39: NCO NC1 ResNett trained on Cifar10. Note that the x-axis is plotted in log-scale.
Ir 0.001
Ir 0.005
0.01
Ir 0.0679
optimizer
1.0
AdamW
Adam
SGDW
SGD
SignumW
10
10a
10-2
102
10-2
102
10-2
102
Signum
NCO
NCO
NCO
NCO
Figure 40: NCO NC2 ResNet9 trained on Cifar10. Note that the x-axis is plotted in log-scale.
Ir 0.001
Ir = 0.005
Ir 0.01
Ir 0.0679
optimizer
AdamW
NC3
SGDW
0.5
SGD
SignumW
10
102
10
102
10 2
102
10 2
102
Signum
NCO
NCO
Figure 41: NCO vs. NC3 on ResNet9 trained on Cifar10. Note that the x-axis is plotted in log-scale.
0.9
0.8
NCO
0.6
AdamiW
0.5
Adam
optimizer AdamW Adam
0.3
10
10 Weight decay
10
10
10
00
10
10 10
Weight decay
Figure 42: NCO (left) and NC3 (right) metrics plotted against weight decay on a VGG9 trained on Cifar10 for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
(pT mive
3.3
weEW
13
1.5
129
12a
8.3
SgrurMi
Skouar
Sgrum
550
10e1
Weigh
Figure 43: NCO and NC3 metrics plotted against weight decay VGG9 trained Cifar10 for Signum and SignumW W (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings with corresponding optimizer.
33

---

Published as a conference paper at ICLR 2026
0.001
Ir= 0.005
Ir= 0.01
Ir 0.0679
optimizer
AdamW
NC1
Adam
10"
SGDW
SGD
SignumW
102
102
102
102
Signum
NCO
NCO
NCO
NCO
Figure 44: NCO vs. NCI on VGG9 trained on Cifarl0. Note that the x-axis is plotted in log-scale.
0.001
0.005
0.01
0.0679
optimizer
AdamW
Adam
18
0.5
SGDW
SGD
SignumW
10-?
10-?
102
10?
10-2
10a
Signum
NCO
NCO
NCO
NCO
Figure 45: NCO vs. NC2 on VGG9 trained on Cifar10. Note that the x-axis is plotted in log-scale
0.001
0.005
0.01
0.0679
optimizer
AdamW
1.0
Adam
0.5
SGDW
SGD
0.0
096
SignumW
10-2
102
10-2
102
10
102
10
102
Signum
NCO
NCO
NCO
NCO
Figure 46: NCO NC3 on VGG9 trained on Cifarl0. Note that the x-axis is plotted in log-scale.
optimizer
0.8
AdamW Adam
0.6
10"
0.4
10-1
optimizer
AdamW
0.2
10
Adam
010
10
10 Weight decay
10
10
10"
10
010-1
10-t 10-2 10-2 10-2 Weight decay
102
10:
Figure 47: NCO (left) and NC3 (right) metrics plotted against weight decay on a ResNet9 trained on MNIST for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
33
1.2
SCOW
merr Sipuure
Sigum
Sigeur
6-2
550
500
30 10 23
1P 10"
032
12-1
26
30
13
16-1
30
Figure 48: NCO and NC3 metrics plotted against weight decay ResNer9 trained MNIST for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
34

---

Published as a conference paper at ICLR 2026
Ir 0.001
Ir 0.005
0.01
Ir 0.0679
optimizer
10"
AdamW
Adam
10-1
SGDW
SGD
10-2
Signumw
10-2
10a
10-2
10a
10
10a
10-2
10a
Signum
NCO
NCO
NCO
NCO
Figure 49: NCO vs. NC1 on ResNet9 trained on MNIST. Note that the x-axis is plotted in log-scale.
Ir 0.001
0.005
0.01
0.0679
optimizer
AdamW
Adam
0.5
SGDW
SGD
S+ 8
0.0
SignumW
10-2
10a
10-2
10a
10-2
10a
10-2
10a
Signum
NCO
NCO
NCO
NCO
Figure 50: NCO vs. NC2 on ResNet9 trained on MNIST. Note that the x-axis is plotted in log-scale.
0.001
Ir 0.005
0.01
0.0679
optimizer
AdamW
NC3
Adam
SGDW
SGD
0o
SignumW
10-2
10a
10-2
10a
10-2
10a
10-2
10a
Signum
NCO
NCO
NCO
NCO
Figure 51: NCO vs. NC3 on ResNet9 trained on MNIST. Note that the x-axis is plotted in log-scale.
1.0
optimizer
10
0.8
AdamW Adam
0.6
0.4
10-1
optimizzrr
AdamW
0.2
10-2
Adam
010-3
10-1
10-1 Weight decay
10->
10-2
109
10:
010
10 Weight decay
10"
10'
Figure 52: NCO (left) and NC3 (right) metries plotted against weight decay on a VGG9 trained on MNIST for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
66
qgt mive
1.2
391
Nagueit Sigrum
0.8
mier
Sprure
232
397
6.1
62
562
10-h 10 We4g1
012-1
15-1
10
1ure Weigh
Figure 53: NCO and NC3 metrics plotted against weight decay on VGG9 trained on MNIST for Signum and SignumW W (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings with corresponding optimizer.
35

---

Published as a conference paper at ICLR 2026
10a
optimizer
AdamW
NC1
Adam
10-1
SGDW
SGD
5x
Reag
SignumW
10-2
10a
10-2
10
10-2
10a
10-2
10
Signum
NCO
NCO
NCO
NCO
Figure 54: NCO vs. NC1 on VGG9 trained on MNIST, Note that the x-axis is plotted in log-scale.
Ir 0.001
0.005
0.01
Ir 0.0679
optimizer
1.0
GRI
AdamW
NC2
Adam
0.5
SGDW
SGD
0.0
SignumW
10-2
10a
10
10a
10-2
10a
10-2
10a
Signum
NCO
NCO
NCO
NCO
Figure 55: NCO NC2 on VGG9 trained on MNIST, Note that the x-axis is plotted in log-scale.
0,001
Ir 0.005
Ir= 0.01
= 0.0679
30th
optimizer
AdamW
NC3
Adam
SGDW
SGD
SignumW
10
10a
10
10a
10 2
10a
10
10a
Signum
NCO
NCO
NCO
Figure 56: NCO vs. NC3 on VGG9 trained on MNIST. Note that the x-axis is plotted in log-scale.
1.0
1.0
1.0 SPoni
0.9
0.9
0.9
0.8
0.8
0.8
10-1
10
10a
0.25
0.50
0.75
1.00
0.5
NC1
NC2
NC3
AdamW
SignumW
Signum
Figure 57: NC4 is largely uncorrelated with NC1-3 across different optimizers and learning rates.
36

---

Published as a conference paper at ICLR 2026
0.8
optimizer
1.4
SGD
NCO
1.3
optimizer
0.6
SGD
1.2
10 4
10
10
10 3
weight weigh
weight_ weight_decay
(a) NCO (left) and NC3 (right) metric for varying values of weight decay on VIT trained with SGD Cifar10.
optimizer
1.0
AdamW
Adam
0.8
2
0.6
0
10
10 10-3 10-2 10
10"
0
10
10
10-3
10
10
10
weight_decay
(b) NCO (left) and NC3 (right) metric for varying values of weight decay ViT trained with Adam and AdamW on Cifar10.
1.0
Validation accuracy
1.0
Train accuracy
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0
10
10
10
10
10
100
0
10
10
10
10
10
10"
weight decay
weight decay
AdamW
SGD
Adam
(c) Training accuracy (left) and validation accuracy (right) varying values weight decay on a VIT trained on C0
37

---

Published as a conference paper at ICLR 2026
E PROOFS
In this section, we will present the proof which is omitted the main text.
Theorem E.1 (Effect of decoupled SGD update NCO). Assume model of the form J(W, ,0,x) Wha (a) is trained using cross- cross-entropy loss with SGD with decoupled weight decay for all parameters W, O. For instance, the last layer weight W has the following update rule:
Vi+1 = Vv + Vw, LCE,
Wi+1 (1- nA) W. V/+1 L+1
where B E =0,,1), 17 O, and À E R. Define the NCO metric
W. 1/
03 = K
Then, for all
a, (1 77)20 og.
In particular, if0 <n>> < then ar decays exponentially to zero:
a₂ = (1 - 7))² ag t-000 0.
Proof. We track the evolution of the row sums of W, and V,. Define
m: w 1/1e RK
q = V/1eRK
By definition of a have
a2
Note that by Lemma E.5
the cross-entropy gradient with respect the last layer satisfies
(Vw, LCE)
for all W.. Consider the momentum update
V 3V, + VW,LCE
Multiplying on the right by 1 and using the above result, we obtain
9++1 = V 1 (5V,+ + Vw, 1 = SV (Vw,LCE) T 1 fq + 0.
Thus 9.+1 = 3q., and by induction
q: 8 qo-
Since Vo = 0, we have qo O, hence
q for all / 2 0.
Consider now the decoupled weight update
W = (1 - n)a W. V
Multiplying the right by 1 gives
m W ([L-nA)W.-nVC 1 1- 111=(111i1 WI- (1-1))m, 77++1
Using q++1 = for all f, we obtain the simple linear recursion
m (1 -77) m.-
Solving this recursion yields
m, = (1 - myJ) m
Substituting the expression for m, into the definition of or gives
(1 m))' 112 - n2) K |lmoll (1 2)
1
K
This establishes the exact formula claimed in the theorem.
38

---

Published as a conference paper at ICLR 2026
Theorem E.2 (Effect of SGD update with coupled weight decay on NCO), Assume model of the form f(W.6, x) = Who (z) is trained using cross-entrops loss with stochastic gradient descens (SGD) and momentum
[0, 1). weight decay X E [0, 1), and learning rate n > 0 sufficiently small. The last-layer weights We are updated according to:
V++1 3V, + Vw, LCE AWE,
(15)
W = Wt - V
where B E {0, =0, 13 O, and X E Then there exists a constant C 2 such that
1 -
miml
Cp2²t 0o
for all 2 0,
(16)
K
where p == max([r+1||r-1) and r+ are the roots of
y² (1 3 1) =
(17)
In particular: nn < 2(1 + B), then p < 1 and the NCO metric au decays exponentially
Proof. We follow the same strategy in the decoupled case: track the evolution of the row sums of V, and W..
From 15
V = 3V, + Vw, LOB + AW,
Right-multiplying by and using Lemma (E.5) we get
V
4++1
(6V: + VW, + +WW)) 1
=3V, + (Vw, LoF) 1 AW.1
=.qq + Am.
Thus
9--1 = Bq + X m.
(18)
From the weight update
W41 = W. Ve+i.
obtain
m = WT 1 =
1 W 1 V/ m. qr/
Using
18
this becomes
m m. nB3 q, Am,) =(1 m7)) m, n1 4:-
(19)
We also have, from the weight update at time 1,
m, m 77
which is just
19 with index shifted by one. Hence
q ?
(m:-1 -m))
(20)
Substitute 20 into
1
m441
(1 171)) m - -733
[m m.)
17
(1 - (πλi m - 33 - m:)
{1 Tπλ) m - 3mm-1 + 3m,
(1 m71) m 8 m
39

---

Published as a conference paper at ICLR 2026
We given m. W. and qu Vi 1 0. Then
0qq + Amo Amo,
and hence from the weight update
mi mg 7/11 mg 7mm (1 772) mo-
The recurrence is linear and homogeneous with constant coefficients. For each coordinate of m,, say (me)a. we have a scalar second-order recursion
(m++1)a (1 17>) (m²)k k (m--1)k-
The characteristic polynomial is
--+3--111 3
with roots r4 and r- given by
+ 3 17 v /1+ - -43
2
Thus each coordinate can be written as
(m.))x = c4,44 + c_
for coefficients C4,k: C- determined by (mo)s and (m))a- Let
max (r+
be the spectral radius of the recursion. Then there exists a constant C' 2 (depending only on 3 X, 1) such that
Imllly S Cp'
and therefore
1 0n K
m C2 p²
1 K
mo 112
C² n²
ax
for some C' 2 which is (16). Finally, for a general quadratic equation 7 br + c= 0, the roots in the unit circle if |c| 1, + b + 0 and 1 - b > 0. Thus it is not difficult to check from the characteristic polynomial that n < 2(1 + 3) implics p < 1.
Note that the above Theorem holds for any model f( ((W00,z) Whh((z) with last layer as linear classifier and with any backbone he parameterized by
However, the dynamics of Adam is more complicated, hence we further restrict the setting to SignGD, special case of Adam, training UFM.
Here, we assume balanced dataset with only one element in each class & E [K]. It is obvious to extend our result multiple elements per class. Hence the total input N K is equal to the number of classes and the UFM loss can be written as
LCE (WH,)) Σ LCE (W,,, en),
where we can decouple the regularization X 2|| 2 ||| into weight decay. know that the UFM
By Zhu et al. (021), Db,
X
max W.H
LEE( Wh,,, Vn.
W
H
has unique global minimum W. H and no strict saddle points. In particular, H UM* for some orthogonal matrix e O(P). To further simplify the analysis, we assume that P = N = K with H M Then have the followings:
40

---

Published as a conference paper at ICLR 2026
Theorem E.3. Consider sign GD with (decoupled) weight decay À > 0 and step size n 0 on the UFM loss
LCE (WH.I)
Σ
LCE (Wh,, Cn),
a=1
where the feature H M is fixed to NC solution and only the weight W is trained:
W u+1 We - ([sign( w, Lce) + X
with initialization Wo e R Rx* We define the covariance matrix C, W.W, and the scalar
ar J) where J 11 Then at will increase monotonically from zero the limit:
(K 2)
lim at (1000
X2
In particular, a, does not vanish as oc.
Proof. By Lemma E.5 we have VLCE W) 3/4 (S Y)H = (S 1 )J
NYK-1 (softmax (WH)- - I) since (softmax (H) - 1)J = 0. Since softmax has range between and 1, we have
sign LVL WH) J 21, that is, the signed gradient is -1 on the diagonal and + elsewhere. Note that this holds for all W e kx The sign GD updates can hence be written as:
W W.
T/
J 21 +AW,
(21)
sign Vw, Lex)
Since sign
is constant, the dynamics collapse onto a scalar Wy:
We w/ (J- - -1),
which has the following recursive form:
Wr+1 (1 -1\)w; 7/, ww 0.
Solve and obtain
W; [ (1 --y-
Recall the definition:
C. W.W, j= 11 and 04 (Cr, J)
K
Since [] 21) T 11 (K 2) K and the factor of 1/K gives (K - 2)2, we have
Oi = (K w
Therefore
an = (K - 2)²
H (1
m (1n=((1))')]]
(K - 2)2
1 (1 11=))))²2
A2
As t + 00, (1- (1-12)' + +0, so
(K (5-2) 2)2
0.00
X2
Theorem E.4. Consider sign GD with (coupled) weight decay X > O and step size nj > on the UFM loss
LCE (WH,I
Σ LIE ew),
n=1
where the feature H M is fixed to an NC solution and only the weight W is trained
W W. misinns ww,,gnn XW.)
with initialization Wo = 0 E R KxK
We define the covariance matrix C, = W.W. and the
scalar J)F where j
* 11 Then there exists learning rate decay scheme
7 = (11) 11000
> such that at t-000
0.
41

---

Published as a conference paper at ICLR 2026
Proof. Throughout the training, apply mathematical induction the structure of W for all t, there exists ae, b, 2 such that
W, (at + b,)1 b,J.
is not hard to see that a = Nv (a. - (K - 1) Note that for t = 0, the signed gradient is the same in the case with decoupled weight decay Theorem 3.3
sign( Vw, LcE XW,) = Sw LCF) = sign(softmax (0) - I) =
21.
Hence, W1 = n J) where = b 17. Since H = M VK-I (I -I- JJ/k),
WH = ((ar + by) b,J) VK - (I- J/K))
1
((ar + by)
brJ (a, + +
VK -1
44 + b1
H 7
VK
where we define 71
bb VK
By Lemma
E.5
and the above expression, the loss gradient becomes:
1
Vw,LCB
(softmax (estmaxxWH)) I)
NVK K
(softmax (7:
NVK K
= (o(-KKKI KI+JJ
where v NVK-1 en/Vk (oVvK-i((K]))
Hence the update weight
NVK-1 2loc+b///k (ECCKK--)KKK-- +(K--1
will also of form
W ++1 (a.-1 + b(+1) b,41J)
Hence the update rule of the signed GD with coupled weight decay can be written as:
0++1 d, + 1 sign ((K 1)4/2 - Xae)
b b, + 17 sign(t'r Ab:)
Then for each fixed n > starting from divided into three phases:
O. let A. (0c+1 des b be), the training can be
1. A, (+7, +1) as long as (K 1) 2 sor and d/ 2 Ab, Note that the oc σोe 13777K (K-) hence u < th, as A, = (+7, +n). Since i,, strictly decreasing with bt. assume is small enough, there exists constant Ti such that Ar; (+1), +7) but AT = (+1, -1) where (K 1) 2 Ao; 2 Ab, Uh.
2. A, = (+n. -n) indicating increases stricity in each step and b, starts to oscillate long as (K - 1) 1/ 2 Xay: each time A, = (+1),- -1]), we have a, b, a:-1 + b,-1 and thus v/ -- Hence th decreases monotonically but not strictly. Similar to above, there exists a constant T2 > Ti such that (K - 1)Yr, 2 dar, but (K 1)4T++1 < daT
3. For t > T2. A, (+n, In) where i) vh becomes constant for A, oscillates between (+1). and (-1), +-y) or ii) the oscillate for A, oscillates between (+1), +-y) and (-7, -1). In wither case, we have max (l(K 1)1/2 Aar 11/2 Abrl} < An as each update will flip the sign.
Hence for each 17. we update T₃ = T2(1) steps until max [I(K 1)4b - Aayl 1:22 Abyl} < An. Next, apply learning rate decay 1f that Any min[((K 1) dars+1 |1 Aby2+1|} < Ar.. Repeat the above argument and find a Ty > such that uh oscillates or remains constant after i. T2 + Ty and hence maxx>Ta+T; [[(K 1) Aae, I1/x -Xb.]] < Any Induction
42

---

Published as a conference paper at ICLR 2026
this argument shows that there exists learning rate decay scheme 17 =n((t) such that max 1) Aarl: t/e Abel
=- 1-000
O, in which case:
a: (ar - (K 1)6e)²
- (Aa: (K -))
S A -2 ((K 1)01 + 1(K 1)12 Xael (K 1)4r
+ (K 1)/12 Abrl)²
A 2- (I(K - 1)1/ - Aaxl + (K 11)/4/ - -bb,)²
S A 2 x2 pax(I(K 11)b dael W -bbal} 0.
1000
Hence 04 = (a: - (K - 1)b,)² 70000 O.
E.I TECHNICAL LEMMATA
Lemma E.5. Let (X, E RAXN x R be a dataset where the labels are written in columns of one-hot vectors. For each pair (x RD x RKK and weight W R define the cross-entropy as:
K
(( del Σ
Yk log (softmax( W [1)) = =log 1+ Σ
exp((w> - Xi
where y = arg maa the class index of x. Let Li (W1) = CE(W,X, Y) be the average cross-entropy loss of the dataset (X, Y). Then the loss gradient (L)(W,)
1 (S- (X- N
VL((W1)
where S = (S),...SN) and s, = softmax( W ix:) for each i. In particular, 1 VLL =
Proof. The expression of the loss gradient comes from simple calculus. The second statement comes from the fact that the LI norms of post-softmax vector and an one-hot vector are both equal to 1, that is,
I Js 1ky, Vi.
Lemma E.6. Assume the weight W. is updated as follows:
V I+ BV, G, + AWW
W = W, W,-nVeH
where G, depends on W t. Define
del. 1
W 1 > 0.
K
Then we have the expression:
(at+1 - 0r) = --2wu 27/- 2dcre + 7/4+1
where wt def. (V.W) j). d def (G.WT j), Pt def (V.VT.J).
Proof. Let C, W, W be the covariance matrix. Notice that a, (Ct, where j == 1/2 11 By update rule W. and V,:
(C,1 C,) -,)==((W, -oW.--Veeis)WW. ( CC))
--(V+11, + W.V. +
43

---

Published as a conference paper at ICLR 2026
Applying the dot product F) on both sides, and denote we def. (V.W7 J). no * (G. J).
V.V, J), we have
a1+1 - a4) = - WI,J) + + (V V
=-2((BV, + G. AW, + m/++1
=-2ww - -2 2λo, 11411
(22)
where in the first line the fact that J is symmetric.
Lemma E.7. Assume X, B E (0, 1) such that log B
The solution of the following ODE:
a(t) = - (f. a
(23)
with initial condition a(0) = 000 > 0 admits the following bound:
(tt) S Coo exp
(Toog
A
log 8
for some absolute constant C > 1.
Proof. Observe that we can write the integral in convolution:
L.
a a(r)dr (o (t), where ((t) = 8
Hence 23 can be written as
a(1) - - (1).
Let ((v(0))))
st (1)dl denote the Laplace transform. Denote
A(s) ({a(t))ss), F(s) (s)(t))ss.
Taking the Laplace transform of both sides:
[(a(()(s) = Xel(o (s)){(ss..
(24)
And by integration by part and the property of convolution,
((a(1)(s) = sA(s) - a(0)) and
* (()){s) = F())A(s).
Hence
*A(s) (00) A())))((s)
Since 3 = (log 3)t we get
1
F(s) = [(B) = Cf clog 976 [s) 8 -log(3))
for s > log(3).
Substitute this back to Eq.
24 and we get:
*A(s) o(0) -
A(s)
log (3)
*A(s)
A(s) = a(0)
log (3)
A(s) 8 8+
= a(0)
$ - log (3)
Lo
lox(8)
[s log [3)
A(s) = a
2
log( (3)
(e-r))(e-r2))
44

---

Published as a conference paper at ICLR 2026
(1)
(1g(18)
4A
where r1: 12
We do partial fractions and matching coefficients gives:
(5)
A
B
A 1,
-log(3) -Ar - Bri.
- (--ri)(s r2)
Since r1 + 12 = (8g( one finds
12
r1
71
T2
Thus
T2
1
A(s) a(0)
12
T1
71
12 T1 12
Recall the inverse of Laplacian transform: --
[tt
nt Therefore,
x-r
- a(l) -- (A())))) a(0)
r2 e
Pat
122 -
1'2 1
Equivalently,
a(t) - a(0) (c + Be A
T2
B
71
(25)
12
where
T1, log(3) + / [lo{(3)] 41
2
Since 3 E (0, set L = log(33 > 0. By the first order approximation,
2A
1²
(log 3)2 47 12 41
+0
L
Hence
-L
21 L
X2
r1, 12
0
I
This gives:
(Aº)
(2
+0
12 -L + o
Plugging T1, r2 into Eq. 25
all) S Ca((0)) Ca(() exp -
for some absolute constant C' > 1. Plug in L log( log --1 finish the proof.
45


Reviews:
[{"review_id": "eFgNgRKwaI", "reviewer": "Reviewer_fTjQ", "summary": "The paper argues that how weight decay is implemented largely determines whether Neural Collapse (NC) emerges. It introduces NC0 as a necessary diagnostic and proves contrasting NC0 dynamics in a stylized model. The paper also validates the mechanism with large-scale experiments, showing NC metrics improve as the coupled component increases, while accuracy remains about the same.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. An actionable mechanism that practitioners can immediately test and reason about.\n\n2. NC0 provides a tractable and provable indicator, enabling convergence and impossibility statements.\n\n3. Value the importance of optimization algorithm choices in NC.", "weaknesses": "1. Modeling gap to real Adam/AdamW. The formal results use SignGD and unconstrained features as a proxy. While the qualitative match to Adam/AdamW is persuasive, the absence of finite-step analysis with (β₁, β₂, ε) leaves open whether corner-casescould break the claimed dynamics.\n\n2. NC0 is necessary, not sufficient. It could happen that NC0→0 but full NC (NC1–NC3) can still fail. Without parallel theory for the other NC metrics, one could over-infer the presence or absence of collapse.\n\n3. External validity across regimes. The empirical case is broad but still focused on common settings. Very large batches, aggressive data augmentation, label smoothing, and heavy regularization may alter gradient noise scales and effective WD, potentially changing the coupling story.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l0E0wWDBOB", "reviewer": "Reviewer_nwnq", "summary": "The authors provide new insight into Neural Collapse (NC). They present NC0, a new necessary condition for NC2 and NC3, and demonstrate that the choice of optimizer impacts neural collapse, particularly the momentum and weight decay parameters and their specific implementation.\nExperimentally, the authors show that non-zero weight decay is necessary for NC0 and NC3 metrics to decrease. They also observe that these NC metrics decrease both by increasing weight decay (for a fixed momentum) and by increasing momentum (for a fixed non-zero weight decay). This behavior is theoretically proven for SGD with momentum and weight decay.\nFinally, they show that neural collapse does not emerge with decoupled weight decay for optimizers like Adam and Signum. They prove this for signed SGD with decoupled weight decay, while also showing that NC0 does decrease for signed SGD when using coupled weight decay.", "questions": "Given that the occurrence of NC is connected to better generalization, how would the authors interpret their result that decoupled weight decay prevents Neural Collapse, in connection with the fact that AdamW is an industry standard due to its performance gain over coupled weight decay?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work investigates Neural Collapse from a novel perspective, supported by both experimentation and theoretical evidence.\n- It opens up for further discussion and deeper investigation in the field.\n- The work is clearly written.\n- The authors critically discuss their findings and limitations.", "weaknesses": "- Some plots are difficult to interpret due to overlapping lines and similar colors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wUpmGhMYFq", "reviewer": "Reviewer_2Q1N", "summary": "The authors address an important literature gap in the understanding of the conditions under which neural collapse emerges. The paper points out that the emergence of neural collapse can largely depend on the choice of the optimizer. In particular, they show that in adaptive optimizers, particularly those of second-order nature, the emergence of NC (in particular the weight-feature alignment NC3) might depend on the way in which weight decay is implemented – whether it is coupled or decoupled. The authors show both empirically and theoretically, that AdamW or SignumW consistently don’t converge to zero/small values of NC3 metric, while their coupled counterparts (as well as SGD with momentum, either coupled or uncoupled) converge to much smaller values. The main analytic tool (also used extensively in experiments) is the so-called NC0 metric (the global bias of the rows of last layer’s weight matrix), which is a necessary condition for the emergence of full NC (especially NC2 and NC3 together). The authors show that this metric converges to a non-zero constant for signGD with decoupled weight decay, while for the coupled weight decay it might converge to zero provided vanishing learning rate. The authors complement these main findings by several other related observations and ablations.", "questions": "-\tIn Figure 1 we see a little increase in the NC3 metric of Adam optimizer for the largest value of weight decay. Do you have a guess why this would be? \n-\tWhy is there such a small variance in the measured metrics in Figure 1 and similar figures, when you average them over a wide variety of learning rates and momentums? \n-\tHave you tried to train Adam with as high values of weight decay as AdamW? What would happen? \n-\tThe NC0 values for SGD in Figure 8 seem weird. Do you have an explanation? \n-\tIn Figure 15 we see that the metrics go up for Adam for large values of weight decay. Why is that? \n-    How big of a difference there is in your opinion between real Adam and SignGD in terms of implicit biases? We know that Adam is known to converge in some regular cases, while SignGD does not seem to be likely to ever converge. For this reason, do you think that SignGD is a good approximation of Adam? \n\n\n**SUMMARY:**\nWhile I think the content of this paper is publishable material and the contributions are insightful, I cannot recommend acceptance at this point due to fair amount of concerns I have. I would recommend the authors to re-structure and re-interpret the work and resubmit. Moreover, since the scope of the paper is a bit modest (particularly because it mostly only concerns the NC3 metric), my recommendation would be to submit to TMLR.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-\tS1: Both theoretical and empirical evidence presented in the paper is convincing (modulo a few caveats mentioned in the weaknesses). The paper makes it clear that the presented phenomenon is indeed happening. Some of the ablations in the appendix (for instance the one with 2000 training epochs) reassure me even more. \n-\tS2: The paper tackles an important topic of better understanding the conditions under which the neural collapse emerges. \n-\tS3: The paper is mostly well written and easy to follow. \n-\tS4: The paper also presents several non-central insights, such as the role of the momentum, analysis of the interpolation between the two Adam variants, discussion of the meaning of the NC metrics. \n-\tS5: The authors seem to be open about the limitations of the paper and discuss them appropriately.", "weaknesses": "-\tW1: If I correctly understand the mathematical essence of the paper, it seems that authors are misinterpreting their results. To me, the main distinction should not be coupled vs. decoupled weight decay but rather first- vs. second-order optimizers. It seems that the crucial parameter that really determines the limiting behavior is not so much the implementation of the weight decay, but rather whether the gradient computation corresponds precisely to the original loss function and thus, whether the zero-sum gradient property from line 263 holds for that optimizer or not. In particular, while authors correctly interpret Theorem 3.3, I believe that Theorem 3.4 should be interpreted as a proof that SignGD with coupled weight decay also doesn’t converge to zero, even in this idealistic ETF features scenario. I think there is a good empirical evidence that Adam itself doesn’t converge to NC too – in Table 2 we see that the best NC0 metric reached by Adam is orders of magnitude higher than those reached by both SGD and SGDW. Similar conclusions can be drawn from Figure 7 where even for pure Adam, NC3 metric seems to converge to positive value well above 0, or Figure 6, where we see that the smallest eigenvalue of Adam converges to small, yet non-zero value. In Figure 5, while NC3 is still improving for Adam, the NC0 seems to have converged. The SGDW, on the other hand, despite the decoupling, does not suffer from the above-mentioned issues, which suggests that the crucial aspect is indeed the division by empirical exponentially averaged variance in second-order optimizers like Adam. Now, the reason why there is such a big empirical difference between Adam and AdamW is that the presence of WD in the internal updates of the momentum and variance parameters seems to alleviate (yet not fully erase) this systematic bias. That’s why the empirical results seem to suggest that the difference between decoupled and coupled weight decay is bigger than that between first and second-order optimizers. Thus, I agree that this difference should be discussed and the paper provides compelling evidence for it, but I don’t think it should be narrated without proper discussion (or even prioritization) of the differences between the optimizer types. \n\n-\tW2: Some of the empirical evidence gives mixed signals and gives the reader an impression that the authors did not properly calibrate the weight decay ranges for which the experiments were conducted. This can be seen by the fact that a lot of weight-decay vs. NC3 plots show concave behavior for the decoupled weight decay variants. This is true in Figure 3, but also in appendix figures 17, 20, 24, 25, 29, 30, 35, 40. This suggests that higher values of weight decay for the decoupled variants could reach as low values of NC3 as the coupled variants. Another strong evidence for this is in Figure 14 for NC0 metric, where we see a huge difference in the metric for the largest weight decay, again suggesting that the weight decay ranges should have been wider on the right end of the plots. While part of this phenomenon could be explained by the quadratic relationship between NC0 and regularization strength, the fact that NC3 metric is normalized properly makes it unclear to what extent are the curves concave because of this, and to what extent because of the actual convergence of NC3 to lower values.\n\n-\tW3: A fair comparison of weight decay and momentum values across experiments, or when comparing coupled and decoupled implementation, is a general issue in the paper. For instance, it is well known and it agrees with Theorem 3.1 of the paper, that the weight decay generally only influences the speed of convergence of NC metrics, but not necessarily whether they converge. Furthermore, it is also well-known that the training converges towards neural collapse also for zero values of weight decay, but the convergence is slower. Therefore, results such as the ones in Figure 4 should be calibrated to account for this, for instance by letting training run until a fixed small training loss is achieved. If the authors only meant that weight decay and momentum only affect the speed of convergence, then it should be more explicitly discussed in the paper, but these results are not novel. \n\n-\tW4: Related to the previous point, I believe that the NC0 metric should be normalized by the norm of the weight matrix. If the NC0 metric remains constant but both the weight matrix and the final layer features grow to infinity (as is known to be the case in the zero weight decay case), than geometrically NC can still be asymptotically approached. Therefore, normalizing NC0 metric would provide for a more fair comparison across various weight decay values. \n\n-\tW5: Finally, the strength of the message and/or contribution of the paper is relatively modest, especially because we see from the experiments that many conclusions the authors draw only hold for NC3 metric, but not for NC1/2 metrics. Thus, the coupling of weight decay / optimizer itself seem to be particularly important only for the NC3 metric. This limits the scope of the results, although I agree that the observation that full NC convergence might depend on the optimizer is still an important and insightful addition to our knowledge. \n\n-\tW6: Some writing aspects should be improved. In particular:\n-    Equation 1 doesn’t use bias in the last layer, but in appendix authors include bias in the definition.\n-\tThe original definition of the NC metrics formulated them for the centered class means. The authors should discuss why they decided to omit this detail in their definitions. This also influences the line 143-144 in the proof where authors talk about the convergence of centered class means. \n-\tLine 147: I don’t see why this should hold. We can always rotate M^* and counter-rotate P accordingly. \n-\tThe constant T in Theorem 3.1 is redundant. \n-\tLine 273: you cite the wrong paper of Jacot. It should be Jacot 2024. \n-\tThe definition of within-class variability in line 637 is wrong.\n-\tThere is a notation inconsistency between the definition of the \\alpha parameter in Theorem 3.3 and its appendix version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)