Cross Entropy versus Label Smoothing: A Neural Collapse Perspective
Authors: Li Guo
George Andriopoulos
Zifan Zhao
Zixuan Dong
Shuyang Ling
Keith W. Ross
Venue: Accepted by TMLR ()

Abstract: Label smoothing loss is a widely adopted technique to mitigate overfitting in deep neural networks. This paper studies label smoothing from the perspective of Neural Collapse (NC), a powerful empirical and theoretical framework which characterizes model behavior during the terminal phase of training. We first show empirically that models trained with label smoothing converge faster to neural collapse solutions and attain a stronger level of neural collapse compared to those trained with cross-entropy loss. Furthermore, we show that at the same level of NC1, models under label smoothing loss exhibit intensified NC2. These findings provide valuable insights into the impact of label smoothing on model performance and calibration. Then, leveraging the unconstrained feature model, we derive closed-form solutions for the global minimizers under both label smoothing and cross-entropy losses. We show that models trained with label smoothing have a lower conditioning number and, therefore, theoretically converge faster. Our study, combining empirical evidence and theoretical results, not only provides nuanced insights into the differences between label smoothing and cross-entropy losses, but also serves as an example of how the powerful neural collapse framework can be used to improve our understanding of DNNs.

Published in Transactions on Machine Learning Research (05/2025)
Cross Entropy versus Label Smoothing: A Neural Collapse Perspective
Li Guo New York University Shanghoi
1g154 @nyu.edu
George Andriopoulos
ga733mmu.
New York University Abu Dhabi
Zifan Zhao
xz43306mmu.edu
New York University Shanghai
Zixuan Dong
zd662@nyu.ed edu
New York University Shanghai
Shuyang Ling
sl36350myv.cm
New York University Shanghoi
Keith Ross
Seithwrosssnyy.. edu
New York University Abu Dhabi
Reviewed on OpenReview: https: / opw.revee....ee..... net/ forum? FEo55EIvGI
Abstract
Label smoothing a widely adopted technique to mitigate overfitting in deep neural networks. This paper studies label smoothing from the perspective of Neural Collapse (NC), powerful empirical and theoretical framework which characterizes model behavior during the terminal phase of training. We first show empirically that models trained with label smoothing converge faster to neural collapse solutions and allain a stronger level of neural collapse compared to those trained with cross-entropy loss. Furthermore, we show that at the same level of NC1, models under label smoothing loss exhibit intensified NC2. These findings provide valuable insights into the impact of label smotthing on model performance and calibration. Then, leveraging the unconstrained feature model, derive closed-form solutions for the global minimizers under both label smoothing and cross-entropy losses. We show that models trained with label smoothing have lower conditioning number and, therefore, theoretically converge faster. Our study, combining empirical evidence and theoretical results, not only provides nuanced insights into the differences between label smoothing and cross-entropy losses, but also serves as an example how the powerful neural collapse framework can be used to improve our understanding of DNNs.
1 Introduction
The effectiveness of a deep neural network (DNN) hinges significantly on the choice of the loss function during training. While cross-entropy loss is one of the most popular choices for classification tasks, many alternatives with improved empirical performance have been proposed. Among these, label smoothing (Szegedy et al.) 2016, Müller et al.) 2019 Lukasik et al. 2020) has emerged as a common technique to enhance the performance of DNNs. Instead supervising the model Training with one-hot. key labels, label smoothing preprocesses the
* *Corresponding authors

---

Published in Transactions on Machine Learning Research (05/2025)
label for each sample by taking a convex combination of the one-hot label and a uniform distribution This procedure is generally understood as means of regularization for improving the model's generalixability.
In this paper, we examine the benefits of label smoothing from the perspective of neural collapse PPapyan ct al. 2020); a new and powerful framework for obtaining an improved understanding of DNNs.
Background and Related Work
Deep neural networks (DNNs) typically consist of a non-linear feature extractor and linear classification layer. Neural collapse, first observed in (Papyan et al., 2020). occurs in DNNs for classification tasks where the data is balanced and cross-cntropy loss is employed. It characterizes the geometric properties of the fcatures produced by the feature extractor and the weight vectors of the classifier during the terminal phase of training (TPT):
NC1: The learned features of samples from the same class approach their respective class means.
NC2: The collapsed features from different classes and the classification weight vectors approach the vertices of a simplex equiangular tight frame (ETF).
NC3: Up to rescaling, the linear classifier weights approach the corresponding class means, giving a self-dual configuration.
Other than cross-cntropy loss, neural collapse phenomena have been observed and studied under the mcan- squared loss as well (Han et al. 2021 Poggio & Liao, 2020 Zhou al.) 2022a). Furthermore, research has
extended neural collapse investigations to scenarios involving imbalanced data (Fang et al., 2021 Hong &
Ling 2023 Thrampoulidis et al. 2022 Dang et al. 2023 Hong & Ling 2023 and cases with a large number
of classes (Jiang et al. 2023).
In addition to being observed empirically, neural collapse has also been proven to arise mathematically. Beginning with MMxon et al. (2020), a series of studies have employed approximation models such as the Unconstrained Feature Models (UFMs) (Mixon et al.) 2020) or layer-peeled models (Fang et al. 2021) to provide theoretical evidence for the emergence of neural collapse. These optimization models simplify a DNN by treating the layt-layer features as free variables to optimize over, which is justified due to the expressiveness of DNNs (Hornik 1991 Cybenko
1989 Lu et al. 2017: Shaham et al. 2018) Theoretical advancements in neural collapse not only enhance our understanding of DNNs but also inspire new techniques to improve their performance in diverse applications, such as imbalanced learning (Xie et al.] 2023 Liu et al. 2023). transfer
learning (Galanti et al.
2021 Li et al. 2022 and continual learning YYu et al. 2022 Yang et al. 2023 etc.
Several studies have explored UFMs under various loss functions and regularization strategies Wojtowytsch
ct al. 2020 Zhu et al.] 2021 Dang et al. 2023 Lu & Steinerberger 2022 Tirer & Bruna 2022 Fier et al.
2023
Yaras et al. 2022; Zhou et al. 2022b). Particularly, Zhou et aZhou et al. 2022b) established that
under the UFM model, the global minimizers for a wide range of loss functions, including cross-entropy loss with one-hot labels and smoothed labels (label smoothing), exhibit the idealized neural collapse properties. Additionally, they demonstrated that the UFM model has a benevolent landscape that enables the global minimizer to be effectively allained using ilerative algorithms. However, their investigation does not provide insights into why label smoothing consistently outperforms cross-entropy loss, or why label smoothing converges faster during training. In this paper, leveraging the neural collapse framework, we conduct an in-depth investigation of training under these loss functions, aiming to explain the reasons behind the observed superiority of label smoothing over cross-entropy loss with one-hot labels.
Our Contributions
Our study begins with a comprehensive empirical comparison between cross-entropy loss with one-hot labels (hercafter referred to as cross-cntropy loss for simplicity) and label smoothing throughout the training process. Specifically, we carefully study how the last layer features and linear classifiers evolve during training, Our findings are follows:
Compared with cross-cntropy loss, models trained with label smoothing exhibit accelerated convergence in terms of training error and neural collapse metrics. Furthermore, they converge more pronounced level of NC1 and NC2.
2

---

Published in Transactions on Machine Learning Research (05/2025)
2. Along with accelerated convergence, label smoothing maintains distinct balance between NC1 and NC2. Notably, as compared with cross-entropy loss, label smoothing results in more pronounced level of NC2 when reaching a comparable level of NCI. We argue that this phenomenon originates from the implicit inductive bias introduced by label smoothing, which equalizes the logits of all non-targel classes and thus promotes the emergence of a simplex ETF structure in both the learned features and classification weight. We posit that the emphasis on NC2 label smoothing, which promotes maximally separable features between classes, enhances the model's generalization performance. Conversely, excessive level of NC1 may cause the features to overly specialize in the training data, hindering the model's ability to generalize effectively.
3. Models trained with smoothed labels exhibit improved calibration (Guo et al. 2017 by implicitly regularizing classification weights and features during training. However, if temperature scaling (Guo et al., 2017) is applied as post-processing to counteract the regularization effect, label smoothing can lead to deteriorated model calibration. This is because excessive level of NC1 under label smoothing can cause the model to be overconfident in its predictions, even when they are incorrect, thereby negatively impacting model calibration.
Of equal importance to the empirical results, we perform mathematical analysis of the convergence properties of the UFM models under cross-entropy loss and label smoothing. While Zhou et al. (Zhou et al. 2022a) demonstrate that, for a broad class of loss functions, including cross-entropy loss and label smoothing, the global minimizers exhibit neural collapse properties, the authors neither derive the exact form of these global minimizers nor thoroughly examine the landscape around them.
1. We first derive closed-form solutions for the global minimizers under both loss functions, which explicitly depend on the smoothing hyperparameter 5.
2. Utilizing these closed-form solutions, we conduct a second-order theoretical analysis of the optimization landscape around their respective global optimizers. Within the UFM context, our mathematical analysis reveals that label smoothing exhibits more well-conditioned landscape around the global minimum, which facilitates the faster convergence observed in our empirical study.
This paper provides significantly deeper understanding of why label smoothing provides better convergence and performance than cross-cntropy loss. Additionally, the paper illustrates how the powerful framework of neural collapse and its associated mathematical models can be employed to gain a more nuanced understanding of the "why" of DNNs. However, we acknowledge that theoretical evidence about how neural collapse impacts model generalization lacking in this study. We hope that our work will inspire future research into the intricate interplay between neural collapse, convergence speed, and model generalizability.
2 Preliminaries
2.1 The Problem Setup
A deep neural network is comprised of two key components: a feature extractor and a linear classifier. The feature extractor Φo(+)) is a nonlinear mapping that maps the input æ to the corresponding feature embodding h = (0) E Rd. Meanwhile, the linear classifier involves a weight matrix W = [[w,, "2, wx] E R and a bias vector R Consequently, the architecture of deep neural network is captured by the following equation:
fe(x) = W (x()) + b,
where θ = (0, W. b) represents the set of all model parameters. In this work, we consider training a deep neural network using a balanced dataset denoted as {( yk)) (ess).sssiiss This dataset consists of samples distributed across K distinct classes, with n samples allocated per class. Here, æ represents the i-th sample from the k-th class, and yhu is a one-hot vector with unity solely in the k-th entry. Our objective is to learn
3

---

Published in Transactions on Machine Learning Research (05/2025)
the parameters θ by minimizing the empirical risk over the total N nK training samples:
K
min e N
Σ (U)()))),) + 2 1
(2)
where 1(:-)) denotes the chosen loss function, and the regularization parameter (i.e., the weight decay parameter).
2.2 Training Losses
To simplify the notation, we use W WT "0(2)) + to represent the network's output logit vector for a given input æ; and p = Softmax(z) to denote the predicted distribution from the model. The cross-entropy between the target distribution y and the predicted distribution p is defined as (ce(y,P) is a one hot vector with value of in the dimension corresponding to the target class. In contrast, label smoothing minimizes the cross-entropy between the smoothed soft label y and the predicted distribution p. denoted as 1CE(3" .P), where the soft. label &d = (1 5)y + (5/K) 1 combines the hard ground truth label y with a uniform distribution over the labels. Here 1x denotes the K-dimensional vector of all ones. The hyper-parameter á determines the degree of smoothing.
Σx & Us log(ps), where y
For simplicity, we use the following formulation to represent cross-entropy loss with and without label smoothing:
IcEly .p) -- :)=
--- yh log(pk),
(3)
where sf (1 8)yk 5 The provided loss corresponds to cross-cntropy loss with one-hot labels when 8 O, and for any other value of 8 (0, 1), represents the loss with label smoothing.
3 Empirical Analysis of Cross-Entropp and Label Smoothing Losses
This section conducts a comprehensive empirical comparison between cross-entropy loss and label smoothing from the perspective of neural collapse. In Section 3.1 we examine the convergence of the model to neural collapse solutions under cross-entropy loss with and without label smoothing. Our results demonstrate that while models under both loss functions converge to neural collapse, label smoothing induces faster convergence and reaches more pronounced level of neural collapse. Beyond convergence rates, Section 3.2 delves into the dynamics of convergence, revealing that label smoothing introduces a bias toward solutions with a symmetric simplex ETF structure, thereby enforcing NC2. In Section 3.3 we attempt to understand the impact of label smoothing on model calibration from the perspective of neural collapse. In addition, we also investigate how the smoothing hyperparameter 8 influences neural collapses during the model training in the appendix B.4
Experiment Setup. We conducted experiments CIFAR-10 KKrihevssy al. 2009 CIFAR-100, STL-10 Coates et al.] 2011). and Tiny ImageNet (Deng et al. 2009). Tiny ImageNet is
subset of ImageNet,
containing 100,000 samples across 200 classes. To ensure consistency with prior studies (Papyan et al. 2020
Zhu et al. 2021 we use ResNet-18 (He et al.) 2016) as the backbone for CIFAR-10 and 2016 for CIFAR-100, STL-10, and Tiny ImageNet.
ResNet-50 (He et al.
To isolate the effects of neural collapse and minimize external influences, we apply standard preprocessing without data augmentation. To comprehensively analyze model behavior during TPT, we extend the training period to 800 epochs for CIFAR-10. 10, CIFAR-100, and STL-10, and 300 epochs for Tiny ImageNet. For all datascts, we use batch size of 128 and train with stochastic gradient descent (SGD) with a momentum of 0.9. The learning rate is initialized at 0.05 and follows a multi-step decay, decreasing by a factor of 0.1 at epochs 100 and 200 for Tiny ImageNet and al epochs 150 and 350 for the other datasets. We use a default weight decay of 5 x 10-4, except for the experiments in Section 3.3 where a weight decay value of 1 x 10-4 used.
Metrics for Measuring NC. We assess neural collapse in the last-layer features and the classifiers using metrics based the properties introduced in Section 1 with metrics similar to those presented in (Papyan)
4

---

Published in Transactions on Machine Learning Research (05/2025)
al.
2020
For convenience, we denote the global mean and class mean of the last-layer features
K
1
hg
hag he
Σ hu (1
(1 S K).
N
Within class variability (NC1) measures the relative magnitude of the within-class covariance matrix Dw = N Ex= Ei=1 (bai
h& (ha: hT) compared to the between-class covariance matrix ΣB = R E (hy (h) - hc) of the last last-layer features. It formulated as:
NC1 trace D
(4)
where D denotes the pseudo inverse ΣB.
Distance to simplex ETF (NC2) quantifies the difference between the product of the classifier weight matrix and the centered class mean feature matrix, and simplex ETF, defined as follows
WH H
IK K [ 1
NC2 = W H VK-1
(5)
where H = hh ha..... hx hal E Rax represents centered class mean matrix.
Self-duality (NC3) measures the distance between the classifier weight matrix W and the centered class-means H:
NC. = W
H
(6)
H
F
It is evident that when NC2 in (5) and NC3 in (6) both reach zero, the matrices W and H form the same simplex ETF up to some scaling factor. Thus, our definitions of NC1-NC3 capture the same concepts the definitions in (Papyan et al. 2020) and (Zhu et al. 2021). We say that neural collapse occurs if NC1, NC2, and NC3 collectively approach zero during the Terminal Phase of Training (TPT).
3.1 Terminal Phase Training under Label Smoothing and Cross-Entropy Loss
In this section, we investigate the distinct behaviors exhibited by models trained under cross-entropy loss with and without label smoothing during TPT. Our experiments are conducted on the CIFAR-10, CIFAR-100, STL-10, and Tiny ImageNet datasets, using a default label smoothing hyperparameter of 0.05. The training dynamics are visualized Figure] 1
The leftmost column in Figure I illustrates the progression of training and testing errors throughout the training process. As expected, models trained under label smoothing exhibit lower testing errors compared to those under cross-entropy loss for all datasets, showcasing an improvement in model generalixability. Furthermore, models trained with label smoothing exhibit faster convergence for training and testing errors. To facilitate a clearer comparison, Table reports training and test errors at differont epochs. Since the network's final layer behaves as nearest centroid classifier (NCC) also include the corresponding training and test errors for the NCC. These results further confirm that models trained with label smoothing exhibit faster reduction in error rates compared those trained with cross-entropy loss. Figure additionally presents the three neural collapse metrics for models trained under cross-entropy loss and label smoothing. While the values of NC3, representing the alignment of W and H, remain consistently low and comparable under both loss functions, models with label smoothing exhibit faster convergence in both NC1 and NC2 and eventually reach lower levels for both NC1 and NC2. In Section B.3 of the appendix, we also compare the training loss of models trained with cross-entrope loss and label smoothing, demonstrating that label smoothing also accelerates convergence in terms of training crror. We provide a theoretical explanation for these empirical findings in Section
the neural collapse solution, we
When the bias is all-zero vector or constant vector, NC₂ implies that the average logit matrix, defined 7 W H + b1 T satisfies Z 1 1x x1)x) some constant a, i.e., Z is simplex ETF.
5

---

Published in Transactions on Machine Learning Research (05/2025)
Error Rate
NC1
NC2
NC3
CE-train error
CE
CE
CE
CIFARI0
Eiror Rate
15:train arror
L5
L5
NC
NC3
0.2
L5-test error
0.2
10
200
400
600
800
200
400
600
800
200
400
600
800
200
400
600
800
10a
1.25
CIFARI0D
Erar Rate
0.75
1.0
0.50
10a
0.75
NC3
0.25
10a
0.50
0.00
10
0.0
800
200
400
800
1.5
STL10
10
1.0
Errer
1.0
0.5
10a
0.5
200
400
600
800
0
200
400
600
800
0
200
400
600
800
200
400
600
800
Tiny Inageliet Erao Rate
1.2
1.00
0.75
0.75
C
1.0
0.50
0.25
0.25
10a
0.6
0.00
200
300
100
200
100
300
Epoch
Epoch
Epoch
Epoch
Figure 1: Neural collapse with cross-entropy loss (CE) and label smoothing (LS). Columns from left to right represent the model's error rate (Train/Testy NC1, NC2, and NC3.
3.2
Label Smoothing Induces Enhanced NC2
CIFAR10
CIFAR100
STL10
Tiny ImageNet
10a
0.15
10a
LS
0.40
10a
LS
0.45
10a
LS
0.45
0.44
10a
10a
+
0.39
+
0:13
10a
0.43
10a
0.38
10
0.42
0.12
0.42
10-1
0.41
L5
0.11
0.37
100
0.40
10a
0.41
0.10
10-1
0.36
10-1
0.39
0.40
0.0
0.1
0.2
0.3
0.4
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.6
0.8
1.0
1.2
NC2
NC2
NC2
NC2
Figure 2: Scatter plots of NC1 vs. NC2 under cross-entropy loss and label smoothing, with colors indicating the test error rate.
Along with the faster convergence to neural collapse under label smoothing, further observe that models trained with label smoothing maintain distinct balance between NC1 and NC2 throughout training. Specifically, we find that for the same level of NC1, label smoothing consistently provides a more pronounced manifestation of NC2 compared to cross-entropy Loss. Figure provides scatter plots of NC1 and NC2 under both loss functions, with the NC metrics recorded every 10 training epochs. Across all four datasets, the data points under label smoothing consistently position the left of those for cross-entropy loss. This observation suggests that at equivalent levels of NC1, label smoothing induces intensified level of NC2.

---

Published in Transactions on Machine Learning Research (05/2025)
Epoch
Train Error (%)
Test Error (%)
NCC Train Error (%)
NCC Test Error (%)
CE
LS
LS
CE
CIFAR10
10
4.84
3.41
21.23
21.60
5.51
3.53
19.67
19.28
20
3.59
3.15
19.67
19.34
3.06
2.49
18.05
17.26
50
2.67
2.28
20.68
17.52
3.75
2.39
18.79
16.86
100
2.27
2.07
17.49
17.26
2.10
2.47
16.90
16.60
CIFAR100
10
41.29
37.37
53.33
54.94
38.84
33.63
53.51
52.13
20
18.23
14.67
52.46
50.94
20.88
11.68
51.57
47.76
50
11.35
55.02
9.02
50.24
100
8.63
7.49
54.21
54.48
7.27
8.26
48.87
48.65
STL10
10
65.74
62.41
07.38
65.96
67.66
64.92
68.01
65.86
20
55.88
47.37
58.18
53.97
64.22
32.20
66.46
56.34
50
0.08
0.02
52.83
40.96
12.16
0.4
50.21
41.81
100
0.06
0.00
45.15
42.11
0.02
0.06
44.24
40.76
Tiny ImageNet
10
51.04
50.67
59.64
60.74
55.84
61.76
62.55
66.95
20
38.30
56.79
42.17
39.85
55.86
54.55
50
27.75
23.14
56.63
56.18
31.21
25.54
54.21
52.36
100
23.14
21.45
57.51
57.42
25.03
19.94
53.48
51.54
Table 1: Training and test error rates based on the final classification layer (left four columns) and nearest centroid decision rule (right four columns) for models trained with cross-entropy (CE) and label smoothing (T.S).
To gain insight into this phenomenon, we now closely examine the formulations of cross-entropy loss and label smoothing. Given an input x, the output logit and the predicted distribution are equal to z == fx) and p = Softmax(2),, respectively. As per Equation 3 assuming the observation æ is from the k-th class, cross-cntropy loss is formulated Ice log(Pk). To minimize cross-cntropy loss, the emphasis is solely on making the logit of the larget class larger than the logits of non-target classes wilhoul constraining the logit. variation among the non-target classes.
On the other hand, label smoothing loss with smoothing hyperparameter 5 is given by
10--((111 K (1) log(pk) +
8
> [log(pi)
LLS =
(7)
K 1if
According to Jenson's incquality, we have
E Sign log(pi) log(pr) =(K 1) ISER
Eist K-T
1- Pk
K-1 < (K 1)
=(K 11)
(8)
K
with equality achieved only pf pv (for 1, k). This implics that label smoothing loss reaches minimum only if the predicted probabilities for non-target classes are all equal. Thus, label smoothing strengthens the inductive bios towards equalizing the logits of the non-target classes. This property aligns with the definition of NC2 in ( explaining why label smoothing loss reinforces NC2. As demonstrated in Section B.1 in the appendix, for deep neural networks with L2 regularization, the convergence of NC₂ as defined in 5) to zero indicates that both the classification weight W and class mean features H converge to a simplex ETF structure. This ETF structure promotes maximally separable features and classifiers, which are inherently more robust to outliers and can theoretically enhance generalization. However, while NC2 is believed to positively impact generalization, the effect of other factors, such as NC1, complicates this relationship. Empirically, we observe slight improvements in generalization for models trained with label smoothing CIFAR-10 and CIFAR-100,

---

Published in Transactions on Machine Learning Research (05/2025)
and more significant improvements on STL-10. Still, due to the interplay between NC1 and NC2, we cannot conclusively establish direct causal link between NC2 and generalization performance. Further theoretical and empirical work on how these factors influence generalization remains a promising area for future research.
3.3 Label Smoothing and Model Calibration
DNN models often suffer from poor calibration, where the assigned probability values to class labels tend to overestimate the actual
0.20
1.6
likelihood of correctness. This issue arises from
CIFAR10
0.15
Post ECE
w--ormmh (h-norm
w-nom Optimal
the high capacity of DNN models, making them prone to overfitting the training data.
FEE
-12
0.10
1.0
0.05
Guo et al. (Guo et al. 2017) introduced the ex- pected calibration error (ECE) as a metric for assessing model calibration. Additionally, they
0.6
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
proposed temperature scaling an effective
0.20
CIFARIDO
w--arrmP--oorr
post-processing technique for improving calibra- tion, which involves dividing a network's logits by a scalar T > before applying softmax, thereby softening (for T > 1) or sharpening (for T < 1) the predicted probability to make it more aligned with the true confidence levels.
ECE
0:15
0.10
0.8
0.05
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Previous studies Müller et al. 2019 have highlighted the effectiveness of label smoothing in improving model calibration. Conversely, our empirical study reveals intriguing findings: without temperature scaling, label smoothing (with properly tuned smoothing paramcter 8)
Figure 3: Plots of ECE Pre/Post temperature scaling (left column), alongside feature and classification vector norms and optimal T (right column) for CIFAR10 and CIFAR100 datasets with varying values of
enhances model calibration compared to cross-entropy loss, bul when lemperalure scaling is applied as post- processing, label smoothing leads to inferior model calibration. This inconsistency motivates us to investigate the underlying reason. In this section, we provide an explanation from the perspective of neural collapse.
Regularization effect of label smoothing on weight and feature norms. We trained 1esNec18 on CIFAR10 and ResNet50 on CIFAR100 using label smoothing with various smoothing hyperparameters ranging from 0 to 0.3. Then, we evaluated models' testing ECE both before and after temperature scaling, as depicted Figure Our observations reveal that compared to cross-entropy loss (8 = 0), label smoothing with properly chosen 8 lead to lower model calibration error (ECF), while larger values of 6 may degrade model calibration. This is because tuning the smoothing hyperparameter has similar effect to tuning temperature T in temperature scaling. Particularly, as shown empirically in the right column Figure 3 and theoretically in Theorem 4.1 increasing 8 decreases both the feature and classification vector norms thereby reducing the confidence level model predictions. Consequently, the corresponding optimal temperature T decreases alongside the increase in In fact, there exists a smoothing hyperparameter 8* for which the optimal T equals 1, making temperature scaling post-processing unnecessary. Selecting d > 8 results in an excessively low confidence level in the model's predictions, leading to notable increase in test ECE.
However, if we apply lemperat scaling to counteract the regularization effect of label smoothing feature and classification weight, models trained with label smoothing demonstrate higher test ECE (Post ECE as shown in the left column of Figure 3 compared to models trained with cross-entropy loss. We further investigate how this phenomenon is attributed to the more pronounced level of NC1 under label smoothing loss.
Thhe feature norm is computed as the average norm each class mean feature, represented by E like hall/K, K. The classification vector norm is determined as the mean norm of the weight vector for each class given by Σx || /I/ where urk E Rd represents the k-th column of the classifier weight. W.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Excessive NC1 adversely impacts model calibration. To investigate the impact of NC1 model calibration, we analyze how miscalibration occurs during training, using CIFAR10 data with cross-entropy loss and label smoothing (ò = 0.05) as examples. We specifically partition the test set into two subgroups: those correctly classified and those incorrectly classified by the model. In Figure 4 the left column illustrates the average cross-entropy loss and the average entropy of the model predictions for both subgroups of test. samples. The middle section of the figure showcases the NC1 metric for the entire test datasct, as well as for each of the subgroups individually. Meanwhile, the right column of Figure presents the training and testing misclassification errors, accompanied by the test set ECE.
12
0:25
NC1
Thitt Toes entropy
10
Test NC1
Test error
Test Loss
correct
8
Test NC1 incorrect
correct
Error Rate/Rest
0.15
Post ECE
That loss incorrect
CE
Test entropy correct
0.10
Test entropy incorrect
0.05
200
600
800
200
400
800
400
800
0.25
Train NC1
Train error
LS Test less/Tost entrocyy entropy
Tost MC1
Enor RaterRest ECE
0.20
Tast
Test
Test NC1 correct
Pre ECE
Test loss correct
NC1 Incorrect
0.15
Post
loss incorrect
NC1
Test entropy correct
Test entropy incorrect
0.05
0.00
200
400
600
800
200
400
600
800
0
200
400
6.00
800
Epoch
Epoch
Epoch
Figure 4: Leftmost plot: average error and entropy for correctly and incorrectly classified test samples. Middle plot: NC1 for the training set, testing set, and correctly/ porretly/incorreee /incorrectly classified testing samples. Right plot: model classification error rate and the test. ECE before and after temperature scaling.
In Figure | models trained with cross-entropy loss and label smoothing show similar trends. For correctly classified test samples, both test loss and average entropy exhibit consistent decreasing trend, indicating the model's increasing confidence in its correct predictions. Conversely, for incorrectly classified test samples, although the average loss starts to increase after 350 epochs, the overall downward trend of entropy suggests the model's growing confidence in its incorrect predictions. Moreover, the middle plot in Figure | indicates that NC1 for both correctly and incorrectly classified test samples consistently decreases during training. For misclassified samples, a smaller NC1 indicates that their feature vectors align more closely with the mean of the incorrectly predicted class, making the model more confident about its incorrect predictions. As a result, the entropy for misclassified test samples continues decrease, and the test ECE starts to rise during TPT. And if we apply temperature scaling to counteract the regularization effect of label smoothing feature and classification vector norms, label smoot loss results in higher ECE due to its stronger NC1.
4 Theoretical Analysis
4.1 Unconstrained Feature Model
Analyzing deep neural networks poses significant challenges because of the non-linearities and complex interactions between layers in the feature mapping h = @0(x). The Unconstrained Feature Model (UFM) Fang et al.) (0211) simplifies DNN models by treating the features of the last layer as free optimization variables. This choice is motivated by the idea that over- wer-parameterized DNN models are able to approximate
any continuous functions (Hornik 1991 Cybenko 1989 Lu et al. 2017 Shaham et al. 2018
Forr misclassified samples, we assess the within-class variance relative the incorrectly predicted class.

---

Published in Transactions on Machine Learning Research (05/2025)
Recall that the dimension of a feature vector h is denoted by d. Let H (huihssss E RaxN be the feature matrix of all training samples with hys denoting the ((k 1)m + i)-th column of H. Under the UFM, we investigate regularized empirical risk minimization, a variant of the formulation in Equation 2
min 1 W.H.b N
E (W hus + b. v + Aw W
AH 2
AB 2
H
where v (1 5) + (k/K)lx is the soft label for class & with smoothing parameter d. Here one-hot vector with the k-th element equal to 1, and Aw.AH. As > 0 are the regularization parameters.
Let Y [ee ee quently, the mairix form of the soft labels can be represented as y (1 o)Y x Ix The empirical risk minimization in (9) has an equivalent matrix form:
_x1p E RKXN represent the matrix form of the (hard) ground truth labels. Conse-
1 min C(W, H.b) == N W,Hbb
CCEE(W ( H +1b Y0)
Aw 2
W
+ Au H
AB 11613 2
(10)
where (W H - bl Yd) computes the cross-entropy column-wiie and takes the sum.
4.2 Theoretical Results
Within the UFM framework, Zhou et al. (hoou et al. 2022b) demonstrate that, under wide range of loss functions, including cross-entropy loss and label smoothing, all the global minimizers satisfy neural collapses properties, and in particular H and W form aligned simplex ETFs. Morcover, they show that every critical point either global minimizer or a strict saddle point, which implies that these global minimizers can be effectively attained through iterative algorithms, However, their work does not provide the exact expression of the global minimizers, nor does it provide landscape analysis (condition number) across different loss functions, which is critical for understanding the model's convergence behavior. In contrast, our work derives the exact solutions of both W and H for UFM models with cross-entropy loss and label smoothing. Furthermore, based on these solutions, we employ conditioning number analysis to closely compare the optimization landscape of the models surounding their respective global minimizers, and thereby provide an explanation for the accelerated convergence observed under label smoothing loss. The proofs of our thcorems can be found in the Appendix.
Let Àz Vr and let a be defined by
(i) if VKNAZ + 6 2 1, then a = =0; 0;
(ii) if VKNAZ + 5 1, then
K
á d= log
log VKNNz +
K [K+1
Theorem 4.1. (Global Optimizer). Assume that the feature dimension d is greater than or equal to the number of classes K, i.e., d 2 K, and the dataset balanced. Then any global optimizer of (W, H. of I0 satisfies the following condition:
There is a semi orthognal matrix PE R (i.e., PTP= Ix) such that:
(i) The classification weight matrix W is given by
(nAa 1/4
W
-
(11)
Aw
(it) The matrir of last-layer features H is given by
Aw
1/4
H =
Va' - -kx1]/K
(12)
ndH
(iit) The bins is vector, f.e. b= 0.
10

---

Published in Transactions on Machine Learning Research (05/2025)
The above theorem provides an explicit closed-form solution for the global minimizer (W. H. b). Notably, our findings indicate that an increase in 6 is associated with decrease in the norm of W and H. This observation aligns closely with the empirical results detailed in Section 3.3
Our observations in Section B reveal that label smoothing loss demonstrates accelerated convergence in terms of the NC metrics and training error. To better understand this phenomenon, we analyze the optimization landscape under the UFM. Note that condition number of the Hessian matrix, representing the ratio of the largest to the smallest cigenvalue, plays a crucial role in convergence rate analysis (Nocedal & Wright 1999 Trefethen & Bau 2022 In the vicinity of the local minimizer, a smaller condition number typically signifies faster convergence rate. We now present our main theoretical result, which, to the best of our knowledge, is the first result providing insight into the convergence rate to the optimal solution under UFM models.
At the global optimizer characterized by Theorem 4. the predicted probability score for class can be expressed as
Ps (p: pr) + Pn
(13)
where Pt and pn represent the predicted probability for the target and non-target classes, at the global optimal solution. Specifically, these probabilities are defined as:
Pt 6 /KK Pn =
(14)
which implics that Pt decreases as the smoothing parameter 8 increases. With this notation, we state the following theorem.
Theorem 4.2. (Optimization Tandscape). Assume the feature dimension d is greater than or equal to the number of classes K (d 2 K) and the dataset is balanced. In addition, assume the regularization parameters and the smoolhing parameter solisfy vKN + 5 < 1. Then, al the global minimmzerr (as provided in Equation 11 12), the condition number for the Hession of the empirical loss with respect to W when H is fized (also the Hession w.r.t. H when W is fized) takes the form
xV Kpp
(15)
Therefore, label smoothing with a larger smoothing parameter å results in smaller values of pt, which in turn leads to smaller condition nemberr
This theorem suggests that the training with smoothed labels results in a better-conditioned optimization landscape in the vicinity of the global minimizers, facilitating faster convergence behavior. It is noteworthy that while the theorem primarily analyzes the local landscape around the global minimizer, experimental results demonstrate that gradient descent with random initialization indeed converges faster to the global solution of neural collapse under label smoothing loss as observed in Section 3.1 and Section B.6
One possible intuition for why label smoothing improves the condition mumber lics in its regularization effect on the predicted probability distribution. Without label smoothing, cross-entropy loss drives the model toward overconfident predictions, assigning nearly all probability mass to the target class and pushing non-target probabilities close to zero. This creates sharp curvature in dimensions corresponding to the target class and flat regions in non-target dimensions, resulting in poorly conditioned optimization landscape with high curvature variations. Label smoothing mitigates this issue by encouraging more uniform probability distribution over non-target classes, preventing gradients from being dominated by the target class prediction. By smoothing the loss landscape and reducing curvature differences across dimensions, label smoothing stabilizes gradient updates and promotes more efficient convergence.
However, it is important to note that while Theorem 4.1 establishes connection between the smoothing parameter 8 and model convergence, it does not directly address how to optimally choose ō to improve model generalization. This remains open question for future research.
"This result follows from the fact that, at the global optimizer, the features of all samples in class & converge the class mean feature, resulting in identical probability scores as given in Equation 13 Furthermore, duc to the simplex equiangular tight frame (ETF) structure of both the class mean features and classifier weight vectors the global minimizer, we obtain uniform probahility distribution across the non-target classes.
11

---

Published in Transactions on Machine Learning Research (05/2025)
5 Discussion
We conducted comprehensive empirical comparison of cross-entropy loss and label smoothing during training. Our results show that models trained with label smoothing exhibit accelerated convergence, both in terms of training error and neural collapse (NC) metrics. Furthermore, they converge to a more pronounced level of NC1 and NC2. Along with the accelerated convergence, we found that label smoothing maintains a distinct balance belween NC1 and NC2. We posit that the emphasis on NC2 in label smoothing enhances the model's 's generalization performance. Additionally, we investigated the impact of label smoothing on model calibration from the perspective of neural collapse, revealing that label smoothing has regularization effect the classifier weight and feature norms, and excessively small NC1 values may adversely affect model calibration.
We performed a mathematical analysis of the convergence properties of the UFM models under CE and LS losses. We first derived closed-form solutions for the global minimizers under both loss functions. Then we conducted a second-order theoretical analysis of the optimization landscape around their respective global optimizers, which reveals that L. exhibits a better-condiitonee optimization landscape around the global minimum, which facilitates the faster convergence observed in our empirical study.
This paper provides significantly deeper understanding of why LS excels in terms of convergence, performance, and model calibration compared to CE loss. Additionally, it illustrates how the powerful framework of neural collapse and its associated mathematical models can be employed to gain more nuanced understanding of the 'why' of DNNs. We expect that these results will inspire future research into the interplay between neural collapse, convergence speed, and model generalizability.
Acknowledgment
This work is submitted in part by the NYU Abu Dhabi Center for Artificial Intelligence and Robotics, funded by Tamkeen under the Research Institute Award CG010.
This work partially supported by Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning at NYU Shanghai. Experimental computation was supported in part through the NYU IT High- Performance Computing resources and services.
References
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215 223. JMLR Workshop and Conference Proceedings, 2011.
George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 2(4))308-311 1989.
Hien Dang, Tho Tran, Stanley Osher, Hung Tran-The, Nhat Ho, and Tan Nguyen. Neural collapse in deep linear networks: from balanced to imbalanced data. ar preprint arXiv:2301000437, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icce, 2009.
Cong Fang, Hangfeng He, Qi Long, and Weijie Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118 (410:31119 2021.
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning, arXiv preprint STT::21122.1122. 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. International conference on machine learning, pp. 1321-1332 PMLR, 2017.
12

---

Published in Transactions on Machine Learning Research (05/2025)
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. Xrxiv preprinl #Xix:2106.02073, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 770-778, 2016.
Wanli Hong and Shuyang Ling. Neural collapse for unconstrained feature model under cross-entropy loss with imbalanced data. arXiv preprint arXiv:2309.0972 09722, 2023.
Kurt Hornik. Approximation capabilities of multilayer feodforward networks. Neural networks, 4(2):251-257, 1991.
Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for a large number of classes. Xrviv preprint orXiv:2310.05357 2023.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Grant Zhihui Zhu, and Qing Qu Principled and efficient. transfer learning of deep models via neural collapse. arXiv preprint arxi:221 12206, 2022.
Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing neural collapse in deep tnigtailedd Icarning. In International Conference on Artificial Intelligence and Statistics, pp- 11534 11544. PMLR, 2023.
Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59:224 241, 2022.
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. Advances in neural information processing systems, 30, 2017.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noisc? In International Conference on Machine Learning, pp. 6448-6458 PMLR, 2020.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. ar preprint arXiw:2011.11619 2020.
Rafacl Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advonces in neuraal information processing systems, 32, 2019.
Alexandru Niculescu-Minn Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pp. 625-632, 2005.
Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(44): 246663, 2020.
Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXto:2101.00012 2020.
Uri Shaham, Alexander Cloninger, and Ronald R Coifman. Provable approximation properties for deep neural networks. Applied and Computational Harmonic Analysis, 44(3):537-55) 557, 2018
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 2818-2826, 2016.
Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. Advances in Neural Information Processing Systems, 35:27225-2777 2022.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Tom Tirer and Joan Bruna. Extended unconstrinndd features model for exploring deep neural collapse. In International Conference on Machine Learning, pp-
PMLR, 2022.
Tom Tirer, Haoxiang Huang, and Jonathan Niles-Weed Perturbation analysis of neural collapse. Tn International Conference on Machine Learning, pp. 34301-34329. PMLR, 2023.
Lloyd N Trefethen and David Bau. Numerical linear algebra, volume 181. Siam, 2022.
Stephan Wojtowytsch et al. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint Xrx::2001 200,20, 2020.
Liang Xie, Yibo Yang, Deng Cai, and Xiaofei He. Neural collapse inspired attraction attrecioon-mpuuliio--baaeeddd loss for imbalanced learning. Neurocomputing, 527:60-T 70, 2023.
Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural collapse inspired feature-classffier aligument for few-shot class incremental learning. arXiv preprint srXh:2302.03004, 2023.
Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu. Neural collapse with normalized features: A geometric analysis over the riemannian manifold. Advances in neural information processing systems, 35: 11547 11560, 2022.
Longhui Yu, Tianyang Hu, Lanqing Hong, Zhen Liu, Adrian Weller, and Weiyang Liu. Continual learning by modeling intra-class variation. arXiv preprint vrXiv:2210053000 2022.
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference Machine Learning, pp. 27179-2172 272022 PMLR, 2022a.
Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. Advonces in Neural Information Processing Systems, 35:31697-31710, 2022b.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820 29834, 2021.
A Appendix
B
Additional Experiments
This section provides supplementary visualizations further support the conclusions of the main paper. In Section B.1
we present more detailed description of the neural collapse metrics, followed by an overview of the experimental setup in iention B.2 Finally, we include additional visualizations to reinforce key findings.
B.1 Metrics for Measuring NC
We introduced the metrics for measuring neural collapse in Section 3 For convenience, we restate them here, providing additional details about the subtle differences between our definitions and those used in prior works (Zhu al. 2021 Zhou al. 2022b). We denote the global mean and classwise means of the last-layer
features
K
hG
hxis
he =
Σ has (1 5 &)
N
where he is the global mean and hg represents the mean of class k. The metrics for neural collapse are then defined follows:
14

---

Published in Transactions on Machine Learning Research (05/2025)
(NC1) Within class variability collapse measures the relative magnitude of the within-class variance Ew Φ E E E ( h₂)² compared to the between-class covariance matrix EB = K Ek=1(h the hh)(hs -ha) of the last-layer features. It is formulated as:
NC1 K trace (2w2ip))
where Σ/B denotes the pscudo inverse of ΣB.
(NC2) Convergence simplex ETF quantifies the difference between the normalized classifier weight matrix and the centered class mean features in comparison a normalized simplex ETF, defined follows:
W H
1 VK-1
1 Ik K k
NC
(16)
W H
where H [1 hG. hx - -hal E Raxk represents centered class mean matrix.
This definition differs slightly from those ZZhu et al. 2021 Zhou et al. 2022b). However, it can be shown that the convergence of NC2 in Equation [16) zero indicates the simultaneous convergence of both W and H towards the simplex ETF structure. Specifically, when the bias b is all-zero vector constant vector, NC₂ defined in [66 approaching zero indicates that the average logit matrix, formulated Z = W H + b1 satisfies the condition Z = a k xxx1]] for some constant a. Remarkably, this matrix aligns with the simplex-encoding label (SEL) matrix introduced in (Thrampoulidis et al. 2022), to a scaling factor a.
From Proposition
C.4
WC have
min (Aw | W1 + Ah H 2 |Z|
where ZZ| represents the nuclear norm of and the equality holds only when H = HY and H Aww/nn W, indicating the self-duality of the class mean feature and the classification vector. Consequently, during the model training with an L2 penalty on both W and H (with the norm of H implicitly penalized by penalizing the model parameters θ), the convergence of NC2 as defined in (16) zero indicates the simultaneous convergence of both W and H towards the simplex ETF structure.
(NC3) Convergence to self-duality measures the distance between the classifier weight matrix W and the centered class-means H:
W
H
NC3 =
(17)
W F
H
F
B.2 Experiment Setup
In this section, we provide more details for reproducing the experiments presented in the paper. We emphasize that all datasets used CIFAR-10, CIFAR-100, STL-10, and Tiny ImageNet are publicly available for academic use. CIFAR-10 and CIFAR-100 are released under the MIT license. All experiments were conducted on a single RTX 3090 GPU with 24GB of memory.
For consistency with prior studies (Papyan al. 2020 Zhu al. 2021), we use ResNet-18 (He al. 2016) the backbone network for CIFAR-10 and ResNet-50 for CIFAR 100, STL-10, and Tiny ImageNet. To isolate behaviors associated with neural collapse while minimizing the influence of other factors, we apply standard preprocessing techniques without any data augmentation during training. The training period is extended to 300 epochs for Tiny ImageNet and 800 epochs for all other datasets to analyze model behavior during the terminal phase training. For all datasets, we use a batch size of 128, stochastic gradient descent with a momentum of 0.9, and an initial learning rate of 0.05, which follows a multi-stop decay schedulo-ddcreassng by a factor of 0.1 epochs 100 and 200 for Tiny ImageNet, and at epochs 150 and 350 for the other three datasets. We considered two different. scenarios for model regularization:
15

---

Published in Transactions on Machine Learning Research (05/2025)
Model with Weight Decay. For the default setting, we trained the models with weight decay regularization set at 5 x 10-1. It's worth noting that the norm of H is implicitly penalized by the weight decay on the model parameters θ
Simulated UFM Model. To simulate the unconstrained feature model, we turn off the weight decay and add an L2 penalty on the classifier parameters W and b and the last-layer feature H to the cross-entropy (CE) or label smoothing (LS) loss. This setting was only used in Section B.6 to validate the thcorctical results.
B.3 Comparing Training Loss Under Cross-Entrops and Label Smoothing
From Theorem 4.2 we establish that label smoothing yields better-condditoned loss landscape, leading to faster convergence. However, due to the different formulations of cross-entropy and label smoothing losses, directly comparing their raw training losses is not mcaningful. Instead, in the main paper, we use the training error rate as a proxy and demonstrate that models trained with label smoothing converge faster than those trained with standard cross-entropy loss.
In this section, we present the training loss under both loss functions, as shown the first row of Figure 5 Since their absolute values are not directly comparable, we also visualize the logarithm of the loss in the second of Figure 5 defined as log(loss min(loss) + +), where min(loss) represents the minimum training loss and 10-3 is small constant added to avoid taking the logarithm of zero. As shown in Figure 5 models trained with label smoothing exhibit faster convergence in terms of log-scaled training loss. For a clearer comparison, Table 2 epochs (or 100 epochs for Tiny ImageNet),, the training loss approaches its minimum value, and the log-scaled loss begins to fluctuate significantly. Therefore, we focus our analysis on the loss behavior before this point, where the trend is more stable and informative.
provides a quantitative comparison of the log-scaled loss. Notably, after 150
CIFAR10
CIFAR100
STL10
Tiny ImageNet
5
5-0
0
6- 0
Train Loss
8 0.05
6 0.05
6=0.05
0.05
1.0
3
0.5
Log-Scaled Train Loss
200
400
200
400
0
200
400
0
50
100
150
8
5-0
5-0 0
5 D
8 0.05
= 0.05
== 0.05
5 0.05
-2
-4-
-2
-4
200 Epoch
400
200
400
0
200
400
0
50
100
150
Epoch
Epoch
Epoch
Figure 5: Comparison of training loss (top row) and log-scaled training loss (bottom row) for models trained with cross-entropy loss (ō = 0) and label smoothing (ó 0.05).
B.4 Impact the Smoothing Hyperparameter
In Soction 3.1 and 3.2 default smoothing hyperparameter of 8 0.05 utilized. This scction explores how the choice of 8 impacts the model's convergence neural collapse and its generalizability Specifically, consider various values for 8 within the interval [0. 1).
16

---

Published in Transactions on Machine Learning Research (05/2025)
CIFAR-10
CIFAR-100
STI-10
Tiny ImageNet
Epoch
CE
LS
CE
LS
CE
LS
CE
LS
10
-1.84
-2.12
0.48
0.33
0.52
0.39
0.71
0.64
20
-2.25
-2.42
-0.46
-0.60
0.38
0.17
0.37
0.28
50
-2.46
-2.62
-0.94
-1.14
-1.45
-3.28
0.02
-0.15
100
-2.67
-2.81
-1.22
-1.31
-3.92
-4.02
-0.17
-0.23
Table 2: Log-scaled training loss at different epochs for models trained with Cross- Cross-Entroppy (CE) and Label Smoothing (LS).
metrics
norn vs
error vs
10-1
NCI
H-nom Winorm
0.25
Nom WIH
CIFARIO
NC2/NC
0.30
0.20
0.05
0.15
0.10
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
10a
NC2
8-
H-norm
W-norm
1.5
CIFAR180
NC2/NC
Nom WWH
NC1
10
11.0
0.6
10a
0.5
2
0.5
0.4
0.0
0.0
0.5
1.0
0.0
0.5
0.0
0.5
1.0
H-nom
10
W-norm
0.60
10a
Norm of MIH
0.55
STL10
MC2IC33
NC1
10a
2
0.50
10
0.4
0.45
0.40
10-2
0
0.35
1.0
0.5
1.0
Figure The effect of the smoothing hyperparameter 8. The columns from left right visualize (a) NC metrics vs. (b) Average norm of classification weight vectors and mean features 6, and (c) Testing performance (test error rate) vs. 8.
Figure B presents the results of the experiments. The experiments yield several important insights. Firstly, the smoothing hyperparameter significantly influences the model's convergence to neural collapse. The curves of NC1 and NC2 function of 8 (2nd column in Figure reveal U-shaped trend, with the levels of NC3 remaining relatively consistent across different values of 5. Exceptionally small and large values of 8 result in reduced collapse for both NC1 and NC2.
Secondly, the average norms of the classifier vectors and the class mean features decrease as increases. This observation aligns with intuition: given features and classifier vectors that form a simplex ETF structure, decrcasing their norms softens the output probabilitics. For LS loss, a higher smoothing parameter 8 corresponds to smoother target labels, and consequently, the features and classification vectors with lower norms achieve close-to-zero label smoothing loss. This observation was also supported by the closed-form
17

---

Published in Transactions on Machine Learning Research (05/2025)
expressions for W and H provided in Theorem 4.1 Thirdly, the trend of the test error also exhibits U-shape, underscoring the necessity of selecting an appropriate 6. When å approaches 1, the nearly uniform smoothed labels do not provide effective training signals to update the parameters. Additionally, the norm of both classification vectors and the last-layer features decrease towards zero, causing the classifier to be dominated by noise, leading to rapid deterioration in model performance. Finally, we observe a surprising robustness both neural collapse metrics and model performance for the choice of [0.02, 0.8). For classification tasks with a larger number of classes, such as CIFAR-100, opting for relatively large å is viable.
B.5 Weight and Feature Regularization Improve Model Calibration
Pre Temperature Scalling
Post Temperature Scaling
1.0
CE
0.8
LS
LS
0.6
0.2
0.2
0.0
0.2
0.5
1.0
0.0
0.6
1.0
Confidence
Conficence
80-
80-
CE
L5
40
2.0
20
20
0.2
1.0
Corfidence
Conficence
Figure 7: Reliability plots with 20 bins (lop row) and the percentage of samples in each bin (bottom row) for models trained with cross-entropy (CE) and label smoothing (LS) losses, both pre- (left column) and post- (right column) temperature scaling. The results are based on the CIFAR-10 dataset.
As mentioned in Section 3.3 label smoothing loss (with properly chosen 8) can improve model calibration. This improvement is due to the regularization effect of label smoothing the classification vector norm and last-layer feature norm, which leads to predictions with lower confidence. To support this, Figure 7 presents 20-bin reliability plot (Niculescu-Mill Mizil & Caruana, 2005) for models trained under CE and LS losses for CIFAR-10, with a default smoothing hyperparameter of 0.05. Additionally, we include a plot showing the percentage of samples in each confidence bin. Clearly, LS loss cxhibits less confidence in its predictions and belter model calibration compared to CE loss without temperature scaling. In the right. column of Figure 7 we employ temperature scaling to calibrate the model, with the hyperparameter T selected through cross-validation. Notably, after temperature scaling, models trained with CE loss achieve even better calibration than those trained with LS loss. This is because temperature scaling counteracts the vegularization effect of label smoothing on the classification vector norm and feature norm. Consequently, T.S loss can result in worse test expected calibration error (ECE) due to an excessive level of NC1.
B.6 Empirical Validation of Theoretical Results
In Theorem 4.1 we derive closed-form solutions for both the mcan feature matrix H and the classifier weight matrix W, both exhibiting a simplex ETF structure. To validate these theoretical findings, we train ResNet18 models on CIFAR10 dataset with varying smoothing parameters To ensure consistency with the UFM model, we adopted the simulated UFM model as introduced in Scction B.2 where weight decay is disabled, and L2 regularization is applied the last-layer features and classification parameters, ensuring exact regularization effects. We set the regularization parameters as follows: Aw 10-3 X 10-6, and
18

---

Published in Transactions on Machine Learning Research (05/2025)
A 10 Under these settings, compare the trained average feature norm and average classification vector norm ||/]| = Σ ////s//KK with their theoretical counterparts derived from Theorem 4.1 which can be represented
|h]
-k=1 1
hall/K
NAH
-
1/4
ndH 1/4
h (K 1)
V(K 1 /
Aw
Aw
Figure illustrates the differences between the trained predictions ||/||| and ||w|| and their theoretical solutions for models trained under LS loss with different The plot shows clear decreasing trend in the relative error between the empirical and theoretical values for both the feature and classification vector norms, with differences reducing to below 0.01 across all cases. These results demonstrate that the empirical findings align with the theoretical predictions, thereby verifying Theorem
0=0.05
5 0.1
Relative Vorm Difference
0.07
Relative error h-norm
w-norm
Difference
0.05
Relative error hinorm
Relative w-norm
Difference
0.04
Relative error h-romm
0.04
0.05
0.04
0.02
0.03
0.02
0.02
0.01
0.01
0.00
0.00
200
400
600
800
200
400
600
800
200
400
600
800
Figure 8: Relative error of classification vector norm results over epochs for ResNet-18 18 on CIFAR1O.
[ and feature norm
h compared to their theoretical
Train Error
NC1
NC2
NC3
0.6
10
6-0 0
Thain Emrr Emrr Rate
10a
0.8
0.4
CIFAR10
0.4
5- D.1
10
0.2
0.2
0.2
0.4
6 0.5
8 0.8
0.0
0.0
250
500
750
0
250
500
750
0
250
500
750
250
500
750
Epoch
Epech
Epoch
Train Error Rato
0.8
101
6=
5 0.05
CIFAR18B
0.6
100
0.6
6 0.2
0.4
0.4
6 0.8
0.2
10a
0.25
0.9
0.00
250
750
250
250
500
750
250
500
Epoch
Epoch
Epoch
Epoch
Figure 9: Comparison model convergence under weight decay regularization. From left to right, the figures display the models' training error rate, NC1, NC2, and NC3 with varying smoothing hyperparameters 6.
According to Theorem 4.2 models trained with label smoothing loss (satisfying v VNNAz + d < 1) converges faster than models trained with cross-entropy loss. In Figure we train ResNet18 and ResNet50 on CIFAR10 and CIFAR100, respectively, with default weight decay value of 5 10-4, Comparing models trained with different smoothing hyperparameters, we observe that for a wide range of values, models with larger 8 converge faster in terms of both training and NC metrics.
19

---

Published in Transactions on Machine Learning Research (05/2025)
Furthermore, investigate model convergence under the simulated UFM model, where weight decay is disabled, and 1.2 regularization is added. As illustrated in Figure 10 across a wide range of such that VKNAz + ò < 1, we observe faster convergence for larger values.
Train Error
NC1
NC2
NC3
10-1
Train Error Rate
10ah
10F
10-2
6=0
CIFAR18
10-2
10-3
0.2
6-0.1
10-1
10
10-2
0.1
0
5 0.0.4
10
0.0
200
800
2000
400
600
800
200
600
800
200
400
600
800
Epoch
Epoch
Epoch
Epoch
10a
10
1.00
10-2
Train Error Rate
10
107
6=0
CIFAR10B
0.75
0.00
10
10a
10-2
0.1.1
0.50
0.2
10:
10a
0.25
10-2
0.0.4
10
10-1
0.00
200
400
600
800
200
400
600
800
0
200
400
600
800
200
400
600
800
Epoch
Figure 10: Comparison of model convergence under simulated UFM model. From left to right, the figures display the models' training error rate, NC1, NC2, and NC3 with varying smoothing hyperparameters 6.
C Proofs
The first key observation of the phenomenon of neural collapse, that is (NC1), refers to a type of collapse that involves the convergence of the feature of samples from the same class a unique mean feature vector. The second key observation of neural collapse, namely (NC2), involves these unique mean feature vectors (after recentering their global mean) they form an equiangular tight frame (ETF), i.e., they share the same pairwise angles and length. Before providing the theoretical poof, first formally define the rank K canonical simplex ETF in the definition below.
C.1 Basics
Definition C.1. (K-simplex ETF) A K-simplex ETF is collection of points in R.d specified by the columns of the matrix
K
IK K k
M =
K
where IK E kxx is the identity matrix and 1x E kK is the ones vector, and P E Bdxk (d 2 K) is partial-orthogonal matrix such that PT pTp= Ix.
Note that the matrix M satisfies:
K K-1 Ix [
M MM
M
Next, prove series of leanmas that will prove crucial upon establishing our main theorems. Lemma C.2. (Young's 's Inequality) Let p.q be positive numbers satisfying we have
p + 1 =1 Then for any a, b E R,
la
|b1
|ab|
<
p
20

---

Published in Transactions on Machine Learning Research (05/2025)
where the equality holds if and only if [a|" 1 The case for p q is just the AM- GM inequality which is [ab] S 2 (a² + 12), where the equality holds if and only if |al [b]
Lemma C.3. For any fixed Z RRXN and > O, we have
min
(I H(P) = -
(18)
2/a
Here [|]. denotes the wuclear norm of Z:
| = Σ (Z trace(E), with Z UEV
min(K.N) denole the singular values of Z, and Z = UEV is the singular value decomposition (SVD) of Z.
where {ok)k=
Proof. Let Z UEV be the SVD of Z. From the fact that UU I, VV I, and trace (A A) =
||A|]; we have
trace (VaUTU>) U2.) + Va trace
( (HaIrT))
-
I trace(2))
2/a
(10 + ||a a //2
2/a
This implies that there exists some W = ///2g//2UT and H = a /= such that | I (I|WI + a:l H), which further indicates that
2ya
|2
2
(I + al H Z H 2/a
min
(19)
On the other hand, for any W T H Z, we have
Z trace (E) -trace(U T ZV) traceeU T HV)
$ 2
W
F + V 2
| 2/a
1
(IWW) + a H
where the first inequality is guaranteed by Young's inequality in Lemma C.2 and equality only holds when
WU = VaHV. The last equality follows because UUT and VV Therefore, we have
1 z WT и 2
|Z]| <
min
[|
H
(20)
Combining the results in
19 and 20 we complete the proof.
Proposition C.4. Consider motrics [HI.... Hal E R and = [Z,,.. Znl E K where H. Raxk and Z, E KxK with N = nK. Let H E. H; and Σ; Z. Then Z W H indicales = W T H. TJ Z is a symmetric matris, then we have
min Z-W и 2/a
(IW + + HHIH
min Z=WTH H 2
(| + on H 1Z
(21)
with the minimum is reached only H. H (for Vi 1, n) and W VonH.
21

---

Published in Transactions on Machine Learning Research (05/2025)
Proof. From Lemma C.2
have ||
= E 2 n H with the equality hold only when H, H,
for any i # Consequently, this yields the following result:
min
(II + a H(())
z=w 2/a (I + an H
min
Z=W H 2/a
Utilizing Lemma C.3 we further deduce:
min Z H 2/a (IWII + aml H |Z
This minimum is achieved only when W VomH, thus completing the proof.
D Proof of Theorem 4.1
D.1 Proof of Theorem 4.1
In this section, we present the proof of Theorem 4.1 in Section 4 which we restate as follows.
Theorem D.1. (Global Optimizer). Assume that the feature dimension is no less than the number of classes K, i.e., d 2 K, and the dataset is balanced. Then any global optimizer of (W, H,b) of
1 1 N WC H b1 Y + Aw |
λH 2
A |bbl
min C(W, H. W. H.b
+
H
(22)
satisfies the following properties:
The classification weight matrix W is given by
AHn Aw
1/4
W
-
(23)
The matria of last-layer feature H con be represented
AAw
1/4
= HY H
Va'P
1x 1/K)
(24)
AHn
The bias b is vector, i.e. b = 0.
Here, P E ga (d 2 K) is partial orthogonal matrix such that P p Ix and a satisfies:
(1) if VKNNZ + 2 1, then a = 0;
(i) iS VKNAZ 0 < 1, then
K
K +1) -k+1))
a = log
VKNNz +8
with Az VW.
Proof. The main idea of proving Theorem 4.1 is first to connect the problem (2) to its corresponding convex counterpart. This allows us to derive the precise form of the global minimizer for the convex optimization problem. Subsequently, we can further characterize the specific structures W and based on the acquired global minimizer.
Connection of 22 to Convex Problem. Let Z T H K represent the output logit matrix
with N = nk and Aw
An. Utilizing Lemma C.3
get:
min Aw W + Au H = AWAA
min w T- Vo
( | + a H
(25)
WTH H=
2 ZlI,,
22

---

Published in Transactions on Machine Learning Research (05/2025)
where |Z|| represent muclear norm of Z. Additionally from Lemma C.3 H HY and W VanH
the minimum is attained only when
An H. Awn
Let Az VxwAH then Equation
22 becomes:
1 1 N
Ab bbb
min L( b)
Nlon(Z + b Yd) + > +
(26)
2,b
2
which is a convex optimization problem.
Characterizing the Optimal Solution of 22) based on the Convex Program (66): We first derive
the exact form of the global minimizer for the convex optimization problem (26 Particularly, we first
establish that the predicted logit vectors within each class collapse to their sample means, i.e., Z ZY
(Lemma D.3 Subsequently, derive the closed-form solution of Z in Lemma D.5
Furthermore, from Lemma C.3. we establish that the minimum in 25 is attained only if H = HY and
W = Ay H. Since Z = W H, combining the above result with Lemma D.5 yields the global minimizer Awn
(W, H, H,b) b)
satisfying:
AHn 1/4 Aw
W
Va'P Iklk/K),
(27)
Aw AHn
1/4
H = HY, H
Va' (Ix 1x1
(28)
and
b= 0,
where P E d 2
2 K) is a partial orthogonal matrix such that P P= IK and a is defined per the specifications in Lemma D.5
Hence, conclude the proof.
D.2 Supporting Lemma
Lemma D.2.
(Optimality Condition) The first-order optimality condition of C(Z, b) in Equation
26 is
N-1 -P - Zl
N- -y - -PP 1N = dob,
(29)
where P is the prediction matrix defoned as
Pki = exp(21s + b)
P= (PahhsskIsIsN
KRN
(30)
(exp(z b), lx)
5 is the matrix of the smoothed soft largel defined as
Y = 1)- + 1Kll, Y= ... ,x1, E RKNN
(31)
K
and 0||
represents the subdifferential of the nuclear norm of Z.
Proof. Consider
1
A
C(2,b) = ICE 2 + +1 Yd) + Az Z11 +
b1|
N
2
in 26
Define
K
((Zbb) N ICE Z + +b yd)
(z + b.,
(32)
=
N
23

---

Published in Transactions on Machine Learning Research (05/2025)
where y (1 -ô)ex + R 1k is the smoothed target. The gradient of Φ is
N (pti - vk.
Pki - vk
Ozki
0b
N
R,
whose matrix form is:
06 (P Y-),
06
1
(- - 1N
8z N
0b
N
Hence the gradient (subgradient) of
0c oz
0C
-P Zz1
N (P- N
-YY")1N + dgb,
ab
where 0| is the subdifferential of the nuclear norm at Z. Thus, (Z,b) is global minimizer of C if its gradient (subgradient) is equal to zero, i.e.
N N - P) Z20 NY PP)N = Agb.
Lemma D.3. Assume that the number of classes K is less than the feature dimension d, L.e., K I d, and the dataset balanced. Then the prediction vectors, formulated as Zki fe( (1 S S K, S $ n) within each class collapse to their sample means 2k:
Zki 2k. 1sisn.
(33)
In other words, the prediction matris Z can be wrillen the Jollowing factorized form:
Z = ZY RRN
(34)
where
Z = [1 kk]:
Y Y= = =x1
Proof. The proof follows from the convexity of the loss function. Recall that the loss function 26 was
given by
C(Z,b) = 662 + Az 211. + A 2 bbb?
2
where ((Z,b) is the Cross-Entrope (CE) or Label Smoothing (LS) loss (depending on the value of the
smoothing parameter 5) as defined in 132 Recalling that {= belong to the same class and z n- D1 Zki their respective sample mean. From Jensen's 's incquality, have
o(Z.b) Kn
1
Σ
a
ICE (Zki bb vi)
K
ICE
Zii + 6), v
k=1
i=1
ICE(* + b. y
K
where the inequality becomes equality only when z k₁.
In the rest of the proof, we employ a permutation argument. Let Zi = [zu,... ... zxd] for 1 1 S n and Z = .z..... Zn]. Let P, represent a distinct permutation of n. Consider Zr, 21Ir,, where Ir, is a
permutation matrix rearranging Z so that the elements Z₁ are ordered according to r,. Then
12r
|| ||
24

---

Published in Transactions on Machine Learning Research (05/2025)
Since ||- is convex function, we deduce
1 Z + = 7
(I
- H
2
n..
where the incquality becomes equality only when Z, = Z(1 S n) or equivalently 2ki = 2x. As a result, it
holds that
C(Z,b)= (( (((,b) + A Z11 + A
2 1b1
1
+ + A
2 <0 + +b, Az /
|16|
K
2
1
A6 |1b1
N lloE (ZY + y Yd) A > /
The global optimality of ((Z.b) implics that Z ZY, i.c., Zki mΦ for s S n.
Lemma D.4. Assume Zui Ek, for 1 < k < K, sis n and (3x, 1x) =0, then it holds that
N-1 (1 (1- +
K
8
Jk
P) = >x 11 112 (Zz (ZZ Z + R
n
(1 I + Jk P) 1x = dob.
8
K
N
where JK E KxK is the ones matrir, 7 = [2,,". ... ,xk] E RR. KxK and R satisfies RZ = 0, ZT R = 0, and |R|| S -1/2
In particular, if Z is of rank K 1, then
N-1 (11 (1- 5) (1)6) +
5 Jk P)
dz
([ZZ))) ) z
K
Proof. Under the assumption, zki = Ebi 1 usis S n, and (Ex, 1x) = 0, have Z ZY and P = PY, where P is defined as
P pp: ... [5]
with P& as the probability vector w.r.t. the EΦ + b. Then the optimality condition in 29 reduces to
N-1 (1- [)- + 5 -J P) p)Y Y E \z0 ZlI..
(35)
Let Z - T be the SVD of Z, then we have:
t| Z| (UVT T + R: R 11, R R = 0},
where
[ZZ 1/2 Z.
UV = =VT=
Since = ZY and YYT_ nIk, we further get:
UV = n 11/2 (ZZ 1)/11/11// ZY.
Then
35 is equivalent to:
-1 (1) (1-6))x 8) K
n 11/2 ( (ZZ ) 1/2
Jk - P Y = >2
2Y R
25

---

Published in Transactions on Machine Learning Research (05/2025)
where R is in the form of such that
R= RY, R
RY (ZY) T nRZ =0,
z R R =0 0, || 1.
This further leads to
Ax n 11/ (ZZ [ZZT)|]
N-1 (1 (1-8)
8) + Jk - P
Z + R
K
where RZ 0, Z R = 0 and
R
In-
For b, it is easy to see that the optimality condition in equation 29
reduces to
n (X - -P) 1N = N (1- = Nb=N-1(Y
J (c-P) 1k
Now, if is of rank K - since 1 =0, the columns of R are parallel to 1g- Moreover, P is a positive left stochastic matrix with PT 1x 1k, and therefore Iς - P is also of rank K 1. The left stochasticity of P further deduces that (Ix- 1 In other words, 1s is in the left mill space of Ix P, which in turn implies R = 0. This leads to
Az - JK - P) Vn K N-1 (1-
(1
z z).
Lemma D.5. Assume that the number of classes K is less than or equal the feature dimension d, i.e., K < d, and the dataset is balanced. Then the global minimizer (Z, b) of (26) satisfies the following properties:
z ZY. Z = a² [IK Jk/K), b 0.
(36)
In Particular, we have
(1) if VKNAZ + 0 2 1, then a 0;
(M) if VKNAz + <1,, then
á = log VKNAz +
K
K xx+1) +1)
Proof. From Lemma D.3 we have Z ZY. Moreover, according to Lemma in Zhou et al, 2022b there
exists constant a such that the global optimum (26 satisfies:
Z (Ix 1/) Y, b=
Equivalently, we have Z = a (IK - 1k k/K). Further, we define P= [pr, ... .Pk] .pk] with py as the probability vector w.r.t. the logit. 2; b. Then,
P JK + (e" 11]I
(37)
K ltea
and
JK (e 1)IK
(1 - I
Jk
P= = (1 8) + Jk
K"
K"
K - -1
K 1 1 ca -6) (Tx [x
K
1 Jk
K
26

---

Published in Transactions on Machine Learning Research (05/2025)
Recall that the first-order optimality condition expressed in Lemma
D.2 indicates that:
1/2 Z R n 1/2 (Zz
(1-00 1 + J - = Ndz
(38)
K
sign(a) /
JK/K) + -IK-
= Niz
(39)
Suppose + 0, then Z has rank K and we have R = 0. Thus the above optimality condition implies:
K- I- ea -6) (Ix (Ix- JK/KKK/K) -iia- Jkk
K
which is equivalent to
K
6 NK))
K K- ea
For 0, the above equation has the following solution:
a = log VNKAz + - K -K+ if VNKA - 11
K
If VNNXz + + 1, then we select a 0, and noting that in that case P K-IJK, we have that the optimality condition in (38 implics that R satisfics
8 8-1 6-1 JK
(1 (1- 51-
NdzR.
We get
R
(1-8)KK +
JK.
Ndz
K
where
R S 11/2 meets the requirement of the optimality condition.
E Proof of Theorem 4.2
In this section of the appendices, we prove Theorem 4.2
Proof. To provide insight into the accelerated convergence of the model under label smoothing loss, we examine the Hossian matrix the empirical loss function, defined as follows:
1 H) FlCE(WT H,Y").
min o (WWH)
(40)
Under commonly used deep learning frameworks, the model parameters are updated iteratively, and is common and often practical to analyze the Hessian matrix with respect to H. W individually rather than considering the full Hessian. Particularly, we demonstrate that the Hessian of the empirical loss concerning W when H is fixed and the Hessian w.r.t. H when W fixed are positive semi-definite at the global optimizer under both cross-entropy and label smoothing losses. Furthermore, we establish that the condition numbers of these Hessian matrices are notably lower under the LS loss in comparison to the CE loss.
Hessian matrix with respect to Z. Let Z = W H E K represent the prediction logit matrix. In the proof of Lemma D.2 we obtained the first-order partial derivatives of the loss Φ:
06
Os bb
(pki
v
Σ Pkk v()
0zki
N
27

---

Published in Transactions on Machine Learning Research (05/2025)
where Pki is the prediction for the i-th sample that belongs to the k-th class and y is the soft label for class k with a smoothing parameter 5. These partial derivatives lead to the corresponding second-order partial derivatives of the loss Φ:
8
(diag(px) PhoPl))
0,V(k, # (1)
Φ 1
8 1 Ozg,0b
(diag(pss) Pk.Pk))
(diag(pas) PkoPki)
26
N
N
R
4.1 Consequently, it follows that
From Theorem
we have the global optimizer satisfies 2 = E and P PE, s k S K, S S n
8 azc.
8
(VI S i, S n). To simply the notation, we denote
dz
D2 = diag(px) PEPE.
(41)
8 Then we have
DR/N.
82
The closed-form solution for P in 37 yields the expression:
K 1 ead (lee 1)ex + 1k)
(42)
To simplify the notation, we denote
Pr = (K /
1+ Pn = -
(43)
where (Px) is the predicted probability for the target (non-target) class at the global optimal solution. Under this notation, we have pt + (K - 1)p, = 1 and Pr. = (p: - p.,);; + P,lk.
For any 1 K, P&P is a positive matrix and the associated Laplacian D₁ positive-semmddennite with all eigenvalues non negative. The smallest eigenvalue of the Laplacian matrix D3 σ1 = 0 with the corresponding eigenvector ei 1x/
Define
82 (Ke, 1x)/[[Ke 1xll.
then we have
Dau2 diag (pk) P&P 02
KKpin U2,
from which we get Kp.pa is an eigenvalue of Dj with the corresponding eigenvector U2.
Consider the outer-product matrix P&P its largest eigenvalue is || with the eigenvector ps Pa/l|px/|- Its null space is of dimension K - 1. Particularly, we can find set of the basis vectors for Null(paP)) as follows:
for... Oк-1 : span(o1, (P)) span(ek. pk)}.
(44)
which means the vectors on and ps span the same 2D space as ey and pk-
Note that
span(es, P&) = span(e,, v2).
where e1 Va the eigenvector for DE-
28

---

Published in Transactions on Machine Learning Research (05/2025)
Then show that 02.... O0-1 0K-1 eigenvectors of D₃ with the corresponding eigenvalue 03 Pn- Particularly, with of orthonormal to both Ps: and we have
D2o: = diag (pk) or P&P 01
= diag (pk)
Pn
for 1 defined in (44
To summarize the unique eigenvalues of the matrix are
01 0, 02 Pri 0g KpiPn
(45)
with Pt and Pn defined in 43
Hessian matrix with respect to H. The gradient of á with respect to has is
do
Ozki do
W do
Thg Thes azk: azyi
Further, we can easily get the corresponding Hessian:
S
8
1 N
W
W
WD&
(46)
ah ki
Oz
ohg
where D3 is the Laplacian matrix as defined in From Theorem
41
4.1 we have
(AHn))
1/4
W
(IK
Aw
For simplicity, we denote a =
then W = a - -K) Under this notation, we
have
(a
WD
- P(Ix - - JK/K)PT
(47)
N
N
)2
PD.PT.
(48)
N
Note that P E Rdxk (d >
K)
is a partial orthogonal matrix. Given the eigenvalues of D; in 45 the
eigenvalues for 47 are:
(a
(a 2
A2 =
Pui
A
N KP:Pn.
(49)
N
with the corresponding multiplicities m(A1) + d K, m(x2) K 2, m(>a) =
From
52 the hessian of with respect to h vec(H) is block diagonal matrix which can be expressed
1 blkdiag( W W WD,WT WD WD& 2h N
(50)
The unique eigenvalues of the Hessian matrix 0' BR are the same provided in 49h. Given that the Hessian
matrix contains a zero cigenvalue, our analysis centers on its condition number within the non-zero eigenvalue space. This is caleulated as follows:
(V)0) = >a/A2 = Kpt
(51)
Considering the formula for pt given in an increase in 8 leads decrease in Pt: subsequently resulting in smaller condition number.
29

---

Published in Transactions on Machine Learning Research (05/2025)
Hessian matrix with respect to W. The gradient of with respect to wi(l = 1, 1, . is
d6
azks 0Φ
Owl
Sw Ozki
Further, we get the corresponding Hessian:
8
Ozks 0 Ozki
Owoiwo
Ow Ow dz
8 dz
e
ethki
k
D& Dult, 1)hgsti
Σ Dull, n
where (U, is (1.1)-th element in Dy and h2 is the k-th column vector in H defined in 24 From the defination of D in (41) we have
86 Owrdwy
n H diag(p:) (p:)) H =fl=''
n H diaag (P.)gg())) H i1
where pi is the average prediction vector defined in the others equal Pn.
42
Particularly, the 1-th element of pt equals pt and
From 24h we have
Aw 1/4 XHn
H
VI
Hence, the Hessian of & w.r.l. the w (W) can be written as:
006
Aw
1/2
a" DDDDDDDDD
(52)
Ow
AHn
where
Dr blkdiag P
P) Ku
(53)
Dn blkdiag( II,
II) E xx²
II
Ix
1x1
(54)
K
and
A1
Ai
S
:
A1
Ak]
(55)
Ax
Ax
with A, diag (p:).
Since P is a partial orthogonal matrix, the condition number of 52 is the same as the condition number of the following matrix:
B= DDSDn
(56)
Subsequently, we proceed to derive the eigenvalues for the above matrix B.
First, considering an eigenvector of B, let A be K matrix such that vec(A). Considering any A
satisfying An or A 0, for the corresponding vec(A) get. D 0 and v Dn 0, wich
30

---

Published in Transactions on Machine Learning Research (05/2025)
further yields Bu 0. Since rank(II) K 1, it follows that 2 0 is eigenvalue of B with multiplicity 2K-11 1.
On the other hand, consider any A that satisfies the conditions:
diag( A) = O, Alk = =
(57)
It is noteworthy that from A1x = 1 0, the corresponding vector satisfies Dnv v. If A further satisfies diag(A) = 0 then Se Pn Consequently, we deduce that
Bv DnSDno DnSv = DuP, Pn
We can conclude that P. is an eigenvalue of B with multiplicity K2 - K - (2K - 1) K² 3K +1. Next, let us consider the matrix A = Idiag(u) where u is a vector satisfying TT 1 = The vectorized form of can denoted as
diag u II diag u ll
= 2 = vec(A)
diag II
where II represents the k-th column of the matrix II. Due to the properties II II and II2 = II, follows that Drv v. Additionally, given u T 1 = 0, we obtain Sv = A2 with A2 = (1 - Pt + px) p- (K 1)pe
K
Consequently, we conclude that A2 (1 - Pt + Pn) Pnt(K-1)pp an cigenvalue of with multiplicity equal to the degrees of freedom of u, which is K K-1 1. K
Lastly, considering vec(II), observe that Dnv Further, have So Kp.P, Hence, we obtain
Bu = DnSDnu= DnSv = KPiPRU.
Therefore, v = vIc an cigenvector of B with cigenvalue A3 = Kp:Pn-
In summary, we identify the distinct eigenvalues of the matrix B as defined in 56 follows:
Ao 0, \ Pn: A = (1 Pt + Pa) Pr (K 1)pt
A KpaPt
(58)
with the corresponding multiplicities 2K -1 K2 3K + 1, - 1, and 1, respectively.
Since the matrix B has zero eigenvalues, we consider its condition number within the non-zero eigenvalue subspace, which given by Aa/A1 Kpp. Consequently, we straightforwardly determine the condition &g number of the Hessian a as defined in 522 is
0) Kpr.
(59)
From the formula of Pe given in 43 evident that increasing 8 leads to a decrease of pt, and consoquently reduction in the condition number.
W,HW By combining 51 and 59 we demonstrate that the Hessian matrices, Vi Ww H)) and ( H),, are positive semi-definite at the global optimizer. Morcover, the condition numbers of these Hessian matrices are notably lower under the label smoothing loss (with < 8 < NKN compared to the cross-entropy loss (with 8 = 0). This observation suggests that the optimization landscape of the empirical loss function is better conditioned around its global minimizer under label smoothing loss, thus completing our proof.
31


Reviews:
[{"review_id": "AHUgq1tNQc", "reviewer": "Reviewer_LSeV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The neural collapse perspective is useful to the community and provides a new perspective on how to analyze label smoothing. \n- The theoretical finding that label smoothing leads to a lower condition number in the solution is also interesting and potentially useful for developing the theory of optimization with DNNs.\n- The paper is very well written and easy to follow. It also provides both empirical and theoretical support for the claims being made.\n\n**Weaknesses**\n- In Figure 1, the paper claims that LS exhibits faster convergence for both training and test errors, but these differences seem very minor. Is there a way to see the difference more clearly? Maybe with a table of the error rate at different epochs?\n- The analysis at the end of Section 3.2 could be made more clear. My understanding is that the second equation is (almost) showing that\n$$\n\\frac{\\delta}{K} \\sum_{l \\neq k} \\log(p_l)\n$$\nachieves its maximum when all $p_1 = p_{l'}$ for all $l,l' \\neq k$. But the two terms aren't exactly the same so some more discussion or clarity on this would be appreciated.\n- While the discussion on model calibration is nice, I feel it distracts a bit from the main message of the paper. To me it seems that the central message is about understanding how LS accelerates learning an NC model. I'm not suggesting the authors must remove this, but it was a bit distracting to me.\n- Theorem 4.2 is unsatisfying. While it does say that LS will result in a better conditioned optimization problem is gives no insight into how much better. Some discussion on the improvement that you get with LS would improve the result.\n-  The terminology \"LS loss\" is confusing. The paper is comparing label smoothing (which is a preprocessing of the data labels) with cross-entropy (the loss function used to train). Based on section 2.2 I believe it would make more sense to claim that this works compares **one-hot encoded labels with smoothed labels and applying cross-entropy loss in both cases**. Label smoothing is really about you're preprocessing the data not about changing the loss.\n\n**Minor Comments/Typos**\n- In Fig. 3 \"optical T\" should be \"optimal T\"\n- In the introduction, when describing the \"LS loss\" the paper state that \"label smoothing introduces a soft target label by blending the hard target label with a uniform distribution...\". The word \"blending\" is quite vague and not necessary. It would be more clear to just say \"LS preprocesses the label for each sample by taking a convex combination of the one-hot label and a uniform distribution.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies label smoothing, a pre-processing technique in training deep neural network (DNN) classifiers. It has been empirically shown that this procedure often leads to DNN models which generalize better than traditional one-hot encoding with cross-entropy loss. This paper focuses specifically on how label smoothing influences neural collapse and the implications this has on our theoretical understanding of DNNs. Empirically, it is shown that DNNs trained on smoothed labels exhibit neural collapse (NC1, NC2 & NC3) faster than DNNs trained on one-hot encoded labels. The paper also presents evidence that for the same level of NC1, models trained on smoothed labels exhibit intensified NC2. A short analysis on how the loss is effected by smoothed labels provides insight for why this might be the case. The paper also gives insight into how enhanced NC1 effects model calibration. Theoretically, under the unconstrained feature model, the paper presents closed form expressions for the global minimizers under both label smoothing and cross-entropy loss. Finally, it proves that label smoothing results in a lower condition number for the output weight matrix which could explain the faster rate of convergence.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "These were also brought up under the weaknesses section, but summarizing here.\n\n- Clearer evidence that LS exhibits faster convergence in both training and test errors. Possibly with a table of the error rate at different epochs instead of a plot.\n- Clarify the analysis at the end of Section 3.2\n- Quantify the improvement in the condition number when using LS over one-hot encoding in Theorem 4.2.\n- Terminology of \"LS loss\".", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K8GTEovBdV", "reviewer": "Reviewer_85QS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and organized. Figures are well-formatted and professional.\n- The finding that label smoothing has greater NC1 and NC2 is interesting. In particular, the explanation of more pronounced NC2 with label smoothing is intuitive and clear.\n- Several practical implications can potentially be drawn from this work, such as tuning the smooth parameter to optimize model calibration and landscape. \n\nWeaknesses:\n- The authors argue that label smoothing can achieve better generalization due to improved NC2. While I find the connection between label smoothing and NC2 convincing, the generalization argument is not, particularly since most of the figures suggest there is little to no difference in error between label smoothing and cross entropy. The only result I could find that suggests otherwise was tucked away in the supplement in Figure 5 for STL10.\n- Similarly, the argument that label smoothing leads to faster convergence also seems unsubstantiated. The provided figures show that the NC metrics converge faster for label smoothing, but this is not the same as the loss/error decreasing faster which is what is generally referred to for convergence. \n- The theoretical result regarding the global solution is incremental, seemingly only providing an explicit value for the norm of weights/features compared to previous work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work provides a comprehensive study of the label smoothing loss from a neural collapse perspective. Empirically, the authors observe that compared to the cross-entropy loss, models trained with the label smoothing loss exhibit greater levels of NC1 (feature collapse) and NC2 (ETF structure), the latter explained by the fact that label smoothing equalizes logits not belonging to the target class. (Observations about ECE). Finally, the authors provide theoretical results that (1) characterize the global solution of the label smoothing loss, and (2) demonstrate that the Hessian condition number around the global solution is lower for the label smoothing loss compared to the cross-entropy loss.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- As mentioned in the weaknesses, some (not all) of the claims are not extremely well-supported by the experimental results, which is why **I currently indicate below that the claims and evidence are not supported. However, if the authors could clarify or provide further results, I will change this.**\n- The connection between label smoothing and calibration can be made more clear. I do understand how label smoothing can play the role of choosing the appropriate temperature by scaling the weight/feature norms. However, I’m confused by why the post temperature calibration is worse with larger label smoothing parameter, wouldn’t the temperature be selected to counteract the increase in delta? If the authors could elaborate on this it would be easier to appreciate.\n- I think Theorem 4.2 can be improved by explicitly stating the decrease in the condition number for a given delta. As presented, one could choose delta to be very close to 0, which wouldn’t be much different from usual cross-entropy, but the theorem would still say this decreases the condition number and does not differentiate from any other delta in the given range. \n- Certain figures (such as Figure 3, 4) have overlap in the legend and axes (e.g., Optimal T and Best T) which is confusing. If the quantity is specified in the axes, it would be better to not repeat in legend (or vice versa).\n- Citations should be fixed to use parentheses form appropriately. \n- Typo in caption of Figure 3, “optical”.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yMkD4cp6Qa", "reviewer": "Reviewer_owR4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper provides a fresh perspective on label smoothing by analyzing it through the framework of NC, which adds theoretical depth to the discussion.\n\n- The paper presents empirical results demonstrating that models trained with label smoothing converge faster and achieve better generalization compared to those trained with cross-entropy loss. It also derives closed-form solutions for global minimizers under both loss functions\n\nWeaknesses:\n\n- The paper presents a theoretical analysis demonstrating that label smoothing leads to a lower conditioning number. However, it lacks an intuitive explanation for why this phenomenon occurs. Providing such insights would greatly enhance the understanding\n\n- The impact of the smoothing hyperparameter $\\delta$ on performance and calibration is mentioned, but further investigation into optimal $\\delta$ values and their implications could provide clearer guidance for practitioners.\n\n- The experiments are primarily conducted on relatively simple/specific datasets such as CIFAR-10, and STL-10. Including a broader range of datasets could enhance the generalizability of the findings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper highlights the advantages of Label Smoothing loss over Cross-Entropy loss in deep neural networks. It demonstrates that LS loss accelerates convergence and improves generalization through enhanced Neural Collapse metrics. Theoretical insights show LS loss leads to a better optimization landscape. Additionally, it addresses LS loss's effects on model calibration, balancing improvements with potential overconfidence.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please check the weakness part above", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides valuable insights into how label smoothing influences Neural Collapse, supported by rigorous empirical evidence and robust theoretical analysis. All reviewers agree that the paper is worth publication at TMLR.

Please incorporate all the comments from the reviewers into the paper for the final camera-ready.
Decision: Accept (poster)