MaxSup: Overcoming Representation Collapse in Label Smoothing
Authors: Yuxuan Zhou
Heng Li
Zhi-Qi Cheng
Xudong Yan
Yifei Dong
Mario Fritz
Margret Keuper
Venue: NeurIPS 2025 oral ()
Keywords: Label Smoothing; Representation Collapse; Loss Decomposition; Error Amplification; Regularization
Primary area: deep_learning

Abstract: Label Smoothing (LS) is widely adopted to reduce overconfidence in neural network predictions and improve generalization. Despite these benefits, recent studies reveal two critical issues with LS. First, LS induces overconfidence in misclassified samples. Second, it compacts feature representations into overly tight clusters, diluting intra-class diversity, although the precise cause of this phenomenon remained elusive. In this paper, we analytically decompose the LS-induced loss, exposing two key terms: (i) a regularization term that dampens overconfidence only when the prediction is correct, and (ii) an error-amplification term that arises under misclassifications. This latter term compels the network to reinforce incorrect predictions with undue certainty, exacerbating representation collapse. To address these shortcomings, we propose Max Suppression (MaxSup), which applies uniform regularization to both correct and incorrect predictions by penalizing the top-1 logit rather than the ground-truth logit. Through extensive feature-space analyses, we show that MaxSup restores intra-class variation and sharpens inter-class boundaries. Experiments on large-scale image classification and multiple downstream tasks confirm that MaxSup is a more robust alternative to LS.

MaxSup: Overcoming Representation Collapse in Label Smoothing
Yuxuan 2..
Heng Li Zhi-Qi Cheng
Xudong Yan Margret Keuper
Yifei Dong"
Mario Fritz
University of Mannheim University of Washington
Meta AI 8 Max Planck Institute for Informaties
P CISPA Helmholtz Center for Information Security
Abstract
Label Smoothing (LS) is widely adopted to reduce overconfidence in neural net- work predictions and improve generalization. Despite these benefits, recent studies reveal two critical issues with LS. First, LS induces overconfidence in misclassified samples. Second, compacts feature representations into overly tight clusters, diluting intra-class diversity, although the precise cause of this phenomenon mained clusive. In this paper, we analytically decompose the LS-induced loss, exposing two key terms: (i) regularization term that dampens overconfidence only when the prediction is correct, and (ii) an error-amplification term that arises under misclassifications. This latter term compels the network to reinforce incorrect predictions with undue certainty, exacerbating representation collapse. To address these shortcomings. form regularization to both correct and incorrect predictions by penalizing the top-1 logit rather than the ground-truth logit. Through extensive feature-space analyses, we show that MaxSup restores intra-class variation and sharpens inter-class bound- aries. Experiments on large-scale image classification and multiple downstream tasks confirm that MaxSup is a more robust alternative to LSS
propose Max Suppression (MaxSup), which applies uni-
Introduction
typically relies on one-hot labels, which implicitly treat different classes as mutually orthogonal. In practice, however, classes often share low-level features [31 or exhibit high-level semantic similarities 3 24 42 rendering the one-hot assumption overly restrictive.
Multi-class classification 19 26
Such mismatch can yield over-confident classifiers and ultimately degrade generalization 9
To moderate overconfidence, Szegedy et al. 334 introduced Label Smoothing (LS). which combines
uniform distribution with the hard ground-truth label, thereby reducing the model's 's certainty
in the primary class. LS has since become prevalent in image recognition [10
22 36 47) and
neural machine translation [ M often boosting accuracy and calibration 23
Yet subsequent work hindering intra-class
indicates that LS can overly compress features into tight clusters
[15
28
41
variability and transferability (S). In parallel, Zhu et (48 found that LS paradoxically fosters
overconfidence in misclassified samples, though the precise mechanism behind this remains uncertain.
In this paper, we reveal that LS's training objective inherently contains an error amplification term. This term pushes the network to reinforce incorrect predictions with exaggerated certainty, yielding
"quual contribution.
IIntershhip at University of Washington.
i "oonesponding author. Assistant Professor, UW Tacoma School of Engineering and Technology.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Space
Crose Entropy
Label Smoothing
NaxSup
Smoothing
MaxSup
Diocline
intra-daass vartation Fresevvelstrr--dee veriadion
Alrisan shomedeon
Distracted sttention
Precie atterdon
Figure 1: Comparison of Label Smoothing (LS) and MaxSup. Left: MaxSup mitigates the intra-class compression induced by LS while preserving class separability. Right: Grad-CAM visualizations show that MaxSup more effectively highlights laas-discrmminaive regions than LS.
highly confident misclassifications and further compressing feature clusters (Section 3.1 Table Building on Zhu et al. [48], we characterize "overconfidence" in terms of the model's top-1 1 prediction, rather than through conventional calibration metrics. Through our analysis, we further show that pumishing the ground-trun logit during misclassification reduces intra-class variation (Table 2 phenomenon corroborated by Grad-CAl CAM visualizations (Figure 2
To overcome these shortcomings, we introduce Max Suppression (MaxSup), a method that retains the beneficial regularization effect of LS while eliminating its error amplification. Rather than penalizing the ground-truth logit, MaxSup focuses on the model's top-1 logit, ensuring a consistent regularization signal regardless of whether the current prediction correct or misclassified. By preserving the ground-truth logit in misclassifications MaxSup sustains richer intra-class variability and sharpens inter-class boundaries. As visualized in Figure this approach mitigates the feature collapse and attention drift often induced by LS, ultimately leading to more robust representations. Through comprehensive experiments in both image classification (Section 4.2) and semantic segmentation (Section 4.3 we show that MaxSup not only alleviates severe intra-class collapse but also consistently boosts top-1 accuracy and robustly enhances downstream transfer performance (Section
Our contributions are summarized as follows:
We perform logit-level analysis of Label Smoothing, revealing how the error amplification term inflates misclassification confidence and compresses features.
We propose Max Suppression (MaxSup). removing detrimental error amplification while preserving LS's beneficial regularization. As shown in extensive ablations, MaxSup allevi- ates intra-class collapse and yields consistent accuracy gains.
We demonstrate superior performance across tasks and architectures, including ResNet, MobileNerV2, and DeiT-S, where MaxSup significantly boosts accuracy on ImageNet and consistently delivers stronger representations for downstream tasks such semantic segmentation and robust transfer learning.
2
Related Work
We first outline mainstream regularization techniques in deep learning, then survey recent advances in Label Smoothing (LS), and finally clarify how our MaxSup diverges from prior variants.
2.1 Regularization
Regularization techniques aim to improve the generalization of deep neural networks by constraining model complexity. Classical methods like & (18) and & [49] impose direct penalties on large or
sparse weights, while Dropout [32) randomly deactivates neurons to discourage over-adaptation. In
the realm of loss-based strategies, Label Smoothing (LS) (34) redistributes a fraction of the label probability mass away from the ground-truth class, thereby improving accuracy and calibration [3] Variants such as Online Label Smoothing (OLS) 45 and Zipf Label Smoothing (Zipf-LS) [1 refine L.S by dynamically adjusting the smoothed labels based on model's evolving predictions. However, they do not fully address the fundamental issue that emerges when the ground-trus -ruth logit is not the highest one (see Section 3.1 Table []. Other loss-based regularizers focus on alternative aspects of the predictive distribution. Confidence Penalty 255 penalizes the model's confidence directly, while Logit Penalty
4 minimizes the global norm logits, a technique reported to enhance class separation [15]. Despite these benefits, Logit Penalty inadvertently shrink intra-class variation,

---

thereby hampering transfer learning (see Section 4.1p. Unlike the aforementioned methods, MaxSup enforces regularization by penalizing only the Togit 2max rather than the ground-trume truth logit agt- In L.S-based approaches, suppressing Eye for misclassified samples can worsen errors, whereas MaxSup applies uniform penalty regardless of whether the model's prediction correct. Consequently, MaxSup avoids the enror amplification effect, retains richer intra-class diversity (see Table 2 and achieves robust transfer performance across diverse datasets and model families (see Table
2.2
Studies on Label Smoothing
Label Smoothing has also been studied extensively under knowledge distillation. For instance, Yuan al.
observed that LS can approximate the effect of teacher-student framework, while Shen investigated its role in such pipelines more systematically. Additionally. Chandrasegaran al. demonstrated that a low-temperature. LS-trained teacher can notably improve distillation outcomes. Concurrently, Kornblith et al. [TS] showed that LS tightens intra-class clusters in the feature space, diminishing transfer performance. From Neural Collapse perspective [46] g LS nudges the model toward rigid feature clusters, as evidenced by the reduced feature variability measured in Xu and Liu 41 Our goal to overcome LS's inherent error amplification effect. Rather than adjusting how the smoothed label distribution is constructed (as in OLS or Zipf-LS), MaxSup directly penalizes the highest logit Zmax- This design ensures consistent regularization even if agt is not the top logit, thereby avoiding the degradation in performance typical of misclassified samples under L.S (see Section B.2). Moreover, MaxSup integrates seamlessly into standard training pipelines, introducing negligible computational overhead beyond substituting the LS term.
43
et al.
30
3 Max Suppression Regularization (MaxSup)
We first partition the training objective into two components: the standard Cross-Entropy (CE) loss and regularization term introduced by Label Smoothing (LS). By expressing LS in terms of logits (Theorem (3.3,, we isolate two key factors: regularization term that controls overconfidence and an error amplification term that enlarges the gap between the ground-truth logit Egt and any higher logits (Theorem 3.4 Equation (5), ultimately degrading performance. To address these issues, we propose Max Suppression Regularization (MaxSup), which applies the penalty to the largest logit Emax rather than Egt (Equation 88. Section 3.2). This shift delivers consistent regularization for both correct and incorrect predictions, preserves intra-class variation, and bolsters inter-class separability. Consequently, MaxSup mitigates the representation collapse found in LS, attains superior ImageNet- 1K accuracy (Table D and improves transferability (Table 2 Table 3 The following sections elaborate on MaxSup's s formulation and integration into the training pipeline.
3.1 Revisiting Label Smoothing
Label Smoothing (LS) is regularization technique designed to reduce overconfidence by softening the target distribution. Rather than assigning probability to the ground-truth class and to all others, LS redistributes fraction a of the probability uniformly across all classes:
Definition 3.1. For standard classification task with classes, Label Smoothing (LS) converts a one-hot label y E R into softened target label E R
5k a y
K'
where y4 = 1 (k=gtt) denotes the ground-truth class. The smoothing factor E [0, 1] reduces the confidence assigned the ground-truth class and distributes &8 to other classes uniformly, thereby mitigating overfitting, enhancing robustness, and promoting better generalization.
To clarify the effect of LS on model training, first decompose the Cross-Entroph (CE) loss into standard CE term and an additional LS-induced regularization term:
Lemma 3.2. Decomposition of Cross-Entropy Loss with Soft Labels.
H( HHy,9) + Las,
(2)
where
Lis = a H 9) - H())a)).
(3)
3

---

Where, q is the predicted probability vector, H.) denotes the Cross Cross-Entropy Entropy, and 1 is the uniform distribution introduced by LS. This shows that LS adds a regularization term, Lis. which smooths the oulput distribution and helps to reduce overfitting. (See Section for a formal proof.)
K
next explicitly express LLs the logit level for further analysis. Theorem 3.3. Logit- Formulation of Label Smoothing Loss.
Building Theorem 3.2
K
TAs a (((zt
Σ 2k
K
where zgt. is the logit corresponding to the ground-truth class, and t L zu is the average logit. Thus, LS penalizes the gap between 2gt and the average logit, encouraging more balanced output distribution and reducing overconfidence. (See Section for the proof.)
The behavior of Lys differs depending on whether Eyt is already the maximum logit. Specifically, depending on whether the prediction is correct (Zyt 2max) incorrect (Zgt Zmax), we can decompose Lis into two parts:
Corollary 3.4. Decomposition of Label Smoothing Loss.
LIS
Egg
+
Σ
Agt
Zn
(S)
K
K
Zm Cage
Zn Zn>22gt
Regularization
Erior amplification
where M and N are the numbers of logits below and above Zgt. respectively (M + N K 1). Note that the error amplification term vanishes when Egt = Amax.
1. Regularization: Penalizes the gap between Egt and any smaller logits, thereby moderating overconfidence.
2. Error amplification: Penalizes the gap between Egt and larger logits, inudvertently increas- ing overconfidence incorrect predictions.
Although LS aims to combat overfitting by reducing prediction confidence, its error amplification component can be detrimental for misclassified samples, as it widens the gap between the ground-trut truth logit Zgt and the incorrect top logit. Concretely:
1. Correct Predictions (Zgt = m ): The error amplification term is zero, and the regular- ization term effectively reduces overconfidence by shrinking the gap between 2gt and any smaller logits.
2. Incorrect Predictions (Zzt + 3max)): LS introduces two potential issues:
Error amplification: Increases the gap between 2gt and larger logits, reinforcing overconfidence in incorrect predictions.
Inconsistent Regularization: The regularization term lowers Zgs yet does not penalize 2mars which further impairs learning.
These issues with LS on misclassified samples have also been systematically observed in prior work [39]. By precisely disentangling these two components (regularization vs. eтror amplification), we can design a more targeted and effective solution.
Ablation Study on LS Components. To isolate the effects of each component in LS, we carefully perform detailed and systematic ablation study ImageNet- 1 K using DeiT-Small model (36 without Mixup or CutMix. As indicated in Table the performance gains from L. stem solely from the regularization term, whereas the eиror amplification term degrades accuracy. contrast, our MaxSup omits the error amplification component and leverages only the beneficial regularization, thereby boosting accuracy beyond that of standard LS. Specifically, Table T that LS's overall improvement can be attributed exclusively to its regularization contribution; the error amplification term consistently reduces accuracy (e.g., to 73.63% or 73.69%) Disabling only the error amplification while retaining the regularization yields a slight but measurable improvement (75.98% vs. 75.91 75.91%). By fully removing error amplification and faithfully preserving the helpful aspects LS, our MaxSup achieves 76.1 12% accuracy, clearly and consistently outperforming L.S This result underscores that MaxSup directly tackles L.Ss fundamental shortcoming by maintaining a consistent and meaningful regularization signal- even when the top-1 prediction is incorreet.

---

Table 1: Ablation on LS components using DeiT-Small on ImageNet-Ik IK (without CutMix Mixup). "Reg- ularization" denotes penalizing logits smaller than Egt; "error amplification" penalizes logits larger than Egt- MaxSup removes error amplification while retaining regularization.
Method
Formulation
Accuracy
Baseline
74.21
Em <aye (gg 2m
Label Smoothing
75.91
Segt (3 kro)
Regularization
Σ 3m 5. (2gt
75.98
error amplification
25yt (Zgt
73.63
+ error amplification
a
2zgt Zmece
73.69
MaxSup
a Zmex
R
1 5
2k
76.12
3.2 Max Suppression Regularization
Building on our analysis in Section 3.1 we find that Label Smoothing (LS) not only impacts correctly classified samples but also influences misclassifications in unintended and harmful ways. Specifically, LS suffers from two main limitations: inconsistent regularization and error amplification. As illustrated in Table [1 L.S penalizes the ground-truth logit Egt even in misclassified examples, needlessly widening the gap between Zgt and the erroneous top-1 logit. To resolve these critical shortcomings, we propose Max Suppression Regularization (MaxSup), which explicitly penalizes the largest logit Zmax rather than Egt. This key design choice ensures uniform regularization across both correct and misclassified samples, effectively eliminating the ertor-amplification issue in LS (Table 1 and preserving the ground-truth logit's 's integrity for more stable, robust learning.
Definition 3.5. Max Suppression Regularization
We define the Cross-Entropy loss with MaxSup as follows:
(s, 9)
q)
IMasSup Max Suppression Loss
(6)
CE with Soft Lahels
CE with Hard Labels
where
MaaeSup
= a H (H( - HHY
and
1/c 1 kmarg max((a) }
so that v'. identifies the model's top-1 prediction and va otherwise. Here, H H q) encourages uniform output distribution to mitigate overeonfidence, while H q) penalizes the current top-1 logit. By shifting the penalty from Zgt (the ground -rrtt logit) to Amar (the highest logit), MaxSup avoids unduly suppressing Zgt when the model misclassifies, thus overcoming Label Smoothing's ss principal shorteoming.
Logit-Level Formulation of MaxSup. Building on the logit-level perspective introduced for LS in Section 3.1
we can express LMasSup as:
a
IMMsSup
Emmxx
K
2k
(8)
where Zmax maxx[z+] is the largest (top-1) logit, and 1 Σk=1 23 is the mean logit. Unlike LS, which penalizes the ground-truth logit Zyt and may worsen errors in misclassified samples, MaxSup shifts the highest logit uniformly. thus providing consistent regularization for both correct and incorrect predictions. As shown in Table this approach eliminates LS's aroo--amlliiiiation issue while preserving the intended overconfidence suppression.
Comparison with Label Smoothing. MaxSup fundamentally differs from LS in handling correct and incorrect predictions. When Zgt = Zmavs both L.S and MaxSup similarly reduce overconfidence. However, when 2g1 # Zmars LS shrinks Egt, widening the gap with the incorrect logit, whereas
5

---

MaxSup penalizes Smar- preserving Zge from undue suppression. As illustrated in Figure 2 this helps the model recover from mistakes more effectively and avoid reinforcing incorrect predictions.
Gradient Analysis. To understand MaxSup's optimization dynamics, we compute its gradients with respect each logit 2/- Specifically.
Maassup Ozk
α 1011-13 R) i arg max
(9)
otherwise.
Thus, the top-1 logit Smar is reduced by a< R) while other logits slightly increase by K In misclassified cases, the ground-truth logit Egt is spared from penalization, avoiding the error- amplification issue seen I.S. For completeness, Appendix A proviids the full gradient derivation. While [39] conducted related gradient analysis of the training loss, it focuses specifically the setting of selective classification, and examines a posthoe logit normalization technique to mitigate confidence calibration issues. However, this approach addresses only the overconfidence problem of label smoothing (LS), without tackling representation collapse. Moreover, our work presents a logit-level reformulation of LS that provides deeper theoretical understanding of why LS amplifies errors.
Behavior Across Different Samples. MaxSup applies dynamic penalty based on the model's s current predictions. For high-confidence, correctly classified examples, it behaves similarly to I.S by reducing overconfidence, effectively mitigating overfitting. In contrast, for misclassified uncertain samples, MaxSup aggressively suppresses the incorrect top-1 I logit, further safeguarding the ground- truth logit agt- This selective strategy preserves a faithful and reliable representation of the true class while actively discouraging error propagation. As shown Section 4.2 and Table this promotes more robust decision boundaries and leads to stronger generalization.
Theoretical Insights and Practical Benefits. MaxSup provides both theoretical and practical ad- vantages over LS. Whereas LS applies a uniform penalty to the ground-trum logit regardless of correctness, MaxSup penalizes only the most confident logit Emax- This dynamic adjustment robustly prevents error accumulation in misclassifications, ensuring more stable convergence. As a result, MaxSup generalizes better and achieves strong performance on challenging datasets. Moreover, as shown in Section 4.1 MaxSup preserves greater intra-class diversity, substantially improving transfer learning (Table and yielding more interpretable activation maps (Figure 2
Experiments
We begin by examining how MaxSup improves feature representations, then evaluate it on large-scale image classification and semantic segmentation tasks. Finally, we visualize class activation maps to illustrate the practical benefits of MaxSup.
4.1 Analysis MaxSup's Learning Benefits
Having established how MaxSup addresses Label Smoothing's (LS) principal shortcomings (Sec- tion 3. we now demonstrate its impact on inter-class separability and intra-class variation- properties essential for accurate classification and effective transfer learning.
4.1.1 Intra-Class Variation and Transferability
Label Smooth- ing (LSS) primarily curbs overconfidence for correctly classified samples but inad- vertently triggers еттог amplification in misclassification.. This uneven penaliza- tion can overly compress intra-class fea- ture representations. By contrast, Max- Sup uniformly penalizes the top-1 logit, whether the prediction is correct or in- correct, thereby eliminating LS's error- amplification effect and preserving finer distinctions within each class.
As noted in Section 3.1
Table 2: Feature quality of ResNet-50 on ImageNet- K.
duviiin t
R²
Method
Train
Val
Train
Val
Baseline
0.311
0.331
0.403
0.445
0.263
0.254
0.469
0.461
OLS
0.271
0,282
0,594
0.571
Zipf-LS
0.261
0.293
0.552
0.479
MaxSup
0.293
0.300
0.519
0.497
Logit Penalty
0.284
0.314
0.645
0.602

---

Table 2 compares intra-class variation (dwanin) and inter-class separability (R²) [13 for ResNet- 50 tramed on ImageNet-IK. IK. Although all investigated regularizers decrease dwithin relative baseline, MaxSup yields the smallest reduction, indicating stronger retention of subtle within-class diverslty-widely associated with enhanced generalization and improved transfer performance.
These benefits are further underscored by the linear-prole transfer accuracy on CIFAR-10 (Table B) While LS and Logit Penalty each boost ImageNet accuracy, both degrade transfer accuracy, likely by suppressing informative and transferable features. By contrast, MaxSup preserves near-baseline performance, implying that it maintains rich discriminative information crucial for downstream tasks, For extended evaluations on diverse datasets, see Table 12 in the appendix.
4.1.2 Connection to Logit Penalty
As detailed in Section g both Label Smoothing (LS) variants and MaxSup impose penalties directly at the logit level, align- ing with the perspective that various regularizers influence a model's representational capacity via distinct logit con- straints [T5] Within this family of techniques. Logit Penalty and Mux both address the maximum logit, yet diverge fundamentally in their specific methods of regularization.
Table 3: Linear Linear-probe transfer accuracy CIFAR-10 10 (higher is better).
Method
Acc.
Baseline
0.814
Label Smoothing
0.746
Logit Penalty
0.724
Logit Penalty minimizes the fz-norm of the entire logit vector, inducing global contraction that can improve class separa- tion but also reduce intra-class diversity. potentially hindering downstream transfer. By contrast, Max focuses exclusively on the top-1 logit, gently nudging it closer to the mean logit. Because only the highest-confidence prediction is penalized, MaxSup avoids the uniform shrinkage observed in Logit Penalty, preserving richer intra-class variation a property essential for robust transfer. Further insights into this behavior can be found in Section where logit-value histograms illustrate how each method affects the logit distribution.
MaxSup
0.810
4.2
Evaluation on ImageNet Classification
Next, we compare Max to standard Label Smoothing (LS) and various LS extensions on the large-scale ImageNet- IK dataset.
4.2.1 Experiment Setup
Model Training Configurationss We evaluate both convolutional (ResNet 10 MobileNetV2 (7
and transformer (DeiT--Smal (36)) architectures on ImageNet [17]. For the ResNet Series, we train for 200 epochs using stochastic gradient descent (SGD) with momentum0.9, weight decay of 1 x 10-4 and a batch size of 2048. The initial learning rate is 0.85 and is annealed via cosine schedule We also test ResNet variants on CIFAR- -100 with a conventional setup: an initial learning rate of 0.I (reduced fivefold at epochs 60, 120, and 160), training for 200 epochs with batch size 128 and weight decay 5 x 10-4, For DeiT-Small, we use the official codebase (36), training from scratch without knowledge distillation isolate MaxSup's 's contribution. CutMix and Mixup disabled to ensure the model optimization objective remains unchanged.
Hyperparameters for Compared Methods.We compare Max Suppression Regularization against a range of L.S extensions, including Zipf Label Smoothing 21 and Online Label Smoothing [45] Where official implementations exist, we adopt them directly; otherwise, we follow the methodologi- cal details provided in each respective paper. Except for any method-specific hyperparameters, all other core training settings remain identical to the baselines. Furthermore, both MaxSup and standard LS employ a linearly increasing a-scheduler for improved training stability (see Section This ensures fair comparison under consistent and reproducible training protocols.
4.2.2 Experiment Results
ConvNet Comparison. Table4 shows results for MaxSup alongside various -obothiig and self-distillation methods on both ImageNet and C1F benchmarks. Across all convolutional
Additional training hyperparameters follow the FFCV scripts at http://gthhhhccommo/m/tt///// See Section B for further details.
7

---

architectures tested, MaxSup consistently delivers the highest top-1 accuracy among labol-smoothing approaches. By contrast, OLS [45 and Zipf-LS [2]] exhibit less stable gains, suggesting their effectiveness may heavily hinge on specific training protocols.
To reproduce OLS and Zipf-LSS, we apply the authors' official codebases and hyperparameters but do not replicate their complete training recipes (c.g., OLS trains for 250 epochs with step-scheduled learning rate of O, and Zipf-LS uses 100 epochs with distinct hyperparameters). Even under these modified settings, Mux remains robust, highlighting its effectiveness across a variety of training schedules unlike the more schedule-sensiteee improvements noted for OLS and Zipf-LS.
Table 4: Performance comparison of classical convolutional networks on ImageNet and CIFAR- 100. results are shown as "mean t std" (pereentage), Bold highlights the best performance; underlined marks the second best. (Methods labeled with indicate code adapted from official repositories; see the text for additional dctails.)
Method
ImageNet
CIFAR-100
ResNet-
ResNet-50
ResNet-101
MobileNetV2
Re
ResNet-50
ResNet-101
MobileNet12
Baseline
69.09 12
76.41=0.10
75.9640.18
71.40:0.12
76.16=0.18
78.69s0.16
79.11#0.21
68.0640.06
Label Smoothing
69.54+0.15
16.91=0.11
77.37+0.15
71.61+0.09
77.05=0.17
78.8840.13
79.19+0.25
69.65+0.08
Zipf-LS
69.31=0.12
76.73=0.17
76.9120.11
71.1640.15
76.21=0.12
78. 78.75a0.21
79.15a0.18
69. 69.39400.08
OL
19.445+0.15
77.21+0.21
17.71+0.17
71.63+0.11
27.35=0.15
78. 78.7940.12 12
79,25+0.15
68.91
MaxSup
69,96+0.13
77.69=0.07
18.1840.12
72.08=0.17
77.82+0.15
79.15=0.13
79.41=0.19
69.88+0.07
Logit Penalty
68.4840.100
76.7340.10
17.2010.15
71.1320.10
76.41=0.15
78.9040.16
78.8940.21
69. 69.4610.08
DeiT Comparison. Table 5 summmarzess per-
Table 5: DeiT-Small top-1 accuracy (%). reported as mean 1 standard deviation. Values in parentheses indicate absolute improvements over the baseline.
formance for DeiT-Small on ImageNet across various regularization strategies. Notably, MaxSup attains top-1 accuracy of 76.49% surpassing standard Label Smoothing by 0.41% %. In contrast, LS variants such as Zipf- LS and OLS offer only minor gains over LS, implying that their heavy reliance on data aug- mentation may limit their upplicability to vi- sion transformers. By outperforming both LS and its variants without additional data manip- ulations, MaxSup demonstrates robust feature enhancement. These findings underscore MaxSup's adaptability to different architectures and emphasize its utility in scenarios where conventional tabelssmoothing methods yield limited benefits.
Method
Mean
Std
Baselinc
74.39
0.19
Label Smoothing
76.08 (+11.69)
0.16
Zipf-LS
75.89 (+1.50)
0.26
OLS
76.16 (+1.777
0.18
MaxSup
76.49 (+2.10)
0.12
Fine-Grained Classification. Beyond large- scale benchmarks like ImageNet, we further evaluate MaxSup on two fine- fine-grained visual recognition tasks: CUB-200-201 [37] and Stanford Cars [16]. These datasets pose unique challenges due subtle inter-class dif- ferences, which often expose the limitations of standard regularization approaches. As shown in Table 66 MaxSup achieves the best perfor- mance across both datasets, surpassing LS and its recent variants. This demonstrates that MaxSup encourages the model to learn more discriminative and semantically rich representations that better capture fine-grained attributes, such as textures and part-level details. The consistent improvements on these benchmarks further validate MaxSup's capacity to generalize across different visual domains and its potential to enhance robustness in recognition scenarios where nuanced feature understanding is critical.
Table 6: Classification on CUB and Cars Datasets.
Method
CUB 37
Cars 16
Baseline
80.88
90.27
LS
81.96
91.64
OL.S
82.33
91.96
Zipf-LS
81.40
90.99
MaxSup
82.53
92.25
Long-Tailed Classificatton. To assess the efffectiveness of MaxSup under data imbalance, we performed experiments on the CIFAR- 10-LT dataset with imbalance ratios of 50 and 100, following the experimental settings described in [35.. The corresponding results are summarized in Table 7 The evaluation compares three setups: Focal Loss, Focal Loss + LS, and Focal Loss + MaxSup. Across all imbalance ratios and splits (val/test), MaxSup consistently outperforms both the baseline and LS in overall accuracy, which jointly reflects the many-shot, medium-shot, and low-shot (minor class) performance. For example, at an imbalance ratio of 50 on the test split, MaxSup achieves 81.4% accuracy, outperforming Focal Loss (76.8%) by 4.6 percentage points, and LS (80.5%) by
8

---

Table 7: Comparison of classification performance (%) across imbalance levels for different loss strategies (Focal Loss vs Label Smoothing (LS) vs MaxSup) on the long-tailed CIFAR- 10 dataset using Resnet-32. Best performances are in bold.
Dataset
Split
Imbalance Ratio
Method
Overall
Many
Medium
Low
Focal Loss
77.4
76.0
89.7
0.0
LT CIFAR- 10 val
50
Label Smoothing
81.2
81.6
77.0
0.0
MaxSup
82.1
82.5
78.1
0.0
Focal Loss
76.8
75.3
90.4
0.0
LT CIFAR-10 test
Label Smoothing
80.5
81.1
75.4
MaxSup
81.4
82.3
73.4
0.0
Focal Loss
75.1
71.8
88.3
0.0
LT CIFAR-10
val
100
Label Smoothing
76.6
80.6
60.7
0.0
MaxSup
77.1
80.1
65.1
0.0
Focal Loss Label Smoothing
74.7
71.6
87.2
0.0
LT CIFAR-10
test
100
76.4
80.8
59.0
0.0
MaxSup 76.4
79.9
62.4
0.0
0.9 percentage points. These results indicate that MaxSup achieves better trade-off between many- and medium-shot accuracy. While does not fully resolve the challenge of imbalanced classification eespecil for minority classes- it shows positive effects and offers promising direction for further extension.
Corrupted Image Classification To evalu- the effectiveness of MaxSup on out-of- distribution (OOD) settings, we also con- ducted experiments on CIFARIO-C bench- mark [12] shown in Table ffolowin settings [I]. Table reports the performance of MaxSup and Label Smoothing (LS) on this benchmark using ResNct-50 as the backbone. Specifically, LS yields better NLL (1.5730 vs. 1.8431), implying more confident proba- bilistic predictions. However, MaxSup achieves better ECE (0.1479 0.1741), indicating better calibration of the predicted confidence scores. These results validate that MaxSup remains effective OOD datasets, achieving performance comparable LS across all three metries.
Table 8: Comparison of Max Label Smoothing (L.S), and standard Cross Entropy (CE) on CIFAR- 10-C. Lower is better. Values show mean(std) across three setups.
Metric
MaxSup
LS
CE
(Curr)
0.362(0.055)
0,359/0.064)
0.354(0.015)
NL. NT.1.(oom)
1.770(0. 103)
(47660.111)
1.819(0.
(Corr) 0.145(0.003)
0,158(0.11))
0.26000.015)
Ablation on the Weight Schedule. We also systematically investigate how different a scheduling strategies impact MaxSup's performance. Empirical results indicate that MaxSup consistently maintains high accuracy across wide range of schedules, further underscoring its robustness against hyperparameter changes. For additional details and discussions, refer to Section
4.3 Evaluation Semantic Segmentation
We further investigate MaxSup's applicabil- ity to downstream tasks by evaluating its performance on semantic segmentation us- ing the widely adopted MMSSemmenttiinn framework [ Specifically, we adopt the Uper- Net [40] architecture with a DeiT-Small back- bone, trained on ADE20K. Models pretrained on I with either MaxSup or Label Smoothing are then fine-tuned under the same cross-entropy objective (Section 4.2.2).
Table 9: Semantic segmentation (multi-scale) on ADE20K using UperNet. All models pretrained ImageNet- K: mloU reported as percentage.
Backbone
Method
mIoU
Baseline
42.1
DeiT-Small
Label Smoothing
42.4 (+0.3)
MaxSup
42.8 (+-0.7)
Table 9 shows that initializing with Max pretrained weights yields an mloU of 42.8%, surpassing the 42.4% achieved by Label Smoothing. This improvement indicates that MaxSup fosters more

---

discriminative feature representations conducive to dense prediction tasks. By more effectively capturing class boundaries and within-class variability, MaxSup promotes stronger segmentation results, underscoring its potential to deliver features that are both transferable and highly robust.
4.4
Visualization via Class Activation Maps
To better understand how MaxSup fundamen- tally differs from Label Smoothing (I.S) in guiding model decisions, we employ Gradient- weighted Class Activation Mapping (Grad-
CAM) (29), which highlights regions most in- fluential for each prediction.
We evaluate DeiT- Deil-Small under three training se- tups: MaxSup (second row), LS (third row), and a baseline with standard cross-entropy (fourth row). As illustrated in Figure Surttted models more effectively suppress background distractions than I.S, which often fixates on un- related objects-stt -ucch as poles in "Bird," tubes in "Goldfish," " and caps in "House Finch." " This behavior reflects LS's artor-enhancement mech- anism, which can misdirect attention.
Sreeching
tiee and Davoohe
The basiing
Gaaaiie lrcal
montay
byld
Moreover, MaxSup retains wider spectrum of salient features, as exemplified in the Shark" and Monkey" images, where -S-trained models of- ten omit crucial semantic details (e-g., fins, tails, or facial contours). These findings align with our analysis in Section[] clearly demonstrating that Max preserves richer intra-class infor- mation. Consequently, MaxSup-trained models produce more accurate and consistent predic-
Figure 2: Grad-CAM 29 visualizations for DeiT- Small models under three training setups: MaxSup (2nd row), Label Smoothing (3rd row), and baseline (4th row). The first row shows the original images. Com- pared to Label Smoothing, MaxSup more effectively filters out non-target regions and highlights essential fea- tures of the target class, reducing instances where the model partially or entirely focuses on irrelevant areas.
tions by effectively leveraging fine- fine-graing grained object cues. Further quantitative Grad-CAM overlay metrics (e.g., precision and recall for target regions) confirm that Max yields more focused and comprehensive activation maps, further underscoring its overall efficacy.
5 Conclusion
We examined the shortcomings of Label Smoothing (LS) and introduced Max Suppression Regular- ization (MaxSup) as a targeted and practical remedy. Our analysis shows that LS can unintentionally heighten overconfidence in misclassified samples by failing to sufficiently penalize incorrect top-1 I logits. In contrast, MaxSup uniformly penalizes the highest logit, regardless of prediction correctness, thereby effectively eliminating LS's error amplification. Extensive experiments demonstrate that MaxSup not only improves accuracy but also preserves richer intra-class variation and enforces sharper inter-class boundaries, leading to more nuanced and transferable feature representations and superior transfer performance. Moreover, class activation maps confirm that MaxSup better attends to salient object regions, reducing focus on irrelevant background elements.
Limitations. Prior work (23] notes that LS-trained teachers may degrade knowledge distillation [13 14], and Guo et al. [8] suggests LS accelerates convergence via improved conditioning. Examining MaxSup's potential role in distillation and its overall impact on training dynamics would clarify these underlying effects. Recent studies [33 7 also show that &g regularization biases final-layer features toward low-rank solutions, raising interesting questions about whether MaxSup behaves similarly.
Impact. In practical applications, MaxSup shows strong promise for systems demanding robust generalization and efficient transfer, and we have not observed any additional adverse effects or trade-offs. By offering researchers and practitioners both clearer understanding of LS's limitations and straightforward, computationally light, and easily integrable method to overcome them, MaxSup may help guide the development of more reliable and interpretable deep learning models.
10

---

Acknowledgments
This work was supported by the University of Washington Faculty Startup Fund, the Carwein- Andrews Fellowship, the UW GSFEI Top Scholar Award, the U.S. DOT PacTTrans sub-center seed funding program, the DFG Research Unit 5336- Learning to Sense (L2S), and the ELSA Eu- ropean Lighthouse Secure and Safe AI funded by the European Union under grant agreement No. 101070617. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or European Commission. Neither the European Union nor the European Commission can be held responsible for them.
We thank the anonymous reviewers for their helpful comments.
References
[1] Duarte M Alves, Nuno M Guerreiro, João Alves, José Pombal, Ricardo Rei, José GC de Souza, Pierre Colombo, and André FT Martins. Steering large language models for machine translation with finetuning and in-context learning. arXiv preprint arXiv:23 13448, 2023.
[2] Keshigeyan Chandrasegaran, Ngoc- Tran, Yunqing Zhao, and Ngai-Man Cheung. Re- visiting label smoothing and knowledge distillation compatibility: What was missing? In Proceedings of the International Conference on Machine Learning (ICML), pages 2890-2916.
PMLR, 2022.
[3] Shiming Chen, Guosen Xie, Yang Liu, Qinmu Peng, Baigui Sun, Hao Li, Xinge You, and Ling Shao. Hsva: Hierarchical semantic-visual adaptation for zero-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 34: 14:16622-16634 2021.
[4] Yann Dauphin and Ekin Dogus Cubuk. Deconstructing the regularization of batchnorm. In International Conference on Learning Representations (ICLR), 2021.
[5] Yutong Feng, Jianwen Jiang, Mingqian Tang, Rong Jin, and Yue Gao. Rethinking supervised pre-training for better downstream transferring. arXiv preprint arXiv:21 10.000144 2021.
[6] Yingbo Gao, Weiyue Wang. Christian Herold, Zijian Yang, and Hermann Ney. Towards better understanding label smoothing neural machine translation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 212-223,
2020.
[7] Connall Garrod and Jonathan Keating. The persistence of neural collapse despite low-rank bias: An analytic perspective through unconstrained features. arXiv preprint arXiv:2410.23169, 2024.
[8] Li Guo, Keith Ross, Zifan Zhao, George Andriopoulos, Shuyang Ling, Yufeng Xu, and Zixuan Dong. Cross entropy versus label smoothing: A neural collapse perspective. arXiv preprint arXiv:2402.03999. 2024.
[9] Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative lcarning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 11020-11029 2020.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[11] Markus Heinonen, Ba-Hien Tran, Michael Kampffmeyer, and Maurizio Filippone. Robust classification by coupling data mollification with label smoothing. In International Conference on Artificial Intelligence and Statistics, pages 4960-4968. PMLR, 2025.
[12] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations (ICLR), 2019.
11

---

[13] Geoffrey Hinton. Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503.02533. 2015.
[14] Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data in class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[15] Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? Advances in Neural Information Processing Systems (NeurIPS), 34:28648- 226622, 2021.
[16] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[17] Alex Krizhevsky, Ily Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[18] Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
[19] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdd/mnistt/ 1998.
[20] Dongkyu Lee, Ka Chun Cheung, and Nevin L Zhang. Adaptive label smoothing with self- knowledge in natural language generation. arXiv preprint arXiv: 2022.
[21] Jiajun Liang, Linze Li, Zhaodong Bing. Borui Zhao, Yao Tang, Bo Lin, and Haoqiang Fan. Efficient one pass self-distillation with zipf's 's label smoothing. In European Conference on Computer Vision (ECCV), pages 104-119. Springer, 2022.
[22] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF International Conference Computer Vision (ICCV), pages 10012-10022,
2021.
[23] Rafael Müller, Simon Korbliith, and Geoffrey E Hinton. When does label smoothing help? Advances Neural Information Processing Systems (NeurIPS). 32, 2019.
[24] Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shot image classification with hierarchical label sets. In Proceedings of the International Conference on Machine Learning (ICML), pages 26342-26362. PMLR, 2023.
[25] Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Reg- ularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017.
[26] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International jounnaatiinooouunall of computer vision, 1
2015.
[27] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4510-4520, 2018
[28] Mert Bulent Sariyildiz, Yannis Kalantidis, Karteek Alahari, and Diane Larlus. No reason for no supervision: Improved generalization supervised models. arXiv preprint vxXi::220.115669 2022.
[29] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. International Journal of Computer Vision 128(2)3336-35 355., October 2019. ISSN 1573- doi: 10. 10.10077/112233--9--122--- s11263- 019-01228-7
URL http: dx doi o0. 1007/
12

---

[30] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang- Tring-Tiig Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study, arXiv preprint arXiv:2104.00676 2021.
[31] Carlos N Silla and Alex A Freitas A survey of hierarchical classification across different application domains. Data mining and knowledge discovery, 22:31 2011.
[32] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting In Journal of Machine Learning Research, volume 15, pages 1958,1958, 2014.
[33] Peter Súkeník, Marco Mondelli, and Christoph Lampert. Neural collapse versus low-rank bias: Is deep neural collapse really optimal? arXiv preprint arXiv:2405. /4468, 2024.
[34] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826, 2016.
[35] Kaihua Tang, Jianqiang Huang. and Hanwang Zhang. Long-tailen classification by keeping the good and removing the bad momentum causal effect. Advances in neural information processing systems, 33:1513-1524, 2020.
[36] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Proceedings of the International Conference on Machine Learning (ICML), pages 10347-10357.
PMLR, 2021.
[37] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 20rds-200-2011 dataset. 2011.
[38] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization, 2022. URL htttps///rxxvv wrw. abs/ 2205.09311
[39] Guoxuan Xia, Olivier Laurent, Gianni Franchi, and Christos-Savve Bouganis. Understand- ing why label smoothing degrades selective classification and how fix it. arXiv preprint 24731:5503.4 2024.
[40] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In European Conference on Computer Vision (ECCV), pages 418-434, 2018.
[41] Jing Xu and Haoxiong Liu. Quantifying the variability collapse of neural networks. In Proceedings of the International Conference on Machine Learning (ICML), pages 38535-38550. PMLR, 2023.
[42] Kai Yi, Xiaoqian Shen, Yunhao Gou, and Mohamed Elhoseiny. Exploring hierarchical graph representation for large-scale zero-shot image classification. In European Conference on Computer Vision, pages 16-132. Springer, 2022.
[43] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang. and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattenn Recognition (CVPR), pages 3903-3911, 2020.
[44] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), pages 818-833. Springer, 2014.
[45] Chang-Bin Zhang, Peng- Tao Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, and Ming-Ming Cheng. Delving deep into label smoothing. IEEE Transactions on Image Processing, 30: 2021.
[46] Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: neural collapse perspective. Advances Neural Information Processing Systems (NeurIPS), 35:31697-31710, 2022.
13

---

[47] Yuxuan Zhou, Wangmeng Xiang. Chao Li, Biao Wang. Xihan Wei, Lei Zhang, Margret Keuper, and Xiansheng Hua. Sp-vit: Learning 2d spatial priors for vision transformers. arXiv preprint arXiv:2206.07662 2022.
[48] Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Cheng-Lin Liu. Rethinking confidence calibration for failure prediction. In European Conference on Computer Vision (ECCV), pages 518-5 536.. Springer, 2022.
[49] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301-320, 2005.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's abstract and introduction outline the main contribu- tions including the identification of Label Smoothing (LS) shortcomings and the proposal of Max Suppression (MaxSup)- and these claims align with the theoretical and experimental sections.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: A dedicated "Limitations" portion (or equivalent discussion) is provided, acknowledging possible extensions (e.g., knowledge distillation scenarios) and other open questions (e.g., interactions with Eg regularization).
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
The paper includes formal statements and proofs in the main text and/or appendix (Leema/TTheoren with proofs in the supplementary material). All assumptions are clearly stated, and references are provided.
Justification:
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results?
Answer: [Yes]
Justification:
The main text and appendix provide training pipelines, hyperparameters, datasets, and references to the code. Full details (batch sizes, learning rates, etc.) are included.
5. Open access to data and code
Question: Does the paper provide open access to the data and code? Answer: [Yes]
Justification: The code released (anonymized needed), and the datasets used (ImageNet, CIFAR, etc.) are publicly available under their respective standard licenses.
6. Experimental setting/details
Question: Does the paper specify all the training and test details necessary to understand the results?
Answer: [Yes]
Justification: Section 4.2.1 and the appendix detail the setup (optimizers, data splits, learning rates, etc.). The authors specify how they selected key hyperparameters.
7. Experiment statistical significance
Question: Does the paper report error bars or statistical significance for the experiments?
Answer: [Yes]
Justification: Tables report "mean + std" from multiple runs, reflecting the variability due to initialization or training seeds. This is shown in all main experimental tables.
8. Experiments compute resources
Question: Does the paper provide sufficient information about compute resources?
Answer: [Yes]
15

---

Justification:
The text or appendix indicates GPU usage (e.g.. ResNet on cluster GPUs), approximate training duration, and other relevant details. Though high-level, suffices to gauge feasibility.
9. Code of ethics
Question: Does the research conform with the NeurIPS Code of Ethics?
Answer: [Yes]
Justification: The work adheres to standard academic norms, uses publicly available datasets, and presents no known ethical concerns or conflicts with the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive and negative societal impacts?
Answer: [Yes]
The "Impact" statement addresses potential benefits (improved accuracy and transfer, leading more robust systems) and acknowledges that misuses are minimal given the method's 's purely algorithmic nature.
Justification:
11. Safeguards
Question: Does the paper describe safeguards for high-risk data or models?
Answer: [NA]
Justification:
The paper does not involve high-risk data (e.g., private user info) or high-risk models (e.g., generative LLMs). Standard ImageNet/CIFAR usage and training code are of no particular misuse risk.
12. Licenses for existing assets
Question: Are the creators of assets properly credited, and the licenses mentioned?
Answer: [Yes]
Justification: The paper cites and credits publicly available code or datasets (ImageNet, CIFAR, etc.) with references their original licenses terms of service.
13. New assets
Question: Are newly introduced assets well documented?
Answer: [NA]
Justification: No new data special code libraries introduced beyond the regular code release. The approach modifies existing code for training frameworks but does not constitute new dataset model asset.
14. Crowdsourcing and research with human subjects
Question:
Are there human subjects or crowdsourcing experiments, with instructions and compensation described?
Answer: [NA]
Justification: The work involves no human subjects or crowdsourcing tasks.
15. Institutional review board (IRB) approvals equivalent
Question: Does the paper discuss IRB approvals for human-subjects research?
Answer: [NA]
Justification: The paper does not involve human subjects; no IRB is necessary.
16. Declaration of LLM usage
Question: Does the paper describe usage of LLMs if it is essential to core methods in this research?
Answer: [Yes]
Justification: We used a Large Language Model (LLM) solely for writing and polishing the paper's 's text. The LLM was not involved in designing, conducting, or analyzing the experiments, nor in developing the core algorithmic contributions.
16

---

A Technical Appendices and Supplementary Material
Technical appendices with additional results, figures, graphs and proofs may be submitted with the paper submission before the full submission deadline (see above), or as separate PDF in the ZIP file below before the supplementary material deadline. There no page limit for the technical appendices.
17

---

A Proof of Lemma 3.2
Proof. We aim to demonstrate the validity of Lemma 3.2, which states:
((s Hy, q) LLS
(10)
where Lr.s a ( 9) ((y.)))))))))))))))))((
Let us proceed with the proof:
We begin by expressing the cross-entropy H(s q):
K
(s, q) =
log 4
(1I)
In the context of label smoothing, sy defined as:
8 (1 a)yx + K
(12)
where a is the smoothing parameter, yi is the original label, and K is the number of classes. Substituting this expression for $Ą into the cross-entropy formula:
H(s = Σ (11 a:)yk= K
log 9k
(13)
Expanding the sum:
(ss, q) -(1 o)
y/ log I
log Qk
(14)
K
We recognize that the first term equivalent (1 ay q), and the second term all Thus:
( q).
1
H(s, q) (1 a) Hy q) all
(
(15)
Rearranging the terms:
H(s q) H(,, +a 1| (# )
(16)
We can now identify Hyy q) the original cross-entropy loss and L,s a (H (#. H(y,g)) as the Label Smoothing loss.
Therefore, we have demonstrated that:
H(s q) = Hyy, q) + LLS
(17)
with LLS defined in the lemma. It is noteworthy that the original cross-entropy loss H q)
remains unweighted by a in this decomposition, which consistent with the statement in I Lemma 3.2
18

---

B Proof of Theorem 3.3
Proof. We aim to prove the equation:
LLS alzgt
2k
(18)
k=1
Let be the smoothed label vector and q be the predicted probability vector. We start with the cross-entropy between s and q:
H(s, 9) Σ sy. log q&
(19)
With label smoothing, 8% (1 a)uk: + R. where y is the one-hot ground truth vector and a is the
smoothing parameter. Substituting this:
H(s q)
Σ [1 k=1
a)yx +
a K
log qu
(20)
Expanding:
H(s, -11 0)
1/ log To
log qk
(21)
K
Since y is one-hot vector, Σ Uk log Qz = log lgt. where gf is the index of the ground truth class:
H(s, q) - (x) log Ogt
log qk
(22)
K
Using the softmax function,
tk E we can express log Ob in terms of logits:
log = 26 - log
(23)
Substituting this into our expression:
H(s q) = (1 ()(=
log
+5) )]
K
K
2
log
e
K
k=1
1=1
K
(1 2)ztt + +(1 - log(
E
(24)
1=1
K
2h a aog
Σ et))
j=1
a
(1 a)
z
log
K
1=1
1=1
19

---

Rearranging:
H(s.q) -zgt + log(
Σ
e +
a
2ggt
2k
(25)
We can identify:
(y, -ggt +log(2 e (cross-entropy for one-hot vector y)
LIS - az - # Ex= zk}
Thus, we have proven:
H(s (y, LLs
(26)
Due to the broad usage of CutMix and Mixup in the training recipe modern Neural Networks, we additionally take their impact into account together with Label Smoothing. Now we additionally prove the case with Cutmix and Mixup:
K
Lis - a((AZget + +11 - A))212)
(27)
K
k=1
With Cutmix and Mixup, the smoothed label becomes:
(1 - aAyaa + (1 - A) +
(28)
K
where y&1 and y1z2 are one-hot vectors for the two ground truth classes from mixing, and is the mixing ratio.
Starting with the cross-entropy:
K Σ
H[s
q)
5k log 9k
(29)
k=
K
Σ
[11
(Ayta + (1 A)yk2)
log q
(30)
K
k=1
K
K
a
- (1 a)
(Ayu1 + (1 - ())w2) log qh-
log
(31)
K
k=
Since yki and yuz are one-hot vectors:
K
H(s = -11- a)llog qee1 + (1 - A) log ggt2)
a
log
(32)
K
k=1
where gil and gl2 are the indices of the two ground truth classes.
Using 9k
e
R d*
ety
express in terms of logits:
K H(s = -(1 -[][[(((2))) - llo(( e +
K +(1 2) log( 2
e>)
(33)
==1
1=1
26
log
ez)]]
(34)
K
j=l
20

---

Simplifying:
H(, -(1 -(1-a) ()(2 (1 2) (1 a)) log
(35)
j=1
K 2h log
E
(36)
K
K
k
= -(11 (Zget + (1 \)2g(2]
a
zk +
log
Σ
e
(37)
K
1=1
/=1
Rearranging:
K
(s, q) -[ + - 2)2g121 + log(
e
(38)
j=1
K
+ a[A2mc1 + (1 - A))2t2
(39)
K
k=1
We can identify:
H(y', .)) = -(Xg01 + (1 - 2)2g(2) + +log( log(E 65)) (cross-entropy for mixed label y')
Lis = a(Aagel + A) : >x EM]
Thus, have proven:
H(s, 9) = Hyy, 9) + Lis
(40)
This completes the proof for both cases of Theorem 3.3
C Gradient Analysis
C.1 New Objective Function
The Cross Entropy with Max Suppression is defined as:
MaaNaup, i (x, y) H yk + K Im=ergmax(q)=-- q: (x)
where H(:,)) denotes the cross-entropy function.
C.2 Gradient Analysis
The gradient of the loss with respect to the logit z; for each class i is derived
MaxSup. yt.4 V:
+ a: l;zargmax(g)
We analyze this gradient under two scenarios:
Scenario 1: Model makes correct prediction
In this case, Max Suppression is equivalent to Label Smoothing. When the model correctly predicts the target class (argmax(g) GT), the gradients are:
21

---

For the target class (GT): MaxSup.t
Q1,GT 00 (1- k))
For non-target classes: 0, MaxSupp Qtii
Scenario 2: Model makes wrong prediction
When the model incorrectly predicts the most confident class (argmax(q) * GT), the gradients
For the target class (GT): GT
MaxSup.t = QI,GT (1 + x)
For non-target classes (not most confident): 0, Maxsup.. = qt,i K
For the most confident non-target class: 0. Massup.
qt.i + a (1 -
The Max Suppression regularization technique implements sophisticated gradient redistribution strategy, particularly effective when the model misclassifies samples. When the model's prediction (argmax(q) differs from the ground truth (GT), the gradient for the incorrectly predicted class is increased by a(1 - }), resulting in argmax(g) = (targmax(g) + a(1 - k) Simultancously, the gradient for the true class is decreased by K giving Max 4,GT (1 x) while for all other classes, the gradient is slightly reduced by f: MMaoSuppt = 4r * This redistribution adds a substantial positive gradient to the misclassified class while slightly reducing the gradients for other classes. The magnitude this adjustment, controlled by the hyperparameter a, effectively penalizes overconfident errors and encourages the model to focus challenging examples. By amplifying the learning signal for misclas aisifiisiii,,, Max Suppression regularization promotes more robust learning from diffieult ambiguous samples.
Algorithm 1 Gradient Descent with Max Suppression (MaxSup)
Require: Training set D {(x(()) ((0))) learning number of itcrations T: smoothing parameter neural network fa(-):
batch size B: total classes K.
1: Initialize network weights 8 (c.g.. randomly).
2: for I = to T do
iteration pricessrr butches 3: for each mini-batch (3), y
D
4: Compute logits: 2 + (x) for each sample in the batch
5: Compute prodicted probabilities: (f)) < softmax(2 (7)
6:
Compute cross-ente crossentropyy koss:
1 8
LCE
Σ H(y0),
B
7: / MaxSup component: penalice the top-1 logit
8:
For each sample j:
(
2(1)
(5)
max KC (L.....)))
K
k-1
LMaxsup
Σ
5 300]
9:
Total loss:
L Les LMLaSSp
10:
Update parameters:
8 + 8- V
11:
end for
12:
end
D Pseudo Code
Algorithm I presents pseudo code illustrating gradient descent with Max Suppression (MaxSup). The main difference from standard Label Smoothing lies in penalizing the highest logit rather than the ground-truth logit.
22

---

E
Robustness Under Different Training Recipes
We assess MaxSup's robustness by testing it under a modified training recipe that reduces total training time and alters the learning rate schedule. This setup models scenarios where extensive training is impractical due to limited resources.
Concretely, we adopt the Torch Vision V1 Weight strategy, reducing the total number of epochs to 90 and replacing the cosine annealing schedule with a step learning-rate scheduler (step size = 30). We also set the initial learning rate to 0.1 and use a batch size of 512. This streamlined recipe aims to reach reasonable accuracy within a shorter duration.
As reported in Table 1O MaxSup continues to deliver strong performance across multiple convolu- tional architectures, generally surpassing Label Smoothing and its variants. Although all methods see a performance decline in this constrained regime, MaxSup remains among the top performers, reinforcing its effectiveness across diverse training conditions.
Table 10:
Performance comparison on ImageNet for various convolutional neural network architec- tures. Results are presented as "mean + std" (percentage). Bold and underlined entries indicate best and second-best best, respectively. (": implementation details adapted from the official repositories.)
Method
ResNet-18
ResNet-50
MobileNetV2
Baseline
69. 10.012
76.4420. 10
76.00=0. 18
71.42+0.12 12
Label Smoothing
69.3820.19
76.65:0.11
77.0120.15
71.40=0.09
Zipi-LS
69.43a0.13
76.8920.17
76.91s0.14
71.2420.16
69.4540.15
76.81=0:21
77.12+0.17
71.29+0.11
MaxSup
69.59++0.3
77.08=0.07
77.33+0.12 12
71.59+0.1 17
Logit Penalty
66.9720.11
74.21=0.16
75.1750.12
70.39:0.14
F
Increasing Smoothing Weight Schedule
Building the intuition that model's confidence naturally grows training progresses. propose a linearly increasing schedule for the smoothing parameter a. Concretely, a is gradually raised from initial value (e.g., 0.1) to higher value (e.g., 0.2) by the end of training. This schedule aims counteract the model's increasing overconfidence, ensuring that regularization remains appropriately scaled throughout.
Experimental Evidence As shown in Table 11 both Label Smoothing and MaxSup benefit from this a scheduler. For Lahel Smoothing, accuracy improves from 75.91% to 76. 16%, while MaxSup sees more pronounced gain, from 76.12% 76.58%. This greater improvement for Max (+0.46%) compared to Label Smoothing (+0.255% corroborates our claim that MaxSup successfully addresses the inconsistent regularization and crror-enhancement issues Label Smoothing during misclassifications.
Table 11:
Effect of an a scheduler on model performance. Here, and denote the current and total epochs, respectively. The baseline model does not involve any label smoothing parameter (a).
Configuration
Formulation
0.1
0.1 + 0.1 &
Remarks
Baseline
74.21
74.21
not used
LS
a
Ege
1 Σx
Zi
75..1
76.16
MaxSup
Emer
1 Ex
2z
76.12
76.58
G Extended Evaluation of Linear Transferability on Different Datasets
To further demonstrate the substantial improvement in feature representation compared to other methods, we further compare the linear transfer accuracies of different methods on a broader range of datasets in Table [2
23

---

Table 12: Validation performance of different methods, evaluated using multinomial logistic regres- sion with 12 regularization. Although Label Smoothing and OLS improve ImageNet accuracy, they substantially degrade transfer accuracy compared MaxSup. Following [15), we selected from 45 logarithmically spaced values between 10- and 10 Notably, the search range is larger than the search range used in Table thus leading to higher overall accuracies on CIFAR10.
Datasets
CIFARIO
CIFARIO0
CUB
Flowers
Foods
Pets
CE
91.74
75.35
70.21
90.96
72.44
92.30
LS
90.14
71.28
64.50
84.84
67.76
91.96
OLS
90.29
73.13
67.86
87.47
69.34
92.21
MaxSup
91.00
73.93
67.29
88.84
70.94
92.93
H Extended Comparison to More Label Smoothing Alternatives
We have included Confidence Penalty 22 and Adaptive Label Smooothing with Self-Knowledge 200 for an extended comparison Table 13 We follow their recommended hyperparameter settings: weight coefficient is set to 0.1 for confidence penalty, and the checkpoint with the highest validation accuracy checkpoint is treated as teacher for Adaptive Label Smoothing.
To further address the novelty of our work, we additionally compared our method to recent approaches that also identify and aim to fix the issues of Label Smoothing on misclassified samples [ 38 Specifically, the method proposed in (38] aims to mitigate overconfidence via logit normalization during training. With its default hyperparameter settings, it achieves an accuracy of 74.32% on ImageNet with a ResNet-50, which is significantly lower than the 76.91% % achieved by standard Label Smoothing. This performance aligns with that of Logit Penalty, which similarly minimizes the global 12-norm of logits and can struggle to match baseline LS performance. We also note that these norm-based methods(6] (38 are often highly sensitive to hyperparameter choices, which can limit their practical applicability.
As shown in 13 MaxSup outperforms all these alternatives on ImageNet with ResNet-50. This aligns with our theoretical analysis effective regularization than penalizing all logits (Confidence Penalty), dynamically smoothing the label distribution with self- affoleneeedge (Adaptive LS), or applying global norm norm-based penalties (Logit Penalty, Logit Normalization).
that selectively penalizing 2mar yields more consistent and
Table 13: Comparison of classic convolutional neural networks ImageNet. Results reported as "mean + std" (percentage). Bold entries highlight the best performance; underlined entries mark the second best. (Methods with denote code adaptations from official repositories; for details.)
Method
ImageNet
ResNet-50
0.10
Label Smoothing
76.91 + 0.11
Zipf-LS"
76. + 0.17
OLS"
7T.23 + 0.21
MaxSup
77.69 + 0.07
Logit Feualty
+ 0.10
Logit Normalization [3*]
74.32
Confidence Penalty
76.58 0.12
Adaptive Label Smoothing"
77.16 + 0.15
Visualization of the Learned Feature Space
To illustrate the differences between Max Suppression Regularization and Label Smoothing, we follow the projection technique of Müller et al. (23). Specifically, we select three semantically related classes and construct an orthonormal basis for the plane intersecting their class templates feature space. We then project each sample's penultimate-layer activation vector onto this plane. To ensure the visual clarity of the resulting plots, we randomly sample 80 images from the training or validation set for each of the three classes.
24

---

Selection Criteria We choose these classes according to two main considerations:
1. Semantic Similarity. We pick three classes that are visually and semantically close.
2. Confusion. We identify a class that the Label Smoothing (LS)-trained model frequently misclassifies and select two additional classes involved in those misclassifications (Figure Figure 4c). Conversely, we also examine a scenario where a class under Max Suppression Is confused with others, highlighting key differences (Figure 3d
Figure
Project on NexSap
Projediote
Profccions
Projecion WanSap
Prgecion:
Projection
Projecionn
Frquccion Impertinnce with
Proectione oit lotel amosthing
Projections wi lae
Project with emoothing
Preedien: Projectione lanel snn
Prrectoo Lshel Smooth
Injeetione Smowthin
Pojjett and Label Smooth
Iropecion::
Projectoon
Irspection
(a) Semantically Similar Classes
(b) Semantically Similar Classes
(c) Confusing Classes (I.S) (d) Confusing Classes (MaxSup)
Figure 3: Visualization of penultimate- caattimmate-lyeerr activations from DeiT-Small (trained with CutMix and Mixup) on the ImageNet validation set. The top shows embeddings for MaxSup-trained model, and the bottom row shows embeddings for Label Smoothing (tane-rained model. In each subfigure, classes either semantically similar or confuxingly labeled. Compared to LS, MaxSup yields more pronounced inter-class separability and richer intra-class diversity, suggesting stronger representation and classification performance.
Observations
As shown Figures and models trained with Max Suppression exhibit:
Enhanced inter-class separability. Distinct classes occupy more clearly separated regions, aligning with improved classification performance.
Greater intra-class variation. Instances within single class are not overly compressed, indicating richer representation of subtle differences.
For instance, images of Schipperke dogs can differ markedly viewpoint, lighting, background, or partial occlusions. Max Suppression preserves such intra-class nuances in the feature space, enabling semantic distances to visually related classes (e.g., Saluki, Grey Fox, or Belgian Sheepdog) to dynamically adjust for each image. Consequently, Max Suppression provides more flexible, fine-grained representation that facilitates better class discrimination.
J Ablation on the Weight Schedule
We conducted ablation study on the a schedule, shown in Table 15 The consistently high accuracy across settings demonstrates the robustness MaxSup. The adaptive schedule, adopted from 200), further highlights the method's integrity and compatibility with principled design choices.
25

---

Projet one NoxS.p
rrjedioo
Pidection Maddip
Iigesion:
Projection wio lotel smoothing
Projections wa label smoothiyy
Wrjject ons we lobel smoothing
Drevccion Prjectione Label Smowthis
Projections Label Smootting
Proections Label Smoothing
Tropedter
Procetion::
Propettion
Insecrion
(a) Semantically Similar Classes
(b) Semantically Similar Classes
(c) Confusing Classes (1.S)
(d) Confusing Classes (MaxSup)
Figure 4: Visualization of the penultimate-kag layer activations for DeiT-Small (trained with CutMix and Mixup) on selected ImageNet classes. The top shows results for MaxSup-trained model; the bottom row shows Label Smoothing (I.S). In (a,b), the model must distinguish semantically similar classes (e-g., Saluki vs. Grey Fox; Tow Truck vs. Pickup), while (c.d) involve confusing categories (c-g., Jean vs. Shoe Shop, Stinkhorn vs. related objects). Compared to LS, MaxSup yields both improved inter-class separability and richer intra-class variation, indicating more robust representation learning.
Table 14: Feature representation metries for a ResNet-50 model trained on ImageNet- reported on both Training and Validation sets. We measure intra-class variation (dwitii) and overall average distance (dwoal). Inter-class separability (R?) is calculated as R dgitia dwithin and R22 are preferred.
Higher values (f) of
duss
dustin f
duul
Method
Train
Val
Train
Val
Train
Val
Baseline
0,3114
0.3313
0.5212
0.5949
0.4025
0,4451
0,2632
0.2543
0.4862
0.4718
0.4690
0,4611
0.2707
7ipf's
0.2932
0.5628
0.5522
MaxSup
0.2926 (+0.03)
0.2998 (+0.05)
0.6081 (+0.12)
0,5962 (+0.12)
0.5188 (+0.05)
0.4972 (+0.04)
Logit Penalty
0,2840
0.3144
0.7996
0.7909
0.6448
0.6024
K Analysis of Computation Efficiency
Beyond the standard cross-entrops operations, MaxSup only requires:
1. A operation determine the largest logit (O(K) complexity),
2. A operation the K-dimensional logit vector, and
3. One subtraction between these scalars.
Table 15: Ablation study on alpha schedules using ResNet50 on ImageNet1
Schedule
0.1 0.1
0=0+0.1 4
0.2 0.1
Adaptive Alpha
77.65
77.62
77.43
77.70
26

---

Eusinn of Logit Values
Leght
Figure 5: Comparison of logit distributions under different regularizers.
Since K (the number of classes) is usually small (e.g., 1000 for ImageNet- IK). this overhead is minimal compared to the overall forward/backward pass of deep network. In Table T6] report the average training time per epoch using a ResNer-50 model on the ImageNet- K dataset.
Table 16: Average training time per epoch on ImageNet-IK with ResNet-50.
Method
CE (One-Hot)
CE + LS
CE + MaxSup
Time/Epoch
min 51
3 min 52
3 min 51
As seen above, the measured run times are nearly identical across all three configurations. Thus, additional cost of MaxSup is negligible compared to the total computation for large-scale training.
L Logits Visualization
As mentioned in Section 4.1.2 all Label Smoothing variants apply a different penalty on the logits. To illustrate the impact of different methods the logits, we plot the histogram of logits of ResNet-50 networks trained with each method over the ImageNet validation set, as shown in Figure 5
27


Reviews:
[{"review_id": "Zp5tKxfn93", "reviewer": "Reviewer_cjw9", "summary": "This paper analyzed the shortcomings of Label Smoothing (newly denoted as error amplification). Building on the foundation, the authors proposed Max Suppression (MaxSup), a modified label smoothing that penalizes only the top-1 logit rather than the ground-truth logit", "questions": "**Q1.** The proposed method operates effectively when the CE with Hard Labels decreases sufficiently to recover $z\\_{gt}$ from $z\\_{max}$ for incorrect samples. How about an imbalanced dataset or out-of-distribution? In those cases, $\\mathcal{L}\\_{MaxSup}$ would not be effective to minor classes\n\n**Q2.** How about integrating the proposed method into other label smoothing methods? It would be synergized enough to increase the model performance\n\n---\n**Things to improve the paper that did not impact the score**\n\n* [line 237, 239] wrong subscript ($\\bar{d}$within)\n\n* [line 269, 272] missing white space (momentum0.9, epochs60)", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n**S1.** This paper figured out the problem of the commonly used method in this field, and developed a simple yet effective method to solve the discovered problem\n\n**S2.** To overcome the simplicity of the proposed method, the authors thoroughly validated the proposed method in various perspectives, such as theorems and proofs, ablation study, gradient analysis, and so on\n\n**S3.** The experimental environment was set up to verify the superiority and versatility of the proposed method sufficiently, and the experimental results were also well analyzed\n\n**Weaknesses**\n\nThere is no weakness in the paper that would justify its rejection", "was_revised": true, "final_justification": "As I referred in the official comment, authors have demonstrated the versatility of the proposed method in various environments, and this result will be helpful to future work in LT and OOD fields. As the authors addressed the concerns about the possibility of extension to various environments, I will maintain my rating of Accept", "rebuttal": "We sincerely thank the reviewer for acknowledging the significance of our contribution and valuable suggestions. We address the mentioned questions in detail, as outlined below: \n\n> **Q1**: The proposed method operates effectively when the CE with Hard Labels decreases sufficiently to recover $z_{gt}$ from $z_{max}$ for incorrect samples. How about an imbalanced dataset or out-of-distribution? In those cases, $L_\\text{maxsup}$ would not be effective to minor classes?\n\nThank you for the insightful question. In our study, we had not explicitly addressed class imbalance or out-of-distribution (OOD). However, we emphasize that **$L_{\\text{maxsup}}$ is orthogonal to existing strategies for handling these challenges**, and can be readily combined with standard technique. For instance, reweighting the total loss by the inverse class frequency can mitigate bias toward majority classes, while entropy-based regularization may enhance robustness under OOD conditions where predictions are often overconfident.\n\nTo specifically address the concern on imbalanced data, we have conducted comprehensive experiments on CIFAR10-LT with varying imbalance ratios (50, and 100) following settings in [1*], as shown in Table R1. The evaluation compares three setups: Focal Loss, Focal Loss + Label Smoothing (LS), and Focal Loss + MaxSup. Across all imbalance ratios and splits (val/test), MaxSup consistently outperforms both the baseline and LS in overall accuracy, which jointly reflects the many-shot, medium-shot, and low-shot (minor class) performance. For example, at an imbalance ratio of 50 on the test split, MaxSup achieves 81.4% accuracy, outperforming Focal Loss (76.8%) by 4.6 percentage points, and LS (80.5%) by 0.9 percentage points. These results indicate that MaxSup is robust under class imbalance, especially in challenging settings with fewer samples in tail classes.\n\nTo address the concern regarding the effectiveness of MaxSup on out-of-distribution (OOD) settings, we also conducted experiments on CIFAR10-C benchmark shown in Table R2 following settings in [2*]. Table R2 reports the performance of MaxSup and Label Smoothing (LS) on this benchmark using ResNet-50 as the backbone. Specifically, LS yields a better NLL (1.5730 vs. 1.8431), implying more confident probabilistic predictions. However, MaxSup achieves a better ECE (0.1479 vs. 0.1741), indicating better calibration of the predicted confidence scores. These results validate that **MaxSup remains effective on OOD datasets, achieving performance comparable to LS across all three metrics**.\n\n**Table R1**: Comparison of overall accuracy (%) (jointly considering many-shot, median-shot, and low-shot top-1 accuracy) for different loss strategies (Focal Loss vs Label Smoothing (LS) vs MaxSup) across datasets, imbalance levels, and backbones. Best performances are in bold.\n\n| **Dataset**           | **Split** | **Imbalance Ratio** | **Backbone** | **Focal Loss** | **+LS**  | **+MaxSup** |\n|-----------------------|-----------|----------------------|--------------|----------------|--------|------------|\n| Long-tailed CIFAR-10  | val       | 50                   | Resnet32     | 77.4           | 81.2   | **82.1**   |\n| Long-tailed CIFAR-10  | test      | 50                   | Resnet32     | 76.8           | 80.5   | **81.4**   |\n| Long-tailed CIFAR-10  | val       | 100                  | Resnet32     | 75.1           | 76.6   | **77.1**   |\n| Long-tailed CIFAR-10  | test      | 100                  | Resnet32     | 74.7           | 76.4   | **76.4**   |\n\n\n**Table R2**: Comparison of MaxSup and Label Smoothing (LS) on CIFAR10-C using Resnet50 as backbone. For all metrics, lower value is better, and best performances are in bold.\n\n| Metric         | MaxSup | LS |\n|----------------|--------|------------------|\n| Error (Corr)   | **0.3951** | **0.3951**           |\n| NLL (Corr)     | 1.8431 | **1.5730**           |\n| ECE (Corr)     | **0.1479** | 0.1741           |\n\n[1*] Tang, K., Huang, J., & Zhang, H. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In NeurIPS 2020, pp. 1513–1524.\n\n[2*] Heinonen, M., Tran, B. H., Kampffmeyer, M., and Filippone, M. \"Robust Classification by Coupling Data Mollification with Label Smoothing.\" AISTATS 2025.\n\n> **Q2**:  How about integrating the proposed method into other label smoothing methods? It would be synergized enough to increase the model performance\n\nThank you for the suggestion. As shown in Table 14 in Appendix L (Ablation on the Weight Schedule), we combined the adaptive scheduler from Adaptive Label Smoothing [18] with our MaxSup method, which indeed led to consistent performance improvements. This indicates that there is potential synergy between MaxSup and existing label smoothing variants.\n\nWe leave a more comprehensive exploration of such combinations—especially with more advanced or task-specific label smoothing techniques—as an interesting direction for future work.\n\n> **\"Things to improve the paper that did not impact the score\"**\n\nWe are glad to resolve these formatting issues in the revised version."}, {"review_id": "ZMPIvEhltK", "reviewer": "Reviewer_wVqR", "summary": "This paper identifies and addresses shortcomings of Label Smoothing (LS), a widely adopted regularization technique. The authors analytically decompose the LS-induced loss, revealing two key components: a beneficial regularization term and a detrimental \"error-amplification\" term that leads to overconfidence in misclassified samples and feature representation collapse. To overcome these issues, the paper proposes Max Suppression (MaxSup). MaxSup applies uniform regularization by penalizing the top-1 logit rather than the ground-truth logit, eliminating the error amplification effect. Through extensive experiments on large-scale image classification (ImageNet) , semantic segmentation , and downstream tasks, MaxSup is shown to be a more robust alternative to LS, consistently reducing overconfidence, preserving richer feature representations, restoring intra-class variation, and sharpening inter-class boundaries.", "questions": "The paper mentions that both MaxSup and standard LS employ a linearly increasing $\\alpha$-scheduler for stability. While the robustness to different schedules is noted, a discussion on optimal strategies for dynamically adjusting $\\alpha$ (similar to adaptive LS variants) specifically for MaxSup could be valuable. Did the authors observe significantly different results when not using $\\alpha$-scheduler. Recently, [2] found that coupling label smoothing and data mollification through schedulers are beneficial. The authors could compare this recent method to really see the effect of the $\\alpha$-scheduler and to disentangle its effect from the main proposed method.\n\nRef:\n[2] Heinonen et al. Robust Classification by Coupling Data Mollification with Label Smoothing. AISTATS 2025.", "limitations": "Please see the Weaknesses. Overall, the main limitations are two fold: (1) the novelty is limited compared to the recent work in the literature, (2) the analysis is mainly based on intuitions and complicated empirical evidence.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-written and organized.\n- The proposed method, MaxSup, is simple and introduces negligible computational overhead, making it easily integratable into existing training pipelines.\n- The paper provides comprehensive experimental results across various architectures (ResNet, MobileNetV2, DeiT-S) and tasks (ImageNet classification, fine-grained classification, semantic segmentation, transfer learning). MaxSup consistently outperforms LS and its variants in terms of top-1 accuracy, preserves intra-class diversity , and improves transferability.\n\n**Weaknesses**\n\n- Indeed, the novelty is quite limited as the issues with LS on misclassified samples have been considered extensively in the prior work [1]. The authors should at least empirically compare this work.\n- While the analytical decomposition of LS is detailed, the theoretical implications of penalizing the top-1 logit beyond its direct effect on error amplification could be further explored. For instance, how does it influence the geometry of the learned feature space in a deeper mathematical sense, beyond the empirical observations of preserving intra-class diversity? At the moment, the authors mainly provide some intuitions and empirical evidence. Although the authors provide some analysis in experiments, they are still complicated and consist of many compounded factors to really confirm the intuitions. It would be great if the authors could provide some simple, controlled experiments on even synthetic data.\n- The paper isolates MaxSup's contribution by disabling CutMix and Mixup. A discussion or additional experiments on how MaxSup interacts when combined with other common regularization techniques and data augmentations (which are often used in practice) could provide more practical insights.\n\nRef:\n[1] Xia et al. Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It. ICLR 2025", "was_revised": true, "final_justification": "The authors provided some additional experiments that addressed some of my concerns. However, my main concern of the novelty compared to [1*] still remains. Thus, I would like to increase the score by one.", "rebuttal": "We sincerely thank the reviewer for the detailed and constructive feedback. In response to the reviewer’s concerns, we address the mentioned concerns below:\n\n> **W1 – Indeed, the novelty is quite limited as the issues with LS on misclassified samples have been considered extensively in the prior work [1]. The authors should at least empirically compare this work.**\n\n### 1. Conceptual distinctions\n\nWe already **cite and briefly discuss** the distinctions with [1*] ([35] in our paper, lines 149–151), and are happy to expand on this comparison in more detail below.\n\nWhile [1*] performs a related gradient analysis, it is conducted in the context of **selective classification** and evaluates a **post-hoc** logit normalization technique (previously suggested in [2*]) to address overconfidence **at inference time**. This approach:\n\n- targets **confidence calibration**, not **representation collapse**,\n- does **not improve classification accuracy**, and\n- relies on **additional hyperparameter tuning**.\n\nIn contrast, our **logit-level reformulation** of the training objective reveals the root cause of **error amplification** in LS. This decomposition leads directly to **MaxSup**, a principled and training-time regularization method that:\n\n- mitigates overconfidence **during training**,\n- improves **classification accuracy**,\n- and **resolves representation collapse**, a critical issue for transferable feature learning.\n\n### 2. Empirical comparison (more results)\n\nTo address the reviewer's concern, we additionally compared to [3*], which proposed to mitigate overconfidence via **logit normalization during training**. With the default hyperparameter, it merely achieves an accuracy of 74.322 (lag far behind the 76.91 accuracy of label smoothing). The performance aligns with that of Logit Penalty [4] in our Table 4, which similarly minimizes the global L2-norm of logits. We also note that **norm-based methods** [1*, 3*, 4] are **highly sensitive** to hyperparameter choices, which limits their practical applicability. We will integrate this expanded comparison in Appendix H in the revised version.\n\n### 3. Highlighting our novelty\n\nOur method is not merely incremental; it represents a **substantial rethinking** of Label Smoothing by:\n\n1. **Theoretically uncovering and explicitly characterizing** the \"error amplification\" mechanism within the LS-induced loss.\n2. **Introducing MaxSup**, a simple yet effective modification of penalizing the top-1 logit rather than the ground-truth logit, thus resolving critical limitations of LS during training.\n3. Demonstrating consistent, robust improvements across various model architectures (CNNs, Vision Transformers), datasets, and tasks (classification, segmentation, fine-grained recognition), highlighting the generalizability and practical value of MaxSup.\n\n\n### References\n\n[1*] Xia et al. Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It. ICLR 2025\n\n[2*] Luís Felipe Prates Cattelan and Danilo Silva. How to fix a broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks. In UAI, 2024.\n\n[3*] Wei, Hongxin, et al. \"Mitigating neural network overconfidence with logit normalization.\" International conference on machine learning. PMLR, 2022.\n\n\n> **W2**: Regarding the theoretical implications of penalizing the top-1 logit **beyond the impact on error amplification**, such as the influence on the geometry of the learned feature space or some simple, controlled experiments on even synthetic data.\n\nWe appreciate the reviewer’s thoughtful comment. While we agree that a deeper theoretical analysis of the geometric effects of penalizing the top-1 logit would be valuable, we respectfully note that this direction **goes beyond the scope of our current study**. Our primary focus is to address the **well-established representation collapse issue** induced by label smoothing [13, 26, 37]. We demonstrate the effectiveness of MaxSup in mitigating this issue through widely adopted qualitative [21] and quantitative [13] evaluations, which we believe are **sufficient for supporting our claims**.\n\nAdditionally, we would like to emphasize that representation collapse is a phenomenon that mainly arises in **complex, real-world data settings**. In contrast, prior work such as Neural Collapse [42] has shown that in **simplified or synthetic settings**—where models achieve near-zero training error—representations often collapse naturally, regardless of the loss function used. Therefore, we believe that **synthetic experiments may not meaningfully reflect the challenges** our method is designed to address.\n\n\n> **W3**: The paper isolates MaxSup's contribution by disabling CutMix and Mixup. A discussion or additional experiments on how MaxSup interacts when combined with other common regularization techniques and data augmentations (which are often used in practice) could provide more practical insights.\n\nWe appreciate the reviewer’s valuable suggestion. In fact, we have  provided **an extended logit-level analysis that incorporates CutMix and Mixup**, which is documented in the **Appendix (lines 641–657)**. As shown in Equation (39), MaxSup can be naturally extended to these settings by replacing the weighted average of two ground-truth logits with the maximum logit value.\n\nAdditionally, we actually performed experiments with CutMix and Mixup that we decided to leave in order to not overload the presentation. Nonetheless, for completeness, we share below the results with CutMix and Mixup enabled on DeiT-Small:\n\n| Model (Touvron et al., 2021) | Method          | Mean  | Std   |\n|-----------------------------|------------------|--------------------------|---------------------------|\n| DeiT-Small                  | Baseline with CutMix&Mixup        | 79.69                    |                            0.11      |\n|                             | Label Smoothing  | 79.81 (+0.12)            |                             0.09  |\n|                             | Zipf-LS          | 79.88 (+0.19)            |                             0.08  |\n|                             | OLS              | 79.95 (+0.27)            |                            0.12  |\n|                             | MaxSup           | 80.16 (+0.47)            |                                 0.09  |\n\nThese results demonstrate that MaxSup continues to provide consistent improvements even when used in conjunction with strong data augmentations like CutMix and Mixup.\n\n> **Q1**: 1) Optimal strategies for dynamically adjusting (similar to adaptive LS variants) specifically for MaxSup. 2) Robustness to $\\alpha$-scheduler. 3) Comparison to [2] which **couples label smoothing and data mollification** through schedulers. \n\nWe appreciate the suggestion to explore optimal strategies for dynamically adjusting MaxSup parameters, such as via adaptive α-schedulers. As shown in Table 14 (Appendix), incorporating the α-scheduler from [18] yields moderate additional improvements. However, our work primarily aims to address the *core limitations* of label smoothing—namely, **error amplification** and **representation collapse**—rather than optimizing performance through scheduler tuning. We view the design of optimal α-scheduling strategies as **orthogonal** and consider it a valuable direction for future work.\n\nImportantly, as demonstrated in Table 8, MaxSup consistently outperforms standard label smoothing **with or without** an α-scheduler. While both methods benefit from scheduling, MaxSup’s gains are **robust** and not reliant on this tuning.\n\nRegarding [2*], we note that while the paper also modifies label smoothing, it tackles a *different problem* by **coupling LS with data mollification**. This coupling leads to a different formulation and focus compared to our method. While a detailed comparison **falls outside the scope of our paper**, we will cite [2*] in the revised version to acknowledge its relevance in the broader context."}, {"review_id": "mMYWZCmMO9", "reviewer": "Reviewer_btTx", "summary": "This paper identifies two critical drawbacks of conventional label smoothing (LS): (1) error amplification—incorrect predictions become overconfident, and (2) representation collapse—within-class feature diversity is overly compressed. By decomposing the LS loss at the logit level, the authors show that the standard LS penalty contains both a “regularization term” (which only activates on correct predictions) and an “error-amplification term” (which activates on mispredictions). To remedy this, they propose Max Suppression (MaxSup): instead of penalizing the ground-truth logit, the loss penalizes the current maximum logit. This simple modification ensures that both correct and incorrect predictions are regularized evenly. Extensive experiments on ImageNet classification, downstream segmentation and fine-tuning tasks, and Grad-CAM visualizations demonstrate that MaxSup consistently improves top-1 accuracy (by 0.4–1.0%), preserves within-class diversity, and boosts segmentation mIoU compared to LS and its variants.", "questions": "- How does MaxSup scale when the number of classes K is extremely large (e.g., ≥1M)? Have you considered approximate or sparse strategies to find the max logit efficiently?\n\n- If the training set contains mislabeled examples, does MaxSup inadvertently amplify noise? Would a confidence-aware or adaptive suppression strategy help?\n\n- Have you tried integrating MaxSup with teacher–student distillation frameworks? How does it affect the quality of distilled student models?", "limitations": "- Evaluated only on vision classification and segmentation; its effectiveness in NLP, multimodal tasks, or reinforcement learning remains untested.\n\n- The theoretical analysis is limited to static logit decomposition; there is no formal study of training dynamics or convergence rates under MaxSup.\n\n- Robustness to extreme label noise, long-tail distributions, adversarial attacks, or online learning scenarios is not assessed.\n\n- Don't add too many underlines, as it affects the overall aesthetics of the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The logit-level decomposition cleanly separates the beneficial and harmful components of LS, revealing an often-overlooked error amplification effect.\n\n- MaxSup requires only a one-line change to the loss function and plugs directly into existing training pipelines without additional architecture or hyperparameter complexity.\n\n- Results cover multiple backbones (ResNet variants, MobileNetV2, DeiT-S), diverse tasks (ImageNet classification, CIFAR linear probes, ADE20K segmentation, fine-grained recognition), and report means ± standard deviations, supporting strong reproducibility.\n\n- Metrics such as within-class variance, R² transfer accuracy, and Grad-CAM maps all indicate that MaxSup maintains richer intra-class representations and focuses more accurately on relevant image regions.\n\n- The appendix and NeurIPS checklist provide full training details, hyperparameter schedules, and a link to code, facilitating community adoption.\n\n\nWeaknesses:\n\n- In the introduction section of the paper, it is stated that the left part of Figure 1 shows a feature collapse. However, in reality, MaxSup also fails to learn class 282 properly, and the features learned for this class are completely scattered—performing even worse than the result of label smoothing.\n\n- Experiments do not cover scenarios with high label noise, severe class imbalance, or adversarial examples—settings where LS variants often differ markedly.\n\n- Evaluation is largely confined to LS variants and logit penalties. It would be stronger to compare against related techniques such as self-knowledge distillation, label perturbation, or mixup-based regularizers.\n\n- Although the authors discuss the weight α schedule in the appendix, a systematic sensitivity analysis across a wide α range (e.g., α→0 or α→1 extremes) is missing.\n\n- MaxSup requires identifying the maximum logit for each example. In tasks with extremely large label sets (e.g., million-class classification or retrieval), the computational and memory overhead of that max operation may become nontrivial.", "was_revised": true, "final_justification": "I appreciate that the authors have provided many detailed experiments to demonstrate the effectiveness of their method across different scenarios. However, the visualization (Figure 1) must be carefully revised. As currently presented, Figure 1 shows that the proposed method performs worse than Label Smoothing, which raises concerns about the validity of MaxSup. Overall, I believe this paper is at the borderline of acceptance—it could go either way. Given the amount of work the authors have put into it, I am inclined toward a borderline accept.", "rebuttal": "We sincerely thank the reviewer for the valuable feedback and deeply appreciate the time and effort invested in reviewing our work. Below, we address the mentioned concerns and suggestions in detail.\n\n> **W1:** Clarification of Figure 1.\n\n- We focus the **severely reduced intra-class variation** caused by label smoothing [13, 26, 37]. This collapse leads to tightly clustered features for each class, which may seem beneficial but is known to **harm downstream tasks** such as transfer learning. **MaxSup explicitly preserves intra-class variation**, as can be observed for classes 281, 282, and 292. \n\n- While class 282 appears more scattered under MaxSup, the **inter-class separation**—particularly between classes 281 and 282—is visibly better (smaller overlapped area) under MaxSup than under ls or ce. This supports the improved classification performance we report and confirms that MaxSup **achieves larger inter-class separation while preserving the intra-class variation**. \n\n> **W2**: Experiments with high label noise, severe class imbalance, or adversarial examples—settings.\n\nWe emphasize that $L_{\\text{maxsup}}$ **is orthogonal to** strategies for handling these challenges, and can be readily combined with them. \n\nFor imbalanced data, we have conducted comprehensive experiments on CIFAR10-LT with varying imbalance ratios (50, and 100) following settings in [1*], as shown in Table R1. Across all imbalance ratios and splits (val/test), MaxSup consistently outperforms both the baseline and LS. \n\nFor out-of-distribution (OOD) settings, we conducted experiments on CIFAR10-C using ResNet-50 shown in Table R2 following settings in [2*]. These results validate that **MaxSup remains effective on OOD datasets, achieving performance comparable to LS**.\n\n**Table R1**: Comparison of overall accuracy (%) (jointly considering many-shot, median-shot, and low-shot top-1 accuracy).\n\n| **Dataset**           | **Split** | **Imbalance Ratio** | **Backbone** | **Focal Loss** | **+LS**  | **+MaxSup** |\n|-----------------------|-----------|----------------------|--------------|----------------|--------|------------|\n| Long-tailed CIFAR-10  | val       | 50                   | Resnet32     | 77.4           | 81.2   | **82.1**   |\n| Long-tailed CIFAR-10  | test      | 50                   | Resnet32     | 76.8           | 80.5   | **81.4**   |\n| Long-tailed CIFAR-10  | val       | 100                  | Resnet32     | 75.1           | 76.6   | **77.1**   |\n| Long-tailed CIFAR-10  | test      | 100                  | Resnet32     | 74.7           | 76.4   | **76.4**   |\n\n**Table R2**: Comparison of MaxSup and LS on CIFAR10-C using Resnet50 as backbone. Lower value is better.\n\n| Metric         | MaxSup | LS |\n|----------------|--------|------------------|\n| Error (Corr)   | **0.3951** | **0.3951**           |\n| NLL (Corr)     | 1.8431 | **1.5730**           |\n| ECE (Corr)     | **0.1479** | 0.1741           |\n\n[1*] Tang, K., Huang, J., & Zhang, H. Long-tailed classification by keeping the good and removing the bad momentum causal effect. NeurIPS 2020.\n\n[2*] Heinonen, M., Tran, B. H., Kampffmeyer, M., and Filippone, M. \"Robust Classification by Coupling Data Mollification with Label Smoothing.\" AISTATS 2025.\n\n> **W3**: Comparison to related techniques such as self-knowledge distillation, label perturbation, or mixup-based regularizers.\n\nWe have trained ResNet-50 with two representative self-knowledge distillation techniques using ResNet-50 on ImageNet, and below are the results:\n\n| Method                    | Accuracy (%)        |\n|---------------------------|---------------------|\n| Baseline                  | 76.41 ± 0.10        |\n| Label Smoothing           | 76.91 ± 0.11        |\n| MaxSup                    | **77.69 ± 0.07**    |\n| TF-KD$_{reg}$ [3*]         | 76.42 ± 0.09        |\n| DDSGD [4*]                 | 77.35 ± 0.09        |\n\nNotably, **DDSGD slows down training by 2×** since it always reuses half of the previous batch.\n\n[3*] Yuan, Li, et al. \"Revisiting knowledge distillation via label smoothing regularization.\" CVPR 2020.\n\n[4*] Shen, Yiqing, et al. \"Self-distillation from the last mini-batch for consistency regularization.\" CVPR 2022.\n\n> **W4**: Sensitivity analysis across a wide α range (e.g., α→0 or α→1 extremes).\n\nWe appreciate the reviewer’s interest in the sensitivity of α. We note that extreme values (e.g., α → 0 or α → 1) are **less informative and rarely used in practice**: As α → 0, the method reduces to standard cross-entropy, which we already include as a baseline; As α → 1, the supervision signal vanishes, leading to  degraded performance.\n\nFor ResNet-50 trained with MaxSup on ImageNet, when α = 0.5, the top-1 accuracy is **76.11%**, while at α = 0.9, it drops sharply to **68.38%**. Our analysis thus focuses on the practical range of α where MaxSup meaningfully improves performance.\n\n> **W5**: In tasks with extremely large label sets, the computational and memory overhead of max operation.\n\nWe appreciate the reviewer's concern on the scalability of MaxSup. However, even in extreme scenarios, the **computation required to find the maximum logit** is **negligible** compared to the **total cost of a forward pass** in modern deep networks.\n\nLet $K$ denote the number of classes. Identifying the maximum logit requires approximately $1 \\times 10^6$ FLOPs for $K = 10^6$. This amounts to only: $0.001 \\text{ GFLOPs}$. By comparison, a single forward pass of ResNet-50 on a 224×224 input image requires about **4 GFLOPs**. Therefore, it adds **virtually no measurable runtime overhead**.\n\n> **Q1**: How does MaxSup scale when the number of classes K is extremely large? \n\nThis is not an issue for MaxSup, please see our response to **W5**.\n\n> **Q2**: If the training set contains mislabeled examples, does MaxSup inadvertently amplify noise? \n\nThank you for the thoughtful question. Since MaxSup penalizes overconfidence in the top-1 prediction **regardless of the ground-truth label**, it does not specifically amplify label noise. In fact, by discouraging overly confident predictions, MaxSup can potentially reduce the harmful impact of mislabeled examples. \n\n> **Q3**: MaxSup with teacher–student distillation frameworks. \n\nFollowing [21], we evaluated the effectiveness of distillation from ResNet-50 to AlexNet on CIFAR-10. Both teacher and student models are trained for 100 epochs using the Adam optimizer with a fixed learning rate of 0.001. The distillation process is conducted with a fixed temperature of 1.0 and an interpolation factor α of 0.1 across all settings:\n\n|                  | Teacher CE | Teacher LS | Teacher MS | Student Baseline | Student from CE | Student from LS | Student from MS |\n|------------------|------------|------------|------------|------------------|------------------|------------------|------------------|\n| **Score**        | 88.76      | 88.96      | 89.47      | 76.66            | 77.95            | 77.41            | 77.84            |\n\nAlthough the student distilled from the teacher using MaxSup (MS) does not outperform the student distilled with cross-entropy (CE), it surpasses the performance of the student distilled with label smoothing (LS). This result aligns with Table 2, which shows that MaxSup effectively mitigates the variation collapse issue.\n\n> **L1**: Effectiveness in NLP, multimodal tasks, or reinforcement learning remains untested. \n\nWe acknowledge the reviewer’s point and agree that a full exploration across domains is valuable future work. While our focus is on computer vision due to the well-documented issues of label smoothing in this domain, we provide preliminary evidence that MaxSup also generalizes to **natural language processing**. Specifically, we train a 12-layer Transformer [1] on the IWSLT 2014 German to English dataset, following the training setup of fairseq repository. The detokenized SacreBLEU scores of 3 runs are compared below. While this enhancement may not appear substantial, it likely stems from the constraints of downstream tasks. Nevertheless, the improvement is statistically significant, as it exceeds the standard deviation.\n\n**Table: Comparison of Label Smoothing and MaxSup on IWSLT 2014 German to English Dataset**\n\n| Model                       | Param. | Method           | BLEU Score     |\n|----------------------------|--------|------------------|----------------|\n| Transformer [1*] | 38M    | Label Smoothing  | 34.4 ± 0.07    |\n|                            |        | MaxSup           | 34.6 ± 0.09    |\n\n[1*] Vaswani, Ashish, et al. \"Attention is all you need.\" Advances in neural information processing systems 30 (2017).\n\n> **L2**: No formal study of training dynamics or convergence rates under MaxSup.\n\nWe appreciate the insightful comment. As briefly mentioned in our Limitation section, Guo et al. [9] demonstrate that label smoothing (LS) accelerates convergence through a conditioning number analysis. A similar approach could be applied to analyze the training dynamics of MaxSup. However, this lies beyond the current scope and is left as a promising direction for future work.\n\n> **L3**: Robustness to extreme label noise, long-tail distributions, adversarial attacks, or online learning scenarios is not assessed.\n\nMaxSup applies the intended regularization when the model makes incorrect predictions, i.e., it consistently suppresses the max logit position even when it is not the class label position. This is independent of the assumption of label noise, or long-tail distributions. For the results on imbalanced dataset and dataset with noisy labels, please refer to our response to **w2**.\n\n> **Paper Formatting Concerns**.\n\nWe appreciate the reviewer's suggestion and are glad to improve this aspect in our revised version."}, {"review_id": "AgbQCllbyq", "reviewer": "Reviewer_1zdf", "summary": "The authors propose a simple modification to the widespread label smoothing regulariser: penalise the max logit rather than the ground truth logit, and term it MaxSup. This design choice is based on the observation that label smoothing does not penalise the logit of the prediction when the model is incorrect during training, but rather the logit of the ground truth label. \n\nThey demonstrate visually and quantitatively that MaxSup improves the inter-class separability of features, whilst preserving intra-class variation, allowing for improved classification accuracy without sacrificing downstream transferability of features.\n\nFinally, they demonstrate modest but consistent improvements in performance over label smoothing over a wide range of experiments.", "questions": "This submission appears to be a revised version of a paper I extensively reviewed previously. The authors had/have addressed earlier concerns effectively, and I am generally satisfied with the current manuscript.\n\nI would like, however, my one concern about discussing related work to be addressed. The authors should also add the underbraces from Eq. 6 to Eq. 2.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths \n- The proposed approach is well motivated by the analysis in Section 3.1. Intuitively, regularising the ground truth logit when the model is wrong is not the best choice, and this is well argued and presented.\n- MaxSup itself is a simple, slot-in replacement for label smoothing. This is a big positive for me, as it allows for easier adoption by practitioners, that don't have to worry about implementing a complex system that may end up being fragile to hyperparameter tuning. \n- Beyond the simplicity of the approach and its motivation, the wide range of experiments supports the idea that MaxSup will give *consistent/reliable* improvements over label smoothing (compared to more complex approaches that may give more impressive numbers on limited experimental setups, but often exhibit fragility when generalised to a broader range of tasks).\n- Experiments on inter-class separability and intra-class variation of features provide additional insight on how MaxSup behaves.\n- The visualisations supplement the quantitative results, providing a better intuition for how MaxSup improves over label smoothing. \n\n## Weaknesses\n- The authors do not adequately address/discuss concurrent work [1]. Although it is briefly mentioned in L150, the fact that it performs a similar training loss analysis for the (different) task of selective classification would best be discussed in Sec. 2.2 Studies on Label Smoothing.\n\n[1] Xia et al. Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix *It ICLR 2025*", "was_revised": true, "final_justification": "I was already mostly satisfied with this submission pre-rebuttal and the authors addressed my only concern about better referencing concurrent work.", "rebuttal": "We are grateful for the reviewer’s thoughtful and constructive feedback. We have carefully considered and incorporated the suggestion into our revision, as detailed below.\n\n> **W1:** The authors do not adequately address/discuss concurrent work [1]. Although it is briefly mentioned in L150, the fact that it performs a similar training loss analysis for the (different) task of selective classification would best be discussed in Sec. 2.2 Studies on Label Smoothing.\n\nWe thank the reviewer for pointing this out and are happy to expand our discussion of [1*] in Section 2.2. While [1*] does conduct a related gradient analysis of the training loss, it focuses specifically on the setting of selective classification, and examines a **posthoc** logit normalization technique to mitigate confidence calibration issues. However, this approach addresses only the overconfidence problem of label smoothing (LS), without tackling representation collapse and improving classification accuracy.\n\nIn contrast, our work presents a logit-level reformulation of LS that provides a deeper theoretical understanding of why LS amplifies errors and leads to representation collapse. This reformulation motivates MaxSup, a principled and effective alternative that directly regularizes the classifier during training, improves classification accuracy, and resolves the representation collapse issue, which is critical for feature learning and downstream generalization.\n\nWe will integrate this expanded comparison into the related work section in the revised version.\n\n\n> **Q1**: The authors should also add the underbraces from Eq. 6 to Eq. 2.\n\nWe are glad to incorporate the reviewer's suggestion into our revised version."}]

Author rebuttal:

Meta-review: The paper received positive reviews, two accepts, and two borderline accepts. Many of the uncertainty calibration techniques use label smoothing and similar probability reshaping ideas. Hence, getting an understanding into the behaviour of such techniques and mitigation of shortcomings could be interesting for the community. AC notes that atleast one of the borderline accept is due to another paper in ICLR’25. AC had a brief look at the ICLR paper and found that though the two are focussed on a similar problem, the insights and mitigation seemed different. 

Label smoothing, and the broader set of output probability reshaping, are techniques already getting used in variety of applications and hence any further insight into its functioning and failure points should be of wider interest to the community. The proposed technique looks a simple and elegant way of doing it, and hence the AC recommends accepting the paper. The AC is also happy to recommend the paper for oral presentation.
Decision: Accept (oral)