Title: MaxSup: Overcoming Representation Collapse in Label Smoothing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Label Smoothing; Representation Collapse; Loss Decomposition; Error Amplification; Regularization
Abstract: Label Smoothing (LS) is widely adopted to reduce overconfidence in neural network predictions and improve generalization. Despite these benefits, recent studies reveal two critical issues with LS. First, LS induces overconfidence in misclassified samples. Second, it compacts feature representations into overly tight clusters, diluting intra-class diversity, although the precise cause of this phenomenon remained elusive. In this paper, we analytically decompose the LS-induced loss, exposing two key terms: (i) a regularization term that dampens overconfidence only when the prediction is correct, and (ii) an error-amplification term that arises under misclassifications. This latter term compels the network to reinforce incorrect predictions with undue certainty, exacerbating representation collapse. To address these shortcomings, we propose Max Suppression (MaxSup), which applies uniform regularization to both correct and incorrect predictions by penalizing the top-1 logit rather than the ground-truth logit. Through extensive feature-space analyses, we show that MaxSup restores intra-class variation and sharpens inter-class boundaries. Experiments on large-scale image classification and multiple downstream tasks confirm that MaxSup is a more robust alternative to LS.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a simple modification to the widespread label smoothing regulariser: penalise the max logit rather than the ground truth logit, and term it MaxSup. This design choice is based on the observation that label smoothing does not penalise the logit of the prediction when the model is incorrect during training, but rather the logit of the ground truth label. \n\nThey demonstrate visually and quantitatively that MaxSup improves the inter-class separability of features, whilst preserving intra-class variation, allowing for improved classification accuracy without sacrificing downstream transferability of features.\n\nFinally, they demonstrate modest but consistent improvements in performance over label smoothing over a wide range of experiments.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n- The proposed approach is well motivated by the analysis in Section 3.1. Intuitively, regularising the ground truth logit when the model is wrong is not the best choice, and this is well argued and presented.\n- MaxSup itself is a simple, slot-in replacement for label smoothing. This is a big positive for me, as it allows for easier adoption by practitioners, that don't have to worry about implementing a complex system that may end up being fragile to hyperparameter tuning. \n- Beyond the simplicity of the approach and its motivation, the wide range of experiments supports the idea that MaxSup will give *consistent/reliable* improvements over label smoothing (compared to more complex approaches that may give more impressive numbers on limited experimental setups, but often exhibit fragility when generalised to a broader range of tasks).\n- Experiments on inter-class separability and intra-class variation of features provide additional insight on how MaxSup behaves.\n- The visualisations supplement the quantitative results, providing a better intuition for how MaxSup improves over label smoothing. \n\n## Weaknesses\n- The authors do not adequately address/discuss concurrent work [1]. Although it is briefly mentioned in L150, the fact that it performs a similar training loss analysis for the (different) task of selective classification would best be discussed in Sec. 2.2 Studies on Label Smoothing.\n\n[1] Xia et al. Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix *It ICLR 2025*\n\nQUESTIONS:\nThis submission appears to be a revised version of a paper I extensively reviewed previously. The authors had/have addressed earlier concerns effectively, and I am generally satisfied with the current manuscript.\n\nI would like, however, my one concern about discussing related work to be addressed. The authors should also add the underbraces from Eq. 6 to Eq. 2.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies two critical drawbacks of conventional label smoothing (LS): (1) error amplification—incorrect predictions become overconfident, and (2) representation collapse—within-class feature diversity is overly compressed. By decomposing the LS loss at the logit level, the authors show that the standard LS penalty contains both a “regularization term” (which only activates on correct predictions) and an “error-amplification term” (which activates on mispredictions). To remedy this, they propose Max Suppression (MaxSup): instead of penalizing the ground-truth logit, the loss penalizes the current maximum logit. This simple modification ensures that both correct and incorrect predictions are regularized evenly. Extensive experiments on ImageNet classification, downstream segmentation and fine-tuning tasks, and Grad-CAM visualizations demonstrate that MaxSup consistently improves top-1 accuracy (by 0.4–1.0%), preserves within-class diversity, and boosts segmentation mIoU compared to LS and its variants.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The logit-level decomposition cleanly separates the beneficial and harmful components of LS, revealing an often-overlooked error amplification effect.\n\n- MaxSup requires only a one-line change to the loss function and plugs directly into existing training pipelines without additional architecture or hyperparameter complexity.\n\n- Results cover multiple backbones (ResNet variants, MobileNetV2, DeiT-S), diverse tasks (ImageNet classification, CIFAR linear probes, ADE20K segmentation, fine-grained recognition), and report means ± standard deviations, supporting strong reproducibility.\n\n- Metrics such as within-class variance, R² transfer accuracy, and Grad-CAM maps all indicate that MaxSup maintains richer intra-class representations and focuses more accurately on relevant image regions.\n\n- The appendix and NeurIPS checklist provide full training details, hyperparameter schedules, and a link to code, facilitating community adoption.\n\n\nWeaknesses:\n\n- In the introduction section of the paper, it is stated that the left part of Figure 1 shows a feature collapse. However, in reality, MaxSup also fails to learn class 282 properly, and the features learned for this class are completely scattered—performing even worse than the result of label smoothing.\n\n- Experiments do not cover scenarios with high label noise, severe class imbalance, or adversarial examples—settings where LS variants often differ markedly.\n\n- Evaluation is largely confined to LS variants and logit penalties. It would be stronger to compare against related techniques such as self-knowledge distillation, label perturbation, or mixup-based regularizers.\n\n- Although the authors discuss the weight α schedule in the appendix, a systematic sensitivity analysis across a wide α range (e.g., α→0 or α→1 extremes) is missing.\n\n- MaxSup requires identifying the maximum logit for each example. In tasks with extremely large label sets (e.g., million-class classification or retrieval), the computational and memory overhead of that max operation may become nontrivial.\n\nQUESTIONS:\n- How does MaxSup scale when the number of classes K is extremely large (e.g., ≥1M)? Have you considered approximate or sparse strategies to find the max logit efficiently?\n\n- If the training set contains mislabeled examples, does MaxSup inadvertently amplify noise? Would a confidence-aware or adaptive suppression strategy help?\n\n- Have you tried integrating MaxSup with teacher–student distillation frameworks? How does it affect the quality of distilled student models?\n\nLIMITATIONS:\n- Evaluated only on vision classification and segmentation; its effectiveness in NLP, multimodal tasks, or reinforcement learning remains untested.\n\n- The theoretical analysis is limited to static logit decomposition; there is no formal study of training dynamics or convergence rates under MaxSup.\n\n- Robustness to extreme label noise, long-tail distributions, adversarial attacks, or online learning scenarios is not assessed.\n\n- Don't add too many underlines, as it affects the overall aesthetics of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies and addresses shortcomings of Label Smoothing (LS), a widely adopted regularization technique. The authors analytically decompose the LS-induced loss, revealing two key components: a beneficial regularization term and a detrimental \"error-amplification\" term that leads to overconfidence in misclassified samples and feature representation collapse. To overcome these issues, the paper proposes Max Suppression (MaxSup). MaxSup applies uniform regularization by penalizing the top-1 logit rather than the ground-truth logit, eliminating the error amplification effect. Through extensive experiments on large-scale image classification (ImageNet) , semantic segmentation , and downstream tasks, MaxSup is shown to be a more robust alternative to LS, consistently reducing overconfidence, preserving richer feature representations, restoring intra-class variation, and sharpening inter-class boundaries.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is well-written and organized.\n- The proposed method, MaxSup, is simple and introduces negligible computational overhead, making it easily integratable into existing training pipelines.\n- The paper provides comprehensive experimental results across various architectures (ResNet, MobileNetV2, DeiT-S) and tasks (ImageNet classification, fine-grained classification, semantic segmentation, transfer learning). MaxSup consistently outperforms LS and its variants in terms of top-1 accuracy, preserves intra-class diversity , and improves transferability.\n\n**Weaknesses**\n\n- Indeed, the novelty is quite limited as the issues with LS on misclassified samples have been considered extensively in the prior work [1]. The authors should at least empirically compare this work.\n- While the analytical decomposition of LS is detailed, the theoretical implications of penalizing the top-1 logit beyond its direct effect on error amplification could be further explored. For instance, how does it influence the geometry of the learned feature space in a deeper mathematical sense, beyond the empirical observations of preserving intra-class diversity? At the moment, the authors mainly provide some intuitions and empirical evidence. Although the authors provide some analysis in experiments, they are still complicated and consist of many compounded factors to really confirm the intuitions. It would be great if the authors could provide some simple, controlled experiments on even synthetic data.\n- The paper isolates MaxSup's contribution by disabling CutMix and Mixup. A discussion or additional experiments on how MaxSup interacts when combined with other common regularization techniques and data augmentations (which are often used in practice) could provide more practical insights.\n\nRef:\n[1] Xia et al. Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It. ICLR 2025\n\nQUESTIONS:\nThe paper mentions that both MaxSup and standard LS employ a linearly increasing $\\alpha$-scheduler for stability. While the robustness to different schedules is noted, a discussion on optimal strategies for dynamically adjusting $\\alpha$ (similar to adaptive LS variants) specifically for MaxSup could be valuable. Did the authors observe significantly different results when not using $\\alpha$-scheduler. Recently, [2] found that coupling label smoothing and data mollification through schedulers are beneficial. The authors could compare this recent method to really see the effect of the $\\alpha$-scheduler and to disentangle its effect from the main proposed method.\n\nRef:\n[2] Heinonen et al. Robust Classification by Coupling Data Mollification with Label Smoothing. AISTATS 2025.\n\nLIMITATIONS:\nPlease see the Weaknesses. Overall, the main limitations are two fold: (1) the novelty is limited compared to the recent work in the literature, (2) the analysis is mainly based on intuitions and complicated empirical evidence.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper analyzed the shortcomings of Label Smoothing (newly denoted as error amplification). Building on the foundation, the authors proposed Max Suppression (MaxSup), a modified label smoothing that penalizes only the top-1 logit rather than the ground-truth logit

STRENGTHS AND WEAKNESSES:
**Strengths**

**S1.** This paper figured out the problem of the commonly used method in this field, and developed a simple yet effective method to solve the discovered problem

**S2.** To overcome the simplicity of the proposed method, the authors thoroughly validated the proposed method in various perspectives, such as theorems and proofs, ablation study, gradient analysis, and so on

**S3.** The experimental environment was set up to verify the superiority and versatility of the proposed method sufficiently, and the experimental results were also well analyzed

**Weaknesses**

There is no weakness in the paper that would justify its rejection

QUESTIONS:
**Q1.** The proposed method operates effectively when the CE with Hard Labels decreases sufficiently to recover $z\_{gt}$ from $z\_{max}$ for incorrect samples. How about an imbalanced dataset or out-of-distribution? In those cases, $\mathcal{L}\_{MaxSup}$ would not be effective to minor classes

**Q2.** How about integrating the proposed method into other label smoothing methods? It would be synergized enough to increase the model performance

---
**Things to improve the paper that did not impact the score**

* [line 237, 239] wrong subscript ($\bar{d}$within)

* [line 269, 272] missing white space (momentum0.9, epochs60)

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03