For Better or For Worse? Learning Minimum Variance Features With Label Augmentation
Authors: Muthu Chidambaram
Rong Ge
Venue: ICLR 2025 Poster (2025)
Keywords: feature learning
mixup
label smoothing
spurious correlations
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Mixup and label smoothing are biased towards learning low variance features in the data, which is empirically verified by looking at the variance of activations in trained neural networks..

Abstract: Data augmentation has been pivotal in successfully training deep learning models on classification tasks over the past decade. An important subclass of data augmentation techniques - which includes both label smoothing and Mixup - involves modifying not only the input data but also the input label during model training. In this work, we analyze the role played by the label augmentation aspect of such methods. We first prove that linear models on binary classification data trained with label augmentation learn only the minimum variance features in the data, while standard training (which includes weight decay) can learn higher variance features. We then use our techniques to show that even for nonlinear models and general data distributions, the label smoothing and Mixup losses are lower bounded by a function of the model output variance. Lastly, we demonstrate empirically that this aspect of label smoothing and Mixup can be a positive and a negative. On the one hand, we show that the strong performance of label smoothing and Mixup on image classification benchmarks is correlated with learning low variance hidden representations. On the other hand, we show that Mixup and label smoothing can be more susceptible to low variance spurious correlations in the training data.

Published as a conference paper at ICLR 2025
FOR BETTER OR FOR WORSE? LEARNING MINIMUM VARIANCE FEATURES WITH LABEL AUGMENTATION
Muthu Chidambaram & Rong Ge Department of Computer Science, Duke University
ABSTRACT
Data augmentation has been pivotal in successfully training deep learning models on classification tasks over the past decade. An important subclass of data aug- mentation techniques which includes both label smoothing and Mixup- involves modifying not only the input data but also the input label during model training. In this work, we analyze the role played by the label augmentation aspect of such methods. We first prove that linear models binary classification data trained with label augmentation learn only the minimum variance features in the data, while standard training (which includes weight decay) can learn higher variance features. We then use our techniques to show that even for nonlinear models and general data distributions, the label smoothing and Mixup losses lower bounded by a function of the model output variance. Lastly, we demonstrate empirically that this aspect of label smoothing and Mixup can be a positive and a negative. On the one hand, we show that the strong performance of label smoothing and Mixup on image classification benchmarks correlated with learning low variance hidden representations. On the other hand, we show that Mixup and label smoothing can more susceptible to low variance spurious correlations in the training data.
INTRODUCTION
The training and fine-tuning procedures for current state-of-the-ar (SOTA) computer vision models rely number of different data augmentation schemes applied in tandem (Yu et al. [022 Wortsman eet al., 2022] Dehghani et al., 2023). While some of these methods involve only transtormations to the input training data such random crops and rotations (Cubuk et al. 2019) non-trivial subset of them also apply transformations to the input training label.
Perhaps the two most widely applied data augmentation methods in this subcategory are label smoothing (Szegedy et al. 2015) and Mixup (Zhang et al. (018). Label smoothing replaces the one-hot encoded labels in the training data with smoothed out labels that assign non-zero probability to every possible class (see Section 2 for a formal definition). Mixup similarly smooths out the training labels, but does so via introducing random convex combinations of data points and their labels. As result, Mixup modifies not only the training labels but also the training inputs. The general principles of label smoothing and Mixup have been extended to several variants, such as Structural Label Smoothing el. 2020). Adaptive Label Smoothing (Wang al. (021). Manifold Mixup (Verma et al. 2019). CutMix Yun et al. 2019). PuzzleMix (Kim et al. 2020). SaliencyMix (Uddin al. 2020), AutoMix
(Liu al. 2021). and Noisy Feature Mixup (Lim al. 2022).
Due to the success of label smoothing and Mixup-based approaches, important question on the theoretical side has been understanding when and why these data augmentations improve model performance. Towards that end, several recent works have studied this problem from the perspectives of regularization (Guo et al.] 2019] (009//Coo Carratino et al.) 2020: Lukasik et al.] 2020 Chidambaram et al.
2021), ), adversarial robustness (Zhang al.] 2020 calibration (Zhang et al. 2021 Chidambaram &
Ge| 2023), feature learning (Chidambaram el al. 2023 You et al. 2023). and sample complexity (Oh
Yun 2023).
Although the connection between the label smoothing and Mixup losses has been noted in both theory and practice (Carratino et al.) 2020), there has not been (to the best of our knowledge) a unifying theoretical perspective why the models trained using these losses exhibit similar behavior. In this paper, we aim to provide such perspective by extending the ideas from previous feature learning analyses to the aspect shared in common between both Mixup and label smoothing: label

---

Published as a conference paper at ICLR 2025
augmentation. Our analysis shows that both Mixup and label smoothing hone in on low variance features in the data, and we demonstrate empirically that this phenomenon occurs in both synthetic settings with spuriously correlated features well as in the training of standard deep learning models for image classification benchmarks.
1.1
MAIN CONTRIBUTIONS
The main message of our paper can be summarized as:
In data distributions where there are low variance and high variance latent features, both label smoothing and Mixup incentivize learning models that correlate more strongly with the low variance features.
We prove this message concretely in Section 3.1 where we characterize linear optimizers of the label smoothing and Mixup losses data in which some dimensions have lower variance than others. In Section 3.2] we also prove weaker analogues of our linear results in the context of general models and multi-class distributions; namely, we show that optimizing the label smoothing and Mixup losses requires decreasing model output variance in way that is not necessarily true for empirical risk minimization (ERM) combined with weight decay.
We verify our theory empirically in Section 4 where show that label smoothing and Mixup do indeed learn lower variance hidden representations (corroborating ideas from the literature on neural collapse (Papyan et al.) (2020) and that this lower variance property correlates with better generalization performance. We hypothesize that, for standard benchmarks, there exist low variance latent features that generalize well as opposed to high-variance, noisy features, and we expect that directly investigating such features would be a fruitful line of future work. However, we also point out that our key message does not directly imply better generalization performance we show via synthetic experiments in Section 4.2 that it is possible for the low variance features to actually be spuriously correlated with the targets (e.g. fixed backgrounds).
1.2 RELATED WORK
Label Smoothing, Since being introduced by Szegedy et al. (2015). ). label smoothing continues to be used train SOTA vision (Wortsman et al. 2022] Liu et al. | [2022), translation (Vaswani et al.) 2017, Team et al., 2022), and multi-modal models Yu et al. (022). Attempts to understand when and why label smoothing is effective can be traced back to Pereyra et al. (2017) and Müller et al.) (2020), which respectively relate label smoothing to entropy regularization and show that it can lead to more closely clustered learned representations. Müller et al. al.] (2020) also show that label smoothing can improve calibration but hurt distillation performance; Zhu et al. 2023 Xia al. (2024) further show that these improvements in calibration do not translate to better selective classification.
On the theoretical side,Lukkee Lukasik et al. (2020) study the relationship between label smoothing and loss correction techniques used to handle le label noise, and show that label smoothing can be effective for mitigating label noise. [iu| 2021) extends this line of work by analyzing how label smoothing can outperform loss correction in the context of memorization of noisy labels. |Wei et al. (2022) further show that negative label smoothing can outperform traditional label smoothing in the presence of label noise. Xu et al. (2020) provide alternative theoretical perspective, showing that label smoothing can improve convergence of stochastic gradient descent.
Mixup. Similar to label smoothing, Mixup (Zhang et al. 2018) and its aforementioned variants have also played an important role in the training of SOTA vision (Wortsman al. [022 Liu et al., 2022), text classification (Sun et al., 2020), translation (Li et al. 2021), and multi-moda models (Hao et al. 2023). often being applied alongside label smoothing. Initial work understanding Mixup studied from the perspective of introducing data-dependent regularization term to the empirical risk (Carratino al. 2020 Zhang al. 2020 Park et al.) 2022), with Zhang et al. (2020) and Park et al. 2022) showing that this regularization effect can lead to improved Rademacher complexity.
On the other hand, Guo et al. (2019) and Chidambaram et al. 2022 show that the regularization terms introduced by the Mixup loss can also have negative impact due to the fact that dixup-augmenter points may coincide with existing training data points, leading to models that fail to minimize the original risk. Mixup has also been studied from the perspective of calibration (Thulasidasan et al.) 2019). with theoretical results (Zhang et al. 202T Chidambaram & Ge 2023) showing that Mixup training can prevent miscalibration issues arising from ERM.
2

---

Published as a conference paper at ICLR 2025
Recently. Oh & Yun 2023) studied Mixup in a similar context to work (binary classification with linear models), and showed that Mixup can significantly improve the sample complexity required to learn the optimal classifier when compared to ERM. Most closely related with our results in this paper, Mixup has been studied from feature learning perspective, with Chidambaram et al. (2023) and Zou et al. (2023) both showing that Mixup training can learn multiple features in a generative, multi-view (Allen-Zhu & Li 2021) data model despite ERM failing to do so.
Neural Collapse. Papyan et al. (222)) showed that the last layer activations of trained neural networks collapse to their class means, in such a way that they are maximally separable. Kornblith et al. (202T); (2024) all investigate the interplay between label smoothing and this effect, with Kornblith et al. (202T) showing that label smoothing can lead to more separable last layer representations (although worse linear transfer performance) and et al. (002);
Zhou et al. (2022); Guo
): Guo et al.)
(2024) showing that it can also lead to faster convergence to these collapsed representations. We corroborate these results with our experiments in Section 4.1 where we show that both Mixup and
label smoothing lead to much lower total variance in last layer activations compared to standard cross-entropy.
General Data Augmentation. General data augmentation techniques have been a mainstay in image classification since the rise of deep learning models for such tasks (Krizhessky et al. (012). As
result, there is ever-growing body of theory (Bishopl 1995 Dao al. 2019 Wu et al. 20200]
Hanin & Sun 2021 Rajput et al. 2019; Yang et al. [022 Wang et al. 2022 Chen et al. 2020 Mei
et al.] 2222)p aimed addressing broad classes of augmentations such as those resulting from group actions (i.e. rotations and reflections). Recently, Shen et al. (2022) also studied a class of linear data augmentations from a feature learning perspective, once again using a data model based on the multi-view data of Allen-Zhu & Li (202T). This work is in the same vein as that Zo el al. (2023) and Chidambaram & Ge (2023), although the augmentation considered is not comparable Mixup.
PRELIMINARIES
Notation. Given E N. we use [n] to denote the set 21. n.. For a vector E I2 and a subset S C [d]. we use vy E |s to denote the restriction of x to only those indices in S, and also use my to denote the hh coordinate of x. The same notation also applies to matrices; ie. for a square matrix T we use T's to denote the restriction of both the rows and columns of T to only those dimensions in S. We use La to denote the identity matrix in Rd. Additionally, we use > to denote the partial order over positive definite matrices and I to denote the identity matrix. We use ||-|| to indicate the Fuclidean norm on Rdd For a function g R + Rhm we use g' (x) to denote the 7th coordinate function of g. We use A k-1 to denote the (k: 1)-dimensional probability simplex in RR For probability distribution ा we use supp(T) to denote its support. Additionally, if π corresponds to the joint distribution of two random variables X and Y (i.e. data and label), we use π and TTy to denote the respective marginals, and T and Ty|x-> to denote the regular conditional distributions. We use Σx to denote the covariance matrix of random variable X E Rd.. Lastly, we use Var(X) to denote Tr(Ex) ) for X E Rdd
We consider the k-class classification setting, in which there is ground-truth -ruth data distribution on Rd x [k]] and our goal is to model the conditional distribution Ty using a learned function g. In our main theoretical results, regression model (although we generalize a weak version of these observations to k classes as well), as in this setting can get a clear handle on the features in the data learned by an optimal model with respect to particular loss. For this case, we will assume that π is supported on E x { and that g parameterized by a weight vector w, i.e. (x) = ow z). where o is the sigmoid function. Throughout this work, we will consider the following three families of losses.
will pay particular attention to the case where h: = 2 and g is logistic
Standard cross-entropy with optional weight decay. The canonical cross- cross-entropy (or negative log-likelhhood) objective in the k-class setting is defined as:
{(g) = Exx, log.9 (X))
(2.1)
Here we have not specified a weight decay term, since we have placed no constraints on the structure of g. On the other hand, for linear binary classification, we can define the binary cross-entropy with optional weight decay as (recalling that Y E (+1))
3 - ( = Exx,Y) |- log you (Y +
2 [|]l
(2.2)
3

---

Published as a conference paper at ICLR 2025
where we have defined the loss in terms of the parameter vector When 8 > 0. (.2 has unique minimizer and we will directly analyze its properties. On the other hand, when 3 (no weight decay), this is longer the case but our results will still apply to the common case of minimizing (2.2 by scaling the max-margin solution, which we define below.
Definition 2.1. [Max-Margin Solution] The max-margin solution w" with respect is defined as:
w" argmin ||w||
wERd
s.t. y (w, 2) 2 for TT..e.
(2.3)
Label smoothing. The cross-entropy defined in (.1 treats the reference distribution that we compare g to as a point mass on the class Y. On the other hand, the label-smoothed cross-entropy is obtained by instead treating the reference distribution as a mixture of point mass on Y and the uniform distribution over [k:]. Namely, the label-smoothed loss with mixing hyperparameter [0, is defined to be:
(g) -x (1 a) logg" (xogg"(xx=0
(X)
(2.4)
And the corresponding binary version is (once again, Y E {+
(w) = -E Exyyy> (1 (1- ()) log In (YX) + a log Iw ( (-YX)
(2.5)
2
Mixup. Similar to label smoothing, Mixup also augments the reference distribution from being point mass on Y to being mixture. However, Mixup also augments the input data as well. In particular, Mixup considers convex combinations of two pairs of points Y1) and (X2, Y2), with mixing weight sampled from distribution D> (which is hyperparameter) whose support is contained in [0, 1]. To simplify notation, we will use X1.n and Y1.x denote multiple inputs Xi,.., and their corresponding labels Y1, .... Yn. and additionally introduce function h defined as:
h(A, 9, X1.,, Y1.2) = A Algg.' (Zx) + (1 - log g (2x)
(2.6)
where Zx AX1 (1 X2
(2.7)
After which we can define the Mixup cross-entropy as:
& (g) Mixi [- g, X12,, Y1:2)]
(2.8)
The corresponding binary version fMIX, (w) of 2.8 is identical except for redefining h to be: h(A, w, 21.2, 21:2) = log g (V Zx) + (1 - A) log 9no (Y2
(2.9)
3 MAIN THEORETICAL RESULTS
All omitted proofs in this section can be found in Section A of the Appendix.
3.1 LINEAR BINARY CLASSIFICATION
We begin first with the binary setting, in which will consider data model where subset of the input dimensions correspond to low variance feature and the complementary dimensions correspond to high variance feature that is more separable. We emphasize that our notion of "feature" here does not correspond to explicit feature vectors that are fixed per class like in prior work, we simply designate subsets of the dimensions as features for simplicity. This data model can be interpreted in multiple ways: depending on the context, we may wish to learn either both of the features present the data or simply hone in on the low variance feature (for example, identifying a stop sign with many different backgrounds).
Although this setting seems simplistic - one may object that the variance differences the input can be handled with suitable www.alliizaioo/w/ttiiiiii the insights from our data model can be applied to learned learuefeettures (i.c. intermediate representations in a deep learning model) where such modifications less straightforward, as we demonstrate in the experiments of Section 4
Our main results in this section show that doing any kind of label augmentation (label smoothing or Mixup) in our low variance/high variance feature setup will lead to model that has only learned the low variance feature, whereas minimizing the binary cross-entropy with non-zero zero weight decay requires learning the high variance feature due its greater separability.
4

---

Published as a conference paper at ICLR 2025
Definition 3.1. [Binary Data Distribution] We consider и to be distribution supported K {+1} where K is a compact subset of IR.. We assume that я is nondegenerate in that it satisfies my (y) > for each E {=1} and that Σx is positive definite, and we also assume E[X] 0. Additionally, designate a subset L S [d] that we refer to as the low variance feature in the data, and we refer to the complement H L as the high variance feature.
For convenience, for a vector Rd we will use U E C to mean that only the dimensions of are non-zero. Definition 3.I treats L and H. as placeholders; this is because our weight decay result insensitive to variance assumptions and only depends on differences in the separability of the dimensions L and H. as we indicate below.
Assumption 3.2. We assume that for every unit vector u² E L, there exists unit vector vr H. and y (z) > y (u", z)) for π-a.c. (z.y).
Assumption 3.2 is strong, but it actually does not imply linear separability, only that the H dimensions in sense better than the L dimensions for classification. We explore simple 2-D distribution illustrating Assumption 3.2 in Appendix C.T
Our first result shows that for (a(w) as defined in 2.2 2p the minimizer w" has a large correlation with the dimensions H. This is of course intuitive given Assumption 3.2. but it is not immediate because the weight decay penalty in (2.2) encourages distributing norm across all of the dimensions
of u (e.g. Σ w, is maximized with respect to the constraint || = by considering uy = 1//Vd).
Theorem 3.3. Let w" be the unique minimizer of ( for B 0 under ι satisfying Assumption 3.2 Then [|wze|| 2 2 1 2 311==11²
Proof Sketch. We can orthogonally decompose the optimal solution w" in terms of unit normal directions u² C and v² H. We show that in this decomposition it must be the case that y (0),x) > y (u", .)) for π-a.c. (x, y), and then we can claim that w must have greater weight associated with v² than u", otherwise we can decrease (ww") by moving weight from u to v".
We cannot immediately extend the result of Theorem 3.3 to the binary cross-entropy (w( without weight decay, since there is no unique minimizer of Eg. However, for linear models trained with gradient descent (as is often done in practice), direction to the max-margin solution (Soudry et al. 2018 Ji & Telgarsky 2020). For this case, the proof technique of Theorem 3.3 readily extends and we obtain the following corollary.
well-known that the learned model converges in
Corollary 3.4. wf is the max-margin solution to fo(w), then the result of Theorem 3.3 still holds.
On the other hand, we will now show that once we introduce variance assumptions on and H. this phenomenon does not occur when minimizing the label smoothing and Mixup losses fr.. and (M In both cases, the optimal solutions have arbitrarily small correlation with the dimensions in H as the distribution of Y concentrates For these results, we require the following assumptions (but no longer need Assumption 3.2)
Assumption 3.5. We assume that EY Xc] & and Σyx. واx for some p 0. Here Σyx, denotes the covariance matrix of Vyx restricted to those rows and columns in H.
The first part of Assumption 3.5 just ensures it is possible to obtain good solution using the dimensions in L while the second part codifies the idea of H being a high (at least non-zero) variance feature. Observe that we have made no direct separability assumptions true that as |_xx, > 0 the class-conditional supports are guaranteed to be lincarly separable in C. This means that we can consider settings where both Assumptions 3.2 and 3.5 are true, and in such settings there will be a clear separation between weight decay and label smoothingMMixup. This kind of separation in the learned decision boundaries is visualized in Appendix C.I which provides intuition for the next two results.
the data, although it is
Theorem 3.6. For E (0, and ग satisfying Assumption 3.5 satisfies ||l E(
every minimizer w" of (I),o
In order for the variance of Y X to go we need the classes to be halanced; however, we can change our assumptions he in terms of the conditional variance of X remove this requirement.
5

---

Published as a conference paper at ICLR 2025
Proof Sketch. We show that (1))a(w)) is strongly convex in w YX. We then use Jensen's inequality to get a lower bound on the loss in terms of this quantity, and show that this lower bound is achievable using only w L as the variance of Y Xy decreases. Then using lower bound on the Jensen gap (Lemma A. and Assumption dimensions remains bounded away from this lower bound.
3.5 we show that any solution that is sufficiently non-zero in the H
Theorem 3.7. For any symmetric Dx that is not a point mass on 0 or and x satisfying Assumption every minimizer w' of EMTX, D2 (w) satisfies ||w < 0(IExxx,ll))
3.5
Proof Sketch. Mixup differs from label smoothing in that we need to condition on À and then show strong convexity of the conditional loss in terms of w Za.- We then get lower bound similar to the label smoothing case, but it is no longer immediate that there exists stationary point w E L that minimizes this lower bound. We prove the existence of such stationary point by considering limiting behavior of the gradient of the loss as wz tends to oo or -oc in each component, after which the rest of the proof follows the label smoothing proof.
3.2 GENERAL MULTI- CLASS CLASSIFICATION
Our results so far have demonstrated separation between standard training (ERM weight decay) and label augmentation (label smoothing and Mixup) in the linear binary classification setting, without explicitly having assume linear separability. The benefit of the linear binary case is that in this case the learning problems are convex in the weight vector w, so we can directly discuss properties of the optimal solutions instead of worrying about optimization dynamics.
Of course, this is no longer true when we pass to nonlinear models, and standard model choices (e.g. neural networks) make it so that it is no longer easy to prove that model has "learned" either Xc or Xx without explicitly analyzing some choice of optimization algorithm (i.e. gradient descent). However, our observations do translate the outputs of any model.
It is obvious that the model output variance should go to zero for any model that achieves the global optimum of the label augmentation losses we have discussed; indeed, this just corresponds to predicting the correct labels (a, 1 with label smoothing and X, 1 - À for every À for Mixup) with probability 1. On the other hand, is less clear that we can make a quantitative statement regarding how bad the loss could be given a certain amount of variance in the model output.
By lifting the techniques the previous subsection, we can prove such quantitative results for both label smoothing and Mixup in the general multi-class setting. The general data distribution we consider for these results is as follows.
Definition 3.8. [Multi-Class Data Distribution] We consider x to be distribution supported on a compact subset of Idd and k > We assume only that π satisfies Ty (и) > for every y and Σx is positive definite.
B x [k] where
We make virtually no assumptions on ı for these results because we will only be proving general lower bounds, which are weaker than the claims regarding the optimal solutions of the previous subsection. For æ as in Definition 3.8 the label smoothing and Mixup results are as follows.
Proposition 3.9. For 0 and any g Rd A k-1, letting OPTLS,a denote the minimum of L.S.ar we have for a universal constant C' > 0:
(1)0(0))) OPT.S. C
Σ TY (B(()) IY v))
(3.1)
Proposition 3.10. For any Dx that is not point mass O or and any g .Rd 4-1 letting MIXMIX, denote the minimum of ExIx,Ds> we have for a universal constant C > 0 that:
EMIX, (x) OPTMIX.D. DA +0 +0 Σ
TY(w))TY (J2) [Var (Blo(Za) 9ı U2, ADJ (3.2)
9/1
The proofs for both Propositions 3.9 follow the same structure; we show that after appropriate conditioning, both losses be broken up into sum of strongly convex conditional losses. The lower bounds in both results show that in order to make progress with respect to the label smoothing or Mixup losses, training algorithm needs push model outputs to be low variance.

---

Published as a conference paper at ICLR 2025
Remark 3.11. We do not prove analogous result to Theorem 3.3 in the general setting because Propositions 3.9 and 3.10 operate directly on the model outputs gX)), and it is not clear how to translate weight norm constraint to this setting without explicitly parameterizing g. Intuitively, however, with a weight norm constraint on it may no longer be optimal to have zero variance predictions, since such predictions may require very large weights depending on the data distribution.
Comparisons to existing results. We are not aware of any existing results on feature learning for label smoothing; prior theoretical work largely focuses on the relationship between label smoothing and learning under label noise, which is orthogonal to the perspective
take in this paper. Although
feature learning results exist for Mixup (Chidambaram et al., 2023 Zou et al. 2023), the pre-existing results constrained the case of mixing using point mass distributions and only prove a separation between Mixup and unregularized ERM, whereas our results work for arbitrary symmetric mixing distributions (that don't 't coincide with ERM) and also separate Mixup from ERM with weight decay, This greater generality comes at the cost of considering only linear models for our feature learning results; both Chidambaram al. (023) and Zou et al. (2023) consider the training dynamics of 2- 2-layer neural networks on non-separable data.
Additionally, our results can be viewed as generalizing the observations made by Chidambaram et al. (2023), which were that Mixup can learn multiple features in the data when doing so decreases the variance of the learned predictor. In our case, we directly show that Mixup will have much larger correlation with the low variance feature in the data as opposed the high variance feature. Our results also do not contradict the observations Zo et al. (2023). which were that Mixup can learn both a "common" " feature and a "rare" feature the data; in their setup the common/rare features are fixed (zero variance) per class and concatenated with high variance noise. We provide a more detailed review of the settings of Chidambaram et al. (2023) and Zou et al. (2023) in Appendix B
4 EXPERIMENTS
We now address the practical ramifications of our theoretical results. In Section we show that the intermediate representations learned by deep learning models trained with label smoothing and Mixup do indeed exhibit significantly lower variance when compared to those learned using just weight decay, and that these lower variance representations correlate with better performance. We also show, however, that this minimum variance feature learning can be detriment by analyzing spurious correlations in the training data in Section 4.2 We also include synthetic experiments directly verifying our theoretical setting in Appendix
All experiments in this section were conducted 2019) for model implementation. All reported results correspond to means over 5 training runs, and shaded regions in the figures correspond to 1 standard deviation bounds.
single A5000 GPU using PyTorch (Paszke et al.
4.1 LEARNED Low VARIANCE FEATURES IN IMAGE CLASSIFICATION
We first consider image classification on the standard benchmarks of CIFAR-IO and CIFAR- 100 (Krizhessky 2009) using ResNets (He et al., 2015); we show results for ResNet-18 in this section and relegate turther experiments for deeper architectures to Appendix C.3 (they follow the same trends). We compare the performance of training using just ERM weight decay to that of ERM + weight decay combined with label smoothing or Mixup; final test error performance is shown in Figure (a) and (d).
Due to compute constraints, we focus known well-performing settings (Zhang et al.) 2018 Müller ct al, 2020) for weight decay, label smoothing, and Mixup. Namely, we take the weight decay parameter to be 5 10-4 4, the label smoothing a parameter to be 0.1, and the mixing distribution for Mixup to be Beta(1, 1) (uniform distribution). We train all models for 200 epochs using a batch size of 1024, a fixed learning rate of 10-3, and AdamW for optimization (Loshchilov & Hutter 2019). We preprocess the training data to have zero mean and unit variance along each channel, and also include random crop and flip augmentations is standard practice for achieving good performance.
Figure [ shows that adding in label smoothing and Mixup to the baseline of ERM weight decay leads to noticeable performance improvements, which corroborates existing results in the literature. The novel aspect of our results is shown in Figure (b), (c), (d), and (e), where track the mean total variance of the penultimate layer activations and output probabilities of each model on the test data over the course of training. The variance values are computed by first computing the total
7

---

Published as a conference paper at ICLR 2025
(a) CIFAR-10 Test Error
(b) CIFAR-I0 Activation Vari-
(c) CIFAR-I0 Output Variance
ance
WHbF Decay
Mengl Decoey
Secre
Miop
Min.
Miai
(d) CIFAR-100 Test Error
(e) CIFAR-100 Activation Vari- ance
(f) CIFAR-100 100 Output Variance
Figure I: ResNet-18 final test errors, penultimate layer activation variances, and output probability variances on CIFAR-10 and CIFAR-100, Activation variance results are shown starting at epoch 25 as early epochs have larger scale oscillations in the computed variance.
variance of the activations/prohabilltis (i.e. the sum of the variance in each dimension) for each class, and then averaging over all classes. Final test error and variance values are provided in Table [ and in Appendix I
Essentially, we measure the average spread of activations and probabilities across classes, which is similar in spirit to what was done by Müller et al. (2020), although we directly look at variances across all classes whereas they look projected clusters of few specific classes. The most telling results are the model activation variances, since is to an extent expected that per-class variance of model outputs should decrease with improvements in test error (this also corresponds to improved model calibration, which is known consequence of training with label smoothing and Mixup). That being said, we further analyze model output variance in Appendix C.4 and show that label smoothing and Mixup decrease total output variance by a larger extent than what can be explained by changes in the target class prediction variance alone.
Overall, our results show that adding either label smoothing or Mixup to the baseline of just weight decay leads to significant decreases in the activation variances, adding credence to the idea that both methods learn low variance features. We note, however, that our results only establish a correlation between this low variance property and better test performance in-depth empirical study to assess a causal relationship between this kind of feature learning and generalization, which we think would be fruitful avenue for future work.
would require significantly more
4.2
Low VARIANCE SPURIOUS CORRELATIONS
We now demonstrate that honing in on low variance features can also be harmful performance via binary classification and multi-class classification tasks in which the training data is modified to have spurious correlations with the target. The introduced spurious correlations are much lower magnitude than the rest of the data, and in that sense they intuitively satisfy both Assumptions 3.2 and 3.5
4.2.1 BINARY CLASSIFICATION WITH PERTURBED TRAINING DATA
For binary classification task, consider reductions of the CIFAR benchmarks binary classification by fixing two classes from each dataset as the positive and negative classes, and replacing their original labels with the labels +1 and -1 respectively. Our experiments are not
8

---

Published as a conference paper at ICLR 2025
CHARSO
0PA039
Snoethng
(a) Weight Decay
(b) Label Smoothing
(c) Mixup
Figure Logistic regression final test errors for various hyperparameter settings on the binary classification versions of CIFAR-1O 10 and CIFAR-100 from Section 4.2.1
sensitive to the choice binary reduction, and for the experiments in this section we fix the -1 class to be class from the original data and the +1 class to be class 1 from the original data.
We preprocess the training data to have mean zero and variance one along every image channel, but do use other augmentations or perform feature extraction using pretrained models. We then "adversarially" modify the training data such that the first value in the tensor representation of each training input is replaced by Ty with y = 0.1 (7 here just needs to be small, we verified the results for y 10 up to 7 0.1). This ensures that the training data is linearly separable in the first dimension of the data, but learning this first dimension requires having larger weight norm due to the goal is to determine whether models trained on the modified training data can learn more than just the single identifying dimension.
scaling by We leave the test data unchanged
We then train logistic regression models both the reduced CIFAR-10 and CIFAR-100 tasks across range of settings for weight decay, label smoothing, and Mixup. We consider 20 uniformly spaced values in [0, 0.1] for the weight decay À parameter and in [0, 0. 0.75] for the label smoothing a parameter, where the upper bound for the label smoothing parameter is obtained from the experiments of Müller al. (2020). For Mixup, we fix the mixing distribution to be the canonical choice of Beta(o, introduced by Zhang et al. (2018) and consider 20 uniformly spaced values in [0, 8] (with corresponding to ERM), which effectively covers the range of Mixup hyperparameter values used in practice. Other hyperparameters the same as before, except
0
learning rate of 5 x 10-3
The results across hyperparameter values are shown in Figure Both the label smoothing and Mixup models have high test error for all settings while weight decay achieves significantly lower error for all A > 0. ERM also has high test error; in this case we differ from the setting of implicit bias results due training with Adam and likely training for the period required for convergence to the margin solution. Furthermore, these results are insensitive to introducing a small amount of weight decay to the label smoothing and Mixup models (i.e. 10 -4 as in the previous subsection).
1dee Stadting
Ldt Seedtin)
CIFAR-10
(b) CIFAR-100
Figure 3: Comparison of norm ratio between first dimension (synthetically modified in the training data) and remaining dimensions (left unchanged) for trained logistic regression weight vector for the 20 different hyperparameter settings of weight decay, label smoothing, and Mixup.
Our results correspond to label smoothing and Mixup learning to use only the spurious, identifying dimension of the training inputs, expect from our theory since this dimension has

---

Published as a conference paper at ICLR 2025
conditional variance. Indeed, we verify this fact empirically in Figure 3 where plot the ratio | /l/ ((d)/1) || (i.e. the ratio between the norm of the trained model weight vector in the first dimension and the remaining dimensions) for each of the trained models.
4.3 SPURIOUS BACKGROUND CORRELATIONS
3
7
3
4
7
3
4
4
8
1
7
6
8
8
6
4
0
4
8
O
5
9 6
4 7
3
0
D
/
7
3
/
7
2
7
8
2 0
5
5
2
7
6
2
9 5 9
4
0
5
6
0
8
8
0
7
2
2
(a) Colored MNIST
(b) Model Performance
Figure 4: Final model test errors over our hyperparameter sweep for the colored MNIST dataset of Section alongside visualization of samples from the dataset.
For our multi-class analogue to Section 4.2. we consider a similar setup to the one used by Arjovsky al. (2020) to motivate the influential invariant risk minimization framework: namely, we construct colored version of the MNIST dataset in which the background pixels for each class replaced with different colors corresponding to the class labels. Unlike Arjovsky et al. (2020), however, we maintain all 10 classes since our theory from Section 3.2 suggests that observations should generalize to the multi-class setting. The colors used for each class background are permuted between the train and the test data, that model that learns to predict only using background pixels cannot achieve good test accuracy.
The only constraints we place on the background colors are that they are distinct across classes and that their intensities (i.e. values in RGB space) small (it suffices consider values bounded by 16). The latter constraint is not necessary for the failure of label smoothing and Mixup, but is necessary for weight decay succeed since ensures that the higher variance feature (the actual digit in the foreground) is generally more separable (due to larger pixel intensities). Note that although there no variance in the background color conditional on class, there is significant variance in the actual pixels per class as the locations of the background pixels change across data points.
For our model setup, we also follow Arjovsky et al. 2020) and consider simple 2-layer feedforward neural network with ReLU activations and a hidden layer size of 2048, this is sufficient to achieve good performance on MNIST and is a tiny enough model that we can efficiently do the same hyperparameter sweep from Section 4.2. Training details remain the same in Section [4.. I except we only train for 20 epochs this is sufficient for the training loss to roughly converge and greatly expedites the hyperparameter sweep.
Test error results across the different hyperparameter settings for weight decay, label smoothing, and We observe the same phenomena before, even in this non-trivial spurious correlation setting: both the label smoothing and Mixup models have high test error all settings while weight decay achieves significantly lower test errr for all À 0.
Mixup are shown in Figure 4
5 CONCLUSION
In this work, we have shown that label augmentation strategies such as label smoothing and Mixup exhibit a variance minimization effect (both in theory and in practice) that leads to lower variance intermediate representations and model outputs, which then correlate with better test performance on image classification benchmarks. A natural follow-up direction to our results is to investigate whether regularizers for encouraging lower variance features can be implemented directly to improve model performance, which would shed light on whether there is a causal relationship between this phenomenon and improved performance.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Although label smoothing and Mixup are used to train and fine-tune large-scale models, our results concerning them in this work have mostly been theoretical and explanatory. As a result, do not anticipate any direct misuse of our results or any broader harmful impacts.
REPRODUCIBILITY STATEMENT
All proof's of the results in this paper can be found in Appendix A contains the code necessary to generate all figures, with instructions on how to run each experiment.
The supplementary material
ACKNOWLEDGMENTS
Rong Ge and Muthu Chidambaram were supported by NSF Award (CAREER) during the completion of this work. Muthu would like thank Annie Tang for thoughtful feedback during the early stages of this project.
and CCF-1845171
REFERENCES
Zeyuan Allen Phien-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization, 2020.
Chris Bishop. Training with noise is equivalent tikhonov regularization. Neural computation, (I::108-166, 1995.
Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regulariza- tion, 2020.
Shuxiao Chen, Edgar Dobriban, and Jane Lee. A group-theoretic framework for data augmentation. Advances in Neural Information Processing Systems, 33:21321-21333 2020.
Muthu Chidambaram and Rong Ge. On the limitations of temperature scaling for distributions with overlaps, 2023.
Muthu Chidambaram, Xiang Wang, Yuzheng Hu, Chenwei Wu, and Rong Ge. Towards understanding the data dependency of mixup-style training. CoRR, abs/2110.076.1) 2021. URL https
//axxiv.org/bb//21000
Muthu Chidambaram, Xiang Wang, Chenwei Wu, and Rong Ge. Provably learning diverse features in multi-view data with midpoint mixup. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference
Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 3563-5599, PMLR, 2023. URL https:// proceedings ml www.s.ppees/vv000 chi ram23a. html
Ekin D. Cubuk, Barret Zoph. Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with reduced search space, 2019.
Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, and Christopher Ré. A kernel theory of modern data augmentation. In International Conference on Machine Learning, pp. 1528-1537. PMLR, 2019.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evei, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetić. Dustin Tran. Thomas Kipf, Mario Lučić, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters, 2023.

---

Published as a conference paper at ICLR 2025
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out- man-ffld regulariza- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3714-3722, 2019.
Li Guo, Keith Ross, Zifan Zhao, George Andriopoulos, Shuyang Ling, Yufeng Xu, and Zixuan Dong, Cross entropy versus label smoothing: A neural collapse perspective, 2024. URL https
Boris Hanin and Yi Sun. How data augmentation affects optimization for linear regression. Advances in Neural Information Processing Systems, 34, 2021.
Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. Mixgen: A new multi-modal data augmentation, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URLpttp://arxix http://..rrga//////aaaaa/ 03385
Ziwei Ji and Matus Telgarsky. Characterizing the implicit bias via a primal-dual analysis, 2020.
Myug-Hyun uu Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pp. 5275-5285. PMLR, 2020.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features?, 2021. URL httttt:tp//xxiii/ http//aaxwvvraaaaaaaaaaaaa abs/ 2000020100.6022222.....2
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https :ttp://ooopssI::122:::
s/api.semanicccchooaar
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep F. Pereira, C.J. Burges, L. Bottou, and K.Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Asso- ciates, Inc., 2012. URL https: http::///occeeedeeeooeoo neurips cc/paper files/paper/ 2012 20127
convolutional neural networks.
Jicheng Li, Pengzhi Gao, Xuanfu Wu, Yang Feng, Zhongjun He, Hua Wu, and Haifeng Wang. Mixup decoding for diverse machine translation. Marie-Frande Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 312-320, Punta Cana, Dominican Republic, November 2021. www./i/n..iniinn-onnen
Association for Computational Linguistics. doi: org 2021. windiiinnneeminimm 1p. 29
URL
htttps://aclanthot
Weizhi Li, Gautam Dasarathy, and Visar Berisha. Regularization via structural label smoothing, 2020.
Soon Hoe Lim, N. Benjamin Erichson, Francisco Utrera, Winnie Xu, and Michael W. Mahoney, Noisy feature mixup. The Tenth International Conference on Learning Representatiinss ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL htttps://oppereeiiiiii
Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s, 2022.
Zicheng Liu, Siyuan Li, Di Wu, Zhiyuan Chen, Lirong Wu, Jianzhu Guo, and Stan Z. Li, Automix: Unveiling the power of mixup. CoRR, abs/2103.11027, 2021. URL httphtts/////.xv.orrr abs/ 2102.111222
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https
//arxiv.org/all wwt.///1../.111111
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In Proceedings of the 37th International Conference on Machine Learning, ICML'20. 20. JMLR.org, 2020.
12

---

Published as a conference paper at ICLR 2025
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Learning with invariances in random features and kernel models. In Conference on Learning Theory, pp. 3351-3418. PMLR, 2021.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help?, 2020.
Junsoo Oh and Chulhee Yun. Provable benefit of mixup for finding optimal decision boundaries. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 11403-264555 PMLR, 23-29 Jul 2023. URL URL wtt.hhtttppppoooooo
ngs ml www...ppeess/222 whtps////2//oh222eee oh23a html
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):
24652-24663, September 2020. ISSN 1091-6490.
doi: 10.10733/mmss
URLhttp:
//dx. doix.dii.co org.lo.rg111.0 www.:/as..ppp..o..momm 221555011111
Chanwoo Park, Sangdoo Yun, and Sanghyuk Chun. A unified analysis of mixed sample data augmentation: A loss function perspective, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Z. Yang. Zach Vetoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. CoRR, abs/1912.01703, 2019.
URL httpttpp///aa...r///aeaeeeeeeeee
01703
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions, 2017.
Shashank Rajput, Zhili Feng. Zachary Charles, Po-Ling Loh, and Dimitris Papailiopoulos. Does data augmentation lead to positive margin? In International Conference on Machine Learning, pp. 5321-5330. PMLR, 2019.
Ruoqi Shen, Sébastien Bubeck, and Suriya Gunasekar, Data augmentation as feature manipulation: a story of desert cows and grass cows, 2022.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1-57, 2018. URL http: /ttp://mmrr... wwt.:oogggpppaaaeee.eeeee patpperrs/v////9 ww9/18-11111 188..888... .ttll
Lichao Sun, Congying Xia, Wenpeng Yin, Tingting Liang, Philip Yu, and Lifang He. Mixup- transformer: Dynamic data augmentation for NI.P tasks. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 3436-3440, Barcelona, Spain (Online), December 2020. International Com- mittee on Computational Linguistics. doi: 10.18653/1122000.oolngg main..33 URL https //aclanchology.. wttp::/..//0222.......
main. 305
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision, 2015.
NLLB Team, Marta R. Costa-jusså, James Cross, Onur Çelebi. Maha Elbayad, Kenneth Heafield, Kevin Hefferman, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews. Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-centers machine translation, 2022.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32: 13388-188899 2019.
13

---

Published as a conference paper at ICLR 2025
A. F. M. Shahab Uddin, Mst. Sirazam Monira, Wheemyung Shin, TaeChoong Chung, and Sung- Bae. Saliencymix: A saliency guided data augmentation strategy for better regularization. CoRR, abs/2006.01791, 2020.
2020. URL/ttps:: www.aaaxivvvv... http://ggggabb/a/oooeoo/e abs/200s...... 20t.//1909.1191111
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, É ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL 1
wwwpeeeddiinng..uusss.... wwt.uiipssccppppp.p..eeep cc/paper Files//apez/2017/
4354 7dee91
pdff
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Joannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, PMLR, 2019. URL
volume 97 of Proceedings of Machine Learning Research, pp. / /vee///eeeeee rma 1 9a. html
httt:://rroceedings mmlr, prwss/ 977
Rui Wang, Robin Walters, and Rose Yu. Data augmentation vs. equivariant networks: A theory of generalization dynamics forecasting. arXiv preprint arXiv:2206.094500 2022.
Yida Wang, Yinhe Zheng, Yong Jiang, and Minlie Huang. Diversifying dialog generation via adaptive label smoothing. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 3507-3520, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021.acl-long 272. UtLhttps: http::///caaaaatooooo/ooo ort.oogg/////// awlallllloong 272
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth not? When label smoothing meets noisy labels. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 13589-23614 PMLR, 17-23 Jul 2022. We wei22b.htn html
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roclofs, Raphael Gontijo-opes, Ari S. Morcos, Hongscok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022.
Sen Wu, Hongyang Zhang, Gregory Valiant, and Christopher Ré. On the generalization effects of linear transformations in data augmentation. In International Conference Machine Learning, pp. 10410-10420. PMLR, 2020.
Guoxuan Xia, Olivier Laurent, Gianni Franchi, and Christos-Savee Bouganis. Towards understanding why label smoothing degrades selective classification and how to fix it, 2024. URL https
//arxiv.org/aaaaa.ll. abs 2403. 14715
Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, and Rong Jin. Towards understanding label smoothing, 2020.
Shuo Yang, Yijun Dong, Rachel Ward, Inderjit S Dhillon, Sujay Sanghavi, and Qi Lei. Sample efficiency of data augmentation consistency regularization. arXiv preprint arXiv:2202.12330 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, 2018.
14

---

Published as a conference paper at ICLR 2025
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization?, 2020.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration, 2021.
Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective, 2022. URL
abbab
2210. 021 92
Fei Zhu, Zhen Cheng, Xu- Yao Zhang, and Cheng-Lin Liu. Rethinking confidence calibration for failure prediction, 2023. URL http:://axxwwwor//aaaaa/ /tggabb///333.....oo/////
Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. The benefits of mixup for feature learning, 2023.
15

---

Published as a conference paper at ICLR 2025
A OMITTED PROOFS
A.1
HELPER LEMMA
The following lemma lower bounds the Jensen gap in terms of the variance of the random variable being considered. We will make repeated use of it in the proofs of the label smoothing and Mixup results.
Lemma A.I. Let o: Rd B. be wwice-differentiable convex function satisfying ILa V2g 12la. Then for any square integrable random variable X on R it follows that:
B[o(X)) 100--6(110 ([E[X]) E [2 Var(X); Var(X)]
(A.1)
Proof. From the assumption of the lemma it follows that (xx) 71 ||0|| /2 is convex and ((x)
1/ | || is concave. Applying Jensen's 's inequality to each of these functions yields A..
A.2 PROOFS FOR SECTION
Theorem 3.3. Let w" be the unique minimizer of (w for B > 0 under π satisfying Assumption
3.2 Then [|w 2 2 11==011?²
Proof. Let us decompose the unique minimizer w' as ~ = au" + _,0,, where u² and v² are orthonormal and satisfy "zz = 0, 22 = (i.e. u" is a normalized version of the L components w² and vr is the same but for the H components). We claim that (u", > (u" .τ) for æ-a.e. (z, y). Indeed, this were not the case then by the assumption on π we could choose an orthonormal vector z" with = = 0 that satisfies (z* .) > y (u", ,x) for æ-..e. (2,y) and decrease the loss of w" by replacing v" with 2=
Now suppose that a > B. Then we claim that replacing a and 3 with 7 = V(a² B2)/2 yields solution with lower loss than w".
To see this, first observe that 2y² = a² + 33, so that the norm of the modified solution is the same as w². This implies that the weight decay penalty term in Eg is unchanged.
Furthermore, we have that E (3, at), and that y 3 > a 7- This follows from the fact that 7 (a + 3)/2 by construction. This, combined with the fact that (u" x) y (u", x), then implies for a-a.e. (I, 3) that:
y (rr" +,""",,) > (on" + -0",,")
(A.2)
Which contradicts the minimality wt. Therefore, we must have a S B, which gives the desired result.
Corollary 3.4. wf is the max-margin solution to lo(w), then the result of Theorem 3.3 still holds.
Proof. We can apply the same decomposition as in the proof of Theorem 3.4 namely, w² ao" So" where w² is the max-margin solution and u" and v* are as before. Suppose again that > B
and let VTa 32)/2. We claim that there exists (0, 7) such that (7 <)u* + yv² satisfies (w. x, 2) for m-.ee. (z, y), which would contradict w being the max-margin solution since || < ||
From the exact same logic in the proof of Theorem 3.4 we have:
y (yu" + "o"",""" (w" 2)
(A.3)
for m-..e v). Since supp(TT) is compact, (A.3) attains minimum > O over supp(n). Similarly, y (u" ,4) attains a finite maximum. By choosing such that (eu", x)) k, we obtain the desired contradiction.
Theorem 3.6. For E (0, 1) and π satisfying Assumption satisfies ||l O(IEyxxccl)
3.5 every minimizer w" of (IS,o
16

---

Published as a conference paper at ICLR 2025
Proof. We follow the outline in the proof sketch. We first observe that there exists M such that it suffices to consider ||=||| I M, this is due to the fact that for a > 0, we have lilljoell-ooo ( = oo, With this in mind, let us use Z to denote w² YX X and use (Z) to denote the loss in terms of this quantity. Then it follows that
0Lss, (Z)
E (2) + a/2].
(A.4)
OZ
8 fs (Z) E (Z) -
(A.5)
a02
where in both cases we applied the dominated convergence theorem, which is justified because L is smooth in Z with bounded derivatives. Now since || & M and the support of X is compact, there exist ? and 72 such that a (2) (71: 72), which implies that LLS, is strongly convex in
022
Z and satisfies the conditions of Lemma A.
By Jensen's inequality, we then have that:
(Z) 2 -1) log o (B[2))
logo (-BB((
(A.6)
2
Since E[Y Xx. A by Assumption 3.5
we can choose such that w'H 0 and E|Z]
0 0-
a a/2), which minimizes the RHS of (A.6) Let us use OPT to denote this minimum. Then by Lemma
A. we have for any w chosen as described:
l.s.. (w) OPT & w² Eyx.cw.
(A.7)
2
On the other hand, if w' is another solution satisfying ||w/H || > e, then Lemma A.I gives
A.Igiies
(ww") OPT 2 7ipe?
(A.8)
2
from which it is clear that for appropriate we cannot have w' be stationary point (p above is the same as Assumption 3.5). That we can take 0 x - 0 follows from (A.7)
Theorem 3.7. For any symmetric Dx that is not point mass on 0 or 1 and a satisfying Assumption 3.5 every minimizer w* of EMIX,D, (w) satisfies ||w < x(
Proof. Let us first outline the overall steps of the proof, and the differences with the label smoothing case.
1. We first show that the loss conditioned on À is strongly convex in w Zx. The conditioning À here is necessary because À random variable, unlike in the label smoothing case. The overall goal here is to use the same argument as for label smoothing. i.e. show that we can achieve the optimal lower bound in terms of B[w" Zx] using only we and letting | + 0.
2.
We cannot explicitly minimize the conditional loss like we did with label smoothing, since it is not possible with fixed choice of w to achieve o (E'e" Zx)) X for every À simultaneously. Instead, we will show that a stationary point of the conditional loss exists that uses only the dimensions of wc.
3. Having shown the above, we can just reuse the same argument as before with Lemma A. to prove the desired result.
Let M denote M with a fixed choice of (i.e. after conditioning on A), and let w Z Then we can compute:
dêMIx. (R)
= E[)(o(Y,R) - 1)M (1 -1 (Y -
(A.9)
ÖR
82 MMMIX (R)
[o RY) + X)) - (Y222)))
(A.10)
2
where again we applied dominated convergence to the expectation with respect to ग. Strong convexity follows from the same consideration label smoothing; namely, we consider ||w|| < M as
17

---

Published as a conference paper at ICLR 2025
long as À is not 0 or 1 and Ty (v) > and this consequently implies (A.10) is lower bounded by some positive real number.
M (R) + ||w||| +
Now by conditional Jensen's inequality, obtain the following lower bound for DM
(, ) 2 Ex [Y,,,, ( [R] Y, Y2DII-
(A.11)
We now show that it is possible to minimize this lower bound while taking WH = 0. This is more difficult than it was in the label smoothing case, because it is longer obvious that R & 0 is sufficient for minimizing the RHS of (A.II) due to the expectation with respect to A. However, we can show the existence of stationary point with WH O, even though we cannot provide an explicit construction.
The idea to consider the limiting behavior of (A.1I) we take the values of w'g to 00 and co. Note that we can take this limit into the expectation with respect to À by dominated convergence again. Let us consider the gradient with respect to w of the RHS of (A.II). To make notation manageable. we will use á₁ = X Y = I]. 22 EE[X Y -]. and a3 R[Zx X, Y1 1, Y2 -1]. We can then explicitly write out the gradient as the expectation with respect of the sum of the following three terms (considering the different cases for Y1, Y2):
V n. By,, (E [R| X, Yi, Y2)] Ty (1)² (a a1) 11 an ( (a(-w" a2) - -1) 0s
+ 2 (1)mT( >(o(w" (az) -1)
- - (1 (o ass) - 1) az.
(A.12)
The first two lines above are obtained from the fact that we can combine terms when Y1 Y2, and the last line is by symmetry. Now we recall that by assumption B|Y Xc] # 0 and E[X] = 0. Thus, WLOG, we can assume that [x[x Y 1] > 0 and E[Xx Y = -1]
i]: < for each index i.
With this in mind, we consider first the case of what happens when the entries we - co. Since wT T aj > O and w ag < O, the first two terms in (2.12) vanish (independent of X). On the other hand, X, we have that the entries of as either strictly negative or strictly positive, with the exceptional case of X = Ty (1) in which dg = 0. the entries of 03 are strictly negative, then (a(-w- ag) 1) - 0 and the coefficient becomes negative, so the third term is positive. Similarly, if is strictly positive, the coefficient is positive
for the third term, there are two cases to consider. Depending
and the third term is still positive. Thus, as wg - oo every entry of (A.12) is positive.
Now by continuity of the gradient, immediately follows that there is some choice of w with only we non-zero such that the expectation of (A.12) with respect to À can be made to be zero. Using this choice of w allows to obtain an that minimizes the RHS of (I.
Similar arguments show that the opposite is true when take we
Now have basically arrived at the same stage as the end of the label smoothing proof. By taking ||Eyxccll + O we can get arbitrary concentration around this optimal R, and by the same logic as the label smoothing proof the result follows.
A.3 PROOFS FOR SECTION 3.2
Proposition 3.9. For 0 and any 9 Idd + A k-1 letting OPTLS, denote the minimum of LLS, we have for a universal constant C' 0:
(1) 2 OPTLS, + C
Ty (ar(B(X)))) v))
(3.1)
y=1
Proof. Let us first define:
(L..ay(0)) -E (1 a) log (X) (X) Y Y -w]
(A.13)
18

---

Published as a conference paper at ICLR 2025
Then decompose (g) as follows:
( my (1) (V)(1,,a,)9)
(A.14)
y=1
Once again, since can restrict attention g( h 1 for some 7 (as the loss goes infinity if g(X) = 1 on a set of positive x T--measure with a >> 0), it is easy to verify that (A.13) is strongly convex in g(X). The desired result then follows by applying Lemma A. with the regular conditional distribution Xy for each term in (A.14).
Proposition 3.10. For any D that is not point mass on 0 and any g IR A letting OPTMIX,D, denote the minimum of Dx have for universul constant C > 0 thut:
DA (g) 2 MIX,D, +C
Σ
Ty(y))T(
Ex [Var (Blg(Zx) 91 U2. ADM] (3.2)
2/1 12=1
Proof. The proof follows an identical structure to that of Proposition 3.9 In particular, we again define the following conditional loss:
1 E[A log. go (Zx) + (1 A) log g (Zx) 91, 32. A].
(A.15)
And then decompose LMIX,DA
h x (0) = Σ
Ty (y)) Ex-D, (CMIX,A.1.
(A.16)
12=1
Since Dx not a point mass on 0 or 1, we can restrict ourselves to (XX [7, 1 > for some y as
before, and strong convexity in o(X) of the conditional loss (A.15) again follows. We then apply Lemma A. to obtain the result.
B COMPARISON TO SETTINGS OF PRIOR WORK
Here review the settings of Chidambaram et al. (2023) and Zou et al. (2023) in greater detail to provide a more precise comparison to the setting in our work.
Setting of Chidambaram et al. (2033). The authors consider multi-view data model inspired by Allen-Zhi -hu & Li (021);; namely, their data distribution π is a multi-class distribution supported on Rra [k] where corresponds the number of patches in each data point. Essentially, for (a,, v) n. we view x as x (x(1),, .2(2), ....,
_,(2),..., ( with each (+() e Rd The purpose of this
partitioning is so that features related to the target class can appear in some patches and noise can appear in the remaining patches. In particular, the authors consider two target features per class (1/,1 and (2)2)) that appear in constant number of signal patches, while all other patches in data point correspond to low magnitude feature noise, i.e. these patches consist of some linear combination of features Va.j for + y. Furthermore, each signal patch has only a single feature (either Ug 1 or 2y22) with random weight 8 such that for any signal patch (P) Buy,1 there is another patch ( (C 3)0y,2 for fixed parameter C. The authors emphasize that model that has the same correlation with both U and 2222 will achieve lower variance, as it will have constant total correlation and be insensitive to the variation in 3.
For this type of data distribution, the authors consider the training dynamics of a two-layer convo- lutional neural network with smoothed ReLU activations and non-trainable second layer weights (not an issue in this case, since second layer weights can be absorbed into the first layer). They analyze training using the empirical cross-entropy, as well as the Mixup cross-entropy for the specific case of mixing distribution DA that is just point mass on 1/2 (Midpoint Mixup). The authors prove that, running gradient descent on the empirical cross-entropy leads (with high probability) to a model that only learns feature for almost all classes, while doing the same for Midpoint Mixup yields model that learns both features per class. The results are asymptotic; the authors consider all hyperparameters in their setup be sufficiently large (even the number of classes k) or small.
19

---

Published as a conference paper at ICLR 2025
Setting Zou et al. (2023). ) Similar to Chidambaram al. (2023) Zou al. (2023) also works in a setting motivated by Allen Zhu & Li (2021) and consider data consisting of patches. However, they focus on binary classification, and their data distribution is supported
x R.d
(1,2). The
data has exactly one (randomly selected) patch x=()) that contains a target feature; Zou et al. (023) delineate one type of target feature u as common and another type u' as rare. There are also up to b other patches in that consist of common features from other classes (i.e. feature noise, like in Chidambaram et al. (2023)). Lastly, different from Chidambaram et al. (2023). Zou et al. (0023)
consider the leftover patches consist of i.i.d. Gaussian noise.
Zou ct al. 2023) also consider the training dynamies of a two-layer convolutional neural network with frozen second layer weights on this type of data distribution, but use a squared activation instead of a smoothed ReLU. Unlike the results Chiidamaara al. (2023) which are stated entirely in terms of whether the features Vy and 2y are learned in a certain sense, Zou et al. (2023) directly prove a lower bound on the test error of models trained using gradient descent the empirical cross-entropy while also showing that the test eπror of models trained on the empirical Mixup cross-entrope is vanishing small at some time step during model training. Essentially, this is due to the non-Mixup models failing to learn the rare feature vf for both classes. Their results apply to Mixup with a mixing distribution DA that is any point mass on (0.5, 1).
Differences in our setting. Both Chidambaram al. 2023) and Zou al. (0233 prove results concerning gradient descent dynamics, whereas our results directly consider the minimizers of the population losses associated with weight decay, label smoothing. and Mixup. Here our choice to work with the population losses is not substantially different from Chidambaram et al. (2023) and Zou et al. (2023), since although they work with the empirical losses they are an asymptotic setting and large swaths of the proofs in both papers rely on concentration of measure arguments. However, we do differ substantially in that our main results apply to linear models this makes our results less practical but technically much simpler than the results of Chidambaaam et al. (2023) and Zou et al. (2023), with the added benefit that we can also handle any symmetric mixing distribution Dx.
Furthermore, since our main results work in this linear setting, we also adopt a much simpler data distribution setup. We do not consider our input data x as partitioned into patches, instead we merely designate some subset of the input dimensions as "low variance" and the complementary subset as "high variance". In this sense, we do not have explicit feature vectors associated with each class y like the Ug 2,, of Chidambaram et al. (2023) or the v, v' Zou et al. 2023
C ADDITIONAL EXPERIMENTS
C.I VISUALIZATION OF DECISION BOUNDARIES
To provide some intuition for Theorems 3.3 3.7 we visualize the decision boundaries of trained logistic regression models on 2-D data. In particular, denoting the classes as usual by y E {-1, +1}, we consider simple data distribution in which the first coordinate distributed uniformly on [y, 10y] and the second coordinate is fixed to be 0.1y.
This data distribution is linearly separable in each coordinate; however, the second coordinate is fixed and thus has no conditional variance. Consequently. Theorems [3.7 p.7deii predict that minimizing the population label smoothing and Mixup losses on this data should lead to learning a model whose decision boundary is aligned with the x-axis (i.e. we only use the second coordinate to determine which class we predict).
To verify this, we visualize the decision boundaries logistic regression models trained on 500 points sampled from this distribution using multiple different settings of weight decay, label smoothing, and Mixup in Figure 5
We train for 500 epochs using full batch SGD with a learning rate of 10 although our results were not sensitive to these choices. We use mixing distribution of Beta a) for Mixup (as is standard), and consider the canonical hyperparameter choices of 5 x 10-4 for weight decay, 0.1 for label smoothing, and Beta(1, 1) for Mixup (Zhang et al. 2018 Müller et al. 2020) in Figure 5 (a) and also check the effect of scaling these hyperparameters in Figure (b) and (c).
As can be seen from the results, the label smoothing and Mixup decision boundaries are much closer to being aligned with the z-axis than the weight decay boundary, with the alignment getting stronger with more extreme choices of the hyperparameters. The fact that the boundaries are not exactly aligned with the a-axis can be attributed to the fact that we not exactly minimizing the population
20

---

Published as a conference paper at ICLR 2025
00
120
14
(a) Canonical Hyperparameters
(b) Scaled Hyperparameters (1)
lopal
(c) Scaled Hyperparameters (2)
Figure 5: Visualization of weight decay, label smoothing, and Mixup decision boundaries. Figure (a) considers canonical hyperparameter choices, and Figures (b) and (c) illustrate the effects of scaling these choices.
loss, since in the finite sample, fixed training horizon setting. That the label smoothing boundary is more aligned to the r-axis than the Mixup boundary is also to be expected, since the Mixup loss introduces randomness in the form of the mixing distribution which makes it more difficult to minimize. Lastly, the fact that the boundaries for both label smoothing and Mixup become more aligned with the r-axxs more extreme hyperparameter values can be intuitively explained by the fact that the loss incurred by predicting a probability close to I for either class increases as we scale the hyperparameters, i.c. we suffer more loss from relying on the high variance first coordinate.
Remark C. We should point out that, while theoretical and empirical results in Figure 5 show clear difference in the types of solutions learned when training linear classifiers using label smoothing. Mixup, and weight decay, it is not always the case that the augmented losses lead different solutions than ERM (with possibly weight decay). Indeed, Chidambaram et al. (2021) and Oh & Yun (2023) both showed that for different settings of Gaussian data, ERM and Mixup can lead to learning the same solution. However, the settings of these prior works don't 't fall within our scope as we consider distributions with compact support, which ends up being an important property for proving our results.
C.2 DIRECT VERIFICATION OF THEORY
Here directly analyze the training of logistic regression models a synthetic data distribution that exactly follows the assumptions of Definition 3.1 the following definition generalizes the 2-D distribution we visualized in Section C.
Definition C.2 (Synthetic Data). We define distribution my parameterized by (where 0 <<> < 1)
on Rdd x 1 with (1) 1/2 and Xy,x|y=y satisfying:
1. First [d/2] dimensions are constant but small. We have 2; yy for i [d/2].
2. Last d - [d2) dimensions are high variance. We have ~ Uniform([y, 100y]) for > |d/2].
21

---

Published as a conference paper at ICLR 2025
In other words, consider data where (conditional the label) the first half of the dimensions fixed (corresponding to C) and the second half are i..dd high variance uniform (corresponding to H). We sample 5000 data points according to Definition C22 with d 10 and y 0.1 and train logistic regression models across a range of settings for weight decay, label smoothing, and Mixup. The choice of y here, well as the range of values for the dimensions in H. is relatively arbitrary; we verified our empirical results hold for different scales of y and ranges for H. The empirical results also do not depend the fact that the L coordinates are zero variance we checked that they still hold when adding a small magnitude uniform noise to the L coordinates.
For model training, consider 20 uniformly spaced values in [0, 0.1] for the weight decay A parameter and in [0, 0.75] for the label smoothing a parameter, where the upper bound for the label smoothing parameter space is obtained from the experiments of Müller al. (2020). For Mixup. fix the mixing distribution to be the canonical choice of Beta(a, a) introduced by Zhang et al. (2018) and consider 20 uniformly spaced values in [0, 8] (with
corresponding to ERM), which effectively covers the range of Mixup hyperparameter values used in practice.
We train all models for 100 epochs using AdamW with the standard hyperparameters of $1 0.9, Bg = 0.999, a learning rate of 5 x 10-3, and batch size of 500. At the end of training, we compute ||w (i.e. the norm of the weight vector the last dimensions) for each trained model. For each model setting, we report the mean and standard deviation of ||w7]| over 5 training runs in Figure
0.08
635
654
3.06
680
633
80
81
83
35
86
(a) Weight Decay
(b) Label Smoothing
(c) Mixup
Figure 6: Final weight norm logistic regression model in the high variance dimensions (lwy |] for various hyperparameter settings on the synthetic data distribution of Section C.2
As can be seen from the results, the weight decay models always have non-trivial values for ||w (even for large values of A), whereas the label smoothing and Mixup models very quickly converge to a ||wx|| of effectively zero their respective hyperparameters move away from the ERM regime
(a = 0 in both cases). This matches the behavior predicted by Theorems 3.3 to 3.7
C.3 FULL RESNET RESULTS
Here collect final test error/variance results for the plots shown in Section 4. and also provide analogous plots and results for ResNet-50 and ResNet- 101 models. In all of the following, we abbreviate weight decay "WD" and label smoothing as "I.S".
22

---

Published as a conference paper at ICLR 2025
C.3.1 RESNET- 18 FINAL RESULTS
Tables and and the experiments in Figure I
show the end-of-training test error, activation variance, and output variance results for
Method
Test Error
Activation Variance
Output Variance
ERM + WD
8.40 + 0.58
198 5
0.133 + 0.008
ERM WD I.
7.96 + 0.30
33.0 + 5.9
0.099 + 0.003
ERM + WD + Mixup
5.90 0.21
18.4 0.4
0.059 + 0.002
Table 1: Final results (mean test error/variance and one standard deviation over runs) for ResNet-18 experiments CIFAR-10.
Method
Test Error
Activation Variance
Output Variance
ERM + WD
31.59 + 0.36
673 = 23
0.406 = 0.007
ERM + WD LS
28.55 4.7
54.7 5.6
0.188 = 0.027
ERM WD Mixup
26.83 0..4
82.2 1.2
0.143 + 0.004
Table 2: Final results (mean test error/variance and standard deviation over runs) for ResNet-18 experiments on CIFAR-100.
C.3.2 RESNET-50 ALL RESULTS
23800
20/00
Wegf DeCRy
Weigh Decip
Maxe
(a) CIFAR-10 Test Error
(b) CIFAR-10 Activation Vari-
(c) CIFAR-10 Output Variance
ance
WHbP Decky
Right Demry Maps
Secey Maxa
030
20
70
(d) CIFAR-100 Test Error
(e) CIFAR-100 Activation Vari- ancc
(1) CIFAR-100 Output Variance
ResNet-50 final test errors, penultimate layer activation variances, and output probability variances on CIFAR-10 and CIFAR-100.
Figure 7:
We train ResNet-50 models under the same hyperparameters as in Section 4.1 except for batch
size of 512 due to memory constraints. ResNet-50 results analogous to those shown in Figure [ are shown in Figure 7 Similarly, final error and variance results are shown in Tables and
We observe that while the same trends hold as in the case of ResNet-18, there is significantly more variance the computed activation variances for Mixup on CIFAR- 10. In this particular case, we found that there were still significant oscillations in activation variances even towards the end of the training horizon. This may in part be attributable to reduced batch size, but we did not investigate this further.
23

---

Published as a conference paper at ICLR 2025
Method
Test Error
Activation Variance
Output Variance
ERM + WD
7.36 + 0.13
551 = 711
0.119 + 0.002
ERM + WD LS
7.24 1 0.26
6.35 = 3.47
0.091 + 0.004
ERM + WD + Mixup
5.41 0.37
95 + 115
0.054 + 0.003
Table 3: Final results (mean test error/variance and one standard deviation over runs) for ResNet-50 experiments on CIFAR-10.
Method
Test Error
Activation Variance
Output Variance
ERM + WD
29.41 + 0.94
310 + 14
0.383 + 0.012
ERM + WD + LS
28.58 + 0.28
13.14 0.20
0.257 + 0.001
ERM+ + WD + Mixup
24.94 1.63
22.95 1.52
0.161 = 0.007
Table 4: Final results (mean test error/variance and one standard deviation over 5 runs) for ResNet-50 experiments on CIFAR-100.
C.3.3 RESNET- 101 ALL RESULTS
We also train ResNet- 101 models under the same hyperparameters in Section 4.1
except once again for a batch size of 512 due to memory constraints. ResNet-101 results analogous to those shown in Figure are shown in Figure Similarly, final error and variance results are shown in Tables and 6
Here see that the variance oscillation behavior that mentioned in Appendix C.3.2 even more pronounced for the CIFAR-10 10 results, suggesting that this behavior is amplified for larger models. Once again, it is not clear what properties of CIFAR-ID 10 lead highly oscillatory activation variances for some initializations, but we again suspect that reduced batch size in training at least plays some role, The CIFAR-100 results remain consistent, although there is still some oscillatory behavior for the Mixup results.
Method
Test Error
Activation Variance
Output Variance
ERM + WD
6.87 1 0.17
491 + 235
0.110 + 0.002
ERM WD HDD+LS LLS
6.90 0.26
2280 2292
0.088 + 0.003
ERM WD Mixup
5.23 0.18
1701 + 1586
0.054 + 0.002
Table 5: Final results (mean test error/variance and one standard deviation over runs) for ResNet-101 experiments on CIFAR-10.
Method
Test Error
Activation Variance
Output Variance
ERM + WD
28.69 + 0.73
283 18
0.373 = 0.008
ERM WD + LS
27.74 0.55
12.01 0.23
0.269 = 0.005
ERM+ + WD + Mixup
23.55 + 0.98
115 + 136
0.157 + 0.01
Table 6: Final results (mean test error/variance and one standard deviation over runs) for ResNet- 101 experiments CIFAR-100.
C.4 OUTPUT PROBABILITY VARIANCE ANALYSIS
A natural explanation for why output probability variance decreases for label smoothing and Mixup in Figures 7 and is is that label smoothing and Mixup improve test error and consequently have less variability in their outputs due to fewer mistakes. Firstly, looking carefully at the label smoothing results in the previous subsections shows this cannot be the full cause, as Table 5Ialee label smoothing leads to worse test error than the baseline of ERM WD but still leads to lower output probability variance.
In Tables
compute the average output probability variance for each class but consider only the probability associated with the target class (i.e. for all points with label y, we compute the variance of the predicted probabilities corresponding to y). In all cases, the target output variance alone leaves a significant fraction of the overall output variance unexplained.
24

---

Published as a conference paper at ICLR 2025
Lesel Smoothing
Lolel Sencattig
000
Dpada
(a) CIFAR-10 10 Test Error
(b) CIFAR-10 Activation Vari-
(c) CIFAR-10 10 Output Variance
ance
096
-0000
030
Maax
(d) CIFAR-100 Test Error
(e) CIFAR-100 Activation Vari- ance
(T) CIFAR-100 Output Variance
Figure 8: ResNet-101 final test errors, penultimate layer activation variances, and output probability variances on CIFAR-I0 and CIFAR-100.
Method
Target Output Variance (CIFAR-10)
Target Output Variance (CIFAR- 100)
ERM + WD
0.066 + 0.004
0.172 = 0.002
ERM WD LS ERM + WD + Mixup
0.050 + 0.002
0.119 = 0.002
0.029 + 0.002
0.087 = 0.002
Table 7: Target output probability variance for ResNet-18.
Method
Target Output Variance (CIFAR-10)
Target Output Variance (CIFAR-100)
ERM + WD
0.059 + 0.001
0.166 + 0.004
ERM WD LS
0.046 0.002
0.131 0.001
ERM + WD Mixup
0.027 + 0.002
0.093 + 0.004
Table 8: Target output probability variance for ResNet-50.
Method
Target Output Variance (CIFAR-10)
Target Output Variance (CIFAR-100)
ERM + WD
0.054 + 0.001
0.162 + 0.002
ERM WD+LS
0.044 + 0.001
0.131 = 0.002
ERM + WD + Mixup
0.026 + 0.002
0.088 = 0.004
Table 9: Target output probability variance for ResNet-101.
25


Reviews:
[{"review_id": "LPZKVJh2jm", "reviewer": "Reviewer_Enmj", "summary": "This manuscript studies the relationship between label augmentation techniques (label smoothing and Mixup) and feature variance. The authors demonstrate that label augmentation in binary classification with a linear model encourages the model to learn low-variance features. In contrast, ERM with weight decay can lead to learning higher-variance features under certain conditions. They also show that the losses of label smoothing and Mixup for nonlinear models are lower-bounded by a function of the model output variance. Additionally, they provide empirical evidence that label augmentation may negatively impact model training.", "questions": "1. Can these results extend to multi-label classification?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This manuscript is well-written, and the theoretical parts are clearly presented.\n2. This manuscript links label smoothing and Mixup to feature variance, offering new insights into the analysis of label augmentation.", "weaknesses": "1. The main claim that “optimizing the empirical risk with weight decay can learn higher variance features” is built on a strong assumption. As the authors note, Assumption 3.2 is indeed strong, suggesting that the $\\\\mathcal{H}$ dimensions are inherently better suited for classification than the $\\\\mathcal{L}$ dimensions. However, it’s straightforward to construct a task that relies more heavily on the $\\mathcal{L}$ dimensions than on the $\\mathcal{H}$ dimensions. For instance, consider a 2-D distribution where the $x$ coordinates for both classes are uniformly ranging from -1 to 1, while the $y$ coordinates are set at -0.1 or 0.1, depending on the class. In fact, by simply reversing the assumption from $y \\\\langle v^*, x \\\\rangle > y \\langle u^*, x \\\\rangle$ to $y \\\\langle u^*, x \\\\rangle > y \\langle v^*, x \\rangle$, the conclusion of Theorem 3.3 changes from $\\\\|w^*_\\\\mathcal{H}\\\\|^2 \\ge \\frac{1}{2} \\\\|w^*\\\\|^2$ to $\\\\|w^*_\\\\mathcal{L}\\\\|^2 \\ge \\frac{1}{2} \\\\|w^*\\\\|^2$.\n2. There is no discussion on why label smoothing or Mixup learn more on low-variance features compared to ERM or ERM with weight decay. In fact, by setting $\\\\alpha$ to 0, the label smoothing loss reduces to the negative log-likelihood loss, allowing one to obtain results similar to those in Theorem 3.6 or Proposition 3.10 for ERM, which also demonstrate that the ERM loss is lower-bounded by a function of the model output variance. Thus, $\\\\alpha$ plays a central role in learning low-variance features, but this important analysis is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fT6K2JSTKQ", "reviewer": "Reviewer_2SFa", "summary": "The paper presents a theoretical analysis of label smoothing and Mixup techniques. The authors investigate the differences between these label augmentation methods and the vanilla approach (i.e., empirical risk minimization with weight decay) regarding their effectiveness in learning features of varying variance. In a binary linear classification problem, they prove that, under certain assumptions, the vanilla method prefers high-variance features, while label augmentation encourages the model to learn low-variance features. This analysis is then extended to more complex settings by showing that the label augmentation loss is lower-bounded by model variance. The theoretical findings are supported by empirical results on both original and modified CIFAR-10, CIFAR-100, and colored MNIST datasets.", "questions": "- In the proof of Corollary 3.4, what happens if the max-margin solution does not include high-variance features? It appears that the proof technique may not apply in this scenario. Could the authors clarify whether this case is impossible or if there are additional conditions to address it?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The theoretical results are insightful and are effectively supported by a variety of well-designed experiments.", "weaknesses": "- It remains unclear why learning low-variance features leads to better generalization in practical scenarios, as this approach can result in poorer generalization on synthetic datasets.\n- The comparison to previous works on feature learning (e.g., Chidambaram et al., 2023; Zou et al., 2023) is insufficient. I encourage the authors to provide a more detailed comparison with these works (e.g. empirical loss vs population loss). Additionally, the notion of \"features\" appears to differ between this work and prior studies; clarifying what \"features\" specifically refers to in this context would enhance the reader’s understanding.\n\n### Minors\n\n- Line 795: “approach of Theorem 3.4” → \"approach of Theorem 3.3\" and I hope the author provides a complete proof of Corollary 3.4 for clarity, even if it largely repeats the proof of Theorem 3.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gmovYjHPsm", "reviewer": "Reviewer_Gh1c", "summary": "In this paper the authors provide theoretical results pertaining to the variance of features in relation to training with label augmentation (Label Smoothing and Mixup). They show that for a binary linear logistic classifier trained on a certain distribution where there is a high and a low variance feature, label augmentation leads optimisation to only consider the low variance feature. For general models and classification they provide a quantative lower bound on how the variance of output probabilities contributes to the loss. The authors then provide some experiments to empirically verify the theoretical results.", "questions": "1. For the CIFAR experiment with the spurious feature, are you training the full model or just probing a frozen ResNet with a linear logistic classifier? Additionally can you clarify the difference between line 65 where you specify \"if the high variance feature satisfies a greater separability\" and the experiment where the low variance spurious feature is more separable.\n2. The experiments all use heavily overparameterised networks. 2048 is somewhat overkill for MNIST and ResNet-18 is not designed for CIFAR but ImageNet. ResNet-18 has over 10 million parameters whilst ResNet-56 which is designed for CIFAR has < 1 million and both achieve similar performance on CIFAR. Can you comment on how overparameterisation/overfitting affects the theory/experiments of the paper? This contrasts to the binary linear scenario where underfitting/model being insufficiently expressive may be more likely(?).\n\nI am happy to raise my score if the stated weaknesses (other than 3 which I feel unfortunately can't be changed) and above questions are addressed.\n\nEdit: as the authors have addressed most of my queries and concerns, and also improved the paper in response to the other reviewers, I have raised my score from 5 to 6.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper provides some new theoretical results that contribute to better understanding of the behaviour of label augmentation approaches. I appreciate work that tries to advance knowledge in deep learning rather than simply chasing benchmark scores so this is a big plus for me.\n- The experiments are well-targeted to verify the theoretical results.", "weaknesses": "I would like to clarify that I did not have time to comb the fine details of the paper or verify the theory, however I believe I have allocated a reasonable amount of time to reading from start to end. I am also not a \"theoretical\" machine learning researcher personally, and I am more familiar with the literature for label smoothing than Mixup, which should help put my later comments in context. I welcome the authors to correct any mistakes/misunderstandings that I make in this review.\n\n1. The presentation of the paper inhibits easy understanding of the theoretical takeaways for a general machine learning audience (i.e. the average researcher using label augmentation). There isn't a single visual aid to illustrate any of the theoretical results, making it both difficult to intuit the theory and understand the key takeaways. For example, for the linear binary classifier, the paper would be greatly aided with a simple 2D illustration showing how the optimal decision boundaries differ between with and without Label Smoothing. Similarly I would suggest an illustrative sketch over 3 classes (visually in the same vein as those in [1]) of how greater variance contributes to loss. I think the text could also be improved by highlighting the key theoretical takeaways in plain English (e.g. in a box/table/in bold), allowing the less mathematically inclined reader to not have to parse through the comparatively dense and notation heavy theory.\n2. Missing related work: there are a few missing references to label smoothing in the context of uncertainty estimation [2,3], neural collapse [4,5] and transfer learning [6].  [4,5,6]  are especially relevant and should be discussed as they directly measure how tight features cluster together after training with label smoothing. [6] also demonstrates that tighter feature clusters lead to **worse** linear probing performance.\n3. Limited insight with regards to generalisation. The paper lacks any direct insight into how label augmentation leads to better generalisation peformance, which is ultimately why these techniques are used in the first place. Although the authors do fully acknowledge this limitation, it doesn't change the fact that this limits the impact of the insights of the paper. \n4. The practical takeaways of the spurious correlation experiments are somewhat unclear. It is not necessarily the case that spurious correlations tend to be lower variance. In the case that they are high variance wouldn't the inverse behaviour manifest? \n\n[1] Muller et al. When Does Label Smoothing Help? 2019\n\n[2] Zhu et al. Rethinking Confidence Calibration for Failure Prediction, 2022\n\n[3] Xia et al. Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It, 2024\n\n[4] Zhou et al. Are All Losses Created Equal: A Neural Collapse Perspective, 2022\n\n[5] Guo et al. Cross Entropy versus Label Smoothing: A Neural Collapse Perspective, 2024\n\n[6] Kornblith et al. Why Do Better Loss Functions Lead to Less Transferable Features? 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xPvgitW6Ip", "reviewer": "Reviewer_LPKu", "summary": "The paper studies feature learning in training with label augmentation techniques like Mixup and label smoothing, etc.. Using a linear binary classification model, and proper conditions on the data distribution, the authors have theoretically shown that in ERM training, the learned model tends to assign more correlations on the high-variance features, while in label augmentation training, it's quite the opposite. They then empirically show that such a behavior can lead to both positive and negative consequences. In particular, they show that the model's learning low variance features correlates to better generalization, while the model's learning only the low variance features, if happening, can  harm the generalization.", "questions": "1. Line 135 on page 3. The identity matrix shouldd be denoted by mathematical expression \"$I_d$\" rather than text \"Id\" right?\n2. In the context of binary classification, the label variable $y$ should be $Y\\in \\{-1, +1\\}$ right? But in that case, the cross-entropy loss defined in equation 2.1 cannot be applied on binary classification. Or, is the paper implicitly regarding the term \"$k$-class classification\" to be distinct from the term \"binary classification\"?\n3. In equation 2.4, I think the second term inside the expectation should be \"$\\dfrac{\\alpha}{k-1}\\sum_{i\\in [k]/{Y}}\\log{g^i(X)}$\".\n4. In the definition of Mixup loss in equation 2.8, are the points pairs $(X_1,Y_1)$ and $(X_2,Y_2)$ sampled independently from $\\pi$? In that context, I would denote it as $(X_{1:2},Y_{1:2})\\overset{\\mathrm{iid}}{\\sim}\\pi$ or something like $(X_{1:2},Y_{1:2})\\sim\\pi\\bigotimes\\pi$\n5. What are the mathematical definitions of low variance features and high variance features? Is it the overall variance? Or in-class variances? Also, how would one justify if a feature is high variance or low? Should there be a predefined threshold? Or maybe even a post-defined threshold, meaning threshold is defined after the groups of high variance feature and low variance features are manually partitioned?\n6. Before assumption 3.2, $\\mathcal{L}$ and $\\mathcal{H}$ are at first defined as subsets of $[d]$, and then regarded as sets of vectors. I understand this is only for the sake of simplicities in notations, but I would suggest that the paper simply uses different notations.\n7. In assumption 3.5, how does the first part ensure that we can obtain good solution using low variance dimensions? First, I think if a data is class-balanced (the marginal distribution on the classes is discrete uniform), and given an early assumption that $\\mathbb{E}[X]=0$, then $\\mathbb{E}[YX_\\mathcal{L}]$ should always be 0. Second, if the data is imbalanced, then this first part might hold true, but then does it mean we are only dealing with class-imbalanced data from now on? Also, even if the data is imbalanced and $\\mathbb{E}[YX_\\mathcal{L}]$ equals 0, it may still be possible to find a good solution.\n8. The assumptions on the data in the setting of linear binary classification don't seem to be so unrealistic. Can the authors provide some toy example experiments to verify the theoretical results more directly?\n9. I was expecting section 3.2 would be a generalization of the results in section 3.1. But in section 3.1, the conditions are about the variances in the data dimension. But in section 3.2, somehow it's all about the variances in the model output. I am aware of the statement made in remark 3.11, but I still don't understand how does this section help us gain more understanding of the link between the model's behavior of learning low variance features and the change in the model's generalization peformance?\n10. Line 368 on page 7. It better to write \"$5\\times 10^{-4}$\" instead of \"$*$\".\n11. Is there and inituitions or insight as of how does the variance of the learned features affect the performance of the model? Without this, it's pretty hard to use the work in this paper as a cornerstone to initiate further studies, such as the mechanism of label augmentation techniques (in other words, why do they work), the limitations and possible flaws of these techniques, how to improve these techniques or design new algorithms, etc.. This is why I feel like the contribution to the research area in this paper is only fair.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The studying of feature learning problem on label smoothing is novel\n2. The paper have uncovered some interesting and new findings about the phenomenons and properties existed in label augmentation techniques like Mixup and label smoothing\n3. The theoretical contributions and their proofs are valid\n4. The experiments are comprehensive, and they show not only the validity of the theoretical results, but also the correlationship between those theoretical results and the impact of the label augmentation techniques on model performance", "weaknesses": "Please refer to the Questions session for details of the weaknesses listed here\n\n1. While the concept of \"high variancce features\" and \"low variance feature\" appear to be the essential components of the theoretical content, the whole paper seems to lack a strict definition of them. Perhaps that's a common knowledge in the field of feature learning, but since this paper is mostly theory-based, it's better to provide more mathematically strict definition of these concepts.\n2. The logic connection from section 3.1 to section 3.2 is not so clear, in fact, from my understanding, they are talking about different things.\n3. Since the main theoretical  results are built based on a simple model, one would love to see first a experiment done on such a toy model to show the validity of the theoretical results in the most direct way.\n4. Some notations and terminologies can be perfected\n5. There lacks an intuitive explanation or an insight behind the connection between learning of low variance features and generalization performance of the trained model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s3bdwdty0n", "reviewer": "Reviewer_tHC2", "summary": "In this research, the authors investigate label augmentation methods such as label smoothing and Mixup within deep learning frameworks. These techniques adjust both the input data and labels to enhance training outcomes. The study delves into how these methods impact feature learning and reduce output variance. The findings indicate that label augmentation helps models develop low-variance features, improving classification results on some benchmarks, though it may increase susceptibility to misleading data patterns. With theoretical backing and empirical results, the paper highlights the strengths and drawbacks of label smoothing and Mixup. The authors conclude that while these approaches bolster model robustness, they may inadvertently promote reliance on low-variance features, affecting model generalization.", "questions": "Is there any theoretical investigation into whether Mixup and label smoothing perform worse in situations involving spurious correlations? I think the limited experimental setups are insufficient to support a counter-intuitive argument compared to the extensive body of previous work on model calibration and noisy label learning.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper rigorously analyzes label smoothing and Mixup with mathematical proofs, building a solid theoretical foundation.", "weaknesses": "1. Overall, the authors' argument that both Mixup and label smoothing are effective in reducing variance is not particularly surprising and has been a well-known perspective for some time. For this reason, Mixup has been used for calibration due to its generalizability to the underlying distribution of training datasets.\n\n2. The authors should conduct experiments with diverse spurious correlation setups. For example, both label smoothing and Mixup are widely used to mitigate noisy label learning, which is one common spurious correlation setup. I believe these previous works contradict the authors' arguments.\n\n[1] When and How Mixup Improves Calibration, ICML 2022 \\\n[2] How Does Mixup Help with Robustness and Generalization?, ICLR 2021 \\\n[3] Does Label Smoothing Mitigate Label Noise?, ICML 2020 \\\n[4] mixup: BEYOND EMPIRICAL RISK MINIMIZATION, ICLR 2018", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)