For Better or For Worse? Learning Minimum Variance Features With Label Augmentation

## Abstract

Abstract Data augmentation has been pivotal in successfully training deep learning models on classification tasks over the past decade. An important subclass of data augmentation techniques - which includes both label smoothing and Mixup - involves modifying not only the input data but also the input label during model training. In this work, we analyze the role played by the label augmentation aspect of such methods. We first prove that linear models on binary classification data trained with label augmentation learn only the minimum variance features in the data, while standard training (which includes weight decay) can learn higher variance features. We then use our techniques to show that even for nonlinear models and general data distributions, the label smoothing and Mixup losses are lower bounded by a function of the model output variance. An important consequence of our results is negative: label smoothing and Mixup can be less robust to spurious correlations in the data. We verify that our theory reflects practice via experiments on image classification benchmarks modified to have spurious correlations.

## 1 Introduction

The training and fine-tuning procedures for current state-of-the-art (SOTA) computer vision models rely on a number of different data augmentation schemes applied in tandem (Yu et al., ; Wortsman et al., ; Dehghani et al., ). While some of these methods involve only transformations to the input training data - such as random crops and rotations (Cubuk et al., ) - a non-trivial subset of them also apply transformations to the input training label.

Perhaps the two most widely applied data augmentation methods in this subcategory are label smoothing (Szegedy et al., ) and Mixup (Zhang et al., ). Label smoothing replaces the one-hot encoded labels in the training data with smoothed out labels that assign non-zero probability to every possible class (see Section 2 for a formal definition). Mixup similarly smooths out the training labels, but does so via introducing random convex combinations of data points and their labels. As a result, Mixup modifies not only the training labels but also the training inputs. The general principles of label smoothing and Mixup have been extended to several variants, such as Structural Label Smoothing (Li et al., ), Adaptive Label Smoothing (Wang et al., ), Manifold Mixup (Verma et al., ), CutMix (Yun et al., ), PuzzleMix (Kim et al., ), SaliencyMix (Uddin et al., ), AutoMix (Liu et al., ), and Noisy Feature Mixup (Lim et al., ).

Due to the success of label smoothing and Mixup-based approaches, an important question on the theoretical side has been understanding when and why these data augmentations improve model performance. Towards that end, several recent works have studied this problem from the perspectives of regularization (Guo et al., ; Carratino et al., ; Lukasik et al., ; Chidambaram et al., ), adversarial robustness (Zhang et al., ), calibration (Zhang et al., ; Chidambaram & Ge, ), feature learning (Chidambaram et al., ; Zou et al., ), and sample complexity (Oh & Yun, ).

Although the connection between the label smoothing and Mixup losses has been noted in both theory and practice (Carratino et al., ), there has not been (to the best of our knowledge) a unifying theoretical perspective on why the models trained using these losses exhibit similar behavior. In this paper, we aim to provide such a perspective by extending the ideas from previous feature learning analyses to the aspect shared in common between both Mixup and label smoothing: label augmentation. Our analysis allows us to identify a clear failure mode shared by both Mixup and label smoothing - that they can ignore higher variance features - which we also verify empirically.

### 1.1 Main Contributions

Our main contributions are three-fold:

- 1.
We introduce a simple class of data distributions in which there exists a low variance feature and a high variance feature, and show that optimizing the label smoothing and Mixup losses for a linear model for such distributions leads to learning only the low variance feature. On the other hand, if the high variance feature satisfies a greater separability condition relative to the low variance feature, optimizing the empirical risk with weight decay successfully learns both features in the data. Our label smoothing and Mixup results are stated under general conditions on both the data and the respective loss hyperparameters; our Mixup results apply to any mixing distribution that does not coincide with ERM, which greatly generalizes the settings considered by prior work on Mixup feature learning (Chidambaram et al., ; Zou et al., ).
- 2.
We show that our techniques extend to the analysis of general model classes, and prove lower bounds for the label smoothing and Mixup losses in terms of the model output variance. These results suggest that optimizing these losses necessarily decreases model output variance, which is not necessarily the case for training with weight decay.
- 3.
We empirically verify our theoretical insights by considering image classification benchmarks modified to (1) have adversarial spurious correlations in the training data and (2) more natural spurious correlations in the form of class-correlated backgrounds. In both cases, we observe that label smoothing and Mixup fail to generalize for the full range of hyperparameters used in practice, while ERM with weight decay does generalize.

### 1.2 Related Work

Label Smoothing. Since being introduced by Szegedy et al. (), label smoothing continues to be used to train SOTA vision (Wortsman et al., ; Liu et al., ), translation (Vaswani et al., ; Team et al., ), and multi-modal models (Yu et al., ). Attempts to understand when and why label smoothing is effective can be traced back to the influential empirical analyses of Pereyra et al. () and Müller et al. (), which respectively relate label smoothing to entropy regularization and show that it can lead to more closely clustered learned representations, with the latter work further showing that these clustered representations can be responsible for decreased distillation performance.

On the theoretical side, Lukasik et al. () study the relationship between label smoothing and loss correction techniques used to handle label noise, and show that label smoothing can be effective for mitigating label noise. Liu () extends this line of work by analyzing how label smoothing can outperform loss correction in the context of memorization of noisy labels. Wei et al. () further show that negative label smoothing can outperform traditional label smoothing in the presence of label noise. Xu et al. () provide an alternative theoretical perspective, showing that label smoothing can improve convergence of stochastic gradient descent. However, to the best of our knowledge, there are no preexisting theoretical works attempting to understand how label smoothing affects feature learning.

Mixup. Similar to label smoothing, Mixup (Zhang et al., ) and its aforementioned variants have also played an important role in the training of SOTA vision (Wortsman et al., ; Liu et al., ), text classification (Sun et al., ), translation (Li et al., ), and multi-modal models (Hao et al., ), often being applied alongside label smoothing. Initial work on understanding Mixup studied it from the perspective of introducing a data-dependent regularization term to the empirical risk (Carratino et al., ; Zhang et al., ; Park et al., ), with Zhang et al. () and Park et al. () showing that this regularization effect can lead to improved Rademacher complexity.

On the other hand, Guo et al. () and Chidambaram et al. () show that the regularization terms introduced by the Mixup loss can also have a negative impact due to the fact that Mixup-augmented points may coincide with existing training data points, leading to models that fail to minimize the original risk. Mixup has also been studied from the perspective of calibration (Thulasidasan et al., ), with theoretical results (Zhang et al., ; Chidambaram & Ge, ) showing that Mixup training can prevent miscalibration issues arising from ERM (even when post-training calibration methods are used with ERM).

Recently, Oh & Yun () studied Mixup in a similar context to our work (binary classification with linear models), and showed that Mixup can significantly improve the sample complexity required to learn the optimal classifier when compared to ERM. Most closely related with our results in this paper, Mixup has been studied from a feature learning perspective, with Chidambaram et al. () and Zou et al. () both showing that Mixup training can learn multiple features in a generative, multi-view (Allen-Zhu & Li, ) data model despite ERM failing to do so. We provide a more detailed comparison to these works when we discuss our main results in Section 3.

General Data Augmentation. General data augmentation techniques have been a mainstay in image classification since the rise of deep learning models for such tasks (Krizhevsky et al., ). As a result, there is an ever-growing body of theory (Bishop, ; Dao et al., ; Wu et al., ; Hanin & Sun, ; Rajput et al., ; Yang et al., ; Wang et al., ; Chen et al., ; Mei et al., ) aimed at addressing broad classes of augmentations such as those resulting from group actions (i.e. rotations and reflections). Recently, Shen et al. () also studied a class of linear data augmentations from a feature learning perspective, once again using a data model based on the multi-view data of Allen-Zhu & Li (). This work is in the same vein as that of Zou et al. () and Chidambaram & Ge (), although the augmentation considered is not comparable to Mixup.

## 2 Preliminaries

Notation. Given $n\in\mathbb{N}$, we use $[n]$ to denote the set $\{1,2,...,n\}$. For a vector $x\in\mathbb{R}^{d}$ and a subset $S\subset[d]$, we use $x_{S}\in\mathbb{R}^{\absolutevalue{S}}$ to denote the restriction of $x$ to only those indices in $S$, and also use $x_{i}$ to denote the $i^{\text{th}}$ coordinate of $x$. The same notation also applies to matrices; i.e. for a square matrix $T$ we use $T_{S}$ to denote the restriction of both the rows and columns of $T$ to only those dimensions in $S$. Additionally, we use $\succ$ to denote the partial order over positive definite matrices and $\mathrm{Id}$ to denote the identity matrix. We use $\norm{\cdot}$ to indicate the Euclidean norm on $\mathbb{R}^{d}$. For a function $g:\mathbb{R}^{n}\to\mathbb{R}^{m}$ we use $g^{i}(x)$ to denote the $i^{\text{th}}$ coordinate function of $g$. We use $\Delta^{k-1}$ to denote the $(k-1)$-dimensional probability simplex in $\mathbb{R}^{k}$. For a probability distribution $\pi$ we use $\mathrm{supp}(\pi)$ to denote its support. Additionally, if $\pi$ corresponds to the joint distribution of two random variables $X$ and $Y$ (i.e. data and label), we use $\pi_{X}$ and $\pi_{Y}$ to denote the respective marginals, and $\pi_{X\mid Y=y}$ and $\pi_{Y\mid X=x}$ to denote the regular conditional distributions. We use $\Sigma_{X}$ to denote the covariance matrix of a random variable $X\in\mathbb{R}^{d}$. Lastly, we use $\mathrm{Var}(X)$ to denote $\Tr(\Sigma_{X})$ for $X\in\mathbb{R}^{d}$.

We consider the $k$-class classification setting, in which there is a ground-truth data distribution $\pi$ on $\mathbb{R}^{d}\times[k]$ and our goal is to model the conditional distribution $\pi_{Y\mid X}$ using a learned function $g$. In our main theoretical results, we will pay particular attention to the case where $k=2$ and $g$ is a logistic regression model (although we generalize a weak version of these observations to $k$ classes as well), as in this setting we can get a clear handle on the features in the data learned by an optimal model with respect to a particular loss. For this case, we will assume that $\pi$ is supported on $\mathbb{R}^{d}\times\{\pm 1\}$ and that $g$ is parameterized by a weight vector $w$, i.e. $g_{w}(x)=\sigma(w^{\top}x)$, where $\sigma$ is the sigmoid function. Throughout this work, we will consider the following three families of losses.

Standard cross-entropy with optional weight decay. The canonical cross-entropy (or negative log-likelihood) objective in the $k$-class setting is defined as:

$$ $\displaystyle\ell(g)=\mathbb{E}_{(X,Y)\sim\pi}[-\log g^{Y}(X)].$ (2.1) $$

Here we have not specified a weight decay term, since we have placed no constraints on the structure of $g$. On the other hand, for linear binary classification, we can define the binary cross-entropy with optional weight decay as (recalling that $Y\in\{\pm 1\}$)

$$ $\displaystyle\ell_{\beta}(w)$ $\displaystyle=\mathbb{E}_{(X,Y)\sim\pi}\left[-\log g_{w}(YX)\right]+\frac{ \beta}{2}\norm{w}^{2},$ (2.2) $$

where we have defined the loss in terms of the parameter vector $w$. When $\beta>0$, (2.2) has a unique minimizer and we will directly analyze its properties. On the other hand, when $\beta=0$ (no weight decay), this is no longer the case - but our results will still apply to the common case of minimizing (2.2) by scaling the max-margin solution, which we define below.

###### Definition 2.1 .

[Max-Margin Solution]
The max-margin solution $w^{*}$ with respect to $\pi$ is defined as:

$$ $\displaystyle w^{*}$ $\displaystyle=\operatorname*{argmin}_{w\in\mathbb{R}^{d}}\norm{w}^{2}$ $\displaystyle\text{s.t. }y\left\langle w,x\right\rangle$ $\displaystyle\geq 1\quad\forall(x,y)\in\mathrm{supp}(\pi).$ (2.3) $$

Label smoothing. The cross-entropy as defined in (2.1) treats the reference distribution that we compare $g(X)$ to as a point mass on the class $Y$. On the other hand, the label-smoothed cross-entropy is obtained by instead treating the reference distribution as a mixture of a point mass on $Y$ and the uniform distribution over $[k]$. Namely, the label-smoothed loss with mixing hyperparameter $\alpha\in[0,1]$ is defined to be:

$$ $\displaystyle\ell_{\mathrm{LS},\alpha}(g)=-\mathbb{E}_{(X,Y)\sim\pi}\bigg{[}(1 -\alpha)\log g^{Y}(X)+\frac{\alpha}{k}\sum_{i=1}^{k}\log g^{i}(X)\bigg{]}.$ (2.4) $$

And the corresponding binary version is (once again, $Y\in\{\pm 1\}$):

$$ $\displaystyle\ell_{\mathrm{LS},\alpha}(w)=-\mathbb{E}_{(X,Y)\sim\pi}\bigg{[}$ $\displaystyle\left(1-\frac{\alpha}{2}\right)\log g_{w}(YX)+\frac{\alpha}{2} \log g_{w}(-YX)\bigg{]}.$ (2.5) $$

Mixup. Similar to label smoothing, Mixup also augments the reference distribution from being a point mass on $Y$ to being a mixture. However, Mixup also augments the input data as well. In particular, Mixup considers convex combinations of two pairs of points $(X_{1},Y_{1})$ and $(X_{2},Y_{2})$, with a mixing weight sampled from a distribution $\mathcal{D}_{\lambda}$ (which is a hyperparameter) whose support is contained in $[0,1]$. To simplify notation, we will use $X_{1:n}$ and $Y_{1:n}$ to denote multiple inputs $X_{1},...,X_{n}$ and their corresponding labels $Y_{1},...,Y_{n}$, and additionally introduce a function $h$ defined as:

$$ $\displaystyle h(\lambda,g,X_{1:2},Y_{1:2})$ $\displaystyle=\lambda\log g^{Y_{1}}(Z_{\lambda})+(1-\lambda)\log g^{Y_{2}}(Z_{ \lambda})$ (2.6) $\displaystyle\text{where}\quad Z_{\lambda}$ $\displaystyle=\lambda X_{1}+(1-\lambda)X_{2}.$ (2.7) $$

After which we can define the Mixup cross-entropy as:

$$ $\displaystyle\ell_{\mathrm{MIX},\mathcal{D}_{\lambda}}(g)=\mathbb{E}_{(X_{1:2} ,Y_{1:2})\sim\pi,\lambda\sim\mathcal{D}_{\lambda}}\left[-h(\lambda,g,X_{1:2},Y _{1:2})\right].$ (2.8) $$

The corresponding binary version $\ell_{\mathrm{MIX},\mathcal{D}_{\lambda}}(w)$ of (2.8) is identical except for redefining $h$ to be:

$$ $\displaystyle h(\lambda,w,x_{1:2},y_{1:2})$ $\displaystyle=\lambda\log g_{w}(Y_{1}Z_{\lambda})+(1-\lambda)\log g_{w}(Y_{2}Z _{\lambda}).$ (2.9) $$

###### Definition 2.1 .

## 3 Main Theoretical Results

All omitted proofs in this section can be found in Section A of the Appendix.

### 3.1 Linear Binary Classification

We begin first with the binary setting, in which we will consider a data model where a subset of the input dimensions correspond to a low variance feature and the complementary dimensions correspond to a high variance feature that is more separable. This setup is intended to be a simplistic model of spurious correlations that appear in image data; one could think of the low variance feature as a fixed background while the high variance feature corresponds to the actual object category of interest.

The goal in such cases is to ensure that we do not only learn the low variance feature, so that generalization is possible even under a shift of this feature (for example, recognizing the same object on different backgrounds). We will show, however, that doing any kind of label augmentation (label smoothing or Mixup) in this low variance/high variance feature setup will lead to a model that has only learned the low variance feature, whereas minimizing the binary cross-entropy with non-zero weight decay requires learning the high variance feature due to its greater separability. The overarching data model for this subsection is as follows.

###### Definition 3.1 .

[Binary Data Distribution]
We consider $\pi$ to be a distribution supported on $K\times\{\pm 1\}$ where $K$ is a compact subset of $\mathbb{R}^{d}$. We assume that $\pi$ is nondegenerate in that it satisfies $\pi_{Y}(y)>0$ for each $y\in\{\pm 1\}$ and that $\Sigma_{X}$ is positive definite, and we also assume $\mathbb{E}[X]=0$. Additionally, we designate a subset $\mathcal{L}\subseteq[d]$ that we refer to as the low variance feature in the data, and we refer to the complement $\mathcal{H}=\mathcal{L}^{c}$ as the high variance feature.

For convenience, for a vector $v\in\mathbb{R}^{d}$, we will use $v\in\mathcal{L}$ to mean that only the $\mathcal{L}$ dimensions of $v$ are non-zero. Definition 3.1 does not actually make any strong assumptions on $\pi$ yet; this is because the weight decay and label augmentation results have different, non-conflicting assumptions that we codify separately. For the former, we need the data to be more separable in the $\mathcal{H}$ dimensions than in the $\mathcal{L}$ dimensions, in the following sense.

###### Assumption 3.2 .

We assume that for every unit vector $u^{*}\in\mathcal{L}$, there exists a unit vector $v^{*}\in\mathcal{H}$ and $y\left\langle v^{*},x\right\rangle>y\left\langle u^{*},x\right\rangle$ for $\pi$-a.e. $(x,y)$.

Assumption 3.2 is strong, but it actually does not imply linear separability, only that the $\mathcal{H}$ dimensions are in a sense better than the $\mathcal{L}$ dimensions for classification. For a simple example to provide intuition for the assumption, one can consider a 2-D distribution in which the $x$ coordinates for both classes are concentrated close to 0, but the $y$ coordinates are either large negative or large positive values depending on the class.

Our first result shows that for $\ell_{\beta}(w)$ as defined in (2.2), the minimizer $w^{*}$ has a large correlation with the dimensions in $\mathcal{H}$. This is of course intuitive given Assumption 3.2, but it is not immediate because the weight decay penalty in (2.2) encourages distributing norm across all of the dimensions of $w$ (e.g. $\sum w_{i}$ is maximized with respect to the constraint $\norm{w}=1$ by considering $w_{i}=1/\sqrt{d}$).

###### Theorem 3.3 .

Let $w^{*}$ be the unique minimizer of $\ell_{\beta}(w)$ for $\beta>0$ under $\pi$ satisfying Assumption 3.2. Then $\norm{w^{*}_{\mathcal{H}}}^{2}\geq\frac{1}{2}\norm{w^{*}}^{2}$.

Proof Sketch. We can orthogonally decompose the optimal solution $w^{*}$ in terms of unit normal directions $u^{*}\in\mathcal{L}$ and $v^{*}\in\mathcal{H}$. We show that in this decomposition it must be the case that $y\left\langle v^{*},x\right\rangle>y\left\langle u^{*},x\right\rangle$ for $\pi$-a.e. $(x,y)$, and then we can claim that $w^{*}$ must have greater weight associated with $v^{*}$ than $u^{*}$, as otherwise we can decrease $\ell_{\beta}(w^{*})$ by moving weight from $u^{*}$ to $v^{*}$.

We cannot immediately extend the result of Theorem 3.3 to the binary cross-entropy $\ell_{0}(w)$ without weight decay, since there is no unique minimizer of $\ell_{0}$. However, for linear models trained with gradient descent (as is often done in practice), it is well-known that the learned model converges in direction to the max-margin solution (Soudry et al., ; Ji & Telgarsky, ). For this case, the proof technique of Theorem 3.3 readily extends and we obtain the following corollary.

###### Corollary 3.4 .

If $w^{*}$ is the max-margin solution to $\ell_{0}(w)$, then the result of Theorem 3.3 still holds.

On the other hand, as alluded to earlier, this phenomenon does not occur when considering the label smoothing and Mixup losses $\ell_{\mathrm{LS},\alpha}$ and $\ell_{\mathrm{MIX},\mathcal{D}_{\lambda}}$. We will show that for both label smoothing and Mixup, the optimal solutions have arbitrarily small correlation with the dimensions in $\mathcal{H}$ as the distribution of $YX$ concentrates. For these results, we require the following assumptions (but no longer need Assumption 3.2).

###### Assumption 3.5 .

We assume that $\mathbb{E}[YX_{\mathcal{L}}]\neq 0$ and $\Sigma_{YX,\mathcal{H}}\succ\rho\mathrm{Id}$ for some $\rho>0$. Here $\Sigma_{YX,\mathcal{H}}$ denotes the covariance matrix of $\Sigma_{YX}$ restricted to those rows and columns in $\mathcal{H}$.

The first part of Assumption 3.5 just ensures it is possible to obtain a good solution using the dimensions in $\mathcal{L}$ while the second part codifies the idea of $\mathcal{H}$ being a high (at least non-zero) variance feature. Observe that we have made no direct separability assumptions on the data, although it is true that as $\norm{\Sigma_{YX,\mathcal{L}}}\to 0$ the class-conditional supports are guaranteed to be linearly separable in $\mathcal{L}$. We now state both results.

###### Theorem 3.6 .

For $\alpha\in(0,1)$ and $\pi$ satisfying Assumption 3.5, every minimizer $w^{*}$ of $\ell_{\mathrm{LS},\alpha}(w)$ satisfies $\norm{w^{*}_{\mathcal{H}}}<O(\norm{\Sigma_{YX,\mathcal{L}}})$.

Proof Sketch. We show that $\ell_{\mathrm{LS},\alpha}(w)$ is strongly convex in $w^{\top}YX$. We then use Jensen’s inequality to get a lower bound on the loss in terms of this quantity, and show that this lower bound is achievable using only $w\in\mathcal{L}$ as the variance of $YX_{\mathcal{L}}$ decreases. Then using a lower bound on the Jensen gap (Lemma A.1) and Assumption 3.5, we show that any solution that is sufficiently non-zero in the $\mathcal{H}$ dimensions remains bounded away from this lower bound.

###### Theorem 3.7 .

For any symmetric $\mathcal{D}_{\lambda}$ that is not a point mass on 0 0 or $1$ and $\pi$ satisfying Assumption 3.5, every minimizer $w^{*}$ of $\ell_{\mathrm{MIX},\mathcal{D}_{\lambda}}(w)$ satisfies $\norm{w^{*}_{\mathcal{H}}}<O(\norm{\Sigma_{YX,\mathcal{L}}})$.

Proof Sketch. The proof follows the same outline as the label smoothing result, but there are now additional difficulties due to mixing inputs. In particular, we need to condition on $\lambda$ and then show strong convexity of the conditional loss in terms of $w^{\top}Z_{\lambda}$. We can then get a lower bound on the overall loss as an expectation with respect to $\lambda$ of this conditional loss. However, it is no longer immediate that there exists a stationary point $w\in\mathcal{L}$ that minimizes this lower bound. We prove the existence of such a stationary point by considering limiting behavior of the gradient of the loss as $w_{\mathcal{L}}$ tends to $\infty$ or $-\infty$ in each component. After establishing a stationary point, we can show as with the label smoothing result that as $\norm{\Sigma_{YX,\mathcal{L}}}\to 0$ we can get a solution with $w\in\mathcal{L}$ that achieves the lower bound while solutions with $w_{\mathcal{H}}$ non-zero are bounded away from the lower bound.

###### Definition 3.1 .

###### Assumption 3.2 .

###### Theorem 3.3 .

###### Corollary 3.4 .

###### Assumption 3.5 .

###### Theorem 3.6 .

###### Theorem 3.7 .

### 3.2 General Multi-Class Classification

Our results so far have demonstrated a separation between standard training (ERM + weight decay) and label augmentation (label smoothing and Mixup) in the linear binary classification setting, without explicitly having to assume linear separability. The benefit of the linear binary case is that in this case the learning problems are convex in the weight vector $w$, so we can directly discuss properties of the optimal solutions instead of worrying about optimization dynamics.

Of course, this is no longer true when we pass to nonlinear models, and standard model choices (e.g. neural networks) make it so that it is no longer easy to prove that a model has “learned” either $X_{\mathcal{L}}$ or $X_{\mathcal{H}}$ without explicitly analyzing some choice of optimization algorithm (i.e. gradient descent). However, our observations do translate to the outputs of any model.

It is obvious that the model output variance should go to zero for any model that achieves the global optimum of the label augmentation losses we have discussed; indeed, this just corresponds to predicting the correct labels ($\alpha,1-\alpha$ with label smoothing and $\lambda,1-\lambda$ for every $\lambda$ for Mixup) with probability 1. On the other hand, it is less clear that we can make a quantitative statement regarding how bad the loss could be given a certain amount of variance in the model output.

By lifting the techniques of the previous subsection, we can prove such quantitative results for both label smoothing and Mixup in the general multi-class setting. The general data distribution we consider for these results is as follows.

###### Definition 3.8 .

[Multi-Class Data Distribution]
We consider $\pi$ to be a distribution supported on $B\times[k]$ where $B$ is a compact subset of $\mathbb{R}^{d}$ and $k>2$. We assume only that $\pi$ satisfies $\pi_{Y}(y)>0$ for every $y$ and $\Sigma_{X}$ is positive definite.

We make virtually no assumptions on $\pi$ for these results because we will only be proving general lower bounds, which are weaker than the claims regarding the optimal solutions of the previous subsection. For $\pi$ as in Definition 3.8, the label smoothing and Mixup results are as follows.

###### Proposition 3.9 .

For $\alpha>0$ and any $g:\mathbb{R}^{d}\to\Delta^{k-1}$, letting $\mathrm{OPT}_{\mathrm{LS},\alpha}$ denote the minimum of $\ell_{\mathrm{LS},\alpha}$, we have for a universal constant $C>0$:

$$ $\displaystyle\ell_{\mathrm{LS},\alpha}(g)\geq\mathrm{OPT}_{\mathrm{LS},\alpha} +C\sum_{y=1}^{k}\pi_{Y}(y)\mathrm{Var}\left(g(X)\mid Y=y\right).$ (3.1) $$

###### Proposition 3.10 .

For any $\mathcal{D}_{\lambda}$ that is not a point mass on 0 0 or $1$ and any $g:\mathbb{R}^{d}\to\Delta^{k-1}$, letting $\mathrm{OPT}_{\mathrm{MIX},\mathcal{D}_{\lambda}}$ denote the minimum of $\ell_{\mathrm{MIX},\mathcal{D}_{\lambda}}$, we have for a universal constant $C>0$ that:

$$ $\displaystyle\ell_{\mathrm{MIX},\mathcal{D}_{\lambda}}(g)\geq\mathrm{OPT}_{ \mathrm{MIX},\mathcal{D}_{\lambda}}+C\sum_{y_{1}=1}^{k}\sum_{y_{2}=1}^{k}\pi_{ Y}(y_{1})\pi_{Y}(y_{2})\mathbb{E}_{\lambda}\left[\mathrm{Var}\left(g(Z_{ \lambda})\mid y_{1},y_{2},\lambda\right)\right].$ (3.2) $$

The proofs for both Propositions 3.9 and 3.10 follow the same structure; we show that after appropriate conditioning, both losses can be broken up into a sum of strongly convex conditional losses. The lower bounds in both results show that in order to make progress with respect to the label smoothing or Mixup losses, a training algorithm needs to push model outputs to be low variance.

###### Remark 3.11 .

We do not prove an analogous result to Theorem 3.3 in the general setting because Propositions 3.9 and 3.10 operate directly on the model outputs $g(X)$, and it is not clear how to translate a weight norm constraint to this setting without explicitly parameterizing $g$. Intuitively, however, with a weight norm constraint on $g$ it may no longer be optimal to have zero variance predictions, since such predictions may require very large weights depending on the data distribution.

Comparisons to existing results. We are not aware of any existing results on feature learning for label smoothing; prior theoretical work largely focuses on the relationship between label smoothing and learning under label noise, which is orthogonal to the perspective we take in this paper. Although feature learning results exist for Mixup (Chidambaram et al., ; Zou et al., ), the pre-existing results are constrained to the case of mixing using point mass distributions and only prove a separation between Mixup and unregularized ERM, whereas our results work for arbitrary symmetric mixing distributions (that don’t coincide with ERM) and also separate Mixup from ERM with weight decay. This greater generality comes at the cost of considering only linear models for our feature learning results; both Chidambaram et al. () and Zou et al. () consider the training dynamics of 2-layer neural networks on non-separable data.

Additionally, our results can be viewed as generalizing the observations made by Chidambaram et al. (), which were that Mixup can learn multiple features in the data when doing so decreases the variance of the learned predictor. In our case, we directly show that Mixup will have much larger correlation with the low variance feature in the data as opposed to the high variance feature. Our results also do not contradict the observations of Zou et al. (), which were that Mixup can learn both a “common” feature and a “rare” feature in the data; in their setup the common/rare features are fixed (zero variance) per class and concatenated with high variance noise.

###### Definition 3.8 .

###### Proposition 3.9 .

###### Proposition 3.10 .

###### Remark 3.11 .

## 4 Experiments

We now verify the predictions of our theoretical results in Section 3 across both binary and multi-class experimental setups. All experiments in this section were conducted on a single P100 GPU using PyTorch (Paszke et al., ) for model implementation. All reported results correspond to means over 5 training runs, and shaded regions in the figures correspond to 1 standard deviation bounds.

### 4.1 Spurious Pixels in Binary Classification

Figure: (a) Weight Decay
Refer to caption: extracted/5624271/plots/CIFAR_WD_sweep.png

We first consider the case of logistic regression trained on data that has trivial spurious correlations that do not appear in the test data. For our binary classification task, we consider reductions of the standard CIFAR-10 and CIFAR-100 (Krizhevsky, ) benchmarks to binary classification by fixing two classes from each dataset as the positive and negative classes, and replacing their original labels with the labels $+1$ and $-1$ respectively. This leads to 10000 training examples and 2000 test examples in the case of CIFAR-10, and 1000 training examples and 200 test examples in the case of CIFAR-100. Our experiments are not sensitive to the choice of binary reduction, and for the experiments in this section we fix the $-1$ class to be class 0 0 from the original data and the $+1$ class to be class $1$ from the original data (“airplane” and “automobile” for CIFAR-10, “apple” and “aquarium fish” for CIFAR-100).

We preprocess the training data to have mean zero and variance one along every image channel (and apply the same transform to the test data). To introduce spurious correlations into the training data, we modify the data such that the first value in the tensor representation of each training input (corresponding to the “R” channel of the top left pixel in the original image) is replaced by $y\gamma$ with $\gamma=0.1$ (the choice of $\gamma$ here is not important as long as it is small, we verified the results for $\gamma=10^{-5}$ up to $\gamma=0.1$). This ensures that the training data is trivially linearly separable in the first dimension of the data, but learning this first dimension requires having larger weight norm due to the scaling by $\gamma$. We leave the test data unchanged - our goal is to determine whether models trained on the modified training data can learn more than just the single identifying dimension.

We then train logistic regression models to convergence (100 epochs) on both the reduced CIFAR-10 and CIFAR-100 tasks across a range of settings for weight decay, label smoothing, and Mixup. We consider 20 uniformly spaced values in $[0,0.1]$ for the weight decay $\lambda$ parameter and in $[0,0.75]$ for the label smoothing $\alpha$ parameter, where the upper bound for the label smoothing parameter is obtained from the experiments of Müller et al. (). For Mixup, we fix the mixing distribution to be the canonical choice of $\mathrm{Beta}(\alpha,\alpha)$ introduced by Zhang et al. () and consider 20 uniformly spaced $\alpha$ values in $[0,8]$ (with $\alpha=0$ corresponding to ERM), which effectively covers the range of Mixup hyperparameter values used in practice. We train all models using Adam with the hyperparameters of $\beta_{1}=0.9,\beta_{2}=0.999$, a learning rate of $5*10^{-3}$, and a batch size of 500.

After training, we report the mean test error along with one standard deviation error bounds over five training runs for each model setting in Figure 1. As can be seen from the results, both the label smoothing and Mixup models have high test error for all settings while weight decay achieves a significantly lower test error for all $\lambda>0$. The higher variance in the reported CIFAR-100 test errors can almost certainly be attributed to the fact that our binary version of CIFAR-100 consists of only 1000 training points and 200 points.

Figure: (a) CIFAR-10
Refer to caption: extracted/5624271/plots/CIFAR10_weight_ratios.png

Our results correspond to label smoothing and Mixup learning to use only the spurious, identifying dimension of the training inputs, as we expect from our theory since this dimension has zero conditional variance. Indeed, we verify this fact empirically in Figure 2, where we plot the ratio $\norm{w_{1}}/\norm{w_{[d]\setminus\{1\}}}$ (i.e. the ratio between the norm of the trained model weight vector in the first dimension and the remaining dimensions) for each of the trained models.

### 4.2 Background Shift in Multi-Class Classification

Figure: (a) Shaded MNIST
Refer to caption: extracted/5624271/plots/mnist_grid.jpg

Although the experiments of Section 4.1 cleanly verify our theory, the spurious correlation introduced in the training data is effectively adversarial in nature. We now consider the case of a spurious correlation of greater practical interest: backgrounds that are correlated with the correct class label.

In particular, we consider a very similar setup to the one used by Arjovsky et al. () to motivate the influential invariant risk minimization framework: namely, we construct a shaded version of the MNIST dataset in which the background pixels for each class are replaced with different low intensity colors corresponding to the class labels. Unlike Arjovsky et al. (), however, we maintain all 10 classes since our theory from Section 3.2 suggests that our observations should generalize to the multi-class setting. The colors used for each class background are permuted between the train and the test data, so that a model that learns to predict only using background pixels cannot achieve good test accuracy.

The only constraints we place on the background colors are that they are distinct across classes and that their intensities (i.e. values in RGB space) are small (it suffices to consider values bounded by 16); this makes it difficult to visually distinguish the background colors (hence "shaded MNIST") even though they are obviously different in RGB space. The color intensity constraint is not necessary for the failure of label smoothing and Mixup, but is necessary for weight decay to succeed since it ensures that the higher variance feature (the actual digit in the foreground) is generally more separable (due to larger pixel intensities). Note that although there is no variance in the background color conditional on a class, there is significant variance in the actual pixels per class as the locations of the background pixels change across data points.

For our model setup, we also follow Arjovsky et al. () and consider a simple 2-layer feedforward neural network with ReLU activations and a hidden layer size of $2048$, as this is sufficient to achieve good performance on MNIST and is a tiny enough model that we can efficiently do the same hyperparameter sweep from Section 4.1. We follow the same training procedure as Section 4.1, training until the training loss roughly converges (only 20 epochs are necessary).

Test error results across the different hyperparameter settings for weight decay, label smoothing, and Mixup are shown in Figure 3. We observe the same phenomena as before, even in this non-trivial spurious correlation setting: both the label smoothing and Mixup models have high test error for all settings while weight decay achieves a significantly lower test error for all $\lambda>0$.

## 5 Conclusion

In this work, we have shown that label augmentation strategies such as label smoothing and Mixup exhibit a variance minimization effect that can lead to learning low variance, spurious correlations in the data even when there exists a better but higher variance feature. The main practical takeaway from this is that label smoothing and Mixup do not necessarily always improve the robustness of models, and their failure to do so may be indicative of certain identifying features in the training data (i.e. adversarially introduced patches or background pixels).

Limitations. The main theoretical limitation of work is that our results for general models are only loss lower bounds and do not characterize optimization dynamics. Our experiments that such dynamics for feedforward networks would still reflect our observations, however, and it would be an interesting follow-up to study these dynamics using the techniques of Zou et al. () and Chidambaram et al. (). On the empirical side, our main limitation is that we have demonstrated our theory using only small-scale experiments; a comprehensive empirical evaluation of label smoothing and Mixup in larger scale spurious correlations settings guided by our theory would be a natural avenue for future work.

Broader Impacts. Although label augmentation are used to train and finetune large-scale models, our results concerning them in this work have mostly been theoretical. As a result, we do not anticipate any direct misuse of our results or any broader harmful impacts.

## Acknowledgements

Rong Ge and Muthu Chidambaram are supported by NSF Award DMS-2031849 and CCF-1845171
(CAREER). Muthu would like to thank Annie Tang for thoughtful feedback during the early stages of this project.

## Appendix A Omitted Proofs

### A.1 Helper Lemma

The following lemma lower bounds the Jensen gap in terms of the variance of the random variable being considered. We will make repeated use of it in the proofs of the label smoothing and Mixup results.

###### Lemma A.1 .

Let $\phi:\mathbb{R}^{d}\to\mathbb{R}$ be a twice-differentiable convex function satisfying $\gamma_{1}\mathrm{Id}\prec\nabla^{2}\phi\prec\gamma_{2}\mathrm{Id}$. Then for any square integrable random variable $X$ on $\mathbb{R}^{d}$ it follows that:

$$ $\displaystyle\mathbb{E}[\phi(X)]-\phi(\mathbb{E}[X])\in\left[\frac{\gamma_{1}} {2}\mathrm{Var}(X),\frac{\gamma_{2}}{2}\mathrm{Var}(X)\right].$ (A.1) $$

###### Proof.

From the assumption of the lemma it follows that $\phi(x)-\gamma_{1}\norm{x}^{2}/2$ is convex and $\phi(x)-\gamma_{2}\norm{x}^{2}/2$ is concave. Applying Jensen’s inequality to each of these functions yields (A.1).
∎

###### Lemma A.1 .

###### Proof.

### A.2 Proofs for Section 3.1

See 3.3

###### Proof.

Let us decompose the unique minimizer $w^{*}$ as $w^{*}=\alpha_{1}u^{*}+\alpha_{2}v^{*}$, where $u^{*}$ and $v^{*}$ are orthonormal and satisfy $u^{*}_{\mathcal{H}}=0,v^{*}_{\mathcal{L}}=0$ (i.e. $u^{*}$ is a normalized version of the $\mathcal{L}$ components of $w^{*}$ and $v^{*}$ is the same but for the $\mathcal{H}$ components). We claim that $y\left\langle v^{*},x\right\rangle>y\left\langle u^{*},x\right\rangle$ for $\pi$-a.e. $(x,y)$. Indeed, if this were not the case then by the assumption on $\pi$ we could choose an orthonormal vector $z^{*}$ with $z^{*}_{\mathcal{L}}=0$ that satisfies $y\left\langle z^{*},x\right\rangle>y\left\langle u^{*},x\right\rangle$ for $\pi$-a.e. $(x,y)$ and decrease the loss of $w^{*}$ by replacing $v^{*}$ with $z^{*}$.

Now suppose that $\alpha_{1}>\alpha_{2}$. Then we claim that replacing $\alpha_{1}$ and $\alpha_{2}$ with $\gamma=\sqrt{(\alpha_{1}^{2}+\alpha_{2}^{2})/2}$ yields a solution with lower loss than $w^{*}$.

To see this, first observe that $2\gamma^{2}=\alpha_{1}^{2}+\alpha_{2}^{2}$, so that the norm of the modified solution is the same as $w^{*}$. This implies that the weight decay penalty term in $\ell_{\beta}$ is unchanged.

Furthermore, we have that $\gamma\in(\alpha_{2},\alpha_{1})$, and that $\gamma-\alpha_{2}>\alpha_{1}-\gamma$. This follows from the fact that $\gamma\geq(\alpha_{1}+\alpha_{2})/2$ by construction. This, combined with the fact that $y\left\langle v^{*},x\right\rangle>y\left\langle u^{*},x\right\rangle$, then implies for $\pi$-a.e. $(x,y)$ that:

$$ $\displaystyle y\left\langle\gamma u^{*}+\gamma v^{*},x\right\rangle>y\left \langle\alpha_{1}u^{*}+\alpha_{2}v^{*},x\right\rangle.$ (A.2) $$

Which contradicts the optimality of $w^{*}$. Therefore, we must have $\alpha_{1}\leq\alpha_{2}$, which gives the desired result.
∎

See 3.4

###### Proof.

The result follows from the same decomposition approach of Theorem 3.4 and the definition of the max-margin solution (Definition 2.1). Namely, we can move weight from the max-margin direction in $\mathcal{L}$ to the max-margin direction in $\mathcal{H}$ to decrease the norm while satisfying the separation constraint.
∎

See 3.6

###### Proof.

We follow the outline in the proof sketch. We first observe that there exists $M$ such that it suffices to consider $\norm{w}\leq M$; this is due to the fact that for $\alpha>0$, we have $\lim_{\norm{w}\to\infty}\absolutevalue{\ell_{\mathrm{LS},\alpha}(w)}=\infty$. With this in mind, let us use $Z$ to denote $w^{T}YX$ and use $\ell_{\mathrm{LS},\alpha}(Z)$ to denote the loss in terms of this quantity. Then it follows that

$$ $\displaystyle\partialderivative{\ell_{\mathrm{LS},\alpha}(Z)}{Z}$ $\displaystyle=\mathbb{E}\left[\sigma(Z)-1+\alpha/2\right],$ (A.3) $\displaystyle\partialderivative[2]{\ell_{\mathrm{LS},\alpha}(Z)}{Z}$ $\displaystyle=\mathbb{E}\left[\sigma(Z)(1-\sigma(Z))\right],$ (A.4) $$

where in both cases we applied the dominated convergence theorem, which is justified because $\ell_{\mathrm{LS},\alpha}$ is smooth in $Z$ with bounded derivatives. Now since $\norm{w}\leq M$ and the support of $X$ is compact, there exist $\gamma_{1}$ and $\gamma_{2}$ such that $\partialderivative[2]{\ell_{\mathrm{LS},\alpha}(Z)}{Z}\in(\gamma_{1},\gamma_{2})$, which implies that $\ell_{\mathrm{LS},\alpha}$ is strongly convex in $Z$ and satisfies the conditions of Lemma A.1.

By Jensen’s inequality, we then have that:

$$ $\displaystyle\ell_{\mathrm{LS},\alpha}(Z)\geq\left(\frac{\alpha}{2}-1\right) \log\sigma(\mathbb{E}[Z])-\frac{\alpha}{2}\log\sigma(-\mathbb{E}[Z]).$ (A.5) $$

Since $\mathbb{E}[YX_{\mathcal{L}}]\neq 0$ by Assumption 3.5, we can choose $w$ such that $w_{\mathcal{H}}=0$ and $\mathbb{E}[Z]=\sigma^{-1}(1-\alpha/2)$, which minimizes the RHS of (A.5). Let us use $\mathrm{OPT}$ to denote this minimum. Then by Lemma A.1, we have for any $w$ chosen as described:

$$ $\displaystyle\ell_{\mathrm{LS},\alpha}(w)-\mathrm{OPT}\leq\frac{\gamma_{2}}{2} w^{T}\Sigma_{YX,\mathcal{L}}w.$ (A.6) $$

On the other hand, if $w^{\prime}$ is another solution satisfying $\norm{w^{\prime}_{\mathcal{H}}}>\epsilon$, then Lemma A.1 gives

$$ $\displaystyle\ell_{\mathrm{LS},\alpha}(w^{\prime})-\mathrm{OPT}\geq\frac{ \gamma_{1}\rho\epsilon^{2}}{2},$ (A.7) $$

from which it is clear that for appropriate $\epsilon$ we cannot have $w^{\prime}$ be a stationary point ($\rho$ above is the same as in Assumption 3.5). That we can take $\epsilon\to 0$ as $\norm{\Sigma_{YX,\mathcal{L}}}\to 0$ follows from (A.6).
∎

See 3.7

###### Proof.

Let us first outline the overall steps of the proof, and the differences with the label smoothing case.

- 1.
We first show that the loss conditioned on $\lambda$ is strongly convex in $w^{\top}Z_{\lambda}$. The conditioning on $\lambda$ here is necessary because $\lambda$ is a random variable, unlike $\alpha$ in the label smoothing case. The overall goal here is to use the same argument as for label smoothing, i.e. show that we can achieve the optimal lower bound in terms of $\mathbb{E}[w^{\top}Z_{\lambda}]$ using only $w_{\mathcal{L}}$ and letting $\norm{\Sigma_{YX,\mathcal{L}}}\to 0$.
- 2.
We cannot explicitly minimize the conditional loss like we did with label smoothing, since it is not possible with a fixed choice of $w$ to achieve $\sigma^{-1}(\mathbb{E}[w^{\top}Z_{\lambda}])=\lambda$ for every $\lambda$ simultaneously. Instead, we will show that a stationary point of the conditional loss exists that uses only the dimensions of $w_{\mathcal{L}}$.
- 3.
Having shown the above, we can just reuse the same argument as before with Lemma A.1 to prove the desired result.

Let $\ell_{\mathrm{MIX},\lambda}$ denote $\ell_{\mathrm{MIX},\mathcal{D}_{\lambda}}$ with a fixed choice of $\lambda$ (i.e. after conditioning on $\lambda$), and let $R=w^{\top}Z_{\lambda}$. Then we can compute:

$$ $\displaystyle\partialderivative{\ell_{\mathrm{MIX},\lambda}(R)}{R}$ $\displaystyle=\mathbb{E}\big{[}\lambda(\sigma(Y_{1}R)-1)Y_{1}+(1-\lambda)( \sigma(Y_{2}R)-1)Y_{2}\big{]},$ (A.8) $\displaystyle\partialderivative[2]{\ell_{\mathrm{MIX},\lambda}(R)}{R}$ $\displaystyle=\mathbb{E}\big{[}\lambda\sigma(Y_{1}R)(1-\sigma(Y_{1}R))+(1- \lambda)\sigma(Y_{2}R)(1-\sigma(Y_{2}R))\big{]},$ (A.9) $$

where again we applied dominated convergence to the expectation with respect to $\pi$. Strong convexity follows from the same consideration as label smoothing; namely, we can consider $\norm{w}\leq M$ as $\ell_{\mathrm{MIX},\lambda}(R)\to\infty$ as $\norm{w}\to\infty$ so long as $\lambda$ is not 0 or 1 and $\pi_{Y}(y)>0$, and this consequently implies (A.9) is lower bounded by some positive real number.

Now by conditional Jensen’s inequality, we obtain the following lower bound for $\ell_{\mathrm{MIX},\mathcal{D}_{\lambda}}$:

$$ $\displaystyle\ell_{\mathrm{MIX},\mathcal{D}_{\lambda}}(w)\geq\mathbb{E}_{ \lambda}\left[\mathbb{E}_{Y_{1},Y_{2}}\left[\ell_{\mathrm{MIX},\lambda}( \mathbb{E}\left[R\mid\lambda,Y_{1},Y_{2}\right])\right]\right].$ (A.10) $$

We now show that it is possible to minimize this lower bound while taking $w_{\mathcal{H}}=0$. This is more difficult than it was in the label smoothing case, because it is no longer obvious that $\mathbb{E}[YX_{\mathcal{L}}]\neq 0$ is sufficient for minimizing the RHS of (A.10) due to the expectation with respect to $\lambda$. However, we can show the existence of a stationary point with $w_{\mathcal{H}}=0$, even though we cannot provide an explicit construction.

The idea is to consider the limiting behavior of (A.10) as we take the values of $w_{\mathcal{L}}$ to $-\infty$ and $\infty$. Note that we can take this limit into the expectation with respect to $\lambda$ by dominated convergence again. Let us consider the gradient with respect to $w$ of the RHS of (A.10). To make notation manageable, we will use $a_{1}=\mathbb{E}[X\mid Y=1],a_{2}=\mathbb{E}[X\mid Y=-1]$, and $a_{3}=\mathbb{E}[Z_{\lambda}\mid\lambda,Y_{1}=1,Y_{2}=-1]$. We can then explicitly write out the gradient as the expectation with respect to $\lambda$ of the sum of the following three terms (considering the different cases for $Y_{1},Y_{2}$):

$$ $\displaystyle\gradient_{w}\mathbb{E}_{Y_{1},Y_{2}}\left[\ell_{\mathrm{MIX}, \lambda}(\mathbb{E}\left[R\mid\lambda,Y_{1},Y_{2}\right])\right]$ $\displaystyle=\pi_{Y}(1)^{2}\left(\sigma(w^{\top}a_{1})-1\right)a_{1}$ $\displaystyle\quad-\pi_{Y}(-1)^{2}\left(\sigma(-w^{\top}a_{2})-1\right)a_{2}$ $\displaystyle\quad+2\pi_{Y}(1)\pi_{Y}(-1)\bigg{(}\lambda\big{(}\sigma(w^{\top} a_{3})-1\big{)}$ $\displaystyle\quad\quad\quad-(1-\lambda)\big{(}\sigma(-w^{\top}a_{3})-1\big{)} \bigg{)}a_{3}.$ (A.11) $$

The first two lines above are obtained from the fact that we can combine terms when $Y_{1}=Y_{2}$, and the last line is by symmetry. Now we recall that by assumption $\mathbb{E}[YX_{\mathcal{L}}]\neq 0$ and $\mathbb{E}[X]=0$. Thus, WLOG, we can assume that $\mathbb{E}[X_{\mathcal{L}}\mid Y=1]_{i}>0$ and $\mathbb{E}[X_{\mathcal{L}}\mid Y=-1]_{i}<0$ for each index $i$.

With this in mind, we consider first the case of what happens when the entries $w_{\mathcal{L}}\to\infty$. Since $w^{\top}a_{1}>0$ and $w^{\top}a_{2}<0$, the first two terms in (A.11) vanish (independent of $\lambda$). On the other hand, for the third term, there are two cases to consider. Depending on $\lambda$, we have that the entries of $a_{3}$ are either strictly negative or strictly positive, with the exceptional case of $\lambda=\pi_{Y}(1)$ in which $a_{3}=0$. If the entries of $a_{3}$ are strictly negative, then $\big{(}\sigma(-w^{\top}a_{3})-1\big{)}\to 0$ and the coefficient becomes negative, so the third term is positive. Similarly, if $a_{3}$ is strictly positive, the coefficient is positive and the third term is still positive. Thus, as $w_{\mathcal{L}}\to\infty$ every entry of (A.11) is positive.

Similar arguments show that the opposite is true when we take $w_{\mathcal{L}}\to-\infty$. Now by continuity of the gradient, it immediately follows that there is some choice of $w$ with only $w_{\mathcal{L}}$ non-zero such that the expectation of $\eqref{eq:mixsum}$ with respect to $\lambda$ can be made to be zero. Using this choice of $w$ allows us to obtain an $R$ that minimizes the RHS of (A.10).

Now we have basically arrived at the same stage as the end of the label smoothing proof. By taking $\norm{\Sigma_{YX,\mathcal{L}}}\to 0$ we can get arbitrary concentration around this optimal $R$, and by the same logic as the label smoothing proof the result follows.
∎

###### Proof.

###### Proof.

###### Proof.

###### Proof.

### A.3 Proofs for Section 3.2

See 3.9

###### Proof.

Let us first define:

$$ $\displaystyle\ell_{\mathrm{LS},\alpha,y}(g)=-\mathbb{E}\bigg{[}(1-\alpha)\log g ^{y}(X)+\frac{\alpha}{k}\sum_{i=1}^{k}\log g^{i}(X)\mid Y=y\bigg{]}.$ (A.12) $$

Then we can decompose $\ell_{\mathrm{LS},\alpha}(g)$ as follows:

$$ $\displaystyle\ell_{\mathrm{LS},\alpha}(g)=\sum_{y=1}^{k}\pi_{Y}(y)\ell_{ \mathrm{LS},\alpha,y}(g).$ (A.13) $$

Once again, since we can restrict our attention to $g(X)\in[\gamma,1-\gamma]$ for some $\gamma$ (as the loss goes to infinity if $g(X)=1$ on a set of positive $\pi_{X}$-measure with $\alpha>0$), it is easy to verify that (A.12) is strongly convex in $g(X)$. The desired result then follows by applying Lemma A.1 with the regular conditional distribution $g(X)\mid Y=y$ for each term in (A.13).
∎

See 3.10

###### Proof.

The proof follows an identical structure to that of Proposition 3.9. In particular, we again define the following conditional loss:

$$ $\displaystyle\ell_{\mathrm{MIX},\lambda,y_{1},y_{2}}=-\mathbb{E}[\lambda\log g ^{y_{1}}(Z_{\lambda})+(1-\lambda)\log g^{y_{2}}(Z_{\lambda})\mid y_{1},y_{2}, \lambda].$ (A.14) $$

And we can then decompose $\ell_{\mathrm{MIX},\mathcal{D}_{\lambda}}$ as:

$$ $\displaystyle\ell_{\mathrm{MIX},\lambda}(g)=\sum_{y_{1}=1}^{k}\sum_{y_{2}=1}^{ k}\pi_{Y}(y_{1})\pi_{Y}(y_{2})\mathbb{E}_{\lambda\sim\mathcal{D}_{\lambda}}[ \ell_{\mathrm{MIX},\lambda,y_{1},y_{2}}(g)].$ (A.15) $$

Since $\mathcal{D}_{\lambda}$ is not a point mass on 0 0 or $1$, we can restrict ourselves to $g(X)\in[\gamma,1-\gamma]$ for some $\gamma$ as before, and strong convexity in $g(X)$ of the conditional loss (A.14) again follows. We then apply Lemma A.1 to obtain the result.
∎

###### Proof.

###### Proof.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Can these results extend to multi-label classification?
soundness: 2
strengths: 1. This manuscript is well-written, and the theoretical parts are clearly presented.
2. This manuscript links label smoothing and Mixup to feature variance, offering new insights into the analysis of label augmentation.
summary: This manuscript studies the relationship between label augmentation techniques (label smoothing and Mixup) and feature variance. The authors demonstrate that label augmentation in binary classification with a linear model encourages the model to learn low-variance features. In contrast, ERM with weight decay can lead to learning higher-variance features under certain conditions. They also show that the losses of label smoothing and Mixup for nonlinear models are lower-bounded by a function of the model output variance. Additionally, they provide empirical evidence that label augmentation may negatively impact model training.
weaknesses: 1. The main claim that “optimizing the empirical risk with weight decay can learn higher variance features” is built on a strong assumption. As the authors note, Assumption 3.2 is indeed strong, suggesting that the $\\mathcal{H}$ dimensions are inherently better suited for classification than the $\\mathcal{L}$ dimensions. However, it’s straightforward to construct a task that relies more heavily on the $\mathcal{L}$ dimensions than on the $\mathcal{H}$ dimensions. For instance, consider a 2-D distribution where the $x$ coordinates for both classes are uniformly ranging from -1 to 1, while the $y$ coordinates are set at -0.1 or 0.1, depending on the class. In fact, by simply reversing the assumption from $y \\langle v^*, x \\rangle > y \langle u^*, x \\rangle$ to $y \\langle u^*, x \\rangle > y \langle v^*, x \rangle$, the conclusion of Theorem 3.3 changes from $\\|w^*_\\mathcal{H}\\|^2 \ge \frac{1}{2} \\|w^*\\|^2$ to $\\|w^*_\\mathcal{L}\\|^2 \ge \frac{1}{2} \\|w^*\\|^2$.
2. There is no discussion on why label smoothing or Mixup learn more on low-variance features compared to ERM or ERM with weight decay. In fact, by setting $\\alpha$ to 0, the label smoothing loss reduces to the negative log-likelihood loss, allowing one to obtain results similar to those in Theorem 3.6 or Proposition 3.10 for ERM, which also demonstrate that the ERM loss is lower-bounded by a function of the model output variance. Thus, $\\alpha$ plays a central role in learning low-variance features, but this important analysis is missing.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - In the proof of Corollary 3.4, what happens if the max-margin solution does not include high-variance features? It appears that the proof technique may not apply in this scenario. Could the authors clarify whether this case is impossible or if there are additional conditions to address it?
soundness: 3
strengths: The theoretical results are insightful and are effectively supported by a variety of well-designed experiments.
summary: The paper presents a theoretical analysis of label smoothing and Mixup techniques. The authors investigate the differences between these label augmentation methods and the vanilla approach (i.e., empirical risk minimization with weight decay) regarding their effectiveness in learning features of varying variance. In a binary linear classification problem, they prove that, under certain assumptions, the vanilla method prefers high-variance features, while label augmentation encourages the model to learn low-variance features. This analysis is then extended to more complex settings by showing that the label augmentation loss is lower-bounded by model variance. The theoretical findings are supported by empirical results on both original and modified CIFAR-10, CIFAR-100, and colored MNIST datasets.
weaknesses: - It remains unclear why learning low-variance features leads to better generalization in practical scenarios, as this approach can result in poorer generalization on synthetic datasets.
- The comparison to previous works on feature learning (e.g., Chidambaram et al., 2023; Zou et al., 2023) is insufficient. I encourage the authors to provide a more detailed comparison with these works (e.g. empirical loss vs population loss). Additionally, the notion of "features" appears to differ between this work and prior studies; clarifying what "features" specifically refers to in this context would enhance the reader’s understanding.

### Minors

- Line 795: “approach of Theorem 3.4” → "approach of Theorem 3.3" and I hope the author provides a complete proof of Corollary 3.4 for clarity, even if it largely repeats the proof of Theorem 3.3.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. For the CIFAR experiment with the spurious feature, are you training the full model or just probing a frozen ResNet with a linear logistic classifier? Additionally can you clarify the difference between line 65 where you specify "if the high variance feature satisfies a greater separability" and the experiment where the low variance spurious feature is more separable.
2. The experiments all use heavily overparameterised networks. 2048 is somewhat overkill for MNIST and ResNet-18 is not designed for CIFAR but ImageNet. ResNet-18 has over 10 million parameters whilst ResNet-56 which is designed for CIFAR has < 1 million and both achieve similar performance on CIFAR. Can you comment on how overparameterisation/overfitting affects the theory/experiments of the paper? This contrasts to the binary linear scenario where underfitting/model being insufficiently expressive may be more likely(?).

I am happy to raise my score if the stated weaknesses (other than 3 which I feel unfortunately can't be changed) and above questions are addressed.

Edit: as the authors have addressed most of my queries and concerns, and also improved the paper in response to the other reviewers, I have raised my score from 5 to 6.
soundness: 3
strengths: - The paper provides some new theoretical results that contribute to better understanding of the behaviour of label augmentation approaches. I appreciate work that tries to advance knowledge in deep learning rather than simply chasing benchmark scores so this is a big plus for me.
- The experiments are well-targeted to verify the theoretical results.
summary: In this paper the authors provide theoretical results pertaining to the variance of features in relation to training with label augmentation (Label Smoothing and Mixup). They show that for a binary linear logistic classifier trained on a certain distribution where there is a high and a low variance feature, label augmentation leads optimisation to only consider the low variance feature. For general models and classification they provide a quantative lower bound on how the variance of output probabilities contributes to the loss. The authors then provide some experiments to empirically verify the theoretical results.
weaknesses: I would like to clarify that I did not have time to comb the fine details of the paper or verify the theory, however I believe I have allocated a reasonable amount of time to reading from start to end. I am also not a "theoretical" machine learning researcher personally, and I am more familiar with the literature for label smoothing than Mixup, which should help put my later comments in context. I welcome the authors to correct any mistakes/misunderstandings that I make in this review.

1. The presentation of the paper inhibits easy understanding of the theoretical takeaways for a general machine learning audience (i.e. the average researcher using label augmentation). There isn't a single visual aid to illustrate any of the theoretical results, making it both difficult to intuit the theory and understand the key takeaways. For example, for the linear binary classifier, the paper would be greatly aided with a simple 2D illustration showing how the optimal decision boundaries differ between with and without Label Smoothing. Similarly I would suggest an illustrative sketch over 3 classes (visually in the same vein as those in [1]) of how greater variance contributes to loss. I think the text could also be improved by highlighting the key theoretical takeaways in plain English (e.g. in a box/table/in bold), allowing the less mathematically inclined reader to not have to parse through the comparatively dense and notation heavy theory.
2. Missing related work: there are a few missing references to label smoothing in the context of uncertainty estimation [2,3], neural collapse [4,5] and transfer learning [6].  [4,5,6]  are especially relevant and should be discussed as they directly measure how tight features cluster together after training with label smoothing. [6] also demonstrates that tighter feature clusters lead to **worse** linear probing performance.
3. Limited insight with regards to generalisation. The paper lacks any direct insight into how label augmentation leads to better generalisation peformance, which is ultimately why these techniques are used in the first place. Although the authors do fully acknowledge this limitation, it doesn't change the fact that this limits the impact of the insights of the paper. 
4. The practical takeaways of the spurious correlation experiments are somewhat unclear. It is not necessarily the case that spurious correlations tend to be lower variance. In the case that they are high variance wouldn't the inverse behaviour manifest? 

[1] Muller et al. When Does Label Smoothing Help? 2019

[2] Zhu et al. Rethinking Confidence Calibration for Failure Prediction, 2022

[3] Xia et al. Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It, 2024

[4] Zhou et al. Are All Losses Created Equal: A Neural Collapse Perspective, 2022

[5] Guo et al. Cross Entropy versus Label Smoothing: A Neural Collapse Perspective, 2024

[6] Kornblith et al. Why Do Better Loss Functions Lead to Less Transferable Features? 2021

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Line 135 on page 3. The identity matrix shouldd be denoted by mathematical expression "$I_d$" rather than text "Id" right?
2. In the context of binary classification, the label variable $y$ should be $Y\in \{-1, +1\}$ right? But in that case, the cross-entropy loss defined in equation 2.1 cannot be applied on binary classification. Or, is the paper implicitly regarding the term "$k$-class classification" to be distinct from the term "binary classification"?
3. In equation 2.4, I think the second term inside the expectation should be "$\dfrac{\alpha}{k-1}\sum_{i\in [k]/{Y}}\log{g^i(X)}$".
4. In the definition of Mixup loss in equation 2.8, are the points pairs $(X_1,Y_1)$ and $(X_2,Y_2)$ sampled independently from $\pi$? In that context, I would denote it as $(X_{1:2},Y_{1:2})\overset{\mathrm{iid}}{\sim}\pi$ or something like $(X_{1:2},Y_{1:2})\sim\pi\bigotimes\pi$
5. What are the mathematical definitions of low variance features and high variance features? Is it the overall variance? Or in-class variances? Also, how would one justify if a feature is high variance or low? Should there be a predefined threshold? Or maybe even a post-defined threshold, meaning threshold is defined after the groups of high variance feature and low variance features are manually partitioned?
6. Before assumption 3.2, $\mathcal{L}$ and $\mathcal{H}$ are at first defined as subsets of $[d]$, and then regarded as sets of vectors. I understand this is only for the sake of simplicities in notations, but I would suggest that the paper simply uses different notations.
7. In assumption 3.5, how does the first part ensure that we can obtain good solution using low variance dimensions? First, I think if a data is class-balanced (the marginal distribution on the classes is discrete uniform), and given an early assumption that $\mathbb{E}[X]=0$, then $\mathbb{E}[YX_\mathcal{L}]$ should always be 0. Second, if the data is imbalanced, then this first part might hold true, but then does it mean we are only dealing with class-imbalanced data from now on? Also, even if the data is imbalanced and $\mathbb{E}[YX_\mathcal{L}]$ equals 0, it may still be possible to find a good solution.
8. The assumptions on the data in the setting of linear binary classification don't seem to be so unrealistic. Can the authors provide some toy example experiments to verify the theoretical results more directly?
9. I was expecting section 3.2 would be a generalization of the results in section 3.1. But in section 3.1, the conditions are about the variances in the data dimension. But in section 3.2, somehow it's all about the variances in the model output. I am aware of the statement made in remark 3.11, but I still don't understand how does this section help us gain more understanding of the link between the model's behavior of learning low variance features and the change in the model's generalization peformance?
10. Line 368 on page 7. It better to write "$5\times 10^{-4}$" instead of "$*$".
11. Is there and inituitions or insight as of how does the variance of the learned features affect the performance of the model? Without this, it's pretty hard to use the work in this paper as a cornerstone to initiate further studies, such as the mechanism of label augmentation techniques (in other words, why do they work), the limitations and possible flaws of these techniques, how to improve these techniques or design new algorithms, etc.. This is why I feel like the contribution to the research area in this paper is only fair.
soundness: 3
strengths: 1. The studying of feature learning problem on label smoothing is novel
2. The paper have uncovered some interesting and new findings about the phenomenons and properties existed in label augmentation techniques like Mixup and label smoothing
3. The theoretical contributions and their proofs are valid
4. The experiments are comprehensive, and they show not only the validity of the theoretical results, but also the correlationship between those theoretical results and the impact of the label augmentation techniques on model performance
summary: The paper studies feature learning in training with label augmentation techniques like Mixup and label smoothing, etc.. Using a linear binary classification model, and proper conditions on the data distribution, the authors have theoretically shown that in ERM training, the learned model tends to assign more correlations on the high-variance features, while in label augmentation training, it's quite the opposite. They then empirically show that such a behavior can lead to both positive and negative consequences. In particular, they show that the model's learning low variance features correlates to better generalization, while the model's learning only the low variance features, if happening, can  harm the generalization.
weaknesses: Please refer to the Questions session for details of the weaknesses listed here

1. While the concept of "high variancce features" and "low variance feature" appear to be the essential components of the theoretical content, the whole paper seems to lack a strict definition of them. Perhaps that's a common knowledge in the field of feature learning, but since this paper is mostly theory-based, it's better to provide more mathematically strict definition of these concepts.
2. The logic connection from section 3.1 to section 3.2 is not so clear, in fact, from my understanding, they are talking about different things.
3. Since the main theoretical  results are built based on a simple model, one would love to see first a experiment done on such a toy model to show the validity of the theoretical results in the most direct way.
4. Some notations and terminologies can be perfected
5. There lacks an intuitive explanation or an insight behind the connection between learning of low variance features and generalization performance of the trained model.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: Is there any theoretical investigation into whether Mixup and label smoothing perform worse in situations involving spurious correlations? I think the limited experimental setups are insufficient to support a counter-intuitive argument compared to the extensive body of previous work on model calibration and noisy label learning.
soundness: 3
strengths: 1. This paper rigorously analyzes label smoothing and Mixup with mathematical proofs, building a solid theoretical foundation.
summary: In this research, the authors investigate label augmentation methods such as label smoothing and Mixup within deep learning frameworks. These techniques adjust both the input data and labels to enhance training outcomes. The study delves into how these methods impact feature learning and reduce output variance. The findings indicate that label augmentation helps models develop low-variance features, improving classification results on some benchmarks, though it may increase susceptibility to misleading data patterns. With theoretical backing and empirical results, the paper highlights the strengths and drawbacks of label smoothing and Mixup. The authors conclude that while these approaches bolster model robustness, they may inadvertently promote reliance on low-variance features, affecting model generalization.
weaknesses: 1. Overall, the authors' argument that both Mixup and label smoothing are effective in reducing variance is not particularly surprising and has been a well-known perspective for some time. For this reason, Mixup has been used for calibration due to its generalizability to the underlying distribution of training datasets.

2. The authors should conduct experiments with diverse spurious correlation setups. For example, both label smoothing and Mixup are widely used to mitigate noisy label learning, which is one common spurious correlation setup. I believe these previous works contradict the authors' arguments.

[1] When and How Mixup Improves Calibration, ICML 2022 \
[2] How Does Mixup Help with Robustness and Generalization?, ICLR 2021 \
[3] Does Label Smoothing Mitigate Label Noise?, ICML 2020 \
[4] mixup: BEYOND EMPIRICAL RISK MINIMIZATION, ICLR 2018

Meta-review: The paper studies the effect of label manipulation in classification tasks. It achieves formal results on the effect of two common manipulations, label smoothing and mixup, on the variance of features that are learnt for linear binary classifiers. Then, it studies non-linear models with the same perspective and derives bounds for the model’s output in presence of label smoothing and mixup. A remark of their formal study is that while such label manipulations can be effective for generalization it can potentially make them more prone to learning and predicting based on unwanted correlations. It empirically shows that this is indeed the case.

The reviewers acknowledged the insights the paper offers and the clarity of the writing.

On the other hand, the reviewers criticized several aspects of the formal results including the assumptions, their relevance including a discussion of causality, the notations, and the connections between the different formal statements. They also raised concerns regarding the coverage of prior works particularly on the topics of feature learning and label smoothing.

The authors provided a rebuttal to those points mostly including clarifications regarding their formal results where they also updated the paper accordingly. They further included the missing references in the revised paper.

The reviewers, after discussion with the authors, generally lean towards acceptance.

The AC does not find solid reason to go against the reviewer’s high majority rating and recommends acceptance. But as discussed with 2SFa and others it is important for the paper to repeatedly emphasize that (1) the relation between feature variance and generalization is not studied in this work and a causality is not to be assumed, and (2) low variance features is not shown to be particularly worse for spurious correlations compared to high-variance features.

Decision: Accept (Poster)