Over-Training with Mixup May Hurt Generalization
Authors: Zixuan Liu
Ziqiao Wang
Hongyu Guo
Yongyi Mao
Venue: ICLR 2023 poster (2023)
Keywords: Mixup
Generalization
Overfitting
Regularization

Abstract: Mixup, which creates synthetic training instances by linearly interpolating random sample pairs, is a simple and yet effective regularization technique to boost the performance of deep models trained with SGD. In this work, we report a previously unobserved phenomenon in Mixup raining: on a number of standard datasets, the performance of Mixup-trained models starts to decay after training for a large number of epochs, giving rise to a  U-shaped generalization curve. This behavior is further aggravated when the size of original dataset is reduced. To help understand such a behavior of Mixup, we show theoretically that Mixup training may introduce undesired data-dependent label noises to the synthesized data. Via analyzing a least-square regression problem with a random feature model, we explain why noisy labels may cause the U-shaped curve to occur: Mixup improves generalization through fitting the clean patterns at the early training stage,  but as training progresses, Mixup becomes over-fitting to the noise in the synthetic data. Extensive experiments are performed on a variety of benchmark datasets, validating this explanation.

Published as a conference paper at ICLR 2023
OVER-TRAINING WITH MIXUP MAY HURT GENERALIZATION
Zixuan Liu'; Ziqiao Wang's : Hongyu Guo Yongyi Mao University of Ottawa Naational Research Council Canada {zliu282, zwang286, hongyu. gon,y ymao) Bwwattwa.......... ca
ABSTRACT
Mixup, which creates synthetic training instances by linearly interpolating ran- dom sample pairs, is simple and yet effective regularization technique boost the performance of deep models trained with SGD. In this work, we report a previously unobserved phenomenon in Mixup training: on a number of standard datasets, the performance of Mixup-trained models starts to decay after training for large number of epochs, giving rise to U-shaped generalization curve. This behavior is further aggravated when the size of original dataset is reduced. To help understand such behavior of Mixup. we show theoretically that Mixup training may introduce undesired data-dependent label noises to the synthesized data. Via analyzing a least-square regression problem with random feature model, we ex- plain why noisy labels may cause the U-shaped curve to occur: Mixup improves generalization through fitting the clean patterns at the early training stage, but as training progresses, Mixup becomes over-fitting to the noise in the synthetic data. Extensive experiments are performed on a variety of benchmark datasets, validat- ing this explanation.
INTRODUCTION
Mixup has empirically shown its effectiveness improving the generalization and robust- ness of deep classification models (Zhang et al., 2018; Guo et al., 2019a;b; Thulasidasan et al., 2019; Zhang et al., 2022b). Unlike the vanilla empirical risk minimization (ERM), in which networks with synthetic examples. These examples are created by linearly interpolating both the in- put features and the labels of instance pairs randomly sampled from the original training set.
trained using the original training set, Mixup trains the networks
Best Training
Testing
Owning to Mixup's sim- plicity and its effective- ness boosting the accu- racy and calibration of deep classification models, there has been a recent surge of interest attempting to better understand Mixup's working mechanism, train- ing characteristics, regular- ization potential, and pos- sible limitations (see, e.g.,
mixup d=1
mixup a=1
0.8
0.6
10
50
100
200
400
1000
50
100
200
1000
total num of epachs (log-scaln]
total num of epachs Oog-scale)
Figure 1: Over-training ResNet18 on CIFARIO.
Thulasidasan et al. (2019), Guo et al. (2019a), Zhang et al. (2021), Zhang et al. (2022b)). In this work, further investigate the generalization properties of Mixup training.
We first report previously unobserved phenomenon in Mixup training. Through extensive experi- ments on various benchmarks, we observe that over-training the networks with Mixup may result in significant degradation of the networks' generalization performance. As a result, along the training
"Equal contribution.

---

Published as a conference paper at ICLR 2023
epochs, the generalization performance of the network measured by its testing error may exhibit U-shaped curve. Figure shows such a curve obtained from over-training ResNet18 with Mixup on CI. As can be seen from Figure 1, after training with Mixup for long time (200 epochs), both ERM and Mixup keep decreasing their training loss, but the testing error of the Mixup-trained R gradually increases, while that of the ERM-trained ResNet18 continues to decrease.
Motivated by this observation, we conduct a theoretical analysis, aiming to better understand the aforementioned behavior of Mixup training. We show theoretically that Mixup training may intro- duce undesired data-dependent label noises to the synthesized data. Then by analyzing the gradient- descent dynamics of training a random feature model for a least-square regression problem, we explain why noisy labels may cause the U-shaped curve to occur: under label noise, the early phase of training is primarily driven by the clean data pattern, which moves the model parameter closer to the correct solution. But as training progresses, the effect of label noise accumulates, through itera- tions and gradually over-seighs that of the clean pattern and dominates the late training process. In this phase, the model parameter gradually moves away from the correct solution until it is sufficient apart and approaches a location depending on the noise realization.
2
RELATED WORK
Mixup Improves Generalization After the initial work of Zhang et al. (2018), series of the Mixup's variants have been proposed (Guo et al., 2019a; Verma et al., 2019; Yun et al., 2019; Kim et al., 2020; Greenewald et al., 2021; Han et al., 2022; Sohn et al., 2022). For example, AdaMixup (Guo et al., 2019a) trains an extra network to dynamically determine the interpolation coefficient parameter a. Manifold Mixup (Verma et al., 2019) performs linear mixing the hidden states of the neural networks. Aside from its use in various applications, Mixup's 's working mechanism and it possible limitations are also being explored constantly. For example, Zhang et al. (2021) demonstrate that Mixup yields a generalization upper bound in terms of the Rademacher complexity of the function class that the network fits. Thulasidasan et al. (2019) show that Mixup helps to improve the calibration of the trained networks. Zhang et al. (2022b) theoretically justify that the calibration effect of Mixup correlated with the capacity of the network. Additionally. Guo et al. (2019a) point out a "manifold intrusion" phenomenon in Mixup training where the synthetic data "intrudes" the data manifolds of the real data.
Training on Random Labels, Epoch-Wise Double Descent and Robust Overfitting The thought-provoking work of Zhang et al. (2017) highlights that neural networks are able to fit data with random labels. After that, the generalization behavior on corrupted label dataset has been widely investigated (Arpit et al., 2017: Liu et al., 2020; Feng & Tu, 2021; Wang & Mao, 2022; Liu et al., 2022). Specifically, Arpit et al. (2017) observe that neural networks will learn the clean pattern first before fitting to data with random labels. This is further explained by Arora et al. (2019a) where they demonstrate that in the overparameterization regime, the convergence of loss depends on the projections of labels on the eigenvectors of some Gram matrix, where true labels and random labels have different projections. In a parallel line of research, an epoch-wise double descent behavior of testing loss of deep neural networks is observed in Nakkiran et al. (2020), shortly after the obser- vation of the model-wise double descent (Belkin et al., 2019; Hastie et al., 2022; Mei & Montanari, 2022: Ba et al., 2020). Theoretical works studying the epoch-wise double descent are rather limited to date (Heckel & Yilmaz, 2021: Stephenson & Lee, 2021: Pezeshki et al., 2022), among which Advani et al. (2020) inspires the theoretical analysis of the U-sharped curve of Mixup in this paper. Moreover, robust overfitting (Rice et al., 2020) is also another yet related research line, In particular, robust overfitting is referred to a phenomenon in adversarial training that robust accuracy will first increase then decrease after long training time. Dong et al. (2022) show that robust overfitting is deemed to the early part of epoch-wise double descent due to the implicit label noise induced by adversarial training. Since Mixup training has been connected to adversarial training or adversarial robustness in the previous works (Archambault et al., 2019; Zhang et al., 2021), the work of Dong et al. (2022) indeed motivates us study the label noise induced by Mixup training.
2

---

Published as a conference paper at ICLR 2023
3 PRELIMINARIES
Consider a C-class classification setting with input space A' = Re and label space y := (I.2,...,CY) C}. Let S {.)x,, be training set, where each y; y may also be treated as one-hot vector in P())). the space of distributions over V. Let θ denote the model parameter space, and for each which maps an input feature to distribution in P())). For any pair (x.y) E X x P(V), let E(O, x. denote the loss of the prediction (x) with respect to y. The empirical risk on S is then
θ, let fo : X + [o. 1] denote the predictive function associated with 8,
1 = n Σ (...x.,..)
Rs(0)=
When training with Empirical Risk Minimization (ERM), one sets out to find 0 to minimize this risk. It is evident that if {(-) is taken as the cross-entropy loss, the empirical risk Rs(0) is non-negative, where (0) 0 precisely when fo (x:) = y; for every i 1,2,...,..
In Mixup, instead of using the original training set 5, the training is performed on a synthetic dataset S obtained by interpolating training examples in S. For given interpolating parameter À E [0, 1], let synthetic training set Sx be defined as
Sx = [(xx (1 A)x' Ay + (1-A))) (x,,) (x,,)))
(I)
The optimization objective, or the "Mixup loss", is then
E, R s, (0) = Ex SA
Σ (K
where the interpolating parameter À is drawn from symmetric Beta distribution, Beta(a, a). The default option is to take = 1. In this case, the following can be proved.
Lemma 3.1. Let E(-) be the cross-entropy loss, and is drawn from Beta(1, (1,1) 1) (or the uniform distribution on [0, 1)). Then for all O E A and for any given training set S that is balanced,
C --1
E,R3
(0)
2C
where the equality holds if and only if fx() ý for each synthetic example (ž. E Sx
For 10-class classification tasks, the bound has value 0.45. Then only when the Mixup loss ap- proaches this value, the found solution is near true optimum (for models with adequate capacity).
EMPIRICAL OBSERVATIONS
We conduct experiments using CIFARIO, CIFARI00 100 and SVHN using ERM and Mixup respec- tively. For each of the datasets, we have adopted both the original dataset and some balanced subsets obtained by downsampling the original data for certain proportions. SGD with weight decay is used. At each epoch, we record the minimum training loss up to that epoch, well as the testing accuracy at the epoch achieving the minimum training loss. Results are obtained both for training with data augmentation and for training without. More experimental details given in Appendix A.
4.1 RESULTS OVER-TRAINING WITHOUT DATA AUGMENTATION
For CIFARIO and SVHN, ResNet18 is used, and both the original training sets and subsets with 30% of the original data are adopted. Training is performed for up to 1600 epochs for CIFAR 10, and the results are shown in Figure 2. For both the 30% dataset and the full dataset, we see clearly that after some number of epochs (e.g, epoch 200 for the full dataset), the testing accuracy of the trained network starts decreasing and this trend continues. This confirms that over-training with Mixup hurts the network's 's generalization. One would observe a U-shaped curve, as shown in Figure (right), if we were to plot testing error and include results from earlier epochs. Notably, this
3

---

Published as a conference paper at ICLR 2023
Beal Traaning Lens
Tesling Aloracstt
Feal Treheigoen
Teall Acanes
6.70
mkp g-1
EMM
0.70
EMM
mxpp 3-1
mepp
6.05
07
0.43
6.40
200
402
603
603
2000
280 U430 2433
230 400
620
830
1000
1230
2000
1002
1:40
206
400
020
820
1008
L200
5400 1030
200
400 520
800
2000
200 1903
(a) Train loss (30% data)
(b) Test acc (30% data)
(c) Train loss (100% data)
(d) Test acc (100% data)
(c) 200 epochs
(f) 400 epochs
(g) 800 epochs
(h) 200 epochs
(i) 400 epochs
800 epochs
Training ResNet18 on CIFARIO training set (100% data and 30% data) without data augmentation. Top row: training loss and testing accuracy for ERM and Mixup. Bottom row: loss landscape of the Mixup-trained 8 (where "loss" refers to the empirical risk on the real data) various training epochs; left 3 figures are for the 30% CIFAR10 dataset, and the right 3 are for the full CIFARIO 10 dataset; visualization follows Li et al. (2018)
Figure 2:
phenomenon is not observed in ERM. We also found that over-training with Mixup tends to force the network to learn a solution located at the sharper local minima on the loss landscape, a phenomenon correlated with degraded generalization performance (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016). The results of training ResNet18 on SVHN is presented in Appendix B.I.
ResNet34 is used for the more challenging task CIFARIO0. This choice allows Mixup to drive its loss to lower values, closer to the lower bound given in Lemma 3.1. In this case, we only use the original training set, since downsampling CIFARI00 appears to result high variances in the testing performance. Training is performed for up to 1600 epochs. The results are plotted in Figure 3. The results again confirm that over-training with Mixup hurts the generalization capability of the learned model. A U-shaped testing loss curve (obtained from single trial) is also observed in Figure 3c. Additional results of training ResNet34 on CIFARIO and SVHN are provided Appendix B.
Best Training Lows
62.5
Testing Accuracy (S)
4.5
Dyn.amics Losses [Nixup gm1)
2.4
training
1.0
mixup 1=1
60.0
4,0
loss
2,0
57.5
3.5
3.0
0.8
18 52.5
2.5
14 50.0
2.0
47.5
1.5
0.6
1.0
45.0
ERM
1.0
200
400
800
1200
1600
200
400
800
3200
1600
10a
10:
10a
107
total num of opochs
total num of apechs
opach (log-scale)
(a) Train loss
(b) Test acc
(c) U-shaped curve
Figure 3: Training loss, testing accuracy and U-shaped testing loss curve (subfigure (c), yellow) of training ResNet34 on CIFAR 100 (100% training data) without data augmentation.
4.2 RESULTS ON OVER-TRAINEE WITH DATA AUGMENTATION
The data augmentation methods include "random crop" and "horizontal flip" are applied to training on CIFAR10 and CIFAR100. We train ResNet18 on 10% of the CIFAR10 training set for up to 7000 epochs. The results are given in Figures 4a and 4b. this case, the Mixup-trainee model also produces U-shaped generalization curve. However, while the dataset is downsampled lower
4

---

Published as a conference paper at ICLR 2023
proportion, the turning point of the U-shaped curve nevertheless comes much later compared to the previous experiments where data augmentation is not applied on CIFARIO.
Best Trantig
Textin Drer
2.5
Texling Ore (h)
3.3
2.0
6.3
6.7
1.0
6.3
<00pp g-1
100
220
600
2000
1000
353
2.0
2203 4000
100
225
2000
1000
330
-00
20300 4000
eoochs toale)
(a) Train loss (CIFARIO) (b) Test error (CIFARIO) (c) Train loss (CIFARIO0) (d) Test error (CIFARI00)
Figure 4: (a): Training losses and testing errors of over-training ResNet18 on 10% of the CI- FARI0 training set with data augmentation. (c), Training losses and testing errors of over- training ResNet34 on 10% of the CIFARI00 training set with data augmentation.
The results of over-training ResNet34 on 10% of the CIFARION 100 training set for up to 7000 epochs are given in Figures 4c and 4d, where similar phenomenons are observed.
THEORETICAL EXPLANATION
5.1 MIXUP INDUCES LABEL NOISE
We will use the capital letters X and Y to denote the random variables representing the input feature and output label, while reserving the notations x and y to denote their respective realizations. In particular, we consider each true label y is as a token, in V, not a one-hot vector in P()).. Let X) be the ground-truth conditional distribution of the label Y given input feature X. For simplicity, we also express X)Y as a vector-valued function f X RC,, where (x) & P(Y = -|x= = x) for each dimension y
For simplicity, we consider Mixup with a fixed X E [0, 1]; extension to random À is ttraight-forwan Let and Y be the random variables corresponding to the synthetic feature and synthetic label respectively. Then X a XX + (1 A)X'. Let P(Ý)X) be the conditional distribution of the synthetic label conditioned the synthetic feature, induced by Mixup, namely, P(Ý ,o (X) + (1 (X) for each j. Then for a synthetic feature X, there are two ways to assign to a hard label. The first is based on the ground truth, assigning Y, - arg maxjey (X). ). The second is based the Mixup-induced conditional P( X), assigning V,, A arg maxjey P(Ý j j(X)- When the two assignments disagree, or Yn, * Yi", we say that the Mixup-assigned label Y,, is noisy. Theorem 5.1. For any fixed X, X' and X related by X=XX + (1 X)X' for a fixed À E [0, 1J. the probubility of assigning a noisy label is lower bounded by
P( # Y11 2 TV(P()18), P(Y)))) N
sup [x( - -((1 X01(()+++((0 jey
where TV(,-)) is the total variation (see Appendix D).
Remark 5.1. This lower bound hints that the label noise induced by Mixup training depends on the distribution of original data Px, the convexity of f(X) and the value of X. Clearly, Mixup will create noisy labels with non-zero probability (at least for some X) unless f, is linear for each j.
Remark 5.2. We often consider that the real data are labelled with certainty, iee, maxjey fx(X)= 1 and E fx = 1. Then the probability of assigning noisy label to a given synthetic data can be discussed in three situations: i) if Y & {Y,Y'), where Y could be the same with Y', then Y is noisy label with probability one; ii) if Y E (X,Y'} where Y o Y', then the probability of assigning noisy label is non-zero and depends X; ii)) V = = Y', then Vi= V
As shown in (Arpit et al., 2017; Arora et al., 2019a), when neural networks are trained with a fraction of random labels, they will first learn the clean pattern and then overfit to noisy labels. In Mixup
5

---

Published as a conference paper at ICLR 2023
training, we in fact create much more data, possibly with noisy labels, than traditional ERM training (n² for a fixed A). Thus, one may expect an improved performance (relative to ERM) in the early training phase, due to the clean pattern in the enlarged training set, but performance impairment in the later phase due noisy labels. Specifically if Y,, & {Y, Y" happens with a high chance, phenomenon known as "manifold intrusion" (Guo et al., 2019a), then the synthetic dataset contains too many noisy labels, causing Mixup to perform inferior to ERM.
Theorem 5.1 has implied that, in classification problems, Mixup training induces label noise. Next, we will provide theoretical analysis using regression setup to explain that such label noise may result in the U-shape learning curve. The choice of regression setup in this analysis is due to the difficulty directly analyzing classification problems (under the cross-entropy loss). Such a regression setting may not perfectly explain the U-shaped curve in classification tasks, we however believe that they give adequate insight illuminating such scenarios as well. Such an approach has been taken in most analytic works that study the behaviour of deep learning. For example, Arora et al. (2019b) uses a regression setup to analyze the optimization and generalization property of overparameterized neural networks. Yang al. (2020) theoretically analyze the bias-variance trade- off in deep network generalization using a regression problem.
5.2 REGRESSION SETTING WITH RANDOM FEATURE MODELS
Consider a simple least squares regression problem. Let y R and let f X + y be the ground- truth labelling function. Let (X, Y) be synthetic pair obtained by mixing (X, and (X', Y"). Let Y = s(X) and Z 2 y - Y Then Z can be regarded as noise introduced by Mixup, which may be data-dependent. For example, if f is strongly convex with some parameter p 0, then Z N (A X)[|X X' Given synthesized training dataset 5 = {(X,, } consider a random feature model, 8T (X), where o X + Fd and 0 E Fd, We will consider & fixed and only learn the model parameter 0 using gradient descent the MSE loss
Rg(0)= A
0 Y
2m
where T- = [x(X)), of ((2)..... o(X (x E Rdxm and Ý = 181, Fo....
Pm ER..
For fixed X, Mixup create m = n² synthesized examples. Thus it is reasonable assume m > d (e.g.. under-parameterized regime) in Mixup training. For example, ResNet-50 has less than 30 million parameters while the square of CIFARIO training dataset size larger than 200 million without using other data augmentation techniques. Then the gradient flow, as shown in Liao & Couillet (2018), is
= -
= m
11 क T (
(2)
where 17 is learning rate and क! (/ is the Moore Penrose inverse of कh' (only possible when m > d). Thus, we have the following important lemma.
Lemma 5.1. Let 6" = T and gnoise- = T Z wherein Z = 1Z1, Zz. Zm] E R", the ODE in Eq. (2) has the following closed form form solution
θ: - 8 = (fg - 8* ) n TT + (La - c 1- guoiise
(3)
Remark 5.3. Notably, 6" = + may be seen as the "clean pattern" of the training data. The first term in Eq. (3) decreasing (in norm) and vanishes at t + co. Thus its role is moving 9 towards the clean puttern 0", allowing the model to generalize to unseen data. But it only dominates the dynamics of 0, in the early training phase. The second term, initially O, increases with I and converges to gnoise as t - co. Thus its role is moving 9r towards the "noisy pattern" 0 + gnoise It dominates the later training phase and hence hurts generalization. It is noteworthy that 0- + groise is also the closed-foru solution for the regression problem (under Mixup labels). This suggests that the optimization problem associated with the Mixup loss has a "wrong" solution, but it is possible benefit from only solving this problem partially, using gradient descent without over-training
Noting that the population risk at time step t is
R₂ A Ea, X,Y 1000 (Xx) -11

---

Published as a conference paper at ICLR 2023
and the true optimal risk is
1 = -x [arl|y--1 - -0T
we have the following result.
Theorem 5.2 (Dynamics of Population Risk). Given synthesized dataset S, assume On (0 E2a), 11x(X) I C2/2 for some constant C1 > 0 and [Z| S vC2 for some constant
C2 then we have the following upper bound
R C
Σ
d
(st 0%2) 2n +
C2 (1 (1-o- + +2VG
Pk
where Ç - Ei= max + 0%? ps.
C2) and μt is the 1th eigenvalue of the matrix TTTT.
Remark 5.4. The additive noise Z is usually assumed as zero mean Gaussian in the literature of generalization dynamics analysis (Advani et al., 2020; Pezeshki et al., 2022; Heckel & Yilmaz, 2021), but this would be hardly justifiable in this context. The boundness assumption of 7. in the theorem can however be easily satisfied as long as the output of f is bounded.
Remark 5.5. If we further let & = (i.e. using zero initialization) and assume that the eigenvalues of the matrix T are all equal to p. then the summation part in the bound above can be re- written as Cj ||0*11 e + (C2/p)) p)) (1 e e-ue= then it is clear that the magnitude of the curve is controlled by the norm of t", the norm of the representation, the noise level and μ..
Theorem 5.2 indicates that the population risk will first decrease due to the first term (i.e. (42 + x e22n)t) then it will grow due to the existence of label noises (i.e. C2 (1 -1-e" (1-e-nuxt)) Overall, the population risk will be endowed with a U-shaped curve. Notice that the quantity mpa plays a key role the upper bound, the larger id Tjl,, the earlier comes the turning point of "U".. This may have interesting application, justifying multi-stage training strategy where the learn- ing rate is reduced at each new stage. Suppose that with the initial learning rate, at epoch T, the test error has dropped to the bottom of the U-curve corresponding rate is decreased at this point, then the U-curve corresponding to the new learning rate may have a lower minimum error and its bottom shifted to the right. In this case, the new learning rate allows the testing error to move to the new U-curve and further decay.
this learning rate. If the learning
6 EMPIRICAL VERIFICATION
6.1 A TEACHER-STUDDE TOY SETUP
To empirically verify our theoretical results discussed in Sec- tion 5.2, construct simple teacher-student regression set- ting. The teacher network is a two-layer neural networks with Tanh activation and random weights. It only serves create training data for the student network. Specifically, the training data created by drawing {X i.i.d. from standard Gaus- sian (O, ld.) and passing them to teacher network to obtain labels (Y.)
22
325
208
The student network is also a two-layer neural network with Tanh activation and hidden layer dimension the parameters in the first layer and only train the second layer using the generated training data. Full-batch gradient descent on the MSE loss is used. For the value of X. we consider two cases: a fixed value with À 0.5 and random values drawn from Beta 1) at each epoch. As a comparison, we also present the result of ERM training in an over-parameterized regime (i.e., < d).
Figure 5: Dynamics of testing loss in the toy example.
100. We fix
The testing loss dynamics are presented in Figure 5. We first note that Mixup still outperforms ERM in this regression problem, but clearly, only Mixup training has a U-shaped curve while the testing loss of ERM training converges to a constant value. Furthermore, the testing loss of Mixup training is endowed with U-shaped behavior for both fixed X = 0.5 and random À drawn from Beta(1, 1)-

---

Published as a conference paper at ICLR 2023
This suggests that our analysis of Mixup in Section 5.2 based on a fixed À is also indicative for more general settings of X. Figure 5 also indicates that when is fixed to 0.5, the increasing stage of the U-shaped curve comes earlier than that of À with Beta(1, 1). This is consistent with our theoretical results in Section 5.2. That is, owning to the fact that À with the constant value 0.5 for represents the largest noise level in Mixup, the noise-dominating effect in Mixup training comes earlier.
6.2 USING MIXUP ONLY IN THE EARLY STAGE OF TRAINING
In the previous section, we have argued that Mixup training learns "clean patterns" in the early stage of the training process and then overfits the "noisy patterns" in the later stage. Such a conclusion implies that turning of Mixup after certain number of epochs and returning standard ERM training may prevent the training from overfitting the noises induced by Mixup. We now present results obtained from such a training scheme both CIFARIO 10 and SVHN in Figure 6.
CIFARIO (30%)
(b) CIFARIO (100%)
(c) SVHN (30%)
(d) SVHN (100%)
Figure 6: Switching from Mixup training to ERM training. The number in the bracket is the epoch number where we let a = (i.e. Mixup training becomes ERM training).
Results in Figure clearly indicate that switching from Mixup to ERM at an appropriate time will successfully avoid the generalization degradation. Figure 6 also suggests that switching Mixup to ERM too early may not boost the model performance. In addition, if the switch is too late, memorization of noisy data may already taken effect, which impact generalization negatively. We note that our results here can be regarded as a complement to (Golatkar et al., 2019), where the authors show that regularization techniques only matter during the early phase of learning.
7 FURTHER INVESTIGATION
Impact of Data Size on U-shaped Curve In the over-training experiments without data augmenta- tion, although the U-shaped behavior occurs on both 100% and 30% of the original training data for both CIFARIO and SVHN, we notice that smaller size datasets appear to enable the turning point of the U-shaped curve arrive earlier. We now corroborate this phenomenon with more experimental results, as shown in Figure 7. In this context, appropriate data augmentation can be seen simply expanding the training set with additional clean data. Then the impact data augmentation on the over wver-training dynamics of Mixup arguably via increasing the size of the training set. This explains our observations in Section 4.2 where the turning points in training with data augmentation arrive much later compared those without data augmentation. Those observations also consistent with the results in Figure 7.
It may be tempting consider the application of the usual analysis of generalization dy- namics from the existing liter- ature (Liao & Couillet, 2018; Advani et al., 2020; Stephen- son & Lee, 2021) to the train- ing of Mixup. For example, one can analyze the distribu- tion of the eigenvalues in The- orem 5.2. Specifically, if en- tries in 4 are independent iden-
100% Dacs
50% Data Eaits Duta
32
200
400
1204
383
250
480
630
<ce
CIFARIO
(b) SVHN
Figure 7: Over-training different number of samples. tically distributed with zero mean, then in the limit of d, m + oo with d/m = ? E (0, +00), the eigenvalues {nx)e d follow the Maachenko-Pastenn (MP) distribution (Marčenko & Pastur, 1967),
8

---

Published as a conference paper at ICLR 2023
which is defined as
1 v(+ 1M 7-) 2x
PMP (ul7)
741
14y
where 71 = (1 + 7)². Note that the PMP are only non-zero when μ = or E 7+]. When y close to one, the probability of extremely small eigenvalues is immensely increased. From Theorem 5.2, when μ4z: is small, the second term, governed by the noisy pattern, will badly dominate the behavior of population risk and converge to a larger value. Thus, letting < m will alleviate the domination of the noise term in Theorem 5.2. However, it is important to note that such analysis lacks rigor since the columns in are not independent (two columns might result from linearly combining the same pair of original instances). To apply a similar analysis here, one need to remove or relax the independence conditions on the entries of Φ, for example, by invoking some techniques similar to that developed in Bryson et al. (2021). This is beyond the scope of this paper, and we will to leave it for future study.
Gradient Norm in Mixup Training Does Not Vanish Normally, ERM training obtains zero gra- dient norm at the end of training, which indicates that SGD finds a local minimum. However, We observe that the gradient norm of Mixup training does not converge zero, as shown in Figure 8.
In fact, gradient norm in the Mixup training even increases until converging to a maxi- mum value, opposed zero. When models are trained with ERM on random labels, this in- creasing trend of gradient norm also observed in the previous works (Feng & Tu, ;021; Wang & Mao, 2022). Specifically, in Wang & Mao (2022), such in- creasing behavior is interpreted as sign that the training of SGD enters "memorization regime", and after the overparameteriized neural networks memorize all the noisy labels, the gradient norm (or gradient dispersion in Wang & Mao (2022)) will decrease again until it converges to zero. In Mixup training, since the size of synthetic dataset is usually larger than the number of parameters (i.e., m > d), neural networks may not be able to memorize all the noisy labels in this case. Notice that m is much larger than n² in practice since À is not fixed to a constant.
5.5
Noss 37% Duta
Cight 59% Date
2.2
015
250
255
102
250
202
00.0
(a) Gradient norm on CIFARIO
(b) Gradient norm on SVHN
Figure 8: Dynamics of gradient norm.
Notably, although ERM training able to find a local minimum in the first 130 epochs on CIFARIO, Figure indicates that Mixup training outperforms ERM in the first 400 epochs. Similar observation also holds for SVHN. This result in fact suggests that Mixup can generalize well without converg- ing to any stationary points. Notice that there is related observation in the recent work of Zhang et al. (2022a), where they show that large-scale neural networks generalize well without having the gradient norm vanish during training. Additionally, by switching Mixup training to ERM training, as what we did in Figure 6, the gradient norm will instantly become zero (see Figure 14 in Ap- pendix B.3). This further justifies that the "clean patterns" are already learned by Mixup trained neural networks at the carly stage of training, and the original data may no longer provide any useful gradient signal.
8 CONCLUDING REMARKS
We discovered novel phenomenon in Mixup: over-training with Mixup may give rise to a U-shaped generalization curve. We theoretically show that this is due to the data-dependent label noises intro- duced to the synthesized data, and suggest that Mixup improves generalization through fitting the clean patterns at the early training stage, but over-fits the noise as training proceeds. The effective- ness of Mixup and the fact it works by only partially optimizing its loss function without reaching convergence, as are validated by our analysis and experiments, seem to suggest that the dynamics of the iterative learning algorithm and an appropriate criteria for terminating the algorithm might be more essential than the loss function or the solutions to the optimization problem. Exploration in the space of iterative algorithms (rather than the space of loss functions) may lead to fruitful discoveries.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTI
This work is supported in part by a National Research Council of Canada (NRC) Collaborative R&D grant (AIAD-CORE-07), Ziqiao Wang is also supported in part by the NSERC CREATE program through the Interdisciplinary Math and Artificial Intelligence
project.
REFERENCES
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of gener- alization error in neural networks. Neural Networks, 132:428-446, 2020.
Guillaume P Archambault, Yongyi Mao, Hongyu Guo, and Richong Zhang. Mixup as directional adversarial training. arXiv preprint arXiv: 190:: 06875, 2019.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference Machine Learning, pp. 322-222222222 322-332. PMLR, 2019a.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), ICML, volume 97, pp. 322-332 PMLR, 2019b.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look at memorization in deep networks. In International conference machine learning, pp. 233-242 PMLR, 2017.
Jimmy Ba, Murat Erdogdu, Taiji Suzuki, Denny Wu, and Tianzong Zhang. Generalization of two- layer neural networks: An asymptotic viewpoint. In International Conference on Learning Rep- resentations, 2020. URL https: : /pppreveeeeeee net/ www.mfrrmiioooooo
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical biri-nariance trade-off. Proceedings of the National Academy of Sciences, (16(3)):15449-15555 2019.
Jennifer Bryson, Roman Vershynin, and Hongkai Zhao. Marchenko- pastuur law with relaxed inde- pendence conditions. Random Matrices: Theory and Applications, 10(04):2150040, 2021.
Chengyu Dong, Liyuan Liu, and Jingbo Shang. Label noise in adversarial training: A novel perspective to study robust overfitting. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL pettps://openrevieee net 1twpummmiidd_00mmTX22_
Yu Feng and Yuhai Tu. Phases of learning dynamics in artificial neural networks in the absence or presence of mislabeled data. Machine Learning: Science and Technology, 2(4):043001, 2021.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. Advances in Neural Information Processing Systems, 32, 2019.
Kristjan Greenewald, Anming Gu, Mikhail Yurochkin, Justin Solomon, and Edward Chien. k- mixup regularization for deep learning via optimal transport, 2021. URL https://arxk:
erg/abs/ 610....6000 02933.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regulariza- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3714-3722, 2019a.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Augmenting data with mixup for sentence classifi- cation: An empirical study. CoRR, abs/1905.08941, 2019b. URL htttth/////iiiii .rgg abs/ 1905. 08941.
Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. arXiv preprint arXiv:2202.07/19, 2022.
10

---

Published as a conference paper at ICLR 2023
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949 986, 2022.
Reinhard Heckel and Fatih Furkan Yilmaz. Early stopping in deep networks: Double descent and how eliminate it. In International Conference on Learning Representations, 2021. URL
https: wwwppppeeeeieeeeee ew, net/ www..nn/fooummie/eeeeeoo...
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. Proceedings of the International Conference on Learning Represen- tations, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 91)::-42, 1997.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 1XXvv:6099.000 04836, 2016.
Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning (ICML), 2020.
David A Levin and Yuval Peres. Markov chains and mixing times, volume 107. American Mathe- matical Soc., 2017.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances in neural information processing systems, 31, 2018.
Zhenyu Liao and Romain Couillet. The dynamics of learning: A random matrix approach. In International Conference on Machine Learning, pp. 3072-3081. PMIR, 2018.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning
regularization prevents memorization of noisy labels. Advances neural information processing systems, 33:20331 2020.
Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over- parameterization. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 14153-14172. PMLR, 17-23 Jul 2022.
Shangyun Lu, Bradley Nott, Aaron Olson, Alberto Todeschini, Hossein Vahabi, Yair Carmon, and Ludwig Schmidt. Harder or different? closer look at distribution shift in dataset reproduction. In ICML Workshop on Uncertainty and Robustness Deep Learning. 2020.
Vladimir A Marčenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1(4):457, 1967.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learn- ing Representations, 2020. URL ht htsps: /ttttps//opeeeeeeeeee http///pppeeeeeeeeeeeee/ee net forumrruuuooo ddd.mmidd-
Mohammad Pezeshki, Amartya Mitra, Yoshua Bengio, and Guillaume Lajoie. Multi-scale feature learning dynamics: Insights for double descent. In International Conference on Machine Learn- ing, pp. 17669-17690 PMLR, 2022.
Francesco Pinto, Harry Yang, Ser-Nam Lim, Philip HS Torr, and Puneet Dokania. Regmixup: Mixup regularizer can surprisingly improve accuracy and out distribution robustness. arXiv preprint arXiv:2206. 14502, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 10 classifiers generalize to cifar-10? arXiv preprint arXiv: 1rXiv:/806.0045 1806000416 2018

---

Published as a conference paper at ICLR 2023
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp. 1993-8104, PMLR, 2020.
Jy-yong Sohn, Liang Shang, Hongxu Chen, Jackyun Moon, Dimitris Papailiopooios, and Kangwook Lee. Genlabel: Mixup relabeling using generative models. arXiv preprint arXiv:2201.02554 2022.
Cory Stephenson and Tyler Lee. When and how epochwise double descent happens. arXiv preprint arXiv:2108. /2006, 2021.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Micha- lak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz. and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp- 6438 6447. PMLR, 09-15 Jun 2019.
Zigiao Wang and Yongyi Mao. On the generalization of models trained with SGD: Information- theoretic bounds and implications. In International Conference on Learning Representations, 2022. URL https: / opwnreviewee...ee///ep/ee net.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In ICML, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. Proceed- ings of the JEEF/CVF international conference computer vision, pp. 6023-6032. 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017. URL https: /oooopprree/e ew. net / forum? disdnmdddd-dyyyxys
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond em- pirical risk minimization. In International Conference on Learning Representations, 2018. URL
https::/ /tttps://openreview net/ forum watprummdddddddd---dmeee
Jingzhao Zhang, Haochuan Li, Suvrit Sra, and Ali Jadbabaie. Neural network weights do not con- verge to stationary points: An invariant measure perspective. In International Conference Machine Learning, pp. 26346. PMLR, 2022a.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In International Conference on Learning Representa- tions, 2021. URL https:://// http//pppeeeeeeieeeeeee/eeeeeeee wwtprreweeeweeeeeeeeeeeeeeeeeeee /eeetforuute forum? id=8ykKo0 6dKNo.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. International Conference Machine Learning, pp. 26135-26160.. PMLR, 2022b.
12

---

Published as a conference paper at ICLR 2023
A EXPERIMENTAL SETUPS OF OVER-TRAINING
For any experimental setting such as the training dataset and its size, whether ERM Mixup is used, whether other data augmentation is used, etc.), we define a training "trial" as training process starting from random initialization to a certain epoch In each trial, we record the minimum train- ing loss obtained during the entire training process. The testing accuracy of the model's 's intermediate solution that gives rise that minimum training loss is also recorded. For different trials we gradu- ally increase / so to gradually let the model be over-trained. For each i, repeat the trial for 10 times with different random seeds and collect all the recorded results (minimum training losses and the corresponding testing accuracies). We then compute their averages and standard deviations for all t's. These results are eventually used to plot the line graphs for presentation.
For example, Figure 2a illustrates the results of training ResNet18 on 30% CIFARIO data without data augmentation. The total number of training epochs t, as shown on the horizontal axis, is in- creased from 100 to 1600. For each each point its vertical axis represents the average of the recorded training losses from the 10 repeats. The width of the shade beside each point reflects the corresponding standard deviation.
B ADDITIONAL EXPERIMENTAL RESULTS
B. ADDITIONAL RESULTS OF OVER-TRAINING WITHOUT DATA AUGMENTATION
Besides CIFARIO, ResNet18 is also used for the SVHN dataset.
Training is performed for up to 1000 epochs for SVHN, since we notice that if we continue training ResNet18 on SVHN after 1000 epochs. the variance of the testing accuracy severely increases. The results are presented in Figure 9. Mixup exhibits similar phenomenon as it does for I..AR1O. What differs notably is that over-training with ERM on the original SVHN training set appears to also lead to test accuracy. However, this does not
the 30% SVHN training set!
The Lass
Testby Accenscy No
0.75
CPM
CMM
325
mep
6.18
275
2225
0:5
D1V
0.55
0:5
253
372
1400 1923
s00 ex1 #S1 1990 Liho Ince
0.50
276
450
FS1
1293
1982
158
$45
100
P51 1991 1231
(a) Train loss (30% data)
(b) Test acc (30% data)
(c) Train loss (100% data)
(d) Test acc (100% data)
(e) 200 epochs
(f) 400 epochs
(g) 800 epochs
(h) 200 epochs
(i) 400 epochs
(j) 800 epochs
Figure 9: Results training ResNet18 on SVHN training set (100% data and 30% data) without data augmentation. Top raw: training loss and testing accuracy for ERM and Mixup. Bottom raw: loss landscape of the Mixup-trained R at various training epochs: the left 3 figures are for the 30% SVHN dataset, and the right 3 for the full SVHN dataset.
As for ResNet34, besides CIFARI00, it is also used for both the CIFAR10 and the SVHN datasets.
'This might be related to the epoch-wis. wise double descent behavior ERM training. That is, when over- training ResNet18 on the whole training set with total of 1000 epochs, the network is still in the first stage of over- over-fitting the training data, while when over-training the network on 30% of the training set, the network learns faster on the training data due to the smaller sample size, thus it passes the turning point of the double descent curve earlier.
13

---

Published as a conference paper at ICLR 2023
Training is performed on both datasets for in total 200, 400 and 800 epochs respectively. The results for CIFARIO are shown in Figure 10. For both the 30% dataset and the original dataset, Mixup exhibits similar phenomenon as it does in training ResNet18 CIFARIO. The difference is that over-training ResNet34 with ERM let the testing accuracy gradually increase on both the 30% dataset and the original dataset.
Trekiing
Aoureeyy
2MM
6BM
GMM
6.00
3.50
0.50
0.51
352
450
1961 upedh
cookts
5000l opechs
(a) Train loss (30% data)
(b) Test acc (30% data)
(c) Train loss (100% data) (d) Test acc (100% data)
Figure 10: Results of the recorded training losses and testing accuracies of training ResNet34 on CIFAR 10 training set (100% data and 30% data) without data augmentation.
The results for SVHN are shown in Figure 11. These results are also in accordance with those of training Re on both 30% and 100% of the SVHN dataset.
0.70
Rend Trokiing 1955
Teidine Aive piay
0.55
0.65
Nedt Trikean
960
0.05
0.52
0.50
4. 66.4
35
0.50 201
300
230
210
451
(a) Train loss (30% data)
(b) Test ace (30% data) (c) Train loss (100% data)
(d) Test acc (100% data)
Figure 11: Results of the recorded training losses and testing accuracies of training ResNet34 on SVHN training set (100% data and 30% data) without data augmentation.
In addition, we have trained VGG16 on the CIFARIO training set (100% data and 30% data) for up to in total 1600 epochs without data augmentation. The results are provided in Figure 12. In both cases, over-training VGG16 with either ERM or Mixup can gradually reduce the best achieved training loss. However, the testing accuracy of the Mixup- trained network also decreases, while that of the ERM-trained network has no significant change.
Best Trobring
E4
Teating Accurayy mX)
Rest Tratrin
Text ng Ascanacy (N4
0.75
6.70
6.55
1.5
6.40
26
1921
6.50
232
6:00
1200
1502
400
1250
3952
1:55
230
600
1200
1000
1260
3992
(a) Train loss (30% data)
(b) Test acc (30% data)
(c) Train loss (100% data)
(d) Test acc (100% data)
Figure 12: Results of the recorded training losses and testing accuracies of training VGG16 on CIFARIO training set (30% data and 100% data) without data augmentation.
B.2 RESULTS OF MEAN SQUARE ERROR LOsS WITHOUT DATA AUGMENTATION
We also perform Mixup training experiments using the mean square error (MSE) loss function on both CIFARIO and SVHN datasets. Figure 13 illustrates that the U-shaped behavior observed in
14

---

Published as a conference paper at ICLR 2023
previous experiments is also present when using the MSE loss function. To ensure optimal training, the learning rate is decreased by a factor of 10 at epoch 100 and 150.
4200
6C58
6CR
100
(a) CIFAR1O (30%)
(b) CIFARIO (100%)
(c) SVHN (30%)
(d) SVHN (100%)
Figure 13: Dynamics of MSE during Mixup training.
B.3 GRADIENT NORM VANISHES WHEN CHANGING MIXUP TO ERM
In Figure 8, we can observe that the gradient norm of Mixup training does not diminish at the end of training and can even explode very high value. In contrast, ERM results in a gradient norm of zero at the end of training. Figure 14 illustrates that when switching from Mixup training to ERM training after certain period, the gradient norm will rapidly become zero. This phenomenon occurs because Mixup-trained neural networks have already learned the "clean patterns" and the original data does not provide any useful gradient signal. Therefore, this further supports the idea that the latter stage of Mixup training is primarily focused on memorizing noisy data.
Mouo
Mir Co41
12
64
120
120
120
CIFARIO (30%)
(b) CIFARIO (100%)
(c) SVHN (30%)
(d) SVHI (100%)
Figure 14: Dynamics of gradient norm when changing Mixup training to ERM training.
B.4 VALIDATION RESULTS ON COVARIANCE-SHI SHIFT DATASETS
Recall Figures 2 and 9, have that increase the training epochs (200- - 400 - 800), the local minima on the loss landscape (measure by the real training data) of the Mixup-trained model gradually becomes sharper. To validate the regular pattern of the relationship between the minima flatness and the generalization behavior on the covariate-shift datasets, we have ran some of the Mixup-trained ResNet18 networks and tested their accuracies CIFAR10.1 (Recht et al., 2018), CIFAR10.2 (Lu et al., 2020) and CIFARIO 10-C (Hendrycks & Dietterich, 2019) using Gaussian noise with severity and (denoted by CIFAR10-C- and CIFARIO-C-5). The results of the models pre- trained on 100% CIFAR10 O are given in Figure 15, and the results of the models pre-trained on 30% CIFARIO are given Figure 16.
From the results, it is seen that with training epochs increase, the testing performance on the models on CIFARIO.J 10 and CIFARIO. decreases, taking a similar trend as our results in standard testing sets (i.e., the original CIFAR10 10 testing sets without covariate shift.) But CIFARIO-C, this behaviour not observed. In particular, the performance on CIFARIO-C-5 continues to improve over the training iterations. This seems to suggest that the flatness of empirical-risk loss landscape may impact generalization to covariate-shift datasets in more complex ways, possibly depending on the nature and structure of the covariate shift.
15

---

Published as a conference paper at ICLR 2023
Tecting Acarscy IN
Teedin Accuscy (9))
Teating Accuncy (8)
Tedleg Aocuaay C1)
0PAN10
OPAN10C
eepocte Pre-Triied Networt
#epocht Pre-Toined Netwer
dio
200
Pappcts Fre-Triied Networe
250
tepocho Fre-Trained hirwork
820
(a) CIFARIO.1
(h) CIFAR10.2
(c) CIFAR1O-C-1
(d) CIFARIO-C-5
Figure 15: Models Trainrained on 100% CIFARIO (without data augmentation)
Teating Acsurecy 190
Teating Accu 29
Teating Accu Boy 26
Teaking Accur aoy 26
OFAR10.
OFAR102
65
CFAR19-1
CFAR36-C5
03
02
03
60
14
10
32
30
200
500
350
250
500
550
250
400
656
250
420
056
(a) CIFARIO.1
(b) CIFARI0.2
(c) CIFARIO-C-1
(d) CIFARIO-C-5
Figure 16: Models Pre- Trainee on 30% CIFAR1O (without data augmentation)
B.5 INVESTIGATION OF THE IMPACT OF REGMIXUP IN OVER-TRAINIE
RegMixup variant algorithm of Mixup proposed by (Pinto et al., 2022). For each synthetic example (X. y) formulated by (x. y) and (x', y'), RegMixup minimizes the following loss
(6 x, + c/fCE (0, y)
where (-) denotes the cross-entropy loss, and is non- non-negative. The authors show that RegMixup can improve generalization on both in-distribution and covariate-shift datasets, and that it can also improve the cittr--disttibution robustness. To validate the performance of RegMixup in the over- training scenario, we have trained ResNet18 using RegMixup with a few different settings of 77. The network is trained on CIFAR1O without data augmentation for up to in total 1200 epochs. The results are given in Figure 17.
Testing Accuracy (%)
90.5
ERM
n= 0.1
90.0
n=2
89.5
Mixup
89.0
88.5
88.0
87.5
87.0
200
400
800
1200
total num of epochs
Figure 17: Results of Over- -ver-Trannnn ResNet18 on CIFARIO (without data augmentation) with Reg- 2; Red: ERM; Blue: standard Mixup.
Mixup. Green: 17 0.1; Purple:
16

---

Published as a conference paper at ICLR 2023
The results show that when 2, RegMixup performs nearly identically standard Mixup in the over-training scenario. When = 0.1, RegMixup postpones the presenting of the turning point, and in the large epochs it outperforms standard Mixup. However, the phenomenon that the generalization performance of the trained model degrades with over-training still exists.
C EXPERIMENT SETTINGS FOR THE TEACHER-STUDERS TOY EXAMPLE
We the dimension of the input feature as 10. The teacher network consists of two layers with the activation function Tanh, and the hidden layer has width of 5. Similarly, the student network is two-layer neural network with Tanh, where we train only the second layer and keep the parameters in the first layer fixed. The hidden layer has a dimension of 100 (i.e. d = 100).
To determine the value A, we either draw from a Beta(1,1) distribution in each epoch or fix it to 0.5 in each epoch. We choose 20, which puts in the overparameterized regime where < d, and the underparameterizze regime where m 2 n² d. The learning rate set to 0.1, and we use full-batch gradient descent train the student network with MSE. Here, the term "full-batch" means that the batch size equal to n, enabling us to compare the fixed and random methods fairly.
For additional information, please refer to our code.
C.I ABLATION STUDY: EFFECT OF FIXED MIXING COEFFICIENT
1e-3
2.6
Fixed Fixed
Fixed - 0.3
Fixed = 0.5
2.2
2.0
1.8
1.6
epoch (x150)
Figure 18: Results of the ablation study on X.
In the teacher-student setting, experiment with different fixed values of X, and the results presented in Figure 18. Of particular interest is the observation that as the noise level increases and approaches 0.5, the turning point of testing error occurs earlier. This finding is consistent with theoretical results.
D OMITTED DEFINITIONS AND PROOFS
Definition D.1 (Total Variation). The total variation between two probability measures P and Q TV Q) A supg ((F)) Q(F.), where the supremum is all measurable set F.
Lemma D.1 (Levvin& & Peres, 2017, Proposition 4.2)). Let and Q be two probability distributions on X. If X is countable, then
TV(P.Q) = Σ P(x) Q(x))
ZEN
17

---

Published as a conference paper at ICLR 2023
Lemma D.2 (Coupling Inequality (Levin & Peres, 2017, Proposition 4.7)). Given random vari- ables X and Y with probability distributions P and Q. any coupling P of P. Q satisfies
TV(P,Q) - P(X
D.1 PROOF OF LEMMA 3.1
Proof. We first prove the closed-form of the cross-entropy loss's lower bound. For any two discrete distributions P and Q defined on the same probability space V, the KL divergence of P from Q is defined as follows:
P(y) Q(y)
DkL(P
-E P(y) log
(4)
VEY
It is non-negative and it equals 0 if and only P Q.
Let's denote the jth element in fa( by fo(ar);- By adapting the definition of the cross-entropy loss, we have:
E(O, (x,y) y" log (fo(x))
Σ y log (fo(x),)
K
[f(())
Σ log
yi
(5)
==1
fo(x);
Σ
Vc log
Yc log y:
yi
i=
DKL (xllfo H(y)
2 Hlyy,
where the equality holds if and only if fo(x) = y. Here Hly) := Σ Yr log is the entropy of the discrete distribution y. Particularly in ERM training, since is one-hot, by definition its entropy simply O. Therefore the lower bound of the empirical risk is given follows.
Rs( =
e(0, (x. 20 0
(6)
The equality holds if fo(x;) y, is true for each E [1,2, ... n..
We then prove the lower bound of the expectation of empirical Mixup loss. From Eq. (5), the lower bound of the general Mixup loss for given À is also given by:
((O, (x, ;)) 2 H()))
- Σ Ui log yo
-- log X (1 A) log(1 A)).
(K, is formulated via cross-class mixing. Recall the definition of the Mixup loss,
Rg( A~ Betafaa n
E(0, (x,,))
(8)
we can exchange the computation of the expectation and the empirical average:
1
Rg(0,a)
(x,5))
n A~ Beta(a,
18

---

Published as a conference paper at ICLR 2023
Note that when 1, Beta(a, a) is simply the uniform distribution in the interval [0. 1]: (O, 1).
Using the fact that the probability density of (O, 1) is constantly in the interval [0, 1]. the lower bound of A-BBtta(111)
E
E(0, ))) where y * y' is given by:
E (1))))()),))
E(0, (x,8)) 2-
E A-11(0,1)
(Alog A + (1 - 2) - )))
J (1- A) dd
2
Alog X dd
(10)
log a /// Ada S.'
1 N (fo Add)) (((a)))al di
- A log X
X2 |
10
0.5
Note that if the synthetic example is formulated via in-class mixing, the synthetic label is still one- hot, thus the lower bound of its general loss is O. In balanced C-class training set, with probability G the in-class mixing occurs. Therefore, the lower bound of the overall Mixup loss is given as follows.
1
C-1
a=(0,a=1)
(11)
2C
The equality holds if fo(x) = y is true for each synthetic example (X, E S. This completes the proof.
D.2 PROOF OF THROREM 5.
Proof. By the coupling inequality i.e. Lemma D.2, we have
(V(P())8), (T) s P(T. + Xx)
Since TV(P(L,))), P(YIX)) = TV(P(R)X) ()); then the first inequality is straightfor- ward.
For the second inequality, by Lemma D.1, we have
TV(P( P(7E
)
1
P( = 5|
2
j=1
C
- Σ
[8( - -(11 (x) + +AB(X))
1=1
sup 1
2
[5(x) - ((1 - x)) + -
; 2
This completes the proof.
D.3
PROOF OF LEMMA 5.1
Proof. The ordinary differential equation of Eq. (2) (Newton's law of cooling) has the closed form solution
0, = o (0o T 2 4
(12)
19

---

Published as a conference paper at ICLR 2023
Recall that Y = Y Z.
0 = YY + 2 + (00 - T (Y*+ + Z) =
= Y + क Z + (00 क Y
e m 1000
T Ze m 60
-0" + (0u 0r) ) 2 254
2 pnoise
E m + (L.
which concludes the proof.
D.4 PROOF OF THEOREM 5.2
Proof. We first notice that
Rs =Br,x,y 10 (XX) 2
=Be,,, 100 ((X) nt 6(X) 0 6(() - -111
=Bo,,x (XX) 0² 0'=((X)])
+Ex,y 1 ((x) - -Y1 x, (0f x(x) 0² (X), 0 0 ((X) YY)
<Ex ||0(x)) Bo, 110,- gxr 1|- +R² 22 Eor (00f((X)- 0- ((x) ?/2 Ex, 110-T1 - Y12
- - Be, 100 - 1 R + 2 where the first inequality is by the Cauchy-Schward Schwarz inequality and the second inequality is by the assumption.
C1 2
Ee, 107 -07----11-
1-
(13)
Recall Eq. (3),
0,- 0" (0o
0*)e
2 -00
(Ia e--e-aa _00" 1 Z. 1)
By eigen-decomposition we have
Φ VAVT
m
where {uujd Jd are orthonormal eigenvectors and {un}a Then, for each dimension A;
Jd are corresponding eigenvectors.
-0,5-00) < 2(0,, e 2mpett + 2(1 e nuutt mC2 m misk
Taking expectation over fg for both side, we have
Es, (Or,s 02)² 2( 023)e 2nice + 2(1 - e C2
(14)
14k
Notich that the RHS in Eq 14 first monotonically decreases and then monotonically increases, so the maximum value of RHS is achieved either = 0 or + co. That is.
Eg, 1107
8" 1
2 max + 0 C2 ). pk
(15)
Plugging Eq. 14 and Eq. 15 into Eq. 13, have
C1 Eo, |1 - - 12 R + +2 2
110T goT 1
R. S 2
Ea,
d
O2
<R" + C1 Σ (52 0 20 + (1 e
+ C.C,
k=1
where ( = Ei=n max 62 + 0r2 Ct). This concludes the proof.
me
20


Reviews:
[{"review_id": "bas8dnWGb0", "reviewer": "Reviewer_9xtA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to understand Mixup training, and specifically the training dynamics if mixup is continued to run for much longer than is usual. The paper shows an interesting empirical phenomenon, that overtraining with mixup leads to a U-shaped curve for training error, i.e. training error increases after a point (and also becomes worse than the error of the ERM solution). To demonstrate this, the paper considers ResNet models trained on CIFAR10 and SVHN datasets. \n\nThe paper then provides an intuitive explanation for this phenomenon for the simple regression setting on Gaussian data. Though the setup is simplistic, it brings out the same phenomenon as in the more realistic experiments. The key idea behind why Mixup overfits when overtrained is intuitive too: if the ground truth function f is non-linear, then the Mixup labels do not necessarily correspond to the labels assigned by the ground-truth function (because Mixup does a linear interpolation to assign the labels of the Mixed-up datapoints which would be inconsistent with the labels if f is non-linear). Therefore, Mixup introduces label noise, and if overtrained the model will fit the noise and not generalize as well.\n\n", "summary_of_the_review": "In summary, I think this is an interesting paper and provides valuable insight into a very popular training strategy (Mixup). While it may not directly lead to better training, I think it does provide useful insight into neural network training dynamics more broadly. This aspect is further strengthened by the papers analysis of the overtraining phenomenon in relation to other empirically observed phenomenon in deep learning.", "strength_and_weaknesses": "Strengths:\n\n1. The paper identifies an interesting empirical phenomenon, and the findings appear to be quite robust to different setups.\n2. The theory is clean, and provides a clear explanation which fits the intuition.\n3. The paper relates the observations to a number of recent empirical phenomenon identified in deep learning, such as sharp/flat minima and the gradient norms not converging to zero. This places the paper well in the broader literature, and it could contribute to a growing understanding of neural network training dynamics.\n\nWeaknesses:\n\n1. The finding might have slightly limited practical applicability since the #epochs at which overfitting starts to happen is quite large, and early-stopping is often employed when the model starts to overfit. The main takeaway is probably the more conceptual one of how mixup training behaves.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nh26mciNPQU", "reviewer": "Reviewer_Zo8A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors studied why training with the mixup loss for extra epochs often degrades test classification accuracy. They argued, through empirical observation, theoretical analysis and synthetic experiments, that the mixup loss introduces label noise, and a model may overfit to the label noise during extended training, hence increasing the generalization gap.\n\nThis paper makes several contributions:\n1. it reports an interesting phenomenon in training classifiers and replicates it on multiple datasets;\n2. it shows that extended training with the mixup loss increases the sharpness of the loss landscape;\n3. it provides a possible explanation to the phenomenon by analyzing a regression problem;\n4. it creates a synthetic regression problem which validates the theoretical analysis;\n5. it suggests a practical solution to alleviate the observed problem.", "summary_of_the_review": "To my best knowledge, I believe the paper studies an interesting and potentially important question; both the experiments and the theoretical analysis are sound per se; however, I have concerns regarding the novelty and scope of validity of the reported phenomenon, as well as whether the theoretical analysis actually justifies the authors' hypothesis.", "strength_and_weaknesses": "Strength:\n- the paper studies an interesting phenomenon concerning a popular training loss (or data augmentation method), which has a wide audience base;\n- the paper provides both empirical evidence and theoretical arguments to support its hypothesis;\n- the paper is well-written and easy to follow.\n\nWeakness:\n- On the novelty and validity of the reported phenomenon. With some search I found a paper [1] reporting test accuracies during extended training with ERM or mixup, on benchmark datasets such as CIFAR-10 and CIFAR-100. In [1, Table 13 and 14] only one out of four settings (i.e. CIFAR-100 + RX-50) shows performance degradation with extended mixup training. One significant difference of the current work is that it turns off (other) data augmentation. Could this be the reason why the reported results seem contradictory? And if so, maybe it is proper to state that the observation may only be valid without data augmentation, and contrast with the previous report.\n\n[1] AutoMix: Unveiling the Power of Mixup for Stronger Classifiers. ECCV 2022.\n\n- On the interpretation of the theoretical analysis. If I understand correctly, the gradient flow characterization (eq. (3), Lemma 5.1) is basically the coordinates of $\\theta_t$ as a parametric curve. While $\\theta_t \\neq \\theta^*$, it does not imply that $||\\theta_t - \\theta^*||$ or $R_t$ is a U-shaped function. Theorem 5.2 is an upper bound result on $R_t - R^*$. Without a matching lower bound, or a direct calculation of $\\frac{d R_t}{dt}$, or showing $R_t$ is smaller for some $t$ vs. $t \\to \\infty$, one still cannot justify that $R_t$ will **actually** increase as $t$ goes to infinity.\n\nThere are two additional points regarding the link between the authors' theory and observation that I would raise for discussion:\n\n1. At the end of Sec. 5, the authors claim that their Theorem 5.2 \"explains why reducing the learning rate will make the population risk again decrease in a certain interval\". However, as I understand, in the gradient flow ODE interpretation (eq. (3), Lemma 5.1), changing $\\eta$ simply corresponds to reparametrize the curve, (e.g. reducing $\\eta$ by half corresponds to running $t$ 2x slower), hence a stretching or squeezing of the $R_t$ curve along the x-axis. Therefore it only scales $\\frac{d R_t}{dt}$ but does not change its sign. Could you explain your reasoning and correct me if I am wrong?\n\n2. At the end of Sec. 6.1, the authors claim that the minimum of the MSE loss for a fixed $\\lambda=0.5$ come earlier in training due to increased noise level, which is consistent with their theory. However, as drawing $\\lambda$ from a distribution introduces extra noise to the training, a better ablation study would be to compare the effects of different fixed $\\lambda$, e.g. $\\\\lambda \\\\in \\\\{0.1, 0.2, 0.3, 0.4, 0.5\\\\}$.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FYcqnum_fls", "reviewer": "Reviewer_VPXk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the phenomenon that the predictive performance of a prediction function degrades when overtraining under mixup data augmentation. The authors refer to the loss landscape as the U-shaped curve, where the prediction performance on the test data degrades when the number of epochs is increased. After observing the phenomenon experimentally, the authors present a theory that explains the phenomenon. The authors consider both classification and regression problems, although they focus primarily on classification problems.", "summary_of_the_review": "### Major comments:\nThe problem setting and the theoretical framework proposed in this paper are very simple. While the theoretical analysis could be improved, the problem posed and the experimental observations are intriguing.\n\nThe reason why I feel that the theoretical analysis is incomplete is that it is hard to find the consistency among some of the results presented. For example, the connection between Theorem 5.1 and Theorem 5.2 is not clear. Overall, the results seem to be pieced together with little relevance. Furthermore, there are some ambiguous descriptions and undefined terms.\n\nAlthough I am dissatisfied with some of such details, I will not vote for rejection because it reports findings that are useful for practical use.\n\n- On page 3: In Lemma 3.1, what is $\\theta$ or what is a class of $\\theta$? For example, can the authors write, \"Then, for all $\\theta \\in ???$\"?\n- On page 3: In Lemma 3.1, $\\mathbb{E}_{\\lambda}\\hat{R}(\\theta)$ is a random variable regarding the randomness of $\\lambda$? That is,  does the statement hold with probability one?\n- The figures are hard to read because the characters are very small. It is not kind to people who read the printed paper. \n- While the y-axis of the experimental results for test data in Figure 1 is test error, it is test accuracy in Figure 2. I read a printed paper, so I could not notice the change in the y-axis and was confused.\n- On page 5: In Theorem 5.1, does the bound hold with probability one?\n- On page 6: Is the definition that \"For example, if $f$ is strongly convex with some parameter...\" applied for all $x$?\n- I omit other details but feel that the definition and statement in the theoretical analysis are not rigorous. I would like the authors to clarify them in more detail.\n\n### Minor comments:\n- On page 3: Can we interpreted $\\mathcal{Y}$ as being $\\mathcal{Y} \\in \\mathcal{P}(\\mathcal{Y})$. \n- On page 3: In Lemma 3.1, what is the definition of $\\mathbb{E}_{\\lambda}$?\n", "strength_and_weaknesses": "This paper provides beneficial practical insights. Many practitioners will be interested in this issue related to the mixup and overtraining. The authors' contribution is to raise the issue based on experimental observations and present a simple theoretical framework.\n\nThe theoretical result is straightforward and may not be accurate in some points. Furthermore, I felt that the theoretical validation between the analysis of classification (Section 5.1) and the analysis of regression (Section 5.2) is not consistent. Therefore, there is much room for improvement.\n\nHowever, the problem setting and the approach are interesting. Although I think that the paper's proposed framework is not complete, I am inclined to accept it because I appreciate the challenging attempts.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5X1vpx3Zbz", "reviewer": "Reviewer_gGCe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors point out a previously unknown behaviour of Mixup when overtraining: when the number of training epochs is extremely high, the Mixup performance is inferior to the one of ERM. The authors provide sufficient empirical evidence to show this phenomenon generalizes at least across datasets, and provide an explanation in terms of the labeling noise Mixup induces. They also provide a simplilfied theoretical explanation on a random feature model that exhibits a similar behaviour. As a fix to the problem, they suggest switching off mixup after a few epochs, which is empirically interesting, although further experiments would be required in order to understand its usefulness for more complex setups.", "summary_of_the_review": "The paper is overall interesting, although its practical usefulness is not particularly emphasized. The idea is simple and clear, the explanations are plausible. If the authors could convincingly address the weaknesses I pointed out, I'd be happy to rise the score.", "strength_and_weaknesses": "Strengths:\n- clearly written, easy to follow, good references\n- the phenomenon is novel, although overtraining is not typically a problem in large-scale experimental settings\n- most explanations provided are plausible and supported either by theory or experiments\n\nWeaknesses:\n- Q1: The experiments mostly use ResNets. Does this phenomenon emerge also for other types of architectures? The way overparametrisation plays a role might interact differently with different inductive biases. Could you try showing some results for other convolutional architectures (e.g. ConvNeXt, PyramidNet), some sequential tasks (e.g. using LSTMs)  or Transformers fine-tuning?\n- Q2: The authors refer to the conjecture that flatter minima imply better generalization: for CIFAR-10 and CIFAR-100 covariate-shift datasets are available (e.g. CIFAR-10/100-C, CIFAR-10.1 and CIFAR-10.2). Would it be possible for them to validate their argument about the flatness inducing better generalization on these datasets by running their models and evaluating the accuracy on these datasets? Indeed, whether the Mixup loss is flat or sharp should not bee too relevant, what probably should matter is whether in the loss-landscape of the ERM cross-entropy loss the configuration reached by the optimisation algorithm is flat or not, and this will influence the actual generalization performance.\n- Q3: I understand the authors probably switched off augmentations to disentangle the behaviour of Mixup from that of augmentations. However, the authors claim augmentations would further exasperate the phenomenon observed. Could they provide empirical evidence for this?\n- Q4: It is not clear to me how the teacher-student experiment should be relevant. What is it supposed to tell? Maybe it's just a matter of writing or my own misunderstanding. Could you please elaborate further?\n- Q5: Mixup is a technique used in many SOTA training procedures. The authors seem to suggest it could be useful to switch it off after a few epochs. This creates the issue of having to determine when it should be switched off: presumably, choosing different epochs will produce very different results. Is there a procedure the authors could suggest to do it? Additionally, the effectiveness of this procedure should be validated on more extensive experimental evidence (a few more architectures, and if possible larger datasets (e.g. ImageNet)). \n- Q6: The authors of [1] suggest using the Mixup loss as a regulariser of normal cross-entropy (i.e. the loss is CE(clean samples) + MixupLoss(interpolatedSamples)) in order to fix out-of-distribution detection issues given by the label-smoothing effect of Mixup induced by the randomness of the interpolation parameter.  Given the argument you make about the impact of the label noise and you suggest a fix by switching off Mixup, could you please either empirically or theoretically discuss whether using Mixup as a regularizer of cross-entropy could be a fix to the problem? Would the impact of the noise signal be alleviated or cancelled in this case?\n- Q7: Overtraining is not really a problem for large-scale classification (since the resources requirements are already extremely high that no one would waste them overtraining), however your observations can be helpful in case practitioners observe failure cases of Mixup on smaller datasets. Instead of subsampling larger datasets, could you suggest real-world datasets in which overtraining might happen unintentionally and hence make practitioners actually observe this phenomenon? \n- Q8: Since Mixup already requires cross-validation (given the need of cross-validating the hyperparameter of the Beta distribution), what would prevent practitioners to just use Early Stopping or some other strategy to avoid over-training? Would Early Stopping outperform ERM training?\n[1] RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness; Francesco Pinto, Harry Yang, Ser-Nam Lim, Philip H.S. Torr, Puneet K. Dokania\n\n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster